Mastering Linux: Essential Advanced System Techniques

Highly recommended Amazon books on Linux:

• The Linux Programming Interfaces: A Linux and Unix System Programming Handbook Click => The Linux Programming Interface
• Linux Pocket Guide: Essential Commands Click => Linux Pocket Guide
• Linux Basics for Hackers: Getting Started with Networking, Scripting, and Security in Kali Click => Linux Basics for Hackers
• Linux System Programming: Talking Directly to the Kernel and C Library Click => Linux System Programming
• Unix and Linux System Administration Handbook Click => UNIX and Linux System Administration Handbook
• The Linux Command Line: A Complete Introduction Click => The Linux Command Line
• Practical Guide to Linux Commands, Editors, and Shell Programming Click => Practical Guide to Linux Commands

File and Disk Management

• List files and directories: ls -al
• Change directory: cd /path/to/dir
• Copy file or directory: cp source destination
• Move or rename file or directory: mv source destination
• Delete file: rm filename
• Delete directory: rm -r directoryname
• Display disk usage: df -h
• Display directory space usage: du -sh /path/to/directory
• Create a new empty file: touch filename
• Create a new directory: mkdir directoryname
• List file type and permissions: ls -l
• Change permissions recursively: chmod -R 755 directoryname
• Change owner recursively: chown -R user:group directoryname
• Check disk usage of files and directories: du -ah /path/to/directory
• Find largest files/directories in directory: du -hsx * | sort -rh | head -10
• List open files by processes: lsof
• List mounted filesystems: mount | column -t
• Check filesystem for errors: fsck /dev/sda1
• Format a disk: mkfs -t ext4 /dev/sdx
• Create a new swap file: fallocate -l 1G /swapfile && mkswap /swapfile && swapon /swapfile
• Display free and used space on mounted filesystems: df -T
• Create a hard link to a file: ln /path/to/original /path/to/link
• Synchronize data on disk with memory: sync
• Mount a filesystem: mount /dev/sda1 /mnt/directory
• Unmount a filesystem: umount /mnt/directory
• Display disk partitions: fdisk -l
• Resize a partition: resize2fs /dev/sda1
• Create an ISO image from a directory: genisoimage -o image.iso /path/to/directory
• Benchmark disk performance: hdparm -tT /dev/sdx
• Encrypt a file system: cryptsetup luksFormat /dev/sdx
• List detailed information about file systems: df -i
• Find inodes usage for a directory: find /path/to/directory -printf '%i\n' | sort -u | wc -l
• Monitor file system changes: inotifywait -m /path
• Archive multiple directories into separate archives: tar -czvf archive_name.tar.gz -C /path/to/directory .
• Recover deleted files from a file system: extundelete /dev/sda1 --restore-all
• Adjust the amount of space reserved for system use on a filesystem: tune2fs -m 1 /dev/sda1
• Show disk usage statistics of the file system: iostat -dx /dev/sda1 2 5
• File system conversion (e.g., from ext3 to ext4): tune2fs -O extents,uninit_bg,dir_index /dev/sdX && e2fsck -fDC0 /dev/sdX
• Increase the size of an LVM partition: lvextend -L +10G /dev/mapper/vg-lv && resize2fs /dev/mapper/vg-lv
• Split a large file into multiple smaller files: split -b 100M bigfile segment_prefix
• Combine multiple files into one: cat part_aa part_ab > combined_file
• Create a bootable USB drive: dd if=/path/to/image.iso of=/dev/sdx bs=4M && sync
• Zero out free space to prepare for disk compression: cat /dev/zero > zero.fill; sync; rm zero.fill
• Check and repair a Linux file system: xfs_repair /dev/sdx1
• Resize an XFS file system: xfs_growfs /dev/sdx1
• Create and manage RAID arrays: mdadm --create --verbose /dev/md0 --level=1 --raid-devices=2 /dev/sda1 /dev/sdb1
• Display detailed disk usage by directory: ncdu /path/to/directory
• Check disk I/O statistics and throughput: iotop
• Set disk quotas for users: quotaon -u /dev/sdx1
• Restore a file system to a previous state: btrfs rollback /path/to/snapshot
• Convert existing data between different RAID levels: btrfs balance start -dconvert=raid1 -mconvert=raid1 /mnt
• Link multiple files to a single inode (hardlink entire directories): cp -rl /source /destination
• Securely erase a disk: shred -v -n 1 /dev/sdx
• Optimize the order of data on the disk: defrag /dev/sdx1
• Monitor filesystem changes using audit system: auditctl -w /path/to/directory -p warx -k keyname
• Automatically mount filesystems on boot: echo '/dev/sdx1 /mountpoint filetype defaults 0 0' >> /etc/fstab
• Clone an entire disk to another disk: dd if=/dev/sda of=/dev/sdb bs=64K conv=noerror,sync
• Show detailed partition table: parted /dev/sdx print
• Automatically adjust system swappiness setting: sysctl vm.swappiness=10
• Securely wipe free space on a filesystem: wipefs -a /dev/sdx
• View detailed SMART disk data and health: smartctl -a /dev/sdx
• Backup the MBR (Master Boot Record): dd if=/dev/sda of=/path/to/backup.mbr bs=512 count=1
• Restore the MBR (Master Boot Record): **** dd if=/path/to/backup.mbr of=/dev/sda bs=512 count=1
• Force a filesystem check on the next reboot: touch /forcefsck
• Generate a report of filesystem usage for all mounted filesystems: df -aH
• Identify files with specific permissions: find / -type f -perm 0777
• Change journaling mode on ext4 filesystems: tune2fs -O ^has_journal /dev/sdx
• Sync all buffered changes to disks: blockdev --flushbufs /dev/sdx
• Resize a non-LVM ext4 filesystem while mounted: resize2fs /dev/sdx1
• Check disk alignment for optimal performance: parted /dev/sdx align-check optimal 1
• Resize a GPT partition non-destructively: gdisk /dev/sdx
• Backup a disk image with progress indicator: pv /dev/sda > /path/to/disk.img
• Restore a disk image with progress indicator: pv /path/to/disk.img > /dev/sda
• Find modified files in the last 24 hours: find / -mtime -1
• Monitor real-time file access: inotifywatch -r /path/to/watch
• Convert an ext3 filesystem to ext4 without formatting: tune2fs -O extents,uninit_bg,dir_index /dev/sdx1 && e2fsck -pf /dev/sdx1
• Enable TRIM support on an SSD: fstrim -v /
• List detailed and real-time disk writes and reads: iotop -o
• Create a RAM disk: mount -t tmpfs -o size=1024M tmpfs /mnt/ramdisk
• Benchmark disk speed: dd if=/dev/zero of=/tmp/test1.img bs=1G count=1 oflag=dsync
• Check for unused blocks on a filesystem: fsck -fn /dev/sdx1
• Change the I/O scheduler for a disk: echo cfq > /sys/block/sda/queue/scheduler
• Report file system disk space usage as a tree: du -cha --max-depth=1 /path/to/directory | grep -E "M|G"
• Batch change file extensions: rename 's/.html/.php/' *.html
• Display disk write and read statistics per device and partition: iostat -p /dev/sda
• List UUIDs of all filesystems: blkid
• Optimize and reorganize fragmented files on a disk: e4defrag /dev/sdx1
• Display detailed network disk usage (NFS, CIFS): nfsiostat
• Create a mirrored ZFS pool: zpool create pool mirror /dev/sda /dev/sdb
• Display real-time file system access and statistics: fatrace
• Set up periodic TRIM for SSDs via a cron job: echo '0 1 * * 0 root fstrim -v /' >> /etc/crontab
• Automatically repair filesystem on boot: tune2fs -c 1 /dev/sdx1
• Secure deletion of a file beyond recovery: srm /path/to/file
• Monitor detailed file system changes in real-time: watch -n 1 df -h
• Check filesystem quotas and usage: repquota -a
• Adjust the maximum mount count between two filesystem checks: tune2fs -C 2 -c 50 /dev/sdx1

User and Group Management

• Add new user: useradd -m username
• Delete a user: userdel username
• Add new group: groupadd groupname
• Add user to a group: usermod -aG groupname username
• Change user password: passwd username
• Create a new user with a home directory and specific shell: useradd -m -s /bin/bash username
• Delete a user and remove their home directory: userdel -r username
• Modify an existing user's login name: usermod -l newusername oldusername
• Lock user accounts: usermod -L username
• Unlock a user account: usermod -U username
• Change the user's primary group: usermod -g groupname username
• Add a user to multiple groups: usermod -aG group1,group2 username
• Set or change a user's password expiration information: chage options username
• List all groups a user belongs to: groups username
• Create a new group: groupadd groupname
• Delete a group: groupdel groupname
• Change group name: groupmod -n newname oldname
• Find all files owned by a specific user: find / -user username
• Change the owner and group of a file: chown username:groupname filename
• Set the setuid bit on an executable: chmod u+s /path/to/executable
• Set the setgid bit on an executable: chmod g+s /path/to/executable
• Show all users with UID greater than 1000: awk -F':' '$3>=1000 {print $1}' /etc/passwd
• Backup /etc/passwd, /etc/shadow, and /etc/group: cp /etc/passwd /etc/shadow /etc/group /path/to/backup/
• Restore /etc/passwd, /etc/shadow, and /etc/group: cp /path/to/backup/* /etc/
• Bulk import users from a CSV file using a script:

while IFS=',' read -r userid name
do
  useradd -m -s /bin/bash -c "$name" $userid
done < users.csv

• Setup monitoring for a specific user: auditctl -a always,exit -F arch=b64 -F uid=1001 -S all -k monitor-user
• Check user last login times: lastlog
• Automatically disable inactive user accounts:

#!/bin/bash
INACTIVE_THRESHOLD_DAYS=90
today=$(date +%s)
lastlog --time $INACTIVE_THRESHOLD_DAYS | grep -vE '^Username|^#' | while read user
do
  last_login=$(date -d "$(echo $user | awk '{print $4, $5, $6}')" +%s)
  inactive_days=$(( ($today - $last_login) / 86400 ))
  if [ $inactive_days -ge $INACTIVE_THRESHOLD_DAYS ]; then
    usermod -L $user
    echo "$user has been disabled after $inactive_days days of inactivity."
  fi
done

• Manage user's shell environment globally: Edit /etc/skel/.bashrc or /etc/profile for global shell settings:

  • /etc/skel/.bashrc is used to set up the environment for every new user.
  • /etc/profile can be modified to alter the shell environment for all users.

• Use getent to handle user info in a networked environment:

  • getent passwd username
  • Retrieves the user details from the system databases including NIS, LDAP, or other mechanisms, ensuring that you see the same data that system utilities such as login or email services would see.

• Advanced group management with GIDs:

  • Set a sticky bit on a directory to manage group file creation rights: chmod g+s /path/to/directory
  • Files created in this directory inherit the group ID of the directory, not of the user who created the file.

• Create and manage user templates for different roles:

  • Utilize /etc/skel/ to create role-specific templates for users such as developers, administrators, or sales personnel, pre-configuring environment variables, aliases, and scripts specific to their roles.

• Set default permissions for new files with umask:

  • Set a global umask: echo "umask 027" >> /etc/profile
  • This command sets a default umask of 027, where newly created files will have permissions 750 and directories 750, denying write permissions to the group and any permissions to others.

• Manage user disk quotas:

  • Enable quotas: quotaon /home
  • Edit user quotas: edquota username
  • Report user quotas and usage: repquota -a
  • Disk quotas are essential for managing the disk usage of users, especially in multi-user environments or shared hosting setups.

• Setup user-specific cron jobs:

  • List cron jobs for a user: crontab -u username -l
  • Edit cron jobs for a user: crontab -u username -e
  • This allows you to manage scheduled tasks specific to each user, enabling automation of routine tasks per user basis.

• Configure password complexity and policies: Using PAM (Pluggable Authentication Modules), configure /etc/pam.d/common-password to enforce password policies like minimum length, complexity, and expiration. An example line might be:

password requisite pam_pwquality.so retry=3 minlen=10 maxlen=16 dcredit=-1 ucredit=-1 lcredit=-1 ocredit=-1

• Setup Sudo access for groups:
  • Edit sudoers file safely: visudo
  • Grant sudo access to a group: echo "%admin ALL=(ALL) ALL" >> /etc/sudoers
  • This allows all members of the 'admin' group to execute any command as any user, a crucial setup for administrative users.
• Audit user actions using auditd:
  • Set up auditing for specific commands: auditctl -a always,exit -F path=/usr/bin/passwd -F perm=x -F auid>=1000 -k password_change
  • This rule logs all executions of the passwd command by users, tagging these logs with password_change, useful for security audits.
• Change default shell for users:
  • Change shell to bash for a user: chsh -s /bin/bash username
  • This command changes the login shell for the specified user, which can be necessary for new users or when changing system configurations.
• Manage user session timeouts:
  • Edit the user profile to auto logout after inactivity: echo "TMOUT=600" >> /etc/profile
  • Automatically logs out users after 10 minutes of inactivity, enhancing security, especially on shared or public systems.
• Manage user access files:
  • Edit .bashrc, .profile for alias and environment setup: nano /home/username/.bashrc
  • Personalize user environments upon login, which is beneficial for setting up a customized working environment.
• Mass change ownership within user's home directory:
  • Change ownership of all files in a user's directory: chown -R username:group /home/username
  • Ensures that all files in the user’s home directory are owned by the user, particularly useful after copying or moving files that might retain previous ownership metadata.

Process Management

• View running processes: ps aux
• Kill a process: kill pid
• Kill a process forcefully: kill -9 pid
• Start a job in background: command &
• Bring a job to foreground: fg jobid
• Find a process by name:
  • pgrep -u username processname
  • This command searches for all processes named processname running under the user username.
• Kill all instances of a process by name:
  • pkill processname
  • Safely terminates all processes with the specified name. It's a more refined approach than using killall.
• Check the priority of running processes:
  • ps -eo pid,ni,cmd --sort=ni
  • This lists all processes with their PID, nice value (priority), and command line, sorted by the nice value.
• Change the priority of a running process:
  • renice +10 1234
  • Changes the nice value of the process with PID 1234 to 10, decreasing its priority.
• Limit the CPU usage of a process:
  • cpulimit -p 1234 -l 50
  • Restricts the process with PID 1234 to use only 50% of the CPU. This is useful for preventing resource hogging by certain processes.
• Start a process in a specific namespace:
  • unshare --net --pid --fork bash
  • Starts a bash shell with a new network and PID namespace. This is useful for testing and isolation purposes.
• Monitor all system calls made by a process:
  • strace -p 1234
  • Attaches to the process with PID 1234 and traces system calls, signals, and returns which are crucial for debugging.
• Monitor real-time CPU usage and memory by process:
  • top -b -n 1 | head -20
  • Runs top in batch mode to display the first 20 lines of the top output, showing the most resource-intensive processes.
• Show statistics about I/O operations by a process:
  • iotop -p 1234
  • Displays I/O usage information for process with PID 1234, helpful for diagnosing performance issues related to disk I/O.
• Freeze and unfreeze a process:
  • Freeze: kill -STOP 1234
  • Unfreeze: kill -CONT 1234
  • Pauses and resumes the process with PID 1234. Useful for managing CPU-intensive tasks without terminating them.
• Show a tree of processes:
  • pstree -p
  • Displays a tree of all running processes, including their PIDs, helping to visualize parent-child relationships.
• Detach a process from terminal and run it in background:
  • nohup command &
  • Runs command in the background and makes it immune to terminal hangups, useful for long-running background processes.
• Monitor changes to files by processes:
  • inotifywait -m -r /path/to/watch
  • Monitors real-time filesystem events in the specified directory or file, showing which process is making changes.
• Automatically restart a terminated process:
  • while true; do command; sleep 1; done
  • This loop continuously restarts a command if it exits for any reason, with a delay of 1 second between restarts, useful for maintaining essential services.
• Isolate and manage CPU affinity for a process:
  • taskset -cp 0,2 1234
  • Assigns the process with PID 1234 to run exclusively on CPUs 0 and 2. This is valuable for performance tuning on multi-core systems.
• Monitor and log system calls in real-time:
  • auditctl -a exit,always -F arch=b64 -S execve -k process_tracking
  • Uses the audit daemon to track execution of new processes via the execve system call, tagging logs with process_tracking for easy filtering.
• Show detailed memory usage by process:
  • smem -P processname
  • Provides detailed reports of memory usage including PSS (Proportional Set Size) and USS (Unique Set Size), which help in understanding the actual memory footprint of processes.
• Control group management to limit resource usage:
  • Create a new cgroup: cgcreate -g cpu,memory:Group1
  • Run process in cgroup: cgexec -g cpu,memory:Group1 command
  • Set resource limits: cgset -r cpu.shares=512 Group1
  • Control groups (cgroups) allow you to allocate resources—such as CPU time, system memory, network bandwidth, or combinations of these resources—among user-defined groups of tasks.
• Trace process activity in real-time:
  • stap -e 'probe process("command").function("function_name") { log("info") }'
  • SystemTap provides instrumentation for a live running kernel and user-space applications, allowing you to monitor the operation of systems in fine detail.
• Identify and respond to zombie processes:
  • List zombie processes: ps aux | awk '{ if ($8 == "Z") { print $2 }}'
  • Reap zombies: kill -9 PID
  • Zombie processes remain in the system due to parents not handling their termination correctly; identifying and removing them can free up system resources.
• Dynamic tracing of a specific process:
  • bpftrace -p 1234 -e 'tracepoint:raw_syscalls:sys_enter { printf("%d %s\n", pid, comm); }'
  • BPFtrace is a high-level tracing language for Linux eBPF programs. It's used for advanced tracing of system operations including real-time performance analysis and troubleshooting.
• Limit a process’s maximum number of open file descriptors:
  • prlimit --pid 1234 --nofile=1024:2048
  • Adjusts the limits on the number of files a process can open, with soft and hard limits, useful for managing system resources and ensuring that no single process can exhaust system file descriptors.
• Schedule a command to run relative to a process's life:
  • timeout --foreground 3600 command
  • Runs command for a maximum of one hour (3600 seconds), terminating it afterward if still running. Useful for ensuring that tasks do not overconsume resources or run indefinitely.

Text Manipulation

• oncatenate and display files: cat file1 file2
• Display the beginning of files: head -n 5 file.txt
• Display the end of files: tail -n 5 file.txt
• Search inside files: grep "search string" file
• Stream editor for filtering: sed 's/old/new/g' file
• Sed (Stream Editor) for advanced text transformations: Replace text across multiple files:

find /path/to/files -type f -exec sed -i 's/oldtext/newtext/g' {} +

• Sed (Stream Editor) for advanced text transformations: Delete lines matching a pattern:

sed -i '/pattern_to_match/d' filename

• Awk for complex data extraction and reporting: Sum a column of numbers:

awk '{sum += $1} END {print sum}' file.txt

• Awk for complex data extraction and reporting: Print lines where a field matches a pattern:

awk '$2 ~ /pattern/ {print $0}' file.txt

• Cut to extract sections from each line of files: Extract specific columns from a file:

cut -d ',' -f 1,3 file.csv

• Cut to extract sections from each line of files: Extract specific columns from a file

cut -d ',' -f 1,3 file.csv

• Sort to order or reorder text data: Sort numerically by the second column and reverse the result

sort -k2,2n -r file.txt

• Unique to report or filter out repeated lines in a file: Count unique lines in a sorted file

sort file.txt | uniq -c

• Paste to merge lines of files: Combine lines from two files side by side

paste file1.txt file2.txt

• Tr for character replacement and deletion: Convert lowercase letters to uppercase

cat file.txt | tr '[:lower:]' '[:upper:]'

• Expand and unexpand to convert between tabs and spaces: Convert tabs to spaces

expand -t 4 file.txt

• Join to join lines of two files on a common field: Join two files based on the first field

join -1 1 -2 1 file1.txt file2.txt

• Tee to read from standard input and write to standard output and files: Duplicate output to a file and console:

echo "Hello" | tee output.txt

• NL to number lines of files:

nl -s ': ' file.txt

• Using grep for complex pattern matching: Search for multiple patterns:

grep -e 'pattern1' -e 'pattern2' filename

• Using grep for complex pattern matching: Exclude lines with a pattern:

grep -v 'pattern' filename

• Append line after a match:

sed '/pattern/a "New line after match"' filename

• Modify lines in place without creating a backup:

sed -i '/pattern/s/old/new/g' filename

• Data manipulation with awk Perform arithmetic operations on fields:

awk '{print $1, $2 * $3}' file.txt

• Data manipulation with awk Filter and format output based on condition:

awk '$3 > 100 {printf "%-10s %-10s\n", $1, $3}' file.txt

• tr for character class operations: Delete all digits from the input

echo "example 123" | tr -d '0-9'

• Use tac to reverse the order of lines: Reverse the file content

tac file.txt > reversed_file.txt

• Advanced cut usage: Extract multiple disjoint ranges

cut -d ',' -f1,3-5,7 file

• Combine multiple files with custom delimiters

paste -d ',' file1.txt file2.txt > combined.txt

• Compare two sorted files and display unique lines to the first file

comm -23 file1_sorted.txt file2_sorted.txt

• Sort a file based on multiple keys:

sort -k1,1 -k2,2n file.txt

• Find duplicate lines in a file:

sort file.txt | uniq -d

• Use regex to find lines starting with a specific pattern:

grep '^start' file.txt

• Sum values from multiple files:

    awk '{sum += $1} END {print sum}' file1.txt file2.txt
    ```
    This command sums the first column across both `file1.txt` and `file2.txt`, then prints the total sum at the end.

- **Complex text transformations with `sed`:**
  - **Insert a line before every line matching a pattern:**
    ```bash
    sed '/pattern/i "New line before match"' filename
    ```
    This inserts "New line before match" before every line containing 'pattern'.

- **Filter and process log files with `awk`:**
  - **Extract specific date logs:**
    ```bash
    awk '/2023-05-12/' logfile.log
    ```
    This retrieves all entries from `logfile.log` that include the date "2023-05-12".

- **Utilize `nl` for more than just numbering lines:**
  - **Number non-empty lines only:**
    ```bash
    nl -b a file.txt
    ```
    This command numbers all lines in `file.txt`, but numbers non-empty lines only, aligning the numbers for easier readability.

- **Merge lines of files selectively with `paste`:**
  - **Alternate lines from two files:**
    ```bash
    paste -d '\n' file1.txt file2.txt
    ```
    This interleaves the lines from `file1.txt` and `file2.txt`, creating a new sequence where lines from each file alternate.

- **Advanced sorting with stability and custom fields:**
  - **Stable sort by the third field, ignoring case:**
    ```bash
    sort -f -k3,3 -s file.txt
    ```
    This sorts `file.txt` by the third column in a case-insensitive manner while preserving the order of lines that compare equal (stable sorting).

- **Using `split` for splitting text files into chunks:**
  - **Split a file into chunks based on line count:**
    ```bash
    split -l 1000 file.txt new
    ```
    This splits `file.txt` into multiple files each containing 1000 lines, with names starting with "new".

- **Detailed examination of text files with `less`:**
  - **View and search within compressed logs:**
    ```bash
    zless logfile.gz
    ```
    This opens a compressed log file in `less`, allowing for interactive searching and browsing without decompressing the file.

- **Creating reports or structured outputs using `awk`:**
  - **Generate a CSV output:**
    ```bash
    awk -F ':' '{print $1 "," $2 "," $3}' file.txt
    ```
    This converts a colon-separated file into a CSV format, selecting the first three fields.

These enhanced commands and scenarios offer you a versatile toolkit for handling complex text processing tasks, aiding in data analysis, system monitoring, and configuration management.

System Information Management

• Display current date and time: date
• Show system uptime: uptime
• Display who is logged in: who
• Show memory usage: free -m
• Display CPU information: cat /proc/cpuinfo
• View detailed CPU architecture information: lscpu
• Display real-time CPU activity and statistics: mpstat -P ALL 1
• Check hardware information from the BIOS:dmidecode
• Monitor and display USB devices and their properties:lsusb -tv
• List all block devices and their information:lsblk -a
• Show system boot and shutdown events from the logs:journalctl --list-boots
• Monitor open files by system processes:lsof
• Display detailed network interface statistics:ifstat
• View detailed memory usage and statistics:vmstat 1
• Check the availability of system and hardware sensors:sensors
• Examine and manage kernel ring buffer messages:dmesg | less
• Inspect disk usage by directories and files: ncdu /path/to/directory
• Display swap usage and management details:swapon --show
• Trace system calls and signals related to a process:strace -p PID
• Detailed process tracking with perf:perf stat -p PID
• Analyze system startup performance:systemd-analyze blame
• Display security-related system information:sestatus
• View hardware interrupts distribution across CPUs:cat /proc/interrupts
• Monitor TCP and UDP sockets in real-time:ss -tunap
• Detailed network traffic analysis:iftop
• Report detailed filesystem disk space usage:df -i
• List dynamically loaded libraries for a running process or binary:ldd /path/to/executable
• Check real-time system performance issues:htop
• Examine filesystem attributes and features:tune2fs -l /dev/sda1
• Analyze detailed CPU and memory performance:numastat
• Automate system snapshots for diagnosis:snapper create --description "pre-update"
• Track file system changes in real-time:inotifywait -m /path/to/watch

Package Management

• Update package list (Debian/Ubuntu): apt-get update
• Upgrade packages (Debian/Ubuntu): apt-get upgrade
• Install a package (Debian/Ubuntu): apt-get install packagename
• Remove a package (Debian/Ubuntu): apt-get remove packagename
• Query installed packages (RPM-based): rpm -qa
• Install a package (RPM-based): yum install packagename
• For Debian and Ubuntu (APT-based systems): Hold a package to prevent it from being automatically updated:sudo apt-mark hold package_name
• For Debian and Ubuntu (APT-based systems): List all installed packages and their versions:dpkg -l
• For Debian and Ubuntu (APT-based systems): Find which package owns a file:dpkg -S /path/to/file
• For Debian and Ubuntu (APT-based systems): Add a PPA (Personal Package Archive) and install a package from it:

sudo add-apt-repository ppa:user/ppa-name
sudo apt-get update
sudo apt-get install package_name

• For Debian and Ubuntu (APT-based systems): Upgrade all packages with new configurations non-interactively: sudo DEBIAN_FRONTEND=noninteractive apt-get -y -o Dpkg::Options::="--force-confnew" upgrade
• For Red Hat, CentOS, and Fedora (RPM-based systems with YUM or DNF): List all installed packages sorted by installation date: rpm -qa --last
• For Red Hat, CentOS, and Fedora (RPM-based systems with YUM or DNF): Check for and display detailed security updates: yum updateinfo list security or dnf updateinfo list security
• For Red Hat, CentOS, and Fedora (RPM-based systems with YUM or DNF): Roll back an update: yum history undo last or dnf history undo last
• For Red Hat, CentOS, and Fedora (RPM-based systems with YUM or DNF): Install a local RPM file resolving dependencies from repositories: yum localinstall /path/to/package.rpm or dnf install /path/to/package.rpm
• For Red Hat, CentOS, and Fedora (RPM-based systems with YUM or DNF): Enable or disable multiple repositories at once: yum-config-manager --enable repo1 repo2 or dnf config-manager --set-enabled repo1 repo2

General Tips Across Distributions:

• Find packages that no longer have dependencies:
  • deborphan (Debian/Ubuntu) or package-cleanup --leaves (RPM-based)
  • Identifies orphaned packages that were installed as dependencies and are no longer required by any installed packages.
• Automate package updates with a cron job:
  • Example crontab entry to update at 2 AM daily

0 2 * * * apt-get update && apt-get -y upgrade

• Verify all installed packages for integrity:
  • debsums (Debian/Ubuntu) or rpm -Va (RPM-based)
  • Checks the integrity of installed packages, verifying that files have not been corrupted or tampered with.

Network Configuration and Monitoring

• Display all network interfaces: ifconfig
• Display routing table: route
• Ping a host: ping hostaddress
• Scan network and ports: nmap -sP 192.168.0.0/24
• Configure IP addresses and routes dynamically with ip command: Add an IP address to an interface:

sudo ip addr add 192.168.1.100/24 dev eth0

• Configure IP addresses and routes dynamically with ip command: Delete an IP address from an interface:

sudo ip addr del 192.168.1.100/24 dev eth0

• Add a new routing table entry:

sudo ip route add 192.168.2.0/24 via 192.168.1.1 dev eth0

Network Monitoring Tools:

• Monitor all network traffic by protocol:
  • sudo tcpdump -i eth0
  • Captures and displays all packets on the network interface eth0. This is useful for debugging network issues or monitoring network activity for unauthorized access.
• Real-time network interface monitoring with iftop:
  • sudo iftop -i eth0
  • Displays bandwidth usage on an interface eth0 in real time, providing insights into which hosts are consuming the most bandwidth.
• Analyze network latency and packet loss with mtr:
  • mtr google.com
  • Combines the functionality of traceroute and ping to provide a detailed, real-time report of the network path to a specified host, along with network latency and packet loss.

Advanced Configuration Techniques:

• Set up Network Bonding:
  • Configure bonding driver:

echo "alias bond0 bonding" | sudo tee /etc/modprobe.d/bonding.conf
echo "options bonding mode=4 miimon=100" | sudo tee -a /etc/modprobe.d/bonding.conf

- **Create a bond interfaces configuration:**

sudo ifconfig bond0 192.168.1.5 netmask 255.255.255.0 up

• Manage Network Bridge:
  • Set up a bridge interface:

sudo ip link add name br0 type bridge
sudo ip link set dev eth0 master br0
sudo ip addr add 192.168.1.101/24 brd + dev br0
sudo ip link set dev br0 up

Security Monitoring:

• Detect ARP spoofing with arpwatch:
  • sudo arpwatch -i eth0
  • Monitors ARP traffic on a network interface eth0, reporting any changes, which can help detect ARP spoofing attacks.

Text Searching

• Find files by name: find / -name filename
• Search for a string in files: grep "string" /path/*
• Advanced pattern search with ack: ack "pattern"

Advanced awk Usage for Text Searching:

• Search for records with specific field conditions:
  • awk '$1 == "status" && $2 >= 200 {print $0}' filename
  • Searches for lines where the first field is 'status' and the second field is 200 or greater.
• Pattern matching across multiple lines:
  • awk '/start_pattern/,/end_pattern/' filename
  • Prints all records from 'start_pattern' to 'end_pattern', useful for extracting logs or data blocks.

Using sed for Selective Text Searching and Manipulation:

• Print only lines matching a specific pattern:
  • sed -n '/pattern/p' filename
  • Searches for 'pattern' and prints only those lines, suppressing other output.
• Delete lines matching a specific pattern:
  • sed '/pattern/d' filename
  • Removes all lines that match 'pattern' from the output.

Combining Tools for Enhanced Searches:

• Combine find and grep for deep directory searches:
  • find /path/to/dir/ -type f -exec grep -H 'pattern' {} \;
  • Finds files under /path/to/dir/ and executes grep on each to search for 'pattern', displaying files where the pattern occurs.
• Use sort and uniq for frequency analysis:
  • grep 'pattern' filename | sort | uniq -c | sort -nr
  • Searches for 'pattern', sorts the results, counts unique occurrences, and sorts them numerically in reverse for frequency analysis.

Utilizing ripgrep (rg) for High-Performance Searches:

• Recursive search with ripgrep:
  • rg 'pattern' /path/to/dir/
  • A modern alternative to grep that is faster and includes recursive and ignore pattern capabilities by default.
• Search for lines not matching a pattern with ripgrep:

  • rg -v 'pattern' filename

  • Searches within 'filename' for lines that do not contain 'pattern'.

  • grep -rnw '/path/to/dir/' -e 'pattern'

  • Searches for 'pattern' recursively within the directory /path/to/dir/, displaying the line numbers and file names where the pattern is found.

• Search for multiple patterns (OR condition):
  • grep -e 'pattern1' -e 'pattern2' filename
  • Searches for lines that match either 'pattern1' or 'pattern2' in 'filename'.
• Exclude specific patterns during searches:
  • grep 'pattern' filename | grep -v 'exclude_pattern'
  • Searches for 'pattern' but excludes lines that contain 'exclude_pattern'.
• Use Perl-compatible regular expressions (PCRE):
  • grep -P '^\d{3}\-\d{2}\-\d{4} filename
  • Uses Perl regex to match specific patterns, such as a Social Security number format.

System Monitoring

• Monitor real time system performance: top
• Interactive process viewer: htop
• Monitor disk I/O: iostat
• Monitor network: netstat

Log Monitoring and Management:

• Real-time log monitoring with tail and grep:
  • tail -f /var/log/syslog | grep 'error'
  • Monitors the system log in real time for new entries containing the word "error," helping to spot issues as they occur.
• Using logwatch for daily log analysis:
  • logwatch --detail high --range 'today' --service all
  • Generates a detailed report of the day's logs from all services, helping you to quickly review what happened on the server each day.

Resource Usage and Process Tracking:

• Using sar for historical performance data:
  • sar -u 1 3
  • Samples system CPU activity three times at one-second intervals. sar can also report on I/O, network traffic, and other vital statistics, storing them for later analysis.
• Advanced memory usage reporting with smem:
  • smem -r
  • Provides a report of memory usage that includes Proportional Set Size (PSS), which better reflects the actual memory used in a shared memory environment.

Advanced Monitoring Tools:

• Using sysdig for deep system analysis:
  • sudo sysdig -c topprocs_cpu
  • Captures and displays system calls and other system events. sysdig can filter, decode, and display the information so that you can pinpoint performance or security issues.
• Network packet analysis with tcpdump:
  • sudo tcpdump -i eth0 -n -s 0 port 80
  • Captures and displays header information of packets on a network interface, filtered by port (e.g., HTTP traffic on port 80). Useful for low-level network debugging or security monitoring.

Alerting and Notification Systems:

• Setting up monit for automatic system recovery:
  • Configure monit to monitor services like Apache, MySQL, and custom applications. Monit can automatically restart these services if they crash or become unresponsive.
• Automate monitoring with cron and alerting scripts:

  • A typical setup might involve a cron job that runs a script every hour to check the health of a service or system metric, sending an alert (via email, SMS, or a messaging system) if certain thresholds are exceeded.

  • htop

  • An interactive process viewer that provides a detailed, color-coded overview of CPU, memory, swap usage, and process details. It allows you to manage processes (e.g., killing or renicing) directly within the interface.

• Monitor disk input/output with iotop:
  • sudo iotop
  • Tracks disk I/O usage by process, which is crucial for diagnosing performance bottlenecks related to disk activity.
• Network traffic monitoring with nethogs:
  • sudo nethogs eth0
  • Monitors network traffic used by each process in real-time, sorted by interface. Useful for identifying processes that are using significant network bandwidth.

Others

• Change file permissions: chmod 755 filename
• Change file owner: chown user:group filename
• Create symbolic link: ln -s /path/to/file /path/to/symlink
• Archive files: tar -czvf archive.tar.gz /path/to/directory
• Extract archive: tar -xzvf archive.tar.gz

Conclusion

Navigating the complexities of Linux system management requires a robust set of tools and an understanding of deep system commands. From manipulating text and managing packages to configuring networks and monitoring system performance, the commands and techniques discussed provide the necessary foundation for advanced Linux administration. By integrating these practices into your daily management routines, you can enhance system efficiency, improve security, and ensure that you are equipped to handle the challenges of managing modern Linux environments. Stay curious, keep learning, and use these advanced techniques to take your Linux system administration skills to the next level.

Earn Money by Reviewing Apps on Your Phone

Looking for a way to earn some extra cash? Check out WriteAppReviews.com! You can get paid to review apps on your phone. It’s a simple and fun way to make money from the comfort of your home.

Get Paid To Use Facebook, Twitter and YouTube

Check out payingsocialmediajobs.com! Online Social Media Jobs That Pay $25 - $50 Per Hour. No Experience Required. Work At Home.

Start Working & Earning Online

Discover how to become an 'Online Assistant' and get paid to do freelance work, tasks & projects from home on behalf of companies.

7 Minutes Daily - New Work From Home Offer

Ordinary People Are Generating Online Paychecks With Just 7 Minutes A Day!

Affiliate Disclosure

This blog contains affiliate links.