Build a working mental model of Linux, from the kernel and the shell to files, permissions, processes, systemd, performance, containers, and troubleshooting.
Use ← → / Space to navigate · I contents · O overview · T theme · F fullscreen
ls -l /etc?One short command touches the shell, the process model, system calls, the filesystem, and the terminal.
ls as an alias, function, builtin, then in PATH.type -a ls shows what will actually run.fork(); the child calls execve("/usr/bin/ls").ls asks the kernel via openat, getdents64, and statx.strace ls -l /etc prints every call and its result.ls writes to file descriptor 1 (stdout).ls formats differently when stdout is not a terminal.wait()s, reaps the child, and stores the status in $?.Click any line to jump there · press I to return to this page
The kernel, the boot process, and the shell you will use for everything else.
What Linux actually is, and how a machine gets from power-on to a login prompt.
The CPU enforces a privilege boundary; programs request kernel services through a narrow interface.
read, write, openat, execve, clone, mmap, socket.ENOENT or EACCES.Most behavior you debug on Linux is either a system call result or a kernel policy decision. strace shows the first; /proc, /sys, and dmesg reveal the second.
Knowing which component provides a feature tells you where to look when it breaks.
| Component | Common examples | Role |
|---|---|---|
| Kernel | Linux 5.x / 6.x | Hardware, processes, memory, filesystems, network |
| C library | glibc, musl | System call wrappers, DNS resolution, locale, threads |
| Init system | systemd (PID 1) | Starts services, manages dependencies, collects logs |
| Core utilities | GNU coreutils, BusyBox | ls, cp, cat, chmod, and friends |
| Shell | bash, dash, zsh | Interactive command line and scripting |
| Package manager | apt, dnf, apk | Installs, upgrades, and verifies software |
Alpine uses musl and BusyBox, so a binary or a command option that works on Ubuntu may behave differently in an Alpine container.
Distributions differ in packaging, defaults, release cadence, and support lifetime.
| Family | Examples | Packages | Notes |
|---|---|---|---|
| Debian | Debian, Ubuntu | .deb via apt | Ubuntu LTS: 5 years standard support |
| Red Hat | RHEL, Rocky, AlmaLinux, Fedora, Amazon Linux 2023 | .rpm via dnf | SELinux enforcing by default (AL2023: permissive) |
| Alpine | Alpine Linux | .apk via apk | musl + BusyBox; very small container images |
| Arch | Arch Linux | pacman | Rolling release |
cat /etc/os-release # distribution name and version
uname -r # running kernel version
hostnamectl # host, OS, kernel, architectureContainer images share the host kernel: uname -r inside a container reports the host’s kernel, while /etc/os-release reports the image’s distribution.
Each stage hands control to the next; knowing the order tells you where a boot failure lives.
root=UUID=….default.target./etc/fstab entries, brings up network and services.getty on consoles, sshd for remote access.cloud-init to apply user data and keys.Inspect a boot with journalctl -b, dmesg -T, systemd-analyze blame, and cat /proc/cmdline.
Files, devices, and kernel state are reached through paths and file descriptors.
/dev/dev/nvme0n1./dev/tty, /dev/null, /dev/urandom./proc: processes and kernel state./sys: devices, drivers, and kernel objects./proc/meminfo asks the kernel, not a disk.A file descriptor is a small integer that refers to an open file, pipe, socket, or device. 0 = stdin, 1 = stdout, 2 = stderr. See yours with ls -l /proc/$$/fd.
Most kernel behavior is adjustable at runtime; persistence needs a configuration file.
uname -r # kernel version
cat /proc/cmdline # boot parameters
sysctl net.ipv4.ip_forward # read
sudo sysctl -w vm.swappiness=10 # change now (lost on reboot)
echo 'vm.swappiness = 10' | sudo tee /etc/sysctl.d/90-tuning.conf
sudo sysctl --system # reload all sysctl files
lsmod | head # loaded modules
modinfo overlay # module details
sudo modprobe br_netfilter # load a module
dmesg -T | tail # kernel messagessysctl keys map to files under /proc/sys: vm.swappiness is /proc/sys/vm/swappiness.sysctl -w is temporary; files in /etc/sysctl.d/ persist across reboots./etc/modules-load.d/ loads them at boot.dmesg shows the kernel ring buffer: hardware errors, OOM kills, filesystem problems.Editing the kernel line in GRUB is the standard way into a broken system.
# GRUB menu: press e, edit the "linux" line
linux /vmlinuz-6.8.0 root=UUID=… ro systemd.unit=rescue.target
# Ctrl-X (or F10) boots once with the change
cat /proc/cmdline # running kernel parameters
# make a change permanent
# Debian/Ubuntu: edit /etc/default/grub
# (GRUB_CMDLINE_LINUX), then:
sudo update-grub
# RHEL family:
sudo grubby --update-kernel=ALL \
--args="console=ttyS0,115200"
sudo grubby --info=DEFAULT| Parameter | Effect |
|---|---|
systemd.unit=rescue.target | Single-user mode with basic services |
systemd.unit=emergency.target | Minimal shell, root mounted read-only |
rd.break | Stop inside the initramfs (RHEL root-password reset) |
init=/bin/bash | Skip systemd; remount root with mount -o remount,rw / |
console=ttyS0,115200 | Kernel output to the serial console (cloud) |
fsck.mode=force | Force a filesystem check at boot |
Anyone with console access can use these to become root. Protect the console and, where required, set a GRUB password.
The kernel reports hardware, memory, filesystem, and network problems that applications never log.
$ dmesg -T --level=err,warn | tail # human timestamps, problems only
$ journalctl -k -b -p warning # same, from the journal
# example lines (abridged)
[Thu Oct 16 10:02:11] EXT4-fs error (device nvme1n1): ext4_find_entry: inode #2: reading directory lblock 0
[Thu Oct 16 10:05:42] Out of memory: Killed process 4121 (java) total-vm:8123456kB, anon-rss:3912345kB
[Thu Oct 16 10:07:03] nf_conntrack: table full, dropping packet
[Thu Oct 16 10:09:30] I/O error, dev sdb, sector 123456 op 0x0:(READ)
[Thu Oct 16 10:11:15] nfs: server fs-01.internal not responding, still tryingEXT4-fs error: filesystem corruption or a failing device; may lead to a read-only remount.Out of memory: Killed process: the OOM killer chose a victim; note the name and RSS.table full, dropping packet: connection tracking is full — new connections silently fail.I/O error, dev …: the disk or volume returned errors; check health and the cloud console.not responding, still trying: an NFS mount is hung; processes using it sit in D state.dmesg -T times can drift after suspend; the journal is more reliable.Wrong time breaks TLS, Kerberos, one-time passwords, and log correlation.
timedatectl # local time, UTC, time zone, NTP status
sudo timedatectl set-timezone UTC
chronyc tracking # offset from reference, stratum
chronyc sources -v # configured servers and reachability
# /etc/chrony.conf (RHEL) or /etc/chrony/chrony.conf (Debian/Ubuntu)
server 169.254.169.123 prefer iburst # Amazon Time Sync Service
pool 2.pool.ntp.org iburst
date -u; date +%s # UTC and Unix epoch secondsManual pages are versioned with the software installed on this machine.
| Section | Contents |
|---|---|
| 1 | User commands: man 1 ls |
| 2 | System calls: man 2 open |
| 3 | Library functions: man 3 printf |
| 5 | File formats: man 5 crontab, man 5 fstab |
| 7 | Overviews: man 7 signal, man 7 namespaces |
| 8 | Administration: man 8 mount, man 8 ip |
man -k keyword (or apropos) searches page descriptions.man: /pattern searches, n jumps to the next match, q quits.cmd --help gives a quick summary; help cd documents bash builtins.tldr cmd (if installed) shows common examples.crontab(1) vs crontab(5).The interface you will use for everything else in this deck.
Most shell surprises come from expansion happening before the command runs.
PATH.cd, export, and source must be builtins: they change the shell itself.type -a cmd and command -v cmd show what will run.bash is the usual interactive shell on servers./bin/sh is dash on Debian and Ubuntu, not bash.[[ ]] or arrays fail under plain sh.echo $SHELL shows your login shell; ps -p $$ shows the shell you are actually running.
Only exported variables are copied into child processes.
NAME=world # shell variable (no spaces around =)
echo "Hello, $NAME"
export APP_ENV=prod # exported to child processes
env | grep APP_ # show environment
APP_ENV=dev ./run.sh # set only for this one command
PATH="$HOME/bin:$PATH" # prepend a directory
unset NAME
source ./env.sh # run in the current shell
./env.sh # runs in a child; changes are lost~/.bash_profile (or ~/.profile); interactive non-login shells read ~/.bashrc.ps.Every process starts with three file descriptors; the shell can rewire them before running a command.
| Syntax | Meaning |
|---|---|
cmd > out.txt | Write stdout to a file (truncate first) |
cmd >> out.txt | Append stdout |
cmd 2> err.txt | Write stderr to a file |
cmd > all.log 2>&1 | stdout to file, then stderr to the same place |
cmd &> all.log | Bash shorthand for both streams |
cmd < in.txt | Read stdin from a file |
cmd1 | cmd2 | stdout of cmd1 becomes stdin of cmd2 |
cmd | tee file | Write to the screen and a file at the same time |
cmd 2>/dev/null | Discard error messages |
Order matters: cmd 2>&1 > f sends stderr to the terminal, because stderr copies stdout before stdout is redirected to the file.
Scripts, &&, ||, if, and CI pipelines all depend on exit codes.
grep -q ERROR app.log && echo "found errors"
mkdir /data || echo "mkdir failed"
false; echo "exit=$?" # exit=1
if systemctl is-active --quiet nginx; then
echo "nginx is running"
fi
curl -fsS https://example.com > /dev/null
echo $? # non-zero on HTTP >= 400 with -f| Code | Meaning |
|---|---|
| 0 | Success |
| 1–2 | General error / misuse (convention) |
| 126 | Found but not executable |
| 127 | Command not found |
| 128 + N | Killed by signal N |
| 130 | Ctrl-C (SIGINT = 2) |
| 137 | SIGKILL (9), often OOM kill |
| 143 | SIGTERM (15) |
A pipeline returns the exit code of its last command. set -o pipefail makes it fail if any stage fails.
Unquoted expansions are split on whitespace and then treated as glob patterns.
"double": expands $var and $(cmd), prevents splitting.'single': completely literal."$var" unless you deliberately want splitting.rm $file with file="my report.txt" tries to remove two files.* any string, ? one character, [a-z] a set.{a,b} and {1..5} generate words even if no files exist.nullglob is set.$(cmd) inserts command output; prefer it to backticks.$((n + 1)) performs integer arithmetic.${var:-default} uses a default when unset or empty.${file%.log} strips a suffix; ${path##*/} keeps the basename.Logs and configuration are text; these two tools answer most “where is it?” questions.
grep -rn "timeout" /etc/nginx/ # recursive, with line numbers
grep -i "error" app.log | tail -n 20 # case-insensitive
grep -E " 5[0-9]{2} " access.log | wc -l # count HTTP 5xx lines
grep -v '^#' sshd_config | grep -v '^$' # drop comments and blanks
grep -C 3 "panic" app.log # 3 lines of context
find /var/log -name "*.log" -mtime +7 # older than 7 days
find / -xdev -type f -size +500M 2>/dev/null # big files, one fs
find /srv -type f -perm -o+w # world-writable files
find . -name "*.tmp" -print0 | xargs -0 rm -f # safe with spacesgrep -E enables extended regular expressions; -F matches fixed strings quickly.find tests are combined with an implicit AND; -o means OR.-xdev stops find from crossing into other mounted filesystems.-print0 with xargs -0 handles file names with spaces or newlines.find … -print first; add -delete only after checking the list.Small tools combine in pipelines to answer questions about large files.
# top 10 client IPs in an nginx access log
awk '{print $1}' access.log | sort | uniq -c | sort -rn | head
# status-code distribution (field 9 in combined log format)
awk '{print $9}' access.log | sort | uniq -c | sort -rn
# regular users from /etc/passwd
awk -F: '$3 >= 1000 {print $1, $7}' /etc/passwd
sed -n '100,120p' big.log # print a line range
sed -i.bak 's/^Port 22$/Port 2222/' sshd_config # edit with backup
cut -d, -f1,3 data.csv # select CSV columns
sort -k2,2n file # numeric sort on column 2
tail -F app.log | grep --line-buffered ERRORawk splits each line into fields $1, $2, … on whitespace or -F.uniq only merges adjacent duplicates, so sort first.sed -i edits in place; the .bak suffix keeps a backup.tail -F keeps following a log even after rotation.jq rather than regular expressions.A few habits save hours on remote servers.
Ctrl-R searches history; !! repeats the last command.!$ reuses the last argument; Alt-. inserts it.Ctrl-A/Ctrl-E jump to line start/end; Ctrl-U clears.less pages output: / search, n next, G end, F follow.watch -n 2 cmd reruns a command and shows changes.column -t aligns whitespace-separated output.cmd | less -S stops long lines from wrapping.tmux or screen keeps work running if SSH drops.tmux new -s work, detach with Ctrl-B d, reattach with tmux a -t work.nohup cmd & survives logout but loses interactive control.A short template prevents the most common scripting accidents.
#!/usr/bin/env bash
set -euo pipefail
LOG_DIR="${1:?usage: $0 <log-dir>}" # required argument
tmp="$(mktemp -d)"
trap 'rm -rf "$tmp"' EXIT # always clean up
log() { printf '%s %s\n' "$(date -Is)" "$*" >&2; }
for f in "$LOG_DIR"/*.log; do
[[ -e "$f" ]] || continue # glob matched nothing
log "compressing $f"
gzip -c -- "$f" > "$tmp/$(basename "$f").gz"
done
mv "$tmp"/*.gz /backup/ || { log "backup failed"; exit 1; }-e: stop on an unhandled error; -u: unset variables are errors; pipefail: pipelines fail if any stage fails.${1:?msg} stops with a usage message when the argument is missing.trap … EXIT cleans up temporary files on success or failure.-- protects against file names starting with -.set -e is ignored inside if conditions and &&/|| lists, so still check critical steps.shellcheck script.sh before committing.Bash has several test syntaxes; [[ ]] is the safest for scripts that require bash.
if [[ -f /etc/app.conf && -r /etc/app.conf ]]; then
echo "config readable"
fi
[[ -z "${VAR:-}" ]] && echo "VAR is empty or unset"
[[ "$file" == *.log ]] # glob match (right side unquoted)
[[ "$ver" =~ ^v[0-9]+\.[0-9]+$ ]] && echo "${BASH_REMATCH[0]}"
(( count > 10 )) && echo "many" # arithmetic comparison
case "${1:-}" in
start|stop) systemctl "$1" myapp ;;
status) systemctl status myapp ;;
*) echo "usage: $0 {start|stop|status}" >&2; exit 2 ;;
esac| Test | True when |
|---|---|
-e path | Exists (any type) |
-f / -d / -L | Regular file / directory / symlink |
-s file | Exists and is not empty |
-r / -w / -x | Readable / writable / executable by you |
-z / -n str | String is empty / not empty |
a -nt b | File a is newer than file b |
-eq -ne -lt -gt | Integer comparison inside [ ] / [[ ]] |
Inside [[ ]], == with an unquoted right side is a glob match, and </> compare strings. Use (( )) or -lt for numbers.
Most automation is a loop over hosts, files, or lines with a function that retries.
for host in web1 web2 web3; do
ssh -o ConnectTimeout=5 "$host" uptime || echo "$host failed" >&2
done
while IFS= read -r line; do # read a file line by line
printf '%s\n' "$line"
done < hosts.txt
servers=(web1 web2 "db 1") # indexed array
echo "${#servers[@]} servers"
for s in "${servers[@]}"; do echo "$s"; done
declare -A port=([http]=80 [https]=443) # associative array
echo "${port[https]}"
retry() { # retry with exponential backoff
local n=0
until "$@"; do
(( ++n >= 5 )) && return 1
sleep $(( 2 ** n ))
done
}
retry curl -fsS http://localhost:8080/healthread -r keeps backslashes; IFS= keeps leading and trailing spaces.cmd | while read … runs the loop in a subshell, so variables set inside are lost; use done < <(cmd)."${arr[@]}" expands each element as one word, even with spaces.$(…).local inside functions to avoid overwriting global variables.These features remove temporary files and make scripts easier to read.
cat <<EOF > /tmp/info.txt # variables expand
host=$(hostname) user=$USER
EOF
cat <<'EOF' > /usr/local/bin/x.sh # quoted: written literally
echo "$HOME is not expanded here"
EOF
grep -c ERROR <<< "$output" # here-string
diff <(sort a.txt) <(sort b.txt) # process substitution
comm -13 <(sort old) <(sort new) # lines only in new
exec > >(tee -a /var/log/job.log) 2>&1 # log the rest of a script
{ date; df -h; free -m; } > report.txt # group output
mkfifo /tmp/pipe # named pipe between processes<<EOF expands variables; <<'EOF' writes the text exactly — use it for scripts and configs.<<-EOF strips leading tabs (not spaces) so heredocs can be indented.<(cmd) gives a command’s output a temporary file name such as /dev/fd/63.exec > >(tee …) sends all later output to the screen and a log file.{ …; } groups commands without starting a subshell; ( … ) starts one.grep, sed, and awk use slightly different regex dialects.
| Pattern | Meaning |
|---|---|
. | Any single character |
^ / $ | Start / end of line |
* | Zero or more of the previous item |
+ ? {2,4} | One or more / optional / range (ERE; BRE: \{2,4\}, GNU also \+) |
[abc] / [^0-9] | One of / none of these characters |
[[:space:]], [[:digit:]] | POSIX classes, portable across tools |
(a|b) | Group and alternation (ERE) |
\d, lookahead | PCRE only: grep -P (GNU tools also accept \s, \w) |
grep 'ab\+c' file # BRE
grep -E 'ab+c' file # ERE
grep -P '\d{3}-\d{4}' file # PCRE (GNU grep)
grep -oE '[0-9]{1,3}(\.[0-9]{1,3}){3}' access.log | sort -u
sed -E 's/(user)=([^ ]+)/\1=***/g' app.log # mask values
awk '/ERROR|FATAL/ {n++} END {print n+0}' app.log
grep -F 'a.b[1]' file # fixed string: no regex at allCloud CLIs, Kubernetes, and modern logs emit JSON; jq turns it into answers.
# instance ID and state as tab-separated text
aws ec2 describe-instances |
jq -r '.Reservations[].Instances[] | [.InstanceId, .State.Name] | @tsv'
# pods that are not Running
kubectl get pods -A -o json |
jq -r '.items[] | select(.status.phase != "Running")
| "\(.metadata.namespace)/\(.metadata.name)"'
# error lines from JSON-lines logs, compact output
jq -c 'select(.level == "error") | {time, msg}' app.jsonl | head
# count by field
jq -r '.status' events.jsonl | sort | uniq -c | sort -rn
jq '.replicas = 3' deploy.json > deploy.new.json # edit a value.field selects; .[] iterates over an array.| pipes values between filters, like the shell.select(cond) keeps matching values.-r prints raw strings without quotes; @tsv/@csv format rows.-c prints one compact JSON value per line.aws --query, kubectl -o jsonpath.From inodes and mounts to RAID, LVM, encryption, NFS, and backups.
Where data lives, how names map to data, and why disks fill up.
The Filesystem Hierarchy Standard (FHS) explains most paths you will see on a server.
| Path | Contents |
|---|---|
/etc | Host-specific configuration |
/usr | Installed programs and libraries (read-mostly) |
/boot | Kernel, initramfs, boot loader files |
/dev | Device files |
/proc, /sys | Kernel interfaces (virtual) |
/run | Runtime state: PID files, sockets (tmpfs) |
| Path | Contents |
|---|---|
/var | Changing data: logs, caches, spools, databases |
/var/log | Log files |
/home, /root | User and root home directories |
/tmp | Temporary files; may be tmpfs and cleaned |
/opt | Self-contained third-party software |
/srv | Data served by this host |
Many distributions have merged /bin, /sbin, and /lib into /usr (“usrmerge”); the old paths remain as symlinks.
An inode holds metadata and data locations; directories map names to inode numbers.
ls -i shows inode numbers; stat file shows details.ln target newnameln -s target link; readlink -f link resolves it.Because data is freed only when nothing holds the inode, deleting a large log file that a process still has open does not free the space.
ls -lThe first character is the file type; three timestamps answer different questions.
| Char | Type |
|---|---|
- | Regular file |
d | Directory |
l | Symbolic link |
c | Character device |
b | Block device |
p | Named pipe (FIFO) |
s | Unix socket |
stat)| Name | Changes when |
|---|---|
| mtime | File content is modified |
| ctime | Inode changes: content writes or metadata (mode, owner, links) — not creation time |
| atime | File is read (often reduced by relatime/noatime) |
| btime | Birth time, if the filesystem records it |
Linux has one directory tree; each filesystem is mounted at a directory in it.
lsblk -f # disks, partitions, fs types, UUIDs
findmnt # mount tree
findmnt /var # what is mounted at /var?
sudo mount /dev/nvme1n1 /data
sudo umount /data # fails if a process is using it
sudo fuser -vm /data # who is using it?
# /etc/fstab
# device mountpoint type options dump pass
UUID=3f1c...a9 /data xfs defaults,nofail 0 2
sudo findmnt --verify # check fstab before rebooting
sudo mount -a # mount everything in fstabUUID= or labels in fstab: device names like /dev/sdb can change.nofail lets the system boot even if a data disk is missing.noexec, nosuid, nodev, ro.Local, virtual, layered, and network filesystems all use the same VFS interface.
| Type | Typical use | Notes |
|---|---|---|
| ext4 | General-purpose default (Debian, Ubuntu) | Mature; can grow online, shrink offline |
| xfs | Default on RHEL family; large files | Grows online; cannot be shrunk |
| btrfs | Snapshots, subvolumes, checksums | Copy-on-write; default on Fedora desktops |
| tmpfs | /run, /dev/shm, often /tmp | Stored in RAM/swap; lost on reboot |
| overlay | Container image layers | Union of read-only lower and writable upper layers |
| nfs | Shared storage, e.g. AWS EFS | Network latency; hangs if the server disappears |
| proc, sysfs | Kernel interfaces | No disk storage; contents generated on read |
Storage is layered: device → partition table → (optional LVM) → filesystem → mount.
/dev/sda, /dev/nvme0n1p1, /dev/xvda.lsblk shows the tree; blkid shows UUIDs and types.parted, fdisk, gdisk.pvs, vgs, lvs summarize each layer.lvextend -r -L +10G /dev/vg0/data grows the LV and filesystem.sudo growpart /dev/nvme0n1 1 extends partition 1.sudo resize2fs /dev/nvme0n1p1sudo xfs_growfs / (takes the mount point).Take a snapshot before resizing or repartitioning production disks.
“No space left on device” (ENOSPC) has more than one cause.
df -h # space per filesystem
df -i # inode usage per filesystem
# largest directories, staying on one filesystem
sudo du -xh --max-depth=1 / 2>/dev/null | sort -h | tail
sudo lsof +L1 # deleted files still held open
sudo du -sh /var/lib/docker /var/log/journal
journalctl --disk-usage
sudo journalctl --vacuum-size=500Mdf -h near 100%. Find big directories with du.df -i near 100% while df -h looks fine — millions of small files.df is full but du cannot find it. Restart or signal the process holding the file.df but invisible to du.logrotate handles classic log files; journald manages its own storage.
# /etc/logrotate.d/myapp
/var/log/myapp/*.log {
daily
rotate 14
compress
delaycompress
sharedscripts
missingok
notifempty
create 0640 myapp myapp
postrotate
systemctl kill -s HUP myapp.service
endscript
}
sudo logrotate -d /etc/logrotate.d/myapp # dry runcreate + postrotate: rename the file and tell the app to reopen its log.copytruncate: copy then truncate in place — works for apps that cannot reopen, but can lose lines.delaycompress keeps the newest rotated file readable.SystemMaxUse= in /etc/systemd/journald.conf.Choose the compression and copy tool by speed, ratio, and resumability.
tar -czf backup.tgz /etc/nginx # create gzip archive
tar -tzf backup.tgz | head # list contents
tar -xzf backup.tgz -C /tmp/restore # extract elsewhere
tar --zstd -cf data.tar.zst /srv/data
rsync -avh --progress src/ host:/dst/ # trailing / matters
rsync -avh --delete src/ dst/ # mirror (deletes extras!)
scp file.txt user@host:/tmp/
sha256sum backup.tgz > backup.tgz.sha256
sha256sum -c backup.tgz.sha256| Tool | Speed | Ratio |
|---|---|---|
| gzip | Fast | Moderate; universal |
| zstd | Very fast | Good; tunable levels |
| xz | Slow | High |
| bzip2 | Slow | Moderate; legacy |
rsync src/ dst/ copies the contents of src; rsync src dst/ creates dst/src. Test --delete with -n (dry run) first.
stat shows the metadata that permission checks, backups, and build tools rely on.
$ stat /etc/hosts
File: /etc/hosts
Size: 221 Blocks: 8 IO Block: 4096 regular file
Device: 259,1 Inode: 1048611 Links: 1
Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root)
Access: 2026-10-16 09:12:44.123456789 +0700
Modify: 2026-09-30 14:02:10.000000000 +0700
Change: 2026-09-30 14:02:10.000000000 +0700
Birth: 2026-09-30 14:02:10.000000000 +0700
$ stat -c '%n %U:%G %a %s bytes' /etc/hosts
/etc/hosts root:root 644 221 bytesSize is the file length; Blocks counts 512-byte units actually allocated — sparse files have fewer.Device and Inode uniquely identify the file on this host.Links is the hard-link count; for a directory it includes . and subdirectories.Access: (0644/…) shows the mode in octal and symbolic form.stat -c with format codes is ideal for scripts.One command shows how disks, partitions, LVM, and mounts fit together.
$ lsblk -f
NAME FSTYPE UUID FSAVAIL FSUSE% MOUNTPOINTS
nvme0n1
├─nvme0n1p1 xfs 3f1c…a9 12.4G 38% /
└─nvme0n1p128 vfat 7B77-95E7 8.7M 13% /boot/efi
nvme1n1 LVM2_member aZ3k…Qp
└─data-app xfs 9e2d…11 80.1G 20% /srv/app
$ findmnt -o TARGET,SOURCE,FSTYPE,OPTIONS /srv/app
TARGET SOURCE FSTYPE OPTIONS
/srv/app /dev/mapper/data-app xfs rw,noatime,attr2,inode64,noquotaLVM2_member marks a physical volume; its logical volumes appear underneath./dev/mapper/vg-lv and /dev/vg/lv (both names work).MOUNTPOINTS column means the filesystem is not mounted.findmnt shows the options actually in effect, which may differ from fstab.RAID, LVM, repair, encryption, network storage, and backups.
RAID keeps a system running through disk failure; it does not protect against deletion or corruption.
| Level | Min disks | Usable capacity | Survives | Typical use |
|---|---|---|---|---|
| RAID 0 | 2 | 100% (n disks) | None: any failure loses data | Scratch space, maximum throughput |
| RAID 1 | 2 | 50% (mirror) | All but one disk of the mirror | Boot and OS disks |
| RAID 5 | 3 | n − 1 disks | One disk | Read-heavy storage; slow rebuilds on large disks |
| RAID 6 | 4 | n − 2 disks | Two disks | Large arrays where rebuilds take long |
| RAID 10 | 4 | 50% | One disk per mirror pair | Databases: good random write performance |
Cloud block volumes are already replicated inside one availability zone. Software RAID 0 across volumes is sometimes used for throughput; protection still comes from snapshots and backups.
Linux md RAID is mature, portable between machines, and visible in /proc/mdstat.
sudo mdadm --create /dev/md0 --level=1 --raid-devices=2 /dev/sdb /dev/sdc
cat /proc/mdstat # state and resync progress
sudo mdadm --detail /dev/md0
# replace a failed disk
sudo mdadm --manage /dev/md0 --fail /dev/sdb --remove /dev/sdb
sudo mdadm --manage /dev/md0 --add /dev/sdd # rebuild starts
# persist the array definition
sudo mdadm --detail --scan | sudo tee -a /etc/mdadm/mdadm.conf # Debian
# RHEL: /etc/mdadm.conf
sudo update-initramfs -u # RHEL: sudo dracut -f[UU] in /proc/mdstat means both mirror members are up; [U_] means one is missing.recovery = 42.0% shows a rebuild in progress — performance is reduced until it finishes.mdadm --monitor (or the mdmonitor service) to email or alert on failures.LVM separates filesystems from physical disks so capacity can change without downtime.
sudo pvcreate /dev/sdb /dev/sdc # 1. mark disks as PVs
sudo vgcreate data /dev/sdb /dev/sdc # 2. pool them in a VG
sudo lvcreate -n app -L 50G data # 3. carve a 50 GB LV
sudo mkfs.xfs /dev/data/app
sudo mount /dev/data/app /srv/app
sudo vgextend data /dev/sdd # add a disk to the pool
sudo lvextend -r -l +100%FREE data/app # grow LV + filesystem
sudo pvmove /dev/sdb # move data off a disk
sudo vgreduce data /dev/sdb # then remove it
sudo pvs; sudo vgs; sudo lvs -a -o +devices # summaries-L 50G sets an absolute size; -l +100%FREE uses all free extents.-r (--resizefs) resizes the filesystem in the same step.pvmove migrates data online, useful for replacing a disk.A classic snapshot keeps original copies of blocks that change after it is taken, so it starts small and grows.
# classic snapshot with 5 GB of space for changes
sudo lvcreate -s -n app-snap -L 5G data/app
sudo lvs data # Data% shows how full the snapshot is
# consistent backup from the snapshot
sudo mkdir -p /mnt/snap
sudo mount -o ro,nouuid /dev/data/app-snap /mnt/snap # nouuid: xfs only
sudo tar -czf /backup/app.tgz -C /mnt/snap .
sudo umount /mnt/snap && sudo lvremove -y data/app-snap
# roll back: merge the snapshot into the origin
# deferred to next activation if the origin is in use
sudo lvconvert --merge data/app-snap
# thin provisioning
sudo lvcreate --type thin-pool -L 100G -n pool data
sudo lvcreate -V 50G --thin -n vol1 data/poolData%.Swap gives the kernel somewhere to move cold memory; it cannot fix a workload that needs more RAM.
swapon --show # active swap areas
free -h
sudo fallocate -l 2G /swapfile # or: dd if=/dev/zero ...
sudo chmod 600 /swapfile
sudo mkswap /swapfile
sudo swapon /swapfile
echo '/swapfile none swap sw 0 0' | sudo tee -a /etc/fstab
sysctl vm.swappiness # 60 is a common default
vmstat 1 # si/so: pages swapped in/outsi/so) means the working set does not fit in memory.zram creates compressed swap in RAM; Fedora enables it by default.Repair tools must run on unmounted filesystems; running them on a mounted one causes damage.
dmesg -T | grep -iE 'ext4-fs error|xfs.*(corrupt|shutdown)'
findmnt -o TARGET,OPTIONS /srv # did it become "ro"?
sudo umount /srv
sudo e2fsck -f /dev/sdb1 # ext4: force a full check
sudo xfs_repair -n /dev/sdb1 # xfs: dry run, report only
sudo xfs_repair /dev/sdb1 # xfs: repair
sudo tune2fs -l /dev/sdb1 | grep -iE 'state|errors|mount count'
# root filesystem: boot with fsck.mode=force, or repair from rescue mediaerrors=remount-ro turns read-only on errors to prevent more damage.dmesg mean hardware, not the filesystem.xfs_repair reports a dirty log, mount and unmount once to replay it; use -L only as a last resort (it discards the log).LUKS encrypts a block device; the filesystem on top works normally once unlocked.
sudo cryptsetup luksFormat /dev/sdb1 # destroys existing data
sudo cryptsetup open /dev/sdb1 secure # → /dev/mapper/secure
sudo mkfs.ext4 /dev/mapper/secure
sudo mount /dev/mapper/secure /secure
sudo cryptsetup luksDump /dev/sdb1 # version, key slots, cipher
sudo cryptsetup luksAddKey /dev/sdb1 # add a second passphrase/key
sudo cryptsetup luksHeaderBackup /dev/sdb1 --header-backup-file sdb1.hdr
# /etc/crypttab name device key options
secure UUID=8a1e… /etc/keys/sdb1.key luks
# /etc/fstab
/dev/mapper/secure /secure ext4 defaults,nofail 0 2systemd-cryptenroll), or network unlock.NFS shares a directory over the network; AWS EFS is NFSv4.1.
# server: /etc/exports
/srv/share 10.0.0.0/24(rw,sync,root_squash)
sudo exportfs -ra # apply changes
sudo exportfs -v # active exports
# client
sudo mount -t nfs4 -o hard,nconnect=4 fs01:/srv/share /mnt/share
# /etc/fstab
fs01:/srv/share /mnt/share nfs4 hard,_netdev,nofail,nconnect=4 0 0
nfsstat -m # mount options in effect
nfsstat -c # client RPC stats, retransmissionshard (default) retries forever: processes hang in D state when the server is gone, but data stays safe.soft returns errors after timeouts and can corrupt data for writers.root_squash maps client root to nobody; UIDs must match across hosts._netdev waits for the network at boot; nconnect opens several TCP connections (kernel 5.3+).ESTALE): the file or export changed on the server; remount.Snapshots, replicas, and RAID protect against different failures than backups do.
export RESTIC_REPOSITORY=s3:s3.amazonaws.com/my-backups/web1
export RESTIC_PASSWORD_FILE=/etc/restic/password
restic init # once
restic backup /etc /srv/app --exclude-caches
restic snapshots
restic forget --keep-daily 7 --keep-weekly 4 --keep-monthly 6 --prune
restic check # verify repository integrity
restic restore latest --target /tmp/restore --include /etc/nginx
# databases: dump first, then back up the dump
pg_dump -Fc mydb > /backup/mydb.dumpControl who can do what, then harden the host against misuse.
Who can do what, and how to grant only what is needed.
The kernel checks numeric UIDs and GIDs; names are looked up in databases such as /etc/passwd.
# /etc/passwd name:x:UID:GID:comment:home:shell
alice:x:1001:1001:Alice:/home/alice:/bin/bash
nginx:x:998:998::/var/lib/nginx:/usr/sbin/nologin
id alice # uid, gid, groups
getent passwd alice # works with LDAP/SSSD too
sudo useradd -m -s /bin/bash bob
sudo usermod -aG docker bob # -a: append, keep other groups
sudo passwd -l bob # lock password login
sudo userdel -r bob # remove user and homeUID_MIN in /etc/login.defs)./etc/shadow, readable only by root.nologin as their shell.newgrp).-rwxr-x--- correctlyThree classes (user, group, other) × three rights (read, write, execute).
-rwxr-x--- 1 app app 4096 Oct 16 10:00 deploy.sh
└┬┘└┬┘└┬┘
│ │ └─ other: --- 0
│ └──── group: r-x 4+1 = 5
└─────── user: rwx 4+2+1 = 7 → mode 750
chmod 640 config.yml # rw-r-----
chmod u+x,g-w script.sh # symbolic form
chmod -R u=rwX,go= private/ # X: execute only on dirs
chown app:app /srv/app
chown -R app: /srv/app # group = app's login groupr: read content; w: modify content; x: execute.r: list names; x: enter and access entries by name.w (with x): create, rename, and delete entries./a/b/c, you need x on every directory in the path.These features explain many “why does this file have that permission?” questions.
| Feature | Numeric | Effect | Example |
|---|---|---|---|
| umask | 022, 027 | Removes bits from new files/dirs | 022 → files 644, dirs 755 |
| setuid | 4xxx | Executable runs as the file’s owner | /usr/bin/passwd |
| setgid (file) | 2xxx | Executable runs with the file’s group | Rare today |
| setgid (dir) | 2xxx | New files inherit the directory’s group | Shared team directories |
| sticky (dir) | 1xxx | Only owners can delete their files | /tmp is drwxrwxrwt |
| ACL | + in ls -l | Extra per-user/group entries | setfacl -m u:bob:r file |
Audit setuid binaries with find / -xdev -perm -4000 -type f. An unexpected setuid file is a security finding.
sudo runs a command as another user (root by default) according to rules in sudoers.
visudo; it checks syntax before saving./etc/sudoers.d/.deploy ALL=(root) /usr/bin/systemctl restart myapp.sudo -l lists what you are allowed to run.sudo -i starts a root login shell; sudo -s keeps more of your environment.sudo -u postgres psql runs as another user./var/log/auth.log / /var/log/secure.sudo echo x > /etc/file fails: your shell, not root, opens the file.echo x | sudo tee /etc/file (or tee -a to append).Root is split into capabilities; Mandatory Access Control adds rules even root must follow.
CAP_NET_BIND_SERVICE for ports below 1024.CAP_SYS_ADMIN is very broad — treat it like root.getcap /usr/bin/ping; setcap cap_net_bind_service=+ep ./server.AmbientCapabilities= and CapabilityBoundingSet=.getenforce, ls -Z, ps -Z show mode and labels.ausearch -m avc -ts recent.restorecon -Rv /path; toggle booleans with setsebool -P.aa-status lists loaded profiles and modes.Do not disable SELinux or AppArmor to make something work. Read the denial, then fix the label, boolean, or profile.
SSH provides encrypted shells, file transfer, and tunnels; configure both client and server deliberately.
ssh-keygen -t ed25519 -C "alice@laptop"
ssh-copy-id alice@server
# ~/.ssh/config
Host bastion
HostName 203.0.113.10
User alice
Host app-*
ProxyJump bastion
User ec2-user
ssh -L 5432:db.internal:5432 bastion # local port forward
ssh -D 1080 bastion # SOCKS proxyPasswordAuthentication no and PermitRootLogin no in sshd_config.sudo sshd -t before reloading.~/.ssh and authorized_keys must not be group/world-writable (use 700/600), or sshd ignores them.Each rule says who may run what, on which host, as which user.
# sudo visudo -f /etc/sudoers.d/deploy
# who host=(as user) [tag:] commands
deploy ALL=(root) NOPASSWD: /usr/bin/systemctl restart myapp, \
/usr/bin/systemctl --no-pager status myapp
%ops ALL=(ALL:ALL) ALL
Cmnd_Alias PKG = /usr/bin/apt-get update, /usr/bin/apt-get upgrade
alice ALL=(root) PKG
Defaults use_pty
Defaults logfile=/var/log/sudo.log
Defaults:deploy !requiretty
$ sudo -l -U deploy # what deploy may run
$ sudo visudo -c # syntax check all files%name refers to a group; ALL=(ALL:ALL) ALL is full root.less, find -exec) grant a root shell — hence --no-pager above.sudoedit to let users edit specific files safely./etc/sudoers.d/ must not contain . or end in ~, or they are skipped.Reduce attack surface, enforce policy, and detect change.
Hardening is a repeatable process, not a one-time list of tweaks.
systemctl list-unit-files --state=enabled.ss -tulpn should hold no surprises.oscap) or Lynis (lynis audit system).Apply hardening through code (Ansible, image builds) so every new host starts compliant and drift is visible.
Use a drop-in file and validate before reloading.
# /etc/ssh/sshd_config.d/10-hardening.conf
PermitRootLogin no
PasswordAuthentication no
KbdInteractiveAuthentication no
PubkeyAuthentication yes
AuthenticationMethods publickey
MaxAuthTries 3
LoginGraceTime 30
AllowGroups ssh-users
X11Forwarding no
AllowAgentForwarding no
ClientAliveInterval 300
ClientAliveCountMax 2
$ sudo sshd -t # syntax check
$ sudo sshd -T | grep -i passwordauth # effective value
$ sudo systemctl reload ssh # RHEL: sshdInclude of sshd_config.d/*.conf sits near the top of the main file, so drop-ins win.sshd -T prints the effective configuration — trust it over reading files.AllowGroups makes access explicit; everyone else is refused.Match blocks.An SSH certificate authority signs short-lived user keys; servers trust the CA, not each key.
# once: create the user CA (keep the private key offline or in a vault)
ssh-keygen -t ed25519 -f user_ca -C "user CA"
# sign alice's key for 8 hours, allowed principals: alice, deploy
ssh-keygen -s user_ca -I alice@corp -n alice,deploy -V +8h id_ed25519.pub
# → id_ed25519-cert.pub (ssh uses it automatically)
ssh-keygen -L -f id_ed25519-cert.pub # inspect validity and principals
# servers: /etc/ssh/sshd_config.d/20-ca.conf
TrustedUserCAKeys /etc/ssh/user_ca.pub
AuthorizedPrincipalsFile /etc/ssh/principals/%ussh-keygen -h) remove “trust on first use” prompts.RevokedKeys; tools like Vault or Teleport automate issuing.PAM decides how login, sudo, sshd, and other services authenticate users and set up sessions.
# /etc/pam.d/sshd (excerpt, varies by distribution)
auth required pam_faillock.so preauth
auth include common-auth # Debian; RHEL: password-auth
account required pam_nologin.so
account include common-account
password include common-password
session required pam_limits.so
session optional pam_motd.so
ls /etc/pam.d/ # one file per service
man pam.conf # syntaxauth (who are you), account (are you allowed now), password (change secrets), session (set up and clean up).required (must pass, keep going), requisite (stop on failure), sufficient (success ends the stack), optional.pam_limits applies /etc/security/limits.conf — why it affects logins but not systemd services.authselect on RHEL, pam-auth-update on Debian) instead of hand-editing shared stacks.Even key-only SSH hosts have local passwords for console and sudo.
# /etc/security/faillock.conf
deny = 5
unlock_time = 900
sudo faillock --user alice # failures
sudo faillock --user alice --reset # unlock
# /etc/security/pwquality.conf
minlen = 14
minclass = 3
sudo chage -l alice # aging information
sudo chage -E 2026-12-31 contractor # account expiry date
sudo usermod -L olduser # lock the password
sudo usermod -s /usr/sbin/nologin olduser| Field | Meaning |
|---|---|
| 2 | Hash: $y$ yescrypt, $6$ SHA-512; ! or * = locked |
| 3 | Last change (days since 1970-01-01) |
| 4 / 5 | Minimum / maximum days between changes |
| 6 | Warning days before expiry |
| 7 | Inactive days allowed after expiry |
| 8 | Account expiration date |
Current NIST guidance favors long passwords and MFA over forced periodic rotation. Remove or expire accounts as soon as people leave.
Put these in /etc/sysctl.d/ and test them; some interfere with routers, containers, or debugging.
| Setting | Value | Purpose |
|---|---|---|
kernel.kptr_restrict | 2 | Hide kernel addresses from users |
kernel.dmesg_restrict | 1 | Only privileged users can read the kernel log |
kernel.yama.ptrace_scope | 1 | Processes can only trace their own children |
kernel.unprivileged_bpf_disabled | 1 | Block eBPF for unprivileged users |
fs.protected_symlinks / _hardlinks | 1 | Stop link attacks in shared directories like /tmp |
fs.suid_dumpable | 0 | No core dumps from setuid programs |
net.ipv4.tcp_syncookies | 1 | Survive SYN floods |
net.ipv4.conf.all.accept_redirects | 0 | Ignore ICMP redirects |
net.ipv4.conf.all.rp_filter | 1 (strict) / 2 (loose) | Drop spoofed sources; strict breaks asymmetric routing |
Container and Kubernetes hosts need net.ipv4.ip_forward = 1; do not copy a desktop hardening profile onto a node blindly.
Allow only what the host serves, and keep established connections working.
# /etc/nftables.conf (RHEL: /etc/sysconfig/nftables.conf)
flush ruleset
table inet filter {
chain input {
type filter hook input priority 0; policy drop;
ct state established,related accept
ct state invalid drop
iif "lo" accept
meta l4proto { icmp, ipv6-icmp } accept
tcp dport 22 ip saddr 10.0.0.0/8 accept
tcp dport { 80, 443 } accept
}
}
sudo nft -c -f /etc/nftables.conf # check syntax only
sudo systemctl enable --now nftables
sudo nft list rulesetinet tables cover IPv4 and IPv6 together.ufw, or firewalld — not several.fail2ban watches logs for repeated failures and temporarily blocks the source address.
# /etc/fail2ban/jail.local
[DEFAULT]
bantime = 1h
findtime = 10m
maxretry = 5
ignoreip = 127.0.0.1/8 10.0.0.0/8
[sshd]
enabled = true
backend = systemd
sudo systemctl enable --now fail2ban
sudo fail2ban-client status sshd
sudo fail2ban-client set sshd unbanip 203.0.113.50jail.local, not jail.conf, so upgrades keep your settings.backend = systemd reads the journal on hosts without /var/log/auth.log.ignoreip to avoid locking yourself out.The Linux audit system logs file access, command execution, and identity changes in the kernel.
# /etc/audit/rules.d/50-custom.rules
-w /etc/passwd -p wa -k identity
-w /etc/shadow -p wa -k identity
-w /etc/sudoers -p wa -k sudoers
-w /etc/sudoers.d/ -p wa -k sudoers
-w /etc/ssh/sshd_config -p wa -k sshd
-a always,exit -F arch=b64 -S execve -F euid=0 -k root-exec
# /etc/audit/rules.d/99-finalize.rules (loaded last)
-e 2
sudo augenrules --load # build and load the rules
sudo auditctl -l # rules in effect
sudo ausearch -k identity -i --start today
sudo aureport --auth --summary-w path -p wa: watch writes and attribute changes on a file or directory.-k tags events so ausearch -k can find them.-S execve -F euid=0: log every program run as root.-e 2 (last file, own line) makes rules immutable until reboot; audit rules allow no trailing comments.Know when system files change, and never leave credentials where other processes can read them.
aide --init builds a database of hashes and attributes (Ubuntu: aideinit).aide --check reports added, removed, and changed files.rpm -Va or debsums -s.ps./proc/<pid>/environ and often dumped in crash logs.HISTCONTROL=ignorespace.0600 owned by the service user.LoadCredential=db:/etc/myapp/db.pass → $CREDENTIALS_DIRECTORY/db.Most real-world compromises use known vulnerabilities with available fixes.
# Debian / Ubuntu
apt list --upgradable
sudo unattended-upgrade --dry-run -d
cat /var/run/reboot-required 2>/dev/null
# RHEL family / Amazon Linux 2023
dnf updateinfo list --security
sudo dnf upgrade --security
sudo needs-restarting -r # reboot needed?
sudo needs-restarting -s # services to restart
# containers and images
trivy image myapp:1.4.2
trivy fs --scanners vuln /srv/appProcesses, services, and the packages they come from.
How programs run, communicate, and stop.
The same program can run as many processes, each with its own identity and resources.
ulimit) and scheduling priority./proc/<pid>/.ps -eLf or top -H shows threads.New processes are copies of their parent that then load a new program.
bash (PID 100)
└─ fork() → child PID 101 (copy of bash)
└─ execve("/usr/bin/grep", …) → now grep
└─ exit(0)
→ zombie until bash calls wait()
→ bash reads status into $?
parent exits first?
→ child is reparented to PID 1 (or a subreaper)fork() cheap: memory is copied only when written.wait() — fix the parent.tini or docker run --init.The state explains whether a process is working, waiting, or stuck.
| State | Meaning | What it suggests |
|---|---|---|
| R | Running or runnable | Using or waiting for CPU |
| S | Interruptible sleep | Waiting for an event: normal for idle daemons |
| D | Uninterruptible sleep | Usually waiting on I/O (disk, NFS); cannot be killed; counts toward load |
| Z | Zombie | Exited, not yet reaped by its parent |
| T | Stopped | Paused by Ctrl-Z, SIGSTOP, or a debugger |
| I | Idle kernel thread | Normal; ignore |
ps -eo pid,ppid,stat,etime,cmd --forest | less
ps -eo stat,pid,cmd | awk '$1 ~ /^D/' # processes stuck in D stateExtra STAT letters: s session leader, l multithreaded, + foreground, < high priority, N low priority.
Signals stop, reload, or interrupt processes; handlers decide most of the behavior.
| Signal | No. | Typical meaning |
|---|---|---|
| SIGHUP | 1 | Terminal closed; many daemons reload config |
| SIGINT | 2 | Ctrl-C |
| SIGQUIT | 3 | Ctrl-\; may dump core |
| SIGKILL | 9 | Kill immediately; cannot be caught |
| SIGTERM | 15 | Polite stop request (default for kill) |
| SIGCHLD | 17 | A child exited |
| SIGSTOP / SIGCONT | 19 / 18 | Pause / resume; STOP cannot be caught |
kill PID sends SIGTERM; use kill -9 only as a last resort.docker stop sends SIGTERM, then SIGKILL after 10 s by default.terminationGracePeriodSeconds (30 s default) before SIGKILL.CMD npm start runs under sh -c, which may not forward signals; use exec form.Start broad, then drill into one PID.
ps aux --sort=-%mem | head # top memory users
ps aux --sort=-%cpu | head # top CPU users
pgrep -af nginx # PIDs + full command lines
top -o %MEM # or htop
pidstat -u -r 1 5 # per-process CPU and memory
cat /proc/<pid>/status | grep -E 'State|VmRSS|Threads'
ls -l /proc/<pid>/fd | wc -l # open file descriptors
sudo lsof -p <pid> # files and sockets
sudo ss -tnp | grep <pid> # its TCP connections
pstree -p <pid> # children%CPU in ps is a lifetime average; top and pidstat show current rates.top keys: P sort by CPU, M by memory, 1 per-CPU, H threads.lsof reveals which process holds a port, a deleted file, or a mount point.The scheduler shares CPUs between runnable tasks; niceness and I/O class adjust the share.
cmd & runs in the background; jobs lists jobs.Ctrl-Z stops the foreground job; bg resumes it in the background, fg brings it back.nohup cmd & ignores SIGHUP; disown detaches a job from the shell.setsid cmd starts it in a new session.nice -n 10 ./batch.sh, sudo renice -n 5 -p <pid>.ionice -c3 cmd runs at idle I/O priority.ionice -c2 -n7 is best-effort, lowest level.IOReadBandwidthMax= in systemd).Every tool on the previous slides reads these files; you can too.
| Path | What you learn |
|---|---|
/proc/<pid>/cmdline | Full command line (NUL-separated): tr '\0' ' ' < … |
/proc/<pid>/environ | Environment at start — may contain secrets; root/owner only |
/proc/<pid>/status | State, UIDs, VmRSS, threads, capabilities |
/proc/<pid>/fd/ | Open file descriptors as symlinks |
/proc/<pid>/limits | Effective resource limits of that process |
/proc/<pid>/cgroup | Which cgroup (service or container) owns it |
/proc/loadavg, /proc/meminfo | Load and memory as the kernel sees them |
/proc/pressure/{cpu,memory,io} | Pressure Stall Information (PSI) |
The header summarizes the whole system; the table ranks processes.
top - 10:42:01 up 12 days, 3:04, 2 users, load average: 3.12, 2.80, 2.41
Tasks: 241 total, 2 running, 239 sleeping, 0 stopped, 0 zombie
%Cpu(s): 35.2 us, 4.1 sy, 0.0 ni, 58.9 id, 1.2 wa, 0.0 hi, 0.6 si, 0.0 st
MiB Mem : 15832.4 total, 1210.5 free, 4920.3 used, 9701.6 buff/cache
MiB Swap: 2048.0 total, 2048.0 free, 0.0 used. 10589.1 avail Mem
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
4121 app 20 0 8123456 3.7g 28432 S 142.0 23.9 812:33.10 java
981 postgres 20 0 411220 98120 92340 S 6.3 0.6 40:12.55 postgresload average over 1, 5, 15 minutes; compare with CPU count.avail Mem is the realistic free memory, including reclaimable cache.VIRT reserved address space, RES resident memory, SHR shared part of RES.%CPU 142 = about 1.4 CPUs across several threads.TIME+ is total CPU time consumed since start.PR/NI show scheduling priority and nice value.Most columns answer a specific question about the process.
$ ps aux
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
root 1 0.0 0.1 171820 13140 ? Ss Oct04 1:02 /usr/lib/systemd/systemd
app 4121 4.7 23.9 8123456 3880000 ? Sl Oct04 812:33 java -jar api.jar
app 5010 0.0 0.0 0 0 ? Z 10:40 0:00 [worker] <defunct>
root 612 0.0 0.0 0 0 ? D 09:58 0:04 [kworker/u8:2+flush-259:0]
alice 7710 0.0 0.0 9012 3360 pts/0 R+ 10:43 0:00 ps auxVSZ and RSS are in KiB.TTY ? = no terminal (daemon); pts/0 = an SSH or terminal session.Ss = sleeping session leader; Sl = sleeping, multithreaded; R+ = running in the foreground.<defunct> with Z is a zombie — look at its parent with ps -o ppid= -p 5010.[brackets] are kernel threads; a kworker in D points at I/O (here a flush to device 259:0).%CPU is averaged over the whole lifetime, unlike top.Choosing and inspecting the channel helps when one side stops responding.
cmd1 | cmd2: an anonymous pipe with a kernel buffer (64 KiB by default).SIGPIPE.mkfifo creates a named pipe in the filesystem./run/docker.sock, /run/postgresql/.s.PGSQL.5432.ss -xlp lists listening Unix sockets and owners./dev/shm (tmpfs) and System V shared memory: ipcs -m.127.0.0.1 when components may later move to other hosts.Access to a control socket such as /run/docker.sock or /run/containerd/containerd.sock is equivalent to root on the host.
Start, supervise, and log long-running software.
systemd starts services in dependency order, supervises them, and records their logs.
.service: a process to run and supervise..timer: scheduled activation (cron replacement)..socket, .mount, .path: activation by socket, mount, or file change..target: a named group of units, e.g. multi-user.target./usr/lib/systemd/system/: shipped by packages./etc/systemd/system/: administrator files and overrides (wins)./run/systemd/system/: runtime units, lost on reboot.systemctl cat name shows the effective file and drop-ins.Wants= / Requires=: also start these units.After= / Before=: ordering only.systemctl list-dependencies name shows the tree.Starting a service and enabling it at boot are different actions.
| Command | Effect |
|---|---|
systemctl status myapp | State, PID, recent log lines, exit code |
systemctl start | stop | restart myapp | Change state now |
systemctl reload myapp | Ask the service to reload config without restarting |
systemctl enable --now myapp | Start at boot and start now |
systemctl daemon-reload | Re-read unit files after editing them |
systemctl edit myapp | Create a drop-in override in /etc |
systemctl --failed | List units that failed |
systemctl mask myapp | Prevent it from being started at all |
systemctl is-active / is-enabled myapp | Script-friendly checks via exit status |
Run as an unprivileged user, restart on failure, and harden the sandbox.
# /etc/systemd/system/myapp.service
[Unit]
Description=My API
After=network-online.target
Wants=network-online.target
StartLimitIntervalSec=300
StartLimitBurst=5
[Service]
Type=simple
User=myapp
WorkingDirectory=/opt/myapp
EnvironmentFile=/etc/myapp/env
ExecStart=/opt/myapp/bin/server --port 8080
Restart=on-failure
RestartSec=5
LimitNOFILE=65536
NoNewPrivileges=yes
ProtectSystem=strict
ReadWritePaths=/var/lib/myapp
PrivateTmp=yes
[Install]
WantedBy=multi-user.targetType=simple: the started process is the service. notify waits for readiness; oneshot for scripts; forking for legacy daemons.Restart=on-failure restarts after crashes but not after a clean stop.StartLimitBurst → “start request repeated too quickly”; systemctl reset-failed clears it.ProtectSystem=strict makes the filesystem read-only except ReadWritePaths=.systemd-analyze security myapp scores the sandbox.The journal stores structured log entries with metadata such as unit, PID, and boot ID.
journalctl -u myapp -f # follow one unit
journalctl -u myapp --since "10 min ago"
journalctl -u myapp -n 100 --no-pager
journalctl -p err -b # errors since this boot
journalctl -b -1 -e # end of previous boot
journalctl -k # kernel messages
journalctl _PID=1234 # one process
journalctl -u myapp -o json-pretty | less
journalctl --disk-usage-b -1) need persistent storage: /var/log/journal must exist or Storage=persistent./var/log/syslog (Debian) or /var/log/messages (RHEL).--since/--until with exact timestamps when correlating incidents.Both run jobs on a schedule; timers integrate with logging, dependencies, and resource limits.
# crontab -e (min hour day-of-month month day-of-week)
*/5 * * * * /opt/bin/sync.sh >> /var/log/sync.log 2>&1
0 3 * * 0 /opt/bin/weekly-report.sh
# /etc/systemd/system/sync.timer
[Timer]
OnCalendar=*:0/5
Persistent=true
[Install]
WantedBy=timers.target
# sync.service holds ExecStart=/opt/bin/sync.sh
systemctl enable --now sync.timer
systemctl list-timers
systemd-analyze calendar "Mon *-*-* 03:00"| cron | systemd timer | |
|---|---|---|
| Logs | Redirect yourself or mail | Journal, per unit |
| Missed runs | Skipped (anacron helps) | Persistent=true catches up |
| Environment | Minimal PATH | Defined in the unit |
| Overlap | Can start twice | Not while still running |
| Limits | None built in | cgroup limits, sandboxing |
“Works in my shell, fails in cron” is usually the minimal environment: use absolute paths and set variables explicitly.
Targets group units into system states; analysis tools show where boot time goes.
multi-user.target: normal server state.graphical.target: adds a desktop.rescue.target: single-user with basic services.emergency.target: minimal shell, root read-only.systemctl get-default / set-default multi-user.target.systemd-analyze blame: slowest units.systemd-analyze critical-chain: what blocked boot.systemd.unit=rescue.target to the kernel line in GRUB to boot into rescue./etc/fstab entry → emergency mode.The status output already answers most first questions.
$ systemctl status myapp
● myapp.service - My API
Loaded: loaded (/etc/systemd/system/myapp.service; enabled; preset: disabled)
Drop-In: /etc/systemd/system/myapp.service.d
└─override.conf
Active: activating (auto-restart) (Result: exit-code) since Thu 2026-10-16 10:41:55 +07; 3s ago
Process: 9921 ExecStart=/opt/myapp/bin/server --port 8080 (code=exited, status=1/FAILURE)
Main PID: 9921 (code=exited, status=1/FAILURE)
CPU: 412ms
Oct 16 10:41:55 web1 server[9921]: error: open /etc/myapp/env: permission deniedLoaded: the unit file path, whether it is enabled at boot, and the vendor preset.Drop-In: overrides that change the unit — check them first.activating (auto-restart): it failed and systemd will try again after RestartSec.code=exited, status=1: the program exited with 1; code=killed, status=9/KILL means a signal.Overrides live in /etc and survive package upgrades.
$ sudo systemctl edit myapp
# opens an editor and saves to
# /etc/systemd/system/myapp.service.d/override.conf
[Service]
Environment=LOG_LEVEL=debug
MemoryMax=1G
ExecStart=
ExecStart=/opt/myapp/bin/server --port 9090
$ systemctl cat myapp # unit + all drop-ins, in order
$ systemctl show myapp -p MemoryMax -p ExecStart
$ sudo systemctl restart myapp
$ sudo systemctl revert myapp # remove all local overridesExecStart= must be cleared with an empty assignment before setting a new value.systemctl edit reloads systemd for you; manual edits need daemon-reload.systemctl edit --full copies the whole unit to /etc — upgrades no longer reach it.systemctl show prints the effective values, including defaults.A few lines in a unit remove most of what a compromised service could reach.
| Directive | Effect |
|---|---|
NoNewPrivileges=yes | setuid binaries and file capabilities cannot raise privileges |
ProtectSystem=strict | Whole filesystem read-only except ReadWritePaths= |
ProtectHome=yes | /home, /root, /run/user are inaccessible |
PrivateTmp=yes | Private /tmp and /var/tmp for the service |
PrivateDevices=yes | Only pseudo devices such as /dev/null are available |
CapabilityBoundingSet= | Limit capabilities; empty removes them all |
RestrictAddressFamilies=AF_INET AF_INET6 AF_UNIX | Block other socket types |
SystemCallFilter=@system-service | Allow-list typical service syscalls (seccomp) |
DynamicUser=yes | Run as a temporary user created for the service |
MemoryMax=, CPUQuota=, TasksMax= | cgroup resource limits |
Install, update, and trust software reproducibly.
Learn the task, then the syntax for the distribution in front of you.
| Task | Debian/Ubuntu | RHEL/Fedora/AL2023 | Alpine |
|---|---|---|---|
| Refresh metadata | apt update | dnf makecache | apk update |
| Install | apt install nginx | dnf install nginx | apk add nginx |
| Search | apt search nginx | dnf search nginx | apk search nginx |
| Upgrade all | apt upgrade | dnf upgrade | apk upgrade |
| Who owns a file? | dpkg -S /usr/bin/curl | rpm -qf /usr/bin/curl | apk info --who-owns /usr/bin/curl |
| Files in a package | dpkg -L curl | rpm -ql curl | apk info -L curl |
| Which package provides? | apt-file search bin/dig | dnf provides '*/dig' | apk search cmd:dig |
| Hold a version | apt-mark hold pkg | dnf versionlock add pkg (plugin) | apk add pkg=1.2.3-r0 |
Packages are trusted because repository metadata is signed.
/etc/apt/sources.list.d/*.list or .sources, with signed-by= keyrings./etc/yum.repos.d/*.repo with gpgcheck=1.apt-key is deprecated; store keys per repository.unattended-upgrades, dnf-automatic)./var/run/reboot-required; RHEL: needs-restarting -r./var/log/apt/ or dnf history.Most executables load shared libraries at runtime through the dynamic loader.
file /usr/bin/curl # ELF type, architecture, dynamic/static
ldd /usr/bin/curl # shared libraries it needs
type -a python3 # which binary runs
readlink -f "$(command -v java)"
# error: libssl.so.3: cannot open shared object file
ls /etc/ld.so.conf.d/
sudo ldconfig # rebuild the library cache
LD_LIBRARY_PATH=/opt/lib ./app # temporary overridex86-64 binary on an aarch64 host (e.g. Graviton)./usr/local or /opt, not /usr./etc safelyConfiguration changes cause many incidents; validate before reloading.
nginx -t, sshd -t, visudo -c, apachectl configtest.systemd-analyze verify myapp.service.named-checkconf, haproxy -c -f file./etc/nginx/conf.d/, /etc/sysctl.d/, /etc/sudoers.d/.90-myapp.conf) to control order..dpkg-dist or .rpmnew files to review.cp file{,.bak}.etckeeper can version /etc in Git.Version mismatches between hosts cause “works on staging” bugs.
# Debian / Ubuntu
apt-cache policy nginx # installed, candidate, repository priority
apt-cache madison nginx # all available versions
sudo apt install nginx=1.24.0-2ubuntu7
grep " install " /var/log/dpkg.log | tail
# RHEL family
dnf --showduplicates list nginx
dnf info nginx
dnf history # transactions with IDs
dnf history info 42
sudo dnf history undo 42 # roll back a transaction
rpm -q --changelog openssl | headInstalled vs Candidate in apt-cache policy shows whether an upgrade is pending.dnf history records who changed what and when — useful after an incident.Measure before tuning: CPU, memory, disk, and limits.
uptime shows 1-, 5-, and 15-minute averages of tasks that are runnable or in uninterruptible sleep.
nproc).vmstat 1 columns r (runnable) and b (blocked)./proc/pressure/cpu, memory, io (kernel 4.20+).some avg10=12.50 = tasks stalled 12.5% of the last 10 s.Load is a symptom, not a diagnosis. Follow it with CPU breakdown, run queue, and I/O latency.
%Cpu(s) line in topWhere CPU time goes points to the next tool.
| Field | Meaning | If high, check |
|---|---|---|
| us | User-space code | Which process: top, pidstat; profile with perf top |
| sy | Kernel on behalf of processes | Syscall-heavy apps, networking, context switches |
| ni | User code at positive nice | Usually batch jobs; expected |
| id | Idle | — |
| wa | Idle while waiting for I/O | Disk latency: iostat -xz 1; not CPU work |
| hi / si | Hardware / software interrupts | Network-heavy hosts; /proc/interrupts |
| st | Steal: hypervisor gave CPU to others | Noisy neighbor, burstable instance out of credits |
mpstat -P ALL 1 shows per-CPU values: one CPU at 100% with others idle suggests a single-threaded bottleneck.
Linux uses spare RAM as page cache and gives it back when applications need it.
$ free -h
total used free shared buff/cache available
Mem: 15Gi 4.8Gi 1.2Gi 210Mi 9.0Gi 9.8Gi
Swap: 2.0Gi 0B 2.0Gi
$ vmstat 1 5 # si/so = swap in/out per second
$ grep -E 'MemAvailable|Dirty|Slab' /proc/meminfo
$ cat /proc/pressure/memoryavailable estimates memory usable without swapping — watch this, not free.buff/cache is mostly reclaimable page cache.si/so activity means memory pressure.smem, /proc/<pid>/smaps_rollup) divides them.vm.swappiness changes the balance between reclaiming cache and swapping.If reclaim fails, the kernel kills a process to keep the system alive.
oom_score based mainly on memory use.oom_score_adj (-1000 to 1000) biases the choice.OOMScoreAdjust= in the unit.dmesg -T | grep -i -E "killed process|out of memory".journalctl -k --since "1 hour ago".kubectl describe pod shows OOMKilled.memory.max) triggers OOM inside that cgroup, even if the host has free RAM.memory.events counts oom_kill events.iostat shows per-device throughput, latency, and queueing.
$ iostat -xz 1
Device r/s w/s rkB/s wkB/s r_await w_await aqu-sz %util
nvme0n1 120 850 3840.0 54400.0 0.45 12.80 10.9 98.0
$ pidstat -d 1 # which process does the I/O
$ sudo iotop -oPa # interactive, accumulated
$ cat /proc/pressure/ior_await/w_await: average ms per request including queue time — compare with the device’s normal latency.aqu-sz: average queue length; sustained high values mean saturation.%util near 100% means busy, but SSDs and NVMe serve many requests in parallel, so it is not proof of saturation.wa in top plus high await → storage is the bottleneck.A checklist from Brendan Gregg that avoids guessing.
| Resource | Utilization | Saturation | Errors |
|---|---|---|---|
| CPU | mpstat -P ALL 1, top | vmstat r > CPUs, PSI cpu | dmesg machine-check events |
| Memory | free -m (available) | vmstat si/so, PSI memory, OOM kills | dmesg (ECC, OOM) |
| Disk | iostat -xz %util | aqu-sz, await, PSI io | dmesg, smartctl -a |
| Network | sar -n DEV 1 throughput | Drops, ss -ti retransmits, backlog overflows | ip -s link errors |
| File descriptors | /proc/sys/fs/file-nr | Near ulimit -n | EMFILE / ENFILE in logs |
Errors are often the quickest win: a single dmesg line about disk resets or OOM can end an investigation.
From Brendan Gregg’s “Linux Performance Analysis in 60,000 Milliseconds” (Netflix).
uptime # load averages and trend
dmesg -T | tail # OOM kills, I/O errors, hardware
vmstat 1 # r, b, si/so, us/sy/wa/st
mpstat -P ALL 1 # per-CPU balance
pidstat 1 # per-process CPU
iostat -xz 1 # disk latency and saturation
free -m # memory and page cache
sar -n DEV 1 # network throughput per interface
sar -n TCP,ETCP 1 # connections and retransmits
top # overall cross-checkmpstat, pidstat, iostat, and sar come from the sysstat package.sar -f.“Too many open files” (EMFILE) is a limit, not a disk problem.
ulimit -n # soft limit for this shell
ulimit -Hn # hard limit
cat /proc/<pid>/limits # limits of a running process
ls /proc/<pid>/fd | wc -l # current usage
# systemd service
[Service]
LimitNOFILE=65536
TasksMax=4096
# login sessions (PAM): /etc/security/limits.d/90-app.conf
app soft nofile 65536
app hard nofile 65536
cat /proc/sys/fs/file-nr # system-wide: allocated, unused, maxlimits.conf applies to PAM logins, not to systemd services — use LimitNOFILE= there.--ulimit) and its configuration.ulimit -u and the cgroup pids.max (TasksMax=).vmstat quickly shows whether the problem is CPU, memory, or I/O.
$ vmstat 1
procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
r b swpd free buff cache si so bi bo in cs us sy id wa st
3 0 0 1210500 102400 9599200 0 0 4 220 3120 5400 30 5 64 1 0
9 0 0 1201300 102400 9600100 0 0 0 180 4410 9800 85 12 3 0 0
2 6 0 1190300 102400 9601100 0 0 8200 9100 2800 4100 10 5 40 45 0r: runnable tasks. Above the CPU count → CPU saturation (line 2).b: tasks blocked on I/O. Together with high wa → storage bottleneck (line 3).si/so non-zero and sustained → memory pressure and swapping.bi/bo: blocks read/written per second; in/cs: interrupts and context switches.st above a few percent on a VM → the hypervisor is taking CPU away.sysstat records system metrics every 10 minutes by default once collection is enabled.
sudo systemctl enable --now sysstat
# Debian/Ubuntu may also need ENABLED="true" in /etc/default/sysstat
sar -u # CPU today
sar -q # run queue and load averages
sar -r # memory
sar -d -p # disk, with device names
sar -n DEV # network per interface
sar -u -s 02:30:00 -e 03:30:00 # time window
sar -u -f /var/log/sa/sa15 # RHEL: day 15
sar -u -f /var/log/sysstat/sa15 # Debian/Ubuntu: day 15sar works even when monitoring is broken.Benchmarks give a baseline; compare them with what production actually needs.
# random 4k reads, 60 s, bypassing the page cache
fio --name=randread --filename=/data/fio.test --size=2G \
--rw=randread --bs=4k --iodepth=32 --ioengine=libaio \
--direct=1 --runtime=60 --time_based --group_reporting
# network throughput between two hosts
iperf3 -s # on the server
iperf3 -c 10.0.1.5 -P 4 -t 30 # on the client, 4 streams
# CPU and memory load
stress-ng --cpu 4 --vm 2 --vm-bytes 1G --timeout 60s --metrics-brief--direct=1 measures the device, not the page cache.Never point a fio write test at a device or file that holds data — it overwrites it.
Configure and inspect the network stack on a host.
For protocol details, see the Computer Networking A to Z deck; these are the Linux-specific pieces.
ip -br addr # interfaces and addresses
ip route # routing table
ip route get 10.0.5.20 # actual route decision
ss -tulpn # listening TCP/UDP sockets + process
ss -s # socket summary
cat /etc/resolv.conf # resolver config
resolvectl status # systemd-resolved view
getent hosts api.internal # resolve via NSS (uses /etc/hosts)
dig api.internal # asks DNS directly (ignores /etc/hosts)/etc/resolv.conf may point to 127.0.0.53, the systemd-resolved stub./etc/nsswitch.conf decides the lookup order (files, then DNS).getent and applications can disagree with dig.ip changes are temporary; persist them in the distribution’s network config.The kernel’s packet filter is used by host firewalls, Docker, and kube-proxy.
filter: INPUT, FORWARD, OUTPUT chains.nat: PREROUTING (DNAT), POSTROUTING (SNAT/masquerade).sudo iptables -L -n -v --line-numbers, sudo nft list ruleset.iptables-nft, which writes nftables rules.ufw (Ubuntu), firewalld (RHEL).ESTABLISHED,RELATED.conntrack -L, conntrack -S show entries and stats.net.netfilter.nf_conntrack_max.Docker and Kubernetes manage their own chains; edit host rules carefully and test from outside before closing your session.
Tune only after measurement shows the limit is being reached.
sysctl net.core.somaxconn # accept backlog cap
sysctl net.ipv4.ip_local_port_range # ephemeral ports
sysctl net.ipv4.tcp_tw_reuse
nstat -az | grep -i -E 'ListenOverflows|ListenDrops'
ss -ltn # Recv-Q on LISTEN = queued
ss -tan state time-wait | wc -l
ss -ti dst 10.0.5.20 # rtt, cwnd, retrans per socketRecv-Q is the current accept queue and Send-Q is its maximum.ListenOverflows means the app accepts connections too slowly or the backlog is too small.TIME_WAIT sockets are normal; reuse connections (keep-alive, pooling) before tuning.When logs disagree, packets show what actually crossed the interface.
sudo tcpdump -i any -nn port 443
sudo tcpdump -i eth0 -nn host 10.0.5.20 and tcp port 5432
sudo tcpdump -i any -nn 'tcp[tcpflags] & (tcp-syn|tcp-rst) != 0'
sudo tcpdump -i any -nn udp port 53
sudo tcpdump -i eth0 -nn -s 0 -c 10000 -w /tmp/cap.pcap
tcpdump -nn -r /tmp/cap.pcap | head # read a capture later
# inside a container's network namespace
sudo nsenter -t "$pid" -n tcpdump -i eth0 -nn port 8080-nn disables DNS and port-name lookups, which are slow and can add traffic.-i any captures on all interfaces; choose one when you need direction clarity.[S] SYN, [S.] SYN-ACK, [.] ACK, [P.] data, [F.] FIN, [R] reset.-w and open in Wireshark for deeper analysis.-c or a filter; captures may contain credentials and personal data.Containers are Linux processes with isolation and limits. Study this right before Docker.
A container is a normal process tree with kernel isolation features applied.
ps -ef on the host shows container processes.docker inspect -f '{{.State.Pid}}' name gives the host PID./proc/<pid>/cgroup shows the container’s cgroup.nsenter -t <pid> -n ip addr enters its network namespace.Each namespace type isolates one kind of global resource.
| Namespace | Isolates | Example effect |
|---|---|---|
| pid | Process IDs | The app sees itself as PID 1 |
| net | Interfaces, routes, ports, firewall rules | Own eth0 and lo; ports do not clash |
| mnt | Mount table | Own root filesystem and mounts |
| uts | Hostname and domain name | Container has its own hostname |
| ipc | System V IPC, POSIX message queues | Shared memory isolated |
| user | UID/GID mappings | Root inside maps to an unprivileged UID outside (rootless) |
| cgroup | View of the cgroup tree | Sees its own cgroup as the root |
| time | Monotonic and boot clocks (5.6+) | Rarely used; for checkpoint/restore |
lsns lists namespaces; Kubernetes pods share the network namespace between their containers, which is why they can use localhost.
Control groups account for and limit CPU, memory, I/O, and process count for a group of tasks.
cat /sys/fs/cgroup/cgroup.controllers
systemd-cgls # cgroup tree
systemd-cgtop # live usage per cgroup
# inside a cgroup directory
cat cpu.max # "200000 100000" = 2 CPUs
cat cpu.stat # nr_throttled, throttled_usec
cat memory.max # hard limit in bytes or "max"
cat memory.current # current usage
cat memory.events # high, max, oom, oom_kill
cat pids.max| Pod setting | cgroup v2 effect |
|---|---|
| CPU request | cpu.weight (relative share) |
| CPU limit | cpu.max quota → throttling |
| Memory request | Scheduling only (plus memory.min with MemoryQoS) |
| Memory limit | memory.max → OOMKilled |
| PID limit | pids.max |
A CPU limit causes throttling even when the node is idle. Check nr_throttled before blaming the application for latency.
Image layers are read-only; each container adds one writable layer on top.
$ mount | grep overlay
overlay on /var/lib/docker/overlay2/<id>/merged type overlay
(lowerdir=L1:L2:L3, # read-only image layers
upperdir=.../diff, # this container's writes
workdir=.../work)
$ docker diff mycontainer # A added, C changed, D deletedDefaults are reasonable; most risk comes from options that remove isolation.
USER in the Dockerfile).--cap-drop ALL --cap-add NET_BIND_SERVICE.no-new-privileges blocks setuid escalation.--privileged: nearly all capabilities and device access./var/run/docker.sock: equivalent to root on the host.--pid=host, --net=host: remove namespace isolation./, /etc, or /proc.See that a “container” is just kernel features applied to a process.
# Terminal A: new PID, mount, UTS, and network namespaces
sudo unshare --pid --fork --mount-proc --uts --net bash
hostname lab-box && hostname # changes only inside
ps aux # only bash and ps; bash is PID 1
ip link # only lo, and it is DOWN
# Terminal B: observe from the host
pgrep -a unshare
sudo lsns | grep bash
# Add resource limits with a transient cgroup
sudo systemd-run --scope -p MemoryMax=200M -p CPUQuota=20% bash
cat /proc/self/cgroupsystemd-run places the shell in a cgroup with memory and CPU limits.When the image has no shell or tools, use host tools against the container’s namespaces.
pid=$(docker inspect -f '{{.State.Pid}}' web)
# containerd / Kubernetes: crictl inspect <id> | jq .info.pid
sudo nsenter -t "$pid" -n ss -tlnp # host ss, container network
sudo nsenter -t "$pid" -n ip addr
sudo ls /proc/"$pid"/root/etc/ # container filesystem
sudo cat /proc/"$pid"/cgroup # 0::/system.slice/docker-<id>.scope
cg=$(sudo awk -F: '{print $3}' /proc/"$pid"/cgroup)
cat /sys/fs/cgroup"$cg"/memory.current /sys/fs/cgroup"$cg"/memory.max
sudo grep Cap /proc/"$pid"/status # capability bitmasks
capsh --decode=00000000a80425fbnsenter -n enters only the network namespace, so host binaries like ss and tcpdump still work./proc/<pid>/root exposes the container’s root filesystem.kubectl debug with an ephemeral container is the Kubernetes-native alternative.A repeatable method, real cases, labs, and review.
A repeatable method, the right tools, and real cases.
Narrow the scope before you change anything.
Start every incident with: systemctl status, journalctl -u … --since, dmesg -T | tail, df -h, free -m, uptime.
When logs are silent, system calls reveal missing files, denied access, and hangs.
strace -f -e trace=openat,execve ./app 2>&1 | grep -v ENOENT
strace -f -e trace=%file,%network -p <pid>
strace -c -p <pid> # summary: counts and time per syscall
strace -tt -T -p <pid> # timestamps + time spent in each call
strace -f -o /tmp/trace.txt systemctl start myapp
# lower-overhead alternatives (bcc / bpftrace tools)
sudo execsnoop-bpfcc # new processes system-wide
sudo opensnoop-bpfcc -x # failed opensopenat(…) = -1 ENOENT: the file the app expected does not exist.EACCES / EPERM: permissions, SELinux/AppArmor, or capabilities.connect, read, futex) shows where it hangs.strace slows the traced process significantly; use briefly in production.The same errno appears in logs, strace, and error messages.
| errno | Message | First check |
|---|---|---|
| ENOENT | No such file or directory | Path, working directory, missing interpreter (#! line), missing library |
| EACCES | Permission denied | namei -l path, mode bits, noexec mount, SELinux/AppArmor |
| EPERM | Operation not permitted | Needs root or a capability; immutable file (lsattr) |
| ENOSPC | No space left on device | df -h and df -i |
| EMFILE | Too many open files | /proc/<pid>/limits, FD leak |
| EADDRINUSE | Address already in use | ss -ltnp for the port |
| ECONNREFUSED | Connection refused | Nothing listening, or listening on another address |
| EROFS | Read-only file system | Filesystem errors forced remount read-only: dmesg |
df -h shows 100%, while du -sh /* adds up to much less.
ENOSPC.df reports the filesystem as full.du cannot find where the space went.sudo lsof +L1: deleted files still open, with size and PID.df -i: maybe inodes, not blocks, are exhausted.sudo mkdir -p /mnt/root && sudo mount --bind / /mnt/root, then du there./var/lib/docker and the journal.sudo truncate -s 0 /proc/<pid>/fd/<n>.systemctl status shows activating (auto-restart) or failed.
systemctl status myapp — exit code or signal.journalctl -u myapp -n 200 --no-pager.systemctl cat myapp — the effective unit and drop-ins.EADDRINUSE).status=9/KILL, result oom-kill) or a crash (SIGSEGV).Type= mismatch: a forking daemon declared as simple.ExecStart manually as the service user: sudo -u myapp ….daemon-reload if the unit changed, reset-failed, start.journalctl -u myapp -f.top shows high load average with high wa and low us/sy.
ps -eo stat,pid,wchan:32,cmd | awk '$1 ~ /^D/'.iostat -xz 1: high await and aqu-sz.cat /proc/pressure/io.dmesg -T: I/O errors, NFS “server not responding”.The file is 644 and owned by the right user, yet the app still gets EACCES.
namei -l /srv/app/data/config.yml shows every directory.x for the user.sudo -u app cat file.findmnt -T /path (noexec, ro).ls -Z, ausearch -m avc.getfacl; immutable flag: lsattr.x bit.restorecon -Rv for SELinux labels.USER UID with volume ownership.chmod 777: it hides the real problem and creates a new one.The pod restarts with exit code 137 and reason OOMKilled.
kubectl describe pod → Last State: OOMKilled.dmesg: “Memory cgroup out of memory: Killed process …”.memory.events shows oom_kill.emptyDir: medium: Memory) counts toward the limit.-XX:MaxRAMPercentage, --max-old-space-size).Each line is a system call, its arguments, and its result.
$ strace -f -e trace=openat,connect ./app
openat(AT_FDCWD, "/etc/app/config.yml", O_RDONLY) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/opt/app/config.yml", O_RDONLY) = 3
connect(4, {sa_family=AF_INET, sin_port=htons(5432), sin_addr=inet_addr("10.0.5.20")}, 16) = -1 EINPROGRESS
[pid 4122] connect(5, {sa_family=AF_UNIX, sun_path="/run/redis/redis.sock"}, 110) = -1 EACCES (Permission denied)
+++ exited with 1 +++= 3 is a new file descriptor; = -1 ERRNO is a failure.ENOENT while searching several paths can be normal — look at the last attempt.EINPROGRESS is normal for non-blocking connects; follow the later poll result.[pid 4122] lines come from a thread or child because of -f.EACCES on the Redis socket: check its permissions and group.+++ exited with 1 +++ gives the final status.The key has not changed, but the server now rejects it.
ssh -vvv user@host — which keys were offered and why they were refused.journalctl -u ssh (RHEL: sshd), /var/log/auth.log or /var/log/secure.sudo sshd -T for the effective configuration.~/.ssh, or authorized_keys became group/world-writable (StrictModes).AllowGroups; account expired (chage -l). faillock only blocks password logins.authorized_keys wrong after a copy (restorecon -Rv ~/.ssh).ssh-rsa (SHA-1) signatures refused by OpenSSH 8.8+.HTTPS calls from one host fail while the same calls work everywhere else.
timedatectl shows System clock synchronized: no.chronyc tracking shows a large offset.date -u with curl -skI https://example.com | grep -i ^date (-k: the check itself would fail).sudo chronyc makestep.Turn commands into skills you can demonstrate.
Answer each question using only the command line.
cat /etc/os-release; uname -r; nproc; free -h
lsblk -f; df -hT; findmnt /
ip -br addr; ip route; cat /etc/resolv.conf
systemctl --failed; systemctl list-timers
ps -eo pid,user,%cpu,%mem,cmd --sort=-%mem | head
journalctl -p err -b --no-pager | tail -n 20
sudo ss -tulpnCreate a shared directory, then observe signals and file descriptors.
sudo groupadd team
sudo mkdir /srv/shared
sudo chown root:team /srv/shared
sudo chmod 2770 /srv/shared # setgid + rwx for group
sudo usermod -aG team "$USER" # then log in again
touch /srv/shared/a && ls -l /srv/shared
sleep 600 & # start a background job
pid=$!
ls -l /proc/$pid/fd
kill -STOP $pid; ps -o pid,stat,cmd -p $pid # T
kill -CONT $pid; kill $pid; wait $pid; echo $? # 143team?chmod +t)?sleep have open, and where do they point?kill -9 return instead?Turn a script into a supervised service with logs and a schedule.
sudo tee /usr/local/bin/heartbeat.sh >/dev/null <<'EOF'
#!/usr/bin/env bash
echo "heartbeat from $(hostname) at $(date -Is)"
EOF
sudo chmod 755 /usr/local/bin/heartbeat.sh
sudo tee /etc/systemd/system/heartbeat.service >/dev/null <<'EOF'
[Service]
Type=oneshot
ExecStart=/usr/local/bin/heartbeat.sh
DynamicUser=yes
EOF
sudo tee /etc/systemd/system/heartbeat.timer >/dev/null <<'EOF'
[Timer]
OnCalendar=*:0/2
[Install]
WantedBy=timers.target
EOF
sudo systemctl daemon-reload
sudo systemctl enable --now heartbeat.timer
journalctl -u heartbeat -fsystemctl list-timers shows the next run.DynamicUser=yes runs it as a temporary unprivileged user.exit 1) and see the failure in systemctl status.disable --now the timer and remove both files.Use a transient cgroup to see throttling and an OOM kill safely.
# CPU: limit a busy loop to 20% of one CPU
sudo systemd-run --unit=burn -p CPUQuota=20% \
bash -c 'while :; do :; done'
top -p "$(systemctl show -p MainPID --value burn)"
sudo systemctl stop burn
# Memory: allocate past a 100 MB limit
sudo systemd-run --unit=hog -p MemoryMax=100M -p MemorySwapMax=0 \
python3 -c 'b=[]; [b.append(bytearray(10**7)) for _ in range(50)]'
systemctl status hog # result: oom-kill
journalctl -k -n 20 | grep -i oom
sudo systemctl reset-failed hogFiles act as disks, so nothing on the host is at risk.
truncate -s 1G /tmp/d1.img /tmp/d2.img
L1=$(sudo losetup -fP --show /tmp/d1.img)
L2=$(sudo losetup -fP --show /tmp/d2.img)
sudo pvcreate "$L1" "$L2"
sudo vgcreate labvg "$L1" "$L2"
sudo lvcreate -n data -L 1.2G labvg # larger than one "disk"
sudo mkfs.ext4 -q /dev/labvg/data
sudo mkdir -p /mnt/lab && sudo mount /dev/labvg/data /mnt/lab
df -h /mnt/lab
sudo lvextend -r -L +500M labvg/data # grow online
sudo lvcreate -s -n snap -L 200M labvg/data
sudo lvs labvg; df -h /mnt/lab
# clean up
sudo umount /mnt/lab && sudo vgremove -f labvg
sudo pvremove "$L1" "$L2" && sudo losetup -d "$L1" "$L2"
rm /tmp/d1.img /tmp/d2.imglvs -o +devices shows both.lvextend -r grows ext4 while it is mounted and in use.Data% grows as you write to /mnt/lab.lsblk shows loop0, loop1, and the LVs on top.mkfs.xfs and try to shrink: it is refused.Collect evidence first, then decide what to change.
sudo ss -tulpn # exposed services
systemctl list-unit-files --state=enabled --type=service
sudo sshd -T | grep -E '^(permitrootlogin|passwordauthentication|kbdinteractiveauthentication) '
sudo find / -xdev -perm -4000 -type f 2>/dev/null # setuid binaries
awk -F: '$3 == 0 {print $1}' /etc/passwd # UID 0 accounts
sudo awk -F: '$2 == "" {print $1}' /etc/shadow # empty passwords
sysctl kernel.randomize_va_space net.ipv4.tcp_syncookies
getenforce 2>/dev/null || { sudo aa-status --enabled && echo "AppArmor enabled"; }
apt list --upgradable 2>/dev/null | head # or: dnf check-update
sudo lynis audit system --quick # if installedIf not, revisit the section and repeat the lab.
cmd 2>&1 > f still print errors?docker stop until it is killed?Wants= and After= differ?Explain each answer, then prove it with a command.
fsck never run on a mounted filesystem?hard and soft NFS mounts trade off?required vs sufficient mean in PAM?docker.sock equivalent to root?/etc/shadow was modified, and by whom?Use these as quick anchors, then explain them in your own words.
| Term | Short meaning |
|---|---|
| Kernel / syscall | Privileged core / request into it |
| Inode | File metadata and data location, without the name |
| File descriptor | Integer handle to an open file, pipe, or socket |
| PID 1 | First process; adopts orphans and reaps zombies |
| Signal | Asynchronous notification to a process |
| Unit | Anything systemd manages |
| Term | Short meaning |
|---|---|
| Page cache | File data cached in RAM; reclaimable |
| Load average | Runnable + uninterruptible tasks, averaged |
| OOM killer | Kernel mechanism that kills to free memory |
| Namespace | Isolated view of a system resource |
| cgroup | Group of processes with accounting and limits |
| Capability | One slice of root’s privileges |
Quick anchors for the storage and hardening sections.
| Term | Short meaning |
|---|---|
| PV / VG / LV | LVM physical volume / volume group / logical volume |
| RAID | Redundancy across disks; not a backup |
| Snapshot | Point-in-time view; keeps originals of changed blocks |
| LUKS | Standard Linux block-device encryption format |
| RPO / RTO | Acceptable data loss / acceptable restore time |
| root_squash | NFS maps client root to an unprivileged user |
| Term | Short meaning |
|---|---|
| PAM | Pluggable authentication stack used by login services |
| SSH CA | Authority that signs short-lived SSH keys |
| auditd | Kernel audit event logging daemon |
| AIDE | File integrity checker based on a hash database |
| CIS Benchmark | Consensus hardening baseline per OS |
| Least privilege | Grant only the access a task needs |
Use the manual pages on your own system for exact behavior; use these for depth.
man7.org/linux/man-pages/docs.kernel.org/admin-guide/docs.kernel.org/admin-guide/cgroup-v2.htmlsystemd.io and man systemd.service, systemd.execrefspecs.linuxfoundation.org/fhs.shtmlgnu.org/software/bash/manual/brendangregg.com/linuxperf.htmlshellcheck.net