Self-study deck

Linux
from A to Z

Build a working mental model of Linux, from the kernel and the shell to files, permissions, processes, systemd, performance, containers, and troubleshooting.

Use ← → / Space to navigate · I contents · O overview · T theme · F fullscreen

The big picture

What happens when you type ls -l /etc?

One short command touches the shell, the process model, system calls, the filesystem, and the terminal.

1 · The shell parses

  • Bash reads the line, splits words, and expands variables and globs.
  • It looks up ls as an alias, function, builtin, then in PATH.
  • type -a ls shows what will actually run.

2 · A process is created

  • The shell calls fork(); the child calls execve("/usr/bin/ls").
  • The dynamic loader maps shared libraries such as libc.
  • The new process inherits environment, working directory, and file descriptors.

3 · System calls do the work

  • ls asks the kernel via openat, getdents64, and statx.
  • User space never touches the disk directly.
  • strace ls -l /etc prints every call and its result.

4 · VFS and the filesystem

  • The Virtual File System routes calls to ext4, xfs, tmpfs, or another type.
  • Metadata often comes from the page and inode caches, not the disk.
  • Permission checks happen here, in the kernel.

5 · Output reaches you

  • ls writes to file descriptor 1 (stdout).
  • A terminal (pty) displays it; a pipe would feed another process instead.
  • ls formats differently when stdout is not a terminal.

6 · Exit and wait

  • The process exits with a status code.
  • The shell wait()s, reaps the child, and stores the status in $?.
  • Every layer in this deck appears in this one command.
Linux · A to Z

Contents

Click any line to jump there · press I to return to this page

Chapter 1

Foundations

The kernel, the boot process, and the shell you will use for everything else.

Chapter 1 · Part 1.1

Kernel and system basics

What Linux actually is, and how a machine gets from power-on to a login prompt.

Kernel and user space

Two worlds separated by system calls

The CPU enforces a privilege boundary; programs request kernel services through a narrow interface.

Kernel space

  • Schedules CPUs, manages memory, drives devices.
  • Implements filesystems, networking, and security checks.
  • Runs privileged; a kernel bug can crash the whole machine.

User space

  • Shells, daemons, databases, and your applications.
  • Each process gets its own virtual address space.
  • A crash usually kills only that process.

System calls

  • read, write, openat, execve, clone, mmap, socket.
  • Usually called through C library wrappers such as glibc.
  • A failed call returns an errno such as ENOENT or EACCES.

Most behavior you debug on Linux is either a system call result or a kernel policy decision. strace shows the first; /proc, /sys, and dmesg reveal the second.

Anatomy of a system

Linux is a kernel; a distribution is a whole system

Knowing which component provides a feature tells you where to look when it breaks.

ComponentCommon examplesRole
KernelLinux 5.x / 6.xHardware, processes, memory, filesystems, network
C libraryglibc, muslSystem call wrappers, DNS resolution, locale, threads
Init systemsystemd (PID 1)Starts services, manages dependencies, collects logs
Core utilitiesGNU coreutils, BusyBoxls, cp, cat, chmod, and friends
Shellbash, dash, zshInteractive command line and scripting
Package managerapt, dnf, apkInstalls, upgrades, and verifies software

Alpine uses musl and BusyBox, so a binary or a command option that works on Ubuntu may behave differently in an Alpine container.

Distributions

Choosing and identifying a distribution

Distributions differ in packaging, defaults, release cadence, and support lifetime.

FamilyExamplesPackagesNotes
DebianDebian, Ubuntu.deb via aptUbuntu LTS: 5 years standard support
Red HatRHEL, Rocky, AlmaLinux, Fedora, Amazon Linux 2023.rpm via dnfSELinux enforcing by default (AL2023: permissive)
AlpineAlpine Linux.apk via apkmusl + BusyBox; very small container images
ArchArch LinuxpacmanRolling release
cat /etc/os-release      # distribution name and version
uname -r                 # running kernel version
hostnamectl              # host, OS, kernel, architecture

Container images share the host kernel: uname -r inside a container reports the host’s kernel, while /etc/os-release reports the image’s distribution.

Boot process

From power-on to a login prompt

Each stage hands control to the next; knowing the order tells you where a boot failure lives.

1 · Firmware

  • UEFI (or legacy BIOS) runs hardware self-tests.
  • UEFI loads a boot loader from the EFI System Partition.

2 · Boot loader

  • GRUB loads the kernel and the initramfs into memory.
  • It passes the kernel command line, e.g. root=UUID=….

3 · Kernel

  • Initializes CPUs, memory, and built-in drivers.
  • Unpacks the initramfs as a temporary root filesystem.

4 · initramfs

  • Loads storage drivers, LVM, RAID, or encryption support.
  • Finds and mounts the real root, then switches to it.

5 · systemd (PID 1)

  • Starts units in dependency order toward default.target.
  • Mounts /etc/fstab entries, brings up network and services.

6 · Login

  • getty on consoles, sshd for remote access.
  • Cloud images run cloud-init to apply user data and keys.

Inspect a boot with journalctl -b, dmesg -T, systemd-analyze blame, and cat /proc/cmdline.

Everything is a file

One interface for many kinds of objects

Files, devices, and kernel state are reached through paths and file descriptors.

Regular files and directories

  • Data and the names that point to it.
  • Directories are tables of names → inode numbers.
  • Pipes and Unix sockets can also appear as files.

Device files in /dev

  • Block devices: disks such as /dev/nvme0n1.
  • Character devices: /dev/tty, /dev/null, /dev/urandom.
  • Created dynamically by the kernel and udev.

Pseudo filesystems

  • /proc: processes and kernel state.
  • /sys: devices, drivers, and kernel objects.
  • Reading /proc/meminfo asks the kernel, not a disk.

A file descriptor is a small integer that refers to an open file, pipe, socket, or device. 0 = stdin, 1 = stdout, 2 = stderr. See yours with ls -l /proc/$$/fd.

Kernel tuning

sysctl, modules, and boot parameters

Most kernel behavior is adjustable at runtime; persistence needs a configuration file.

uname -r                          # kernel version
cat /proc/cmdline                 # boot parameters

sysctl net.ipv4.ip_forward        # read
sudo sysctl -w vm.swappiness=10   # change now (lost on reboot)
echo 'vm.swappiness = 10' | sudo tee /etc/sysctl.d/90-tuning.conf
sudo sysctl --system              # reload all sysctl files

lsmod | head                      # loaded modules
modinfo overlay                   # module details
sudo modprobe br_netfilter        # load a module
dmesg -T | tail                   # kernel messages

Read the output

  • sysctl keys map to files under /proc/sys: vm.swappiness is /proc/sys/vm/swappiness.
  • sysctl -w is temporary; files in /etc/sysctl.d/ persist across reboots.
  • Modules add drivers or features without rebuilding the kernel; /etc/modules-load.d/ loads them at boot.
  • dmesg shows the kernel ring buffer: hardware errors, OOM kills, filesystem problems.
Kernel command line

Boot parameters and recovery

Editing the kernel line in GRUB is the standard way into a broken system.

# GRUB menu: press e, edit the "linux" line
linux /vmlinuz-6.8.0 root=UUID=… ro systemd.unit=rescue.target
# Ctrl-X (or F10) boots once with the change

cat /proc/cmdline          # running kernel parameters

# make a change permanent
# Debian/Ubuntu: edit /etc/default/grub
# (GRUB_CMDLINE_LINUX), then:
sudo update-grub
# RHEL family:
sudo grubby --update-kernel=ALL \
  --args="console=ttyS0,115200"
sudo grubby --info=DEFAULT

Useful parameters

ParameterEffect
systemd.unit=rescue.targetSingle-user mode with basic services
systemd.unit=emergency.targetMinimal shell, root mounted read-only
rd.breakStop inside the initramfs (RHEL root-password reset)
init=/bin/bashSkip systemd; remount root with mount -o remount,rw /
console=ttyS0,115200Kernel output to the serial console (cloud)
fsck.mode=forceForce a filesystem check at boot

Anyone with console access can use these to become root. Protect the console and, where required, set a GRUB password.

Kernel log

Read dmesg like an incident timeline

The kernel reports hardware, memory, filesystem, and network problems that applications never log.

$ dmesg -T --level=err,warn | tail      # human timestamps, problems only
$ journalctl -k -b -p warning            # same, from the journal

# example lines (abridged)
[Thu Oct 16 10:02:11] EXT4-fs error (device nvme1n1): ext4_find_entry: inode #2: reading directory lblock 0
[Thu Oct 16 10:05:42] Out of memory: Killed process 4121 (java) total-vm:8123456kB, anon-rss:3912345kB
[Thu Oct 16 10:07:03] nf_conntrack: table full, dropping packet
[Thu Oct 16 10:09:30] I/O error, dev sdb, sector 123456 op 0x0:(READ)
[Thu Oct 16 10:11:15] nfs: server fs-01.internal not responding, still trying

What each line means

  • EXT4-fs error: filesystem corruption or a failing device; may lead to a read-only remount.
  • Out of memory: Killed process: the OOM killer chose a victim; note the name and RSS.
  • table full, dropping packet: connection tracking is full — new connections silently fail.
  • I/O error, dev …: the disk or volume returned errors; check health and the cloud console.
  • not responding, still trying: an NFS mount is hung; processes using it sit in D state.
  • dmesg -T times can drift after suspend; the journal is more reliable.
Time

Clocks, time zones, and NTP

Wrong time breaks TLS, Kerberos, one-time passwords, and log correlation.

timedatectl                          # local time, UTC, time zone, NTP status
sudo timedatectl set-timezone UTC

chronyc tracking                     # offset from reference, stratum
chronyc sources -v                   # configured servers and reachability

# /etc/chrony.conf (RHEL) or /etc/chrony/chrony.conf (Debian/Ubuntu)
server 169.254.169.123 prefer iburst # Amazon Time Sync Service
pool 2.pool.ntp.org iburst

date -u; date +%s                    # UTC and Unix epoch seconds

Practical rules

  • Run servers in UTC; convert to local time only for display.
  • Use one NTP client (chrony or systemd-timesyncd), not both.
  • NTP uses UDP port 123; firewalls must allow it to your time source.
  • chrony slews (gradually adjusts) small offsets instead of jumping the clock.
  • Certificates show “not yet valid” when the client clock is behind.
Getting help

Read the manual before searching the web

Manual pages are versioned with the software installed on this machine.

Manual sections

SectionContents
1User commands: man 1 ls
2System calls: man 2 open
3Library functions: man 3 printf
5File formats: man 5 crontab, man 5 fstab
7Overviews: man 7 signal, man 7 namespaces
8Administration: man 8 mount, man 8 ip

Finding answers fast

  • man -k keyword (or apropos) searches page descriptions.
  • Inside man: /pattern searches, n jumps to the next match, q quits.
  • cmd --help gives a quick summary; help cd documents bash builtins.
  • tldr cmd (if installed) shows common examples.
  • The same command name can exist in several sections: crontab(1) vs crontab(5).
Chapter 1 · Part 1.2

Shell and command line

The interface you will use for everything else in this deck.

How the shell works

Parse, expand, then execute

Most shell surprises come from expansion happening before the command runs.

Expansion order (bash)

  • Brace → tilde → parameter, command substitution, arithmetic.
  • Then word splitting on unquoted results.
  • Then pathname expansion (globs), then quote removal.

Finding the command

  • Aliases, functions, and builtins are checked before PATH.
  • cd, export, and source must be builtins: they change the shell itself.
  • type -a cmd and command -v cmd show what will run.

Which shell?

  • bash is the usual interactive shell on servers.
  • /bin/sh is dash on Debian and Ubuntu, not bash.
  • Bash features like [[ ]] or arrays fail under plain sh.

echo $SHELL shows your login shell; ps -p $$ shows the shell you are actually running.

Variables and environment

Shell variables versus environment variables

Only exported variables are copied into child processes.

NAME=world                 # shell variable (no spaces around =)
echo "Hello, $NAME"
export APP_ENV=prod        # exported to child processes
env | grep APP_            # show environment

APP_ENV=dev ./run.sh       # set only for this one command
PATH="$HOME/bin:$PATH"     # prepend a directory
unset NAME

source ./env.sh            # run in the current shell
./env.sh                   # runs in a child; changes are lost

Key rules

  • A child process can never change its parent’s environment.
  • Login shells read ~/.bash_profile (or ~/.profile); interactive non-login shells read ~/.bashrc.
  • cron and systemd services start with a minimal environment, not your interactive one.
  • Never put secrets on the command line: other users can see arguments in ps.
Streams and redirection

stdin, stdout, stderr, and pipes

Every process starts with three file descriptors; the shell can rewire them before running a command.

SyntaxMeaning
cmd > out.txtWrite stdout to a file (truncate first)
cmd >> out.txtAppend stdout
cmd 2> err.txtWrite stderr to a file
cmd > all.log 2>&1stdout to file, then stderr to the same place
cmd &> all.logBash shorthand for both streams
cmd < in.txtRead stdin from a file
cmd1 | cmd2stdout of cmd1 becomes stdin of cmd2
cmd | tee fileWrite to the screen and a file at the same time
cmd 2>/dev/nullDiscard error messages

Order matters: cmd 2>&1 > f sends stderr to the terminal, because stderr copies stdout before stdout is redirected to the file.

Exit status

Every command reports success or failure

Scripts, &&, ||, if, and CI pipelines all depend on exit codes.

grep -q ERROR app.log && echo "found errors"
mkdir /data || echo "mkdir failed"
false; echo "exit=$?"            # exit=1

if systemctl is-active --quiet nginx; then
  echo "nginx is running"
fi

curl -fsS https://example.com > /dev/null
echo $?                          # non-zero on HTTP >= 400 with -f

Common exit codes

CodeMeaning
0Success
1–2General error / misuse (convention)
126Found but not executable
127Command not found
128 + NKilled by signal N
130Ctrl-C (SIGINT = 2)
137SIGKILL (9), often OOM kill
143SIGTERM (15)

A pipeline returns the exit code of its last command. set -o pipefail makes it fail if any stage fails.

Quoting and globbing

Most shell bugs are quoting bugs

Unquoted expansions are split on whitespace and then treated as glob patterns.

Quotes

  • "double": expands $var and $(cmd), prevents splitting.
  • 'single': completely literal.
  • Always write "$var" unless you deliberately want splitting.
  • rm $file with file="my report.txt" tries to remove two files.

Globs and braces

  • * any string, ? one character, [a-z] a set.
  • Globs match file names; they are not regular expressions.
  • {a,b} and {1..5} generate words even if no files exist.
  • An unmatched glob stays literal in bash unless nullglob is set.

Substitution

  • $(cmd) inserts command output; prefer it to backticks.
  • $((n + 1)) performs integer arithmetic.
  • ${var:-default} uses a default when unset or empty.
  • ${file%.log} strips a suffix; ${path##*/} keeps the basename.
Text toolkit 1

Search with grep and find

Logs and configuration are text; these two tools answer most “where is it?” questions.

grep -rn "timeout" /etc/nginx/         # recursive, with line numbers
grep -i "error" app.log | tail -n 20    # case-insensitive
grep -E " 5[0-9]{2} " access.log | wc -l  # count HTTP 5xx lines
grep -v '^#' sshd_config | grep -v '^$' # drop comments and blanks
grep -C 3 "panic" app.log               # 3 lines of context

find /var/log -name "*.log" -mtime +7         # older than 7 days
find / -xdev -type f -size +500M 2>/dev/null  # big files, one fs
find /srv -type f -perm -o+w                  # world-writable files
find . -name "*.tmp" -print0 | xargs -0 rm -f # safe with spaces

Read the commands

  • grep -E enables extended regular expressions; -F matches fixed strings quickly.
  • find tests are combined with an implicit AND; -o means OR.
  • -xdev stops find from crossing into other mounted filesystems.
  • -print0 with xargs -0 handles file names with spaces or newlines.
  • Run find … -print first; add -delete only after checking the list.
Text toolkit 2

Transform with awk, sed, sort, and uniq

Small tools combine in pipelines to answer questions about large files.

# top 10 client IPs in an nginx access log
awk '{print $1}' access.log | sort | uniq -c | sort -rn | head

# status-code distribution (field 9 in combined log format)
awk '{print $9}' access.log | sort | uniq -c | sort -rn

# regular users from /etc/passwd
awk -F: '$3 >= 1000 {print $1, $7}' /etc/passwd

sed -n '100,120p' big.log               # print a line range
sed -i.bak 's/^Port 22$/Port 2222/' sshd_config  # edit with backup
cut -d, -f1,3 data.csv                  # select CSV columns
sort -k2,2n file                        # numeric sort on column 2
tail -F app.log | grep --line-buffered ERROR

Read the commands

  • awk splits each line into fields $1, $2, … on whitespace or -F.
  • uniq only merges adjacent duplicates, so sort first.
  • sed -i edits in place; the .bak suffix keeps a backup.
  • tail -F keeps following a log even after rotation.
  • For JSON, use jq rather than regular expressions.
Productivity

Work faster and survive disconnects

A few habits save hours on remote servers.

History and editing

  • Ctrl-R searches history; !! repeats the last command.
  • !$ reuses the last argument; Alt-. inserts it.
  • Ctrl-A/Ctrl-E jump to line start/end; Ctrl-U clears.
  • Tab completion completes commands, paths, and often options.

Reading output

  • less pages output: / search, n next, G end, F follow.
  • watch -n 2 cmd reruns a command and shows changes.
  • column -t aligns whitespace-separated output.
  • cmd | less -S stops long lines from wrapping.

Sessions

  • tmux or screen keeps work running if SSH drops.
  • tmux new -s work, detach with Ctrl-B d, reattach with tmux a -t work.
  • nohup cmd & survives logout but loses interactive control.
  • Long-running services belong in systemd, not in a terminal.
Bash scripting

Write scripts that fail safely

A short template prevents the most common scripting accidents.

#!/usr/bin/env bash
set -euo pipefail

LOG_DIR="${1:?usage: $0 <log-dir>}"   # required argument
tmp="$(mktemp -d)"
trap 'rm -rf "$tmp"' EXIT             # always clean up

log() { printf '%s %s\n' "$(date -Is)" "$*" >&2; }

for f in "$LOG_DIR"/*.log; do
  [[ -e "$f" ]] || continue           # glob matched nothing
  log "compressing $f"
  gzip -c -- "$f" > "$tmp/$(basename "$f").gz"
done

mv "$tmp"/*.gz /backup/ || { log "backup failed"; exit 1; }

Why each line exists

  • -e: stop on an unhandled error; -u: unset variables are errors; pipefail: pipelines fail if any stage fails.
  • ${1:?msg} stops with a usage message when the argument is missing.
  • trap … EXIT cleans up temporary files on success or failure.
  • Quote every expansion; -- protects against file names starting with -.
  • set -e is ignored inside if conditions and &&/|| lists, so still check critical steps.
  • Run shellcheck script.sh before committing.
Conditions

Tests, comparisons, and case

Bash has several test syntaxes; [[ ]] is the safest for scripts that require bash.

if [[ -f /etc/app.conf && -r /etc/app.conf ]]; then
  echo "config readable"
fi

[[ -z "${VAR:-}" ]] && echo "VAR is empty or unset"
[[ "$file" == *.log ]]          # glob match (right side unquoted)
[[ "$ver" =~ ^v[0-9]+\.[0-9]+$ ]] && echo "${BASH_REMATCH[0]}"
(( count > 10 )) && echo "many"  # arithmetic comparison

case "${1:-}" in
  start|stop) systemctl "$1" myapp ;;
  status)     systemctl status myapp ;;
  *)          echo "usage: $0 {start|stop|status}" >&2; exit 2 ;;
esac

Common tests

TestTrue when
-e pathExists (any type)
-f / -d / -LRegular file / directory / symlink
-s fileExists and is not empty
-r / -w / -xReadable / writable / executable by you
-z / -n strString is empty / not empty
a -nt bFile a is newer than file b
-eq -ne -lt -gtInteger comparison inside [ ] / [[ ]]

Inside [[ ]], == with an unquoted right side is a glob match, and </> compare strings. Use (( )) or -lt for numbers.

Loops, functions, arrays

Structure scripts like small programs

Most automation is a loop over hosts, files, or lines with a function that retries.

for host in web1 web2 web3; do
  ssh -o ConnectTimeout=5 "$host" uptime || echo "$host failed" >&2
done

while IFS= read -r line; do         # read a file line by line
  printf '%s\n' "$line"
done < hosts.txt

servers=(web1 web2 "db 1")          # indexed array
echo "${#servers[@]} servers"
for s in "${servers[@]}"; do echo "$s"; done

declare -A port=([http]=80 [https]=443)   # associative array
echo "${port[https]}"

retry() {                           # retry with exponential backoff
  local n=0
  until "$@"; do
    (( ++n >= 5 )) && return 1
    sleep $(( 2 ** n ))
  done
}
retry curl -fsS http://localhost:8080/health

Details that bite

  • read -r keeps backslashes; IFS= keeps leading and trailing spaces.
  • cmd | while read … runs the loop in a subshell, so variables set inside are lost; use done < <(cmd).
  • "${arr[@]}" expands each element as one word, even with spaces.
  • Functions return a status (0–255); “return values” are printed to stdout and captured with $(…).
  • Use local inside functions to avoid overwriting global variables.
Advanced redirection

Heredocs, here-strings, and process substitution

These features remove temporary files and make scripts easier to read.

cat <<EOF > /tmp/info.txt          # variables expand
host=$(hostname) user=$USER
EOF

cat <<'EOF' > /usr/local/bin/x.sh  # quoted: written literally
echo "$HOME is not expanded here"
EOF

grep -c ERROR <<< "$output"        # here-string

diff <(sort a.txt) <(sort b.txt)   # process substitution
comm -13 <(sort old) <(sort new)   # lines only in new

exec > >(tee -a /var/log/job.log) 2>&1   # log the rest of a script
{ date; df -h; free -m; } > report.txt   # group output

mkfifo /tmp/pipe                   # named pipe between processes

When to use which

  • <<EOF expands variables; <<'EOF' writes the text exactly — use it for scripts and configs.
  • <<-EOF strips leading tabs (not spaces) so heredocs can be indented.
  • <(cmd) gives a command’s output a temporary file name such as /dev/fd/63.
  • exec > >(tee …) sends all later output to the screen and a log file.
  • { …; } groups commands without starting a subshell; ( … ) starts one.
Regular expressions

BRE, ERE, and PCRE in one page

grep, sed, and awk use slightly different regex dialects.

Syntax

PatternMeaning
.Any single character
^ / $Start / end of line
*Zero or more of the previous item
+ ? {2,4}One or more / optional / range (ERE; BRE: \{2,4\}, GNU also \+)
[abc] / [^0-9]One of / none of these characters
[[:space:]], [[:digit:]]POSIX classes, portable across tools
(a|b)Group and alternation (ERE)
\d, lookaheadPCRE only: grep -P (GNU tools also accept \s, \w)
grep    'ab\+c'     file   # BRE
grep -E 'ab+c'      file   # ERE
grep -P '\d{3}-\d{4}' file # PCRE (GNU grep)
grep -oE '[0-9]{1,3}(\.[0-9]{1,3}){3}' access.log | sort -u
sed -E 's/(user)=([^ ]+)/\1=***/g' app.log    # mask values
awk '/ERROR|FATAL/ {n++} END {print n+0}' app.log
grep -F 'a.b[1]' file      # fixed string: no regex at all
JSON on the command line

Query structured output with jq

Cloud CLIs, Kubernetes, and modern logs emit JSON; jq turns it into answers.

# instance ID and state as tab-separated text
aws ec2 describe-instances |
  jq -r '.Reservations[].Instances[] | [.InstanceId, .State.Name] | @tsv'

# pods that are not Running
kubectl get pods -A -o json |
  jq -r '.items[] | select(.status.phase != "Running")
         | "\(.metadata.namespace)/\(.metadata.name)"'

# error lines from JSON-lines logs, compact output
jq -c 'select(.level == "error") | {time, msg}' app.jsonl | head

# count by field
jq -r '.status' events.jsonl | sort | uniq -c | sort -rn

jq '.replicas = 3' deploy.json > deploy.new.json   # edit a value

Core ideas

  • .field selects; .[] iterates over an array.
  • | pipes values between filters, like the shell.
  • select(cond) keeps matching values.
  • -r prints raw strings without quotes; @tsv/@csv format rows.
  • -c prints one compact JSON value per line.
  • Many CLIs also have a built-in query: aws --query, kubectl -o jsonpath.
Chapter 2

Files and storage

From inodes and mounts to RAID, LVM, encryption, NFS, and backups.

Chapter 2 · Part 2.1

Filesystem

Where data lives, how names map to data, and why disks fill up.

Filesystem hierarchy

Where things live

The Filesystem Hierarchy Standard (FHS) explains most paths you will see on a server.

System

PathContents
/etcHost-specific configuration
/usrInstalled programs and libraries (read-mostly)
/bootKernel, initramfs, boot loader files
/devDevice files
/proc, /sysKernel interfaces (virtual)
/runRuntime state: PID files, sockets (tmpfs)

Data

PathContents
/varChanging data: logs, caches, spools, databases
/var/logLog files
/home, /rootUser and root home directories
/tmpTemporary files; may be tmpfs and cleaned
/optSelf-contained third-party software
/srvData served by this host

Many distributions have merged /bin, /sbin, and /lib into /usr (“usrmerge”); the old paths remain as symlinks.

Inodes and links

A file name is not the file

An inode holds metadata and data locations; directories map names to inode numbers.

Inode

  • Stores owner, group, mode, size, timestamps, and block pointers.
  • Does not store the file name.
  • ls -i shows inode numbers; stat file shows details.
  • Each filesystem has a finite number of inodes.

Hard link

  • Another directory entry for the same inode.
  • Must be on the same filesystem; not allowed for directories.
  • Data is freed only when the link count is 0 and no process has it open.
  • ln target newname

Symbolic link

  • A small file containing a path.
  • Can cross filesystems and point to directories.
  • Breaks (“dangles”) if the target moves.
  • ln -s target link; readlink -f link resolves it.

Because data is freed only when nothing holds the inode, deleting a large log file that a process still has open does not free the space.

File types and timestamps

Read the first column of ls -l

The first character is the file type; three timestamps answer different questions.

File types

CharType
-Regular file
dDirectory
lSymbolic link
cCharacter device
bBlock device
pNamed pipe (FIFO)
sUnix socket

Timestamps (stat)

NameChanges when
mtimeFile content is modified
ctimeInode changes: content writes or metadata (mode, owner, links) — not creation time
atimeFile is read (often reduced by relatime/noatime)
btimeBirth time, if the filesystem records it
Mounts

Attaching filesystems to the tree

Linux has one directory tree; each filesystem is mounted at a directory in it.

lsblk -f                 # disks, partitions, fs types, UUIDs
findmnt                  # mount tree
findmnt /var             # what is mounted at /var?

sudo mount /dev/nvme1n1 /data
sudo umount /data        # fails if a process is using it
sudo fuser -vm /data     # who is using it?

# /etc/fstab
# device         mountpoint  type  options            dump pass
UUID=3f1c...a9   /data       xfs   defaults,nofail    0    2

sudo findmnt --verify    # check fstab before rebooting
sudo mount -a            # mount everything in fstab

Practical rules

  • Use UUID= or labels in fstab: device names like /dev/sdb can change.
  • nofail lets the system boot even if a data disk is missing.
  • A bad fstab line can drop the system into emergency mode on the next boot.
  • Security options: noexec, nosuid, nodev, ro.
  • Mounting over a non-empty directory hides its old contents until unmounted.
Filesystem types

Pick the filesystem for the job

Local, virtual, layered, and network filesystems all use the same VFS interface.

TypeTypical useNotes
ext4General-purpose default (Debian, Ubuntu)Mature; can grow online, shrink offline
xfsDefault on RHEL family; large filesGrows online; cannot be shrunk
btrfsSnapshots, subvolumes, checksumsCopy-on-write; default on Fedora desktops
tmpfs/run, /dev/shm, often /tmpStored in RAM/swap; lost on reboot
overlayContainer image layersUnion of read-only lower and writable upper layers
nfsShared storage, e.g. AWS EFSNetwork latency; hangs if the server disappears
proc, sysfsKernel interfacesNo disk storage; contents generated on read
Disks and volumes

Block devices, partitions, and LVM

Storage is layered: device → partition table → (optional LVM) → filesystem → mount.

Devices and partitions

  • Names: /dev/sda, /dev/nvme0n1p1, /dev/xvda.
  • lsblk shows the tree; blkid shows UUIDs and types.
  • GPT is the modern partition table; MBR is legacy.
  • Tools: parted, fdisk, gdisk.

LVM

  • Physical volumes (PV) → volume group (VG) → logical volumes (LV).
  • LVs can be resized and span several disks.
  • pvs, vgs, lvs summarize each layer.
  • lvextend -r -L +10G /dev/vg0/data grows the LV and filesystem.

Grow a cloud disk

  • 1. Resize the volume in the cloud provider.
  • 2. sudo growpart /dev/nvme0n1 1 extends partition 1.
  • 3a. ext4: sudo resize2fs /dev/nvme0n1p1
  • 3b. xfs: sudo xfs_growfs / (takes the mount point).

Take a snapshot before resizing or repartitioning production disks.

Disk usage

Why is the disk full?

“No space left on device” (ENOSPC) has more than one cause.

df -h                    # space per filesystem
df -i                    # inode usage per filesystem

# largest directories, staying on one filesystem
sudo du -xh --max-depth=1 / 2>/dev/null | sort -h | tail

sudo lsof +L1            # deleted files still held open
sudo du -sh /var/lib/docker /var/log/journal

journalctl --disk-usage
sudo journalctl --vacuum-size=500M

Three different “full” states

  • Blocks full: df -h near 100%. Find big directories with du.
  • Inodes full: df -i near 100% while df -h looks fine — millions of small files.
  • Deleted but open: df is full but du cannot find it. Restart or signal the process holding the file.
  • ext4 reserves about 5% of blocks for root by default, so non-root writes can fail earlier.
  • Files hidden under a mount point are counted by df but invisible to du.
Log rotation

Keep logs useful without filling the disk

logrotate handles classic log files; journald manages its own storage.

# /etc/logrotate.d/myapp
/var/log/myapp/*.log {
    daily
    rotate 14
    compress
    delaycompress
    sharedscripts
    missingok
    notifempty
    create 0640 myapp myapp
    postrotate
        systemctl kill -s HUP myapp.service
    endscript
}

sudo logrotate -d /etc/logrotate.d/myapp   # dry run

Choices that matter

  • create + postrotate: rename the file and tell the app to reopen its log.
  • copytruncate: copy then truncate in place — works for apps that cannot reopen, but can lose lines.
  • delaycompress keeps the newest rotated file readable.
  • journald limits: SystemMaxUse= in /etc/systemd/journald.conf.
  • In containers, log to stdout/stderr and let the runtime rotate.
Archives and transfer

tar, compression, and copying data

Choose the compression and copy tool by speed, ratio, and resumability.

tar -czf backup.tgz /etc/nginx       # create gzip archive
tar -tzf backup.tgz | head           # list contents
tar -xzf backup.tgz -C /tmp/restore  # extract elsewhere
tar --zstd -cf data.tar.zst /srv/data

rsync -avh --progress src/ host:/dst/   # trailing / matters
rsync -avh --delete src/ dst/           # mirror (deletes extras!)
scp file.txt user@host:/tmp/

sha256sum backup.tgz > backup.tgz.sha256
sha256sum -c backup.tgz.sha256

Compression trade-offs

ToolSpeedRatio
gzipFastModerate; universal
zstdVery fastGood; tunable levels
xzSlowHigh
bzip2SlowModerate; legacy

rsync src/ dst/ copies the contents of src; rsync src dst/ creates dst/src. Test --delete with -n (dry run) first.

Reading stat

Everything the inode knows

stat shows the metadata that permission checks, backups, and build tools rely on.

$ stat /etc/hosts
  File: /etc/hosts
  Size: 221        Blocks: 8          IO Block: 4096   regular file
Device: 259,1      Inode: 1048611     Links: 1
Access: (0644/-rw-r--r--)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2026-10-16 09:12:44.123456789 +0700
Modify: 2026-09-30 14:02:10.000000000 +0700
Change: 2026-09-30 14:02:10.000000000 +0700
 Birth: 2026-09-30 14:02:10.000000000 +0700

$ stat -c '%n %U:%G %a %s bytes' /etc/hosts
/etc/hosts root:root 644 221 bytes

Read the output

  • Size is the file length; Blocks counts 512-byte units actually allocated — sparse files have fewer.
  • Device and Inode uniquely identify the file on this host.
  • Links is the hard-link count; for a directory it includes . and subdirectories.
  • Access: (0644/…) shows the mode in octal and symbolic form.
  • stat -c with format codes is ideal for scripts.
Reading lsblk

Map devices to filesystems and mount points

One command shows how disks, partitions, LVM, and mounts fit together.

$ lsblk -f
NAME            FSTYPE      UUID          FSAVAIL FSUSE% MOUNTPOINTS
nvme0n1
├─nvme0n1p1     xfs         3f1c…a9         12.4G    38% /
└─nvme0n1p128   vfat        7B77-95E7        8.7M    13% /boot/efi
nvme1n1         LVM2_member aZ3k…Qp
└─data-app      xfs         9e2d…11         80.1G    20% /srv/app

$ findmnt -o TARGET,SOURCE,FSTYPE,OPTIONS /srv/app
TARGET   SOURCE                FSTYPE OPTIONS
/srv/app /dev/mapper/data-app  xfs    rw,noatime,attr2,inode64,noquota

Read the output

  • The tree shows layering: disk → partition → filesystem, or disk → LVM → logical volume.
  • LVM2_member marks a physical volume; its logical volumes appear underneath.
  • LVs appear as /dev/mapper/vg-lv and /dev/vg/lv (both names work).
  • An empty MOUNTPOINTS column means the filesystem is not mounted.
  • findmnt shows the options actually in effect, which may differ from fstab.
Chapter 2 · Part 2.2

Storage operations

RAID, LVM, repair, encryption, network storage, and backups.

RAID levels

Redundancy is not a backup

RAID keeps a system running through disk failure; it does not protect against deletion or corruption.

LevelMin disksUsable capacitySurvivesTypical use
RAID 02100% (n disks)None: any failure loses dataScratch space, maximum throughput
RAID 1250% (mirror)All but one disk of the mirrorBoot and OS disks
RAID 53n − 1 disksOne diskRead-heavy storage; slow rebuilds on large disks
RAID 64n − 2 disksTwo disksLarge arrays where rebuilds take long
RAID 10450%One disk per mirror pairDatabases: good random write performance

Cloud block volumes are already replicated inside one availability zone. Software RAID 0 across volumes is sometimes used for throughput; protection still comes from snapshots and backups.

Software RAID

Manage arrays with mdadm

Linux md RAID is mature, portable between machines, and visible in /proc/mdstat.

sudo mdadm --create /dev/md0 --level=1 --raid-devices=2 /dev/sdb /dev/sdc
cat /proc/mdstat                     # state and resync progress
sudo mdadm --detail /dev/md0

# replace a failed disk
sudo mdadm --manage /dev/md0 --fail /dev/sdb --remove /dev/sdb
sudo mdadm --manage /dev/md0 --add /dev/sdd   # rebuild starts

# persist the array definition
sudo mdadm --detail --scan | sudo tee -a /etc/mdadm/mdadm.conf  # Debian
# RHEL: /etc/mdadm.conf
sudo update-initramfs -u            # RHEL: sudo dracut -f

Read the output

  • [UU] in /proc/mdstat means both mirror members are up; [U_] means one is missing.
  • recovery = 42.0% shows a rebuild in progress — performance is reduced until it finishes.
  • Set up mdadm --monitor (or the mdmonitor service) to email or alert on failures.
  • A degraded array works until the next failure: replace disks promptly.
  • Update the initramfs so the array assembles at boot.
LVM in practice

Create, grow, and move logical volumes

LVM separates filesystems from physical disks so capacity can change without downtime.

sudo pvcreate /dev/sdb /dev/sdc              # 1. mark disks as PVs
sudo vgcreate data /dev/sdb /dev/sdc         # 2. pool them in a VG
sudo lvcreate -n app -L 50G data             # 3. carve a 50 GB LV
sudo mkfs.xfs /dev/data/app
sudo mount /dev/data/app /srv/app

sudo vgextend data /dev/sdd                  # add a disk to the pool
sudo lvextend -r -l +100%FREE data/app       # grow LV + filesystem

sudo pvmove /dev/sdb                         # move data off a disk
sudo vgreduce data /dev/sdb                  # then remove it

sudo pvs; sudo vgs; sudo lvs -a -o +devices  # summaries

Key points

  • -L 50G sets an absolute size; -l +100%FREE uses all free extents.
  • -r (--resizefs) resizes the filesystem in the same step.
  • pvmove migrates data online, useful for replacing a disk.
  • Shrinking needs an ext4 filesystem shrunk first; xfs cannot shrink at all.
  • Keep some free space in the VG for snapshots and emergencies.
LVM snapshots

Point-in-time copies for safe changes and backups

A classic snapshot keeps original copies of blocks that change after it is taken, so it starts small and grows.

# classic snapshot with 5 GB of space for changes
sudo lvcreate -s -n app-snap -L 5G data/app
sudo lvs data                         # Data% shows how full the snapshot is

# consistent backup from the snapshot
sudo mkdir -p /mnt/snap
sudo mount -o ro,nouuid /dev/data/app-snap /mnt/snap   # nouuid: xfs only
sudo tar -czf /backup/app.tgz -C /mnt/snap .
sudo umount /mnt/snap && sudo lvremove -y data/app-snap

# roll back: merge the snapshot into the origin
# deferred to next activation if the origin is in use
sudo lvconvert --merge data/app-snap

# thin provisioning
sudo lvcreate --type thin-pool -L 100G -n pool data
sudo lvcreate -V 50G --thin -n vol1 data/pool

Watch out

  • A classic snapshot becomes invalid if its space fills up — monitor Data%.
  • Snapshots slow down writes to the origin while they exist.
  • Freeze or stop the app first for application-consistent data (databases).
  • Thin pools allow overcommit; running out of pool space is serious — alert early.
  • A snapshot on the same disks is not a backup: copy it elsewhere.
Swap

Swap files, swappiness, and zram

Swap gives the kernel somewhere to move cold memory; it cannot fix a workload that needs more RAM.

swapon --show                     # active swap areas
free -h

sudo fallocate -l 2G /swapfile    # or: dd if=/dev/zero ...
sudo chmod 600 /swapfile
sudo mkswap /swapfile
sudo swapon /swapfile
echo '/swapfile none swap sw 0 0' | sudo tee -a /etc/fstab

sysctl vm.swappiness              # 60 is a common default
vmstat 1                          # si/so: pages swapped in/out

Guidance

  • A small swap area lets the kernel move rarely used pages out of RAM.
  • Constant swap-in/out (si/so) means the working set does not fit in memory.
  • zram creates compressed swap in RAM; Fedora enables it by default.
  • Kubernetes nodes traditionally run without swap; newer versions support it with configuration.
  • Swap files on btrfs need special handling (no copy-on-write).
Filesystem repair

fsck, xfs_repair, and read-only remounts

Repair tools must run on unmounted filesystems; running them on a mounted one causes damage.

dmesg -T | grep -iE 'ext4-fs error|xfs.*(corrupt|shutdown)'
findmnt -o TARGET,OPTIONS /srv       # did it become "ro"?

sudo umount /srv
sudo e2fsck -f /dev/sdb1             # ext4: force a full check
sudo xfs_repair -n /dev/sdb1         # xfs: dry run, report only
sudo xfs_repair /dev/sdb1            # xfs: repair

sudo tune2fs -l /dev/sdb1 | grep -iE 'state|errors|mount count'
# root filesystem: boot with fsck.mode=force, or repair from rescue media

Decision guide

  • ext4 with errors=remount-ro turns read-only on errors to prevent more damage.
  • Check the device first: I/O errors in dmesg mean hardware, not the filesystem.
  • Snapshot or image the volume before repairing important data.
  • xfs: if xfs_repair reports a dirty log, mount and unmount once to replay it; use -L only as a last resort (it discards the log).
  • In the cloud, attach the volume to a healthy instance to repair it.
Disk encryption

LUKS with cryptsetup

LUKS encrypts a block device; the filesystem on top works normally once unlocked.

sudo cryptsetup luksFormat /dev/sdb1         # destroys existing data
sudo cryptsetup open /dev/sdb1 secure         # → /dev/mapper/secure
sudo mkfs.ext4 /dev/mapper/secure
sudo mount /dev/mapper/secure /secure

sudo cryptsetup luksDump /dev/sdb1            # version, key slots, cipher
sudo cryptsetup luksAddKey /dev/sdb1          # add a second passphrase/key
sudo cryptsetup luksHeaderBackup /dev/sdb1 --header-backup-file sdb1.hdr

# /etc/crypttab  name  device          key   options
secure  UUID=8a1e…  /etc/keys/sdb1.key  luks
# /etc/fstab
/dev/mapper/secure  /secure  ext4  defaults,nofail  0  2

Operational notes

  • LUKS2 is the default format; up to 32 key slots allow key rotation without re-encrypting.
  • Losing every key or a damaged header means losing the data — back up the header.
  • Encryption at rest protects stolen disks, not a running, unlocked system.
  • Automatic unlock at boot needs a key file, TPM (systemd-cryptenroll), or network unlock.
  • Cloud volume encryption (e.g. EBS + KMS) is often simpler; LUKS adds a key the provider does not hold.
NFS

Network file storage done safely

NFS shares a directory over the network; AWS EFS is NFSv4.1.

# server: /etc/exports
/srv/share  10.0.0.0/24(rw,sync,root_squash)
sudo exportfs -ra                    # apply changes
sudo exportfs -v                     # active exports

# client
sudo mount -t nfs4 -o hard,nconnect=4 fs01:/srv/share /mnt/share
# /etc/fstab
fs01:/srv/share  /mnt/share  nfs4  hard,_netdev,nofail,nconnect=4  0 0

nfsstat -m                           # mount options in effect
nfsstat -c                           # client RPC stats, retransmissions

Behavior to expect

  • hard (default) retries forever: processes hang in D state when the server is gone, but data stays safe.
  • soft returns errors after timeouts and can corrupt data for writers.
  • root_squash maps client root to nobody; UIDs must match across hosts.
  • _netdev waits for the network at boot; nconnect opens several TCP connections (kernel 5.3+).
  • “Stale file handle” (ESTALE): the file or export changed on the server; remount.
Backup and restore

A backup only counts if you have restored it

Snapshots, replicas, and RAID protect against different failures than backups do.

export RESTIC_REPOSITORY=s3:s3.amazonaws.com/my-backups/web1
export RESTIC_PASSWORD_FILE=/etc/restic/password

restic init                              # once
restic backup /etc /srv/app --exclude-caches
restic snapshots
restic forget --keep-daily 7 --keep-weekly 4 --keep-monthly 6 --prune
restic check                             # verify repository integrity
restic restore latest --target /tmp/restore --include /etc/nginx

# databases: dump first, then back up the dump
pg_dump -Fc mydb > /backup/mydb.dump

Principles

  • 3-2-1: three copies, two media types, one copy off-site or in another account.
  • Define RPO (how much data you can lose) and RTO (how long a restore may take).
  • Back up databases with their own tools or consistent snapshots, not by copying live files.
  • Protect backups from the hosts they back up: separate credentials, immutability (e.g. S3 Object Lock).
  • Schedule restore tests and time them.
Chapter 3

Users and security

Control who can do what, then harden the host against misuse.

Chapter 3 · Part 3.1

Users and permissions

Who can do what, and how to grant only what is needed.

Users and groups

Identity is a number

The kernel checks numeric UIDs and GIDs; names are looked up in databases such as /etc/passwd.

# /etc/passwd  name:x:UID:GID:comment:home:shell
alice:x:1001:1001:Alice:/home/alice:/bin/bash
nginx:x:998:998::/var/lib/nginx:/usr/sbin/nologin

id alice                  # uid, gid, groups
getent passwd alice       # works with LDAP/SSSD too
sudo useradd -m -s /bin/bash bob
sudo usermod -aG docker bob   # -a: append, keep other groups
sudo passwd -l bob            # lock password login
sudo userdel -r bob           # remove user and home

Key facts

  • UID 0 is root and bypasses most permission checks.
  • Regular users usually start at UID 1000 (UID_MIN in /etc/login.defs).
  • Password hashes live in /etc/shadow, readable only by root.
  • Service accounts use nologin as their shell.
  • New group membership applies at the next login (or newgrp).
  • Container volumes show numeric UIDs; a mismatch looks like “wrong owner”.
Permission bits

Read -rwxr-x--- correctly

Three classes (user, group, other) × three rights (read, write, execute).

-rwxr-x---  1 app app 4096 Oct 16 10:00 deploy.sh
 └┬┘└┬┘└┬┘
  │  │  └─ other: ---   0
  │  └──── group: r-x   4+1   = 5
  └─────── user:  rwx   4+2+1 = 7    → mode 750

chmod 640 config.yml        # rw-r-----
chmod u+x,g-w script.sh     # symbolic form
chmod -R u=rwX,go= private/ # X: execute only on dirs
chown app:app /srv/app
chown -R app: /srv/app      # group = app's login group

Files vs directories

  • File r: read content; w: modify content; x: execute.
  • Directory r: list names; x: enter and access entries by name.
  • Directory w (with x): create, rename, and delete entries.
  • Deleting a file depends on the directory’s permissions, not the file’s.
  • To reach /a/b/c, you need x on every directory in the path.
Special bits and defaults

umask, setuid, setgid, sticky bit, and ACLs

These features explain many “why does this file have that permission?” questions.

FeatureNumericEffectExample
umask022, 027Removes bits from new files/dirs022 → files 644, dirs 755
setuid4xxxExecutable runs as the file’s owner/usr/bin/passwd
setgid (file)2xxxExecutable runs with the file’s groupRare today
setgid (dir)2xxxNew files inherit the directory’s groupShared team directories
sticky (dir)1xxxOnly owners can delete their files/tmp is drwxrwxrwt
ACL+ in ls -lExtra per-user/group entriessetfacl -m u:bob:r file

Audit setuid binaries with find / -xdev -perm -4000 -type f. An unexpected setuid file is a security finding.

sudo

Borrow privileges, with an audit trail

sudo runs a command as another user (root by default) according to rules in sudoers.

Configuration

  • Edit with visudo; it checks syntax before saving.
  • Prefer drop-ins in /etc/sudoers.d/.
  • Grant specific commands, e.g. deploy ALL=(root) /usr/bin/systemctl restart myapp.
  • sudo -l lists what you are allowed to run.

Everyday use

  • sudo -i starts a root login shell; sudo -s keeps more of your environment.
  • sudo -u postgres psql runs as another user.
  • Usage is logged to the journal or /var/log/auth.log / /var/log/secure.
  • Avoid shared root passwords; give each admin their own account.

Classic pitfall

  • sudo echo x > /etc/file fails: your shell, not root, opens the file.
  • Use echo x | sudo tee /etc/file (or tee -a to append).
  • Wildcards in sudoers rules can grant far more than intended.
  • Allowing editors or shells via sudo grants full root.
Beyond rwx

Capabilities, SELinux, and AppArmor

Root is split into capabilities; Mandatory Access Control adds rules even root must follow.

Capabilities

  • Fine-grained root powers, e.g. CAP_NET_BIND_SERVICE for ports below 1024.
  • CAP_SYS_ADMIN is very broad — treat it like root.
  • getcap /usr/bin/ping; setcap cap_net_bind_service=+ep ./server.
  • systemd: AmbientCapabilities= and CapabilityBoundingSet=.

SELinux (RHEL family)

  • Labels every file and process; policy decides allowed access.
  • getenforce, ls -Z, ps -Z show mode and labels.
  • Denials: ausearch -m avc -ts recent.
  • Fix labels with restorecon -Rv /path; toggle booleans with setsebool -P.

AppArmor (Ubuntu, Debian)

  • Path-based profiles per program.
  • aa-status lists loaded profiles and modes.
  • Denials appear in the kernel log / journal.
  • Profiles can be set to complain mode while tuning.

Do not disable SELinux or AppArmor to make something work. Read the denial, then fix the label, boolean, or profile.

SSH

Secure remote access

SSH provides encrypted shells, file transfer, and tunnels; configure both client and server deliberately.

ssh-keygen -t ed25519 -C "alice@laptop"
ssh-copy-id alice@server

# ~/.ssh/config
Host bastion
  HostName 203.0.113.10
  User alice
Host app-*
  ProxyJump bastion
  User ec2-user

ssh -L 5432:db.internal:5432 bastion   # local port forward
ssh -D 1080 bastion                    # SOCKS proxy

Server hardening

  • PasswordAuthentication no and PermitRootLogin no in sshd_config.
  • Validate with sudo sshd -t before reloading.
  • Keep one session open while changing SSH settings.
  • ~/.ssh and authorized_keys must not be group/world-writable (use 700/600), or sshd ignores them.
  • Verify host key fingerprints; a changed key can mean a rebuilt host or an attack.
  • Consider SSM Session Manager or similar instead of exposing port 22.
sudoers syntax

Write sudo rules that grant exactly what is needed

Each rule says who may run what, on which host, as which user.

# sudo visudo -f /etc/sudoers.d/deploy
# who   host=(as user)  [tag:] commands
deploy  ALL=(root) NOPASSWD: /usr/bin/systemctl restart myapp, \
          /usr/bin/systemctl --no-pager status myapp
%ops       ALL=(ALL:ALL) ALL

Cmnd_Alias PKG = /usr/bin/apt-get update, /usr/bin/apt-get upgrade
alice      ALL=(root) PKG

Defaults   use_pty
Defaults   logfile=/var/log/sudo.log
Defaults:deploy  !requiretty

$ sudo -l -U deploy          # what deploy may run
$ sudo visudo -c             # syntax check all files

Rules of thumb

  • %name refers to a group; ALL=(ALL:ALL) ALL is full root.
  • If several rules match, the last one wins — order matters.
  • Always give full paths; arguments are matched literally unless wildcards are used.
  • Programs with shell escapes (editors, pagers such as less, find -exec) grant a root shell — hence --no-pager above.
  • Use sudoedit to let users edit specific files safely.
  • Files in /etc/sudoers.d/ must not contain . or end in ~, or they are skipped.
Chapter 3 · Part 3.2

Security hardening

Reduce attack surface, enforce policy, and detect change.

Hardening approach

Start from a baseline, then measure

Hardening is a repeatable process, not a one-time list of tweaks.

Reduce the attack surface

  • Install only needed packages; remove compilers from production images.
  • Disable unused services: systemctl list-unit-files --state=enabled.
  • Close unneeded ports: ss -tulpn should hold no surprises.
  • Prefer minimal images and immutable infrastructure.

Enforce least privilege

  • Individual accounts, no shared root password.
  • Narrow sudo rules and service users per application.
  • Keep SELinux/AppArmor enforcing; sandbox services with systemd.
  • Restrict SSH to keys from known networks or a bastion.

Measure and maintain

  • Use a benchmark: CIS Benchmarks or DISA STIG.
  • Scan with OpenSCAP (oscap) or Lynis (lynis audit system).
  • Patch regularly and track kernel reboots.
  • Log centrally and alert on security events.

Apply hardening through code (Ansible, image builds) so every new host starts compliant and drift is visible.

SSH server hardening

A safe sshd configuration

Use a drop-in file and validate before reloading.

# /etc/ssh/sshd_config.d/10-hardening.conf
PermitRootLogin no
PasswordAuthentication no
KbdInteractiveAuthentication no
PubkeyAuthentication yes
AuthenticationMethods publickey
MaxAuthTries 3
LoginGraceTime 30
AllowGroups ssh-users
X11Forwarding no
AllowAgentForwarding no
ClientAliveInterval 300
ClientAliveCountMax 2

$ sudo sshd -t                 # syntax check
$ sudo sshd -T | grep -i passwordauth   # effective value
$ sudo systemctl reload ssh    # RHEL: sshd

Why it works this way

  • For most keywords sshd uses the first value it reads; Include of sshd_config.d/*.conf sits near the top of the main file, so drop-ins win.
  • sshd -T prints the effective configuration — trust it over reading files.
  • AllowGroups makes access explicit; everyone else is refused.
  • Keep an existing session open and test a new login before closing it.
  • Disable forwarding features you do not use; enable them per user with Match blocks.
SSH certificates

Replace scattered authorized_keys with a CA

An SSH certificate authority signs short-lived user keys; servers trust the CA, not each key.

# once: create the user CA (keep the private key offline or in a vault)
ssh-keygen -t ed25519 -f user_ca -C "user CA"

# sign alice's key for 8 hours, allowed principals: alice, deploy
ssh-keygen -s user_ca -I alice@corp -n alice,deploy -V +8h id_ed25519.pub
# → id_ed25519-cert.pub   (ssh uses it automatically)
ssh-keygen -L -f id_ed25519-cert.pub        # inspect validity and principals

# servers: /etc/ssh/sshd_config.d/20-ca.conf
TrustedUserCAKeys /etc/ssh/user_ca.pub
AuthorizedPrincipalsFile /etc/ssh/principals/%u

Benefits and duties

  • Access expires automatically — no stale keys left on servers.
  • Principals map certificates to accounts or roles.
  • Host certificates (ssh-keygen -h) remove “trust on first use” prompts.
  • Protect the CA private key: whoever holds it can log in everywhere.
  • Revoke with RevokedKeys; tools like Vault or Teleport automate issuing.
PAM

Pluggable Authentication Modules

PAM decides how login, sudo, sshd, and other services authenticate users and set up sessions.

# /etc/pam.d/sshd (excerpt, varies by distribution)
auth       required     pam_faillock.so preauth
auth       include      common-auth        # Debian; RHEL: password-auth
account    required     pam_nologin.so
account    include      common-account
password   include      common-password
session    required     pam_limits.so
session    optional     pam_motd.so

ls /etc/pam.d/                # one file per service
man pam.conf                  # syntax

How a stack works

  • Module types: auth (who are you), account (are you allowed now), password (change secrets), session (set up and clean up).
  • Controls: required (must pass, keep going), requisite (stop on failure), sufficient (success ends the stack), optional.
  • pam_limits applies /etc/security/limits.conf — why it affects logins but not systemd services.
  • Use distribution tools (authselect on RHEL, pam-auth-update on Debian) instead of hand-editing shared stacks.
  • A broken PAM file can lock everyone out — keep a root session open.
Accounts and passwords

Lockout, password quality, and account lifetime

Even key-only SSH hosts have local passwords for console and sudo.

# /etc/security/faillock.conf
deny = 5
unlock_time = 900
sudo faillock --user alice              # failures
sudo faillock --user alice --reset      # unlock

# /etc/security/pwquality.conf
minlen = 14
minclass = 3

sudo chage -l alice                     # aging information
sudo chage -E 2026-12-31 contractor     # account expiry date
sudo usermod -L olduser                 # lock the password
sudo usermod -s /usr/sbin/nologin olduser

/etc/shadow fields

FieldMeaning
2Hash: $y$ yescrypt, $6$ SHA-512; ! or * = locked
3Last change (days since 1970-01-01)
4 / 5Minimum / maximum days between changes
6Warning days before expiry
7Inactive days allowed after expiry
8Account expiration date

Current NIST guidance favors long passwords and MFA over forced periodic rotation. Remove or expire accounts as soon as people leave.

Kernel hardening

sysctl settings that reduce risk

Put these in /etc/sysctl.d/ and test them; some interfere with routers, containers, or debugging.

SettingValuePurpose
kernel.kptr_restrict2Hide kernel addresses from users
kernel.dmesg_restrict1Only privileged users can read the kernel log
kernel.yama.ptrace_scope1Processes can only trace their own children
kernel.unprivileged_bpf_disabled1Block eBPF for unprivileged users
fs.protected_symlinks / _hardlinks1Stop link attacks in shared directories like /tmp
fs.suid_dumpable0No core dumps from setuid programs
net.ipv4.tcp_syncookies1Survive SYN floods
net.ipv4.conf.all.accept_redirects0Ignore ICMP redirects
net.ipv4.conf.all.rp_filter1 (strict) / 2 (loose)Drop spoofed sources; strict breaks asymmetric routing

Container and Kubernetes hosts need net.ipv4.ip_forward = 1; do not copy a desktop hardening profile onto a node blindly.

Host firewall

A default-deny nftables baseline

Allow only what the host serves, and keep established connections working.

# /etc/nftables.conf  (RHEL: /etc/sysconfig/nftables.conf)
flush ruleset
table inet filter {
  chain input {
    type filter hook input priority 0; policy drop;
    ct state established,related accept
    ct state invalid drop
    iif "lo" accept
    meta l4proto { icmp, ipv6-icmp } accept
    tcp dport 22 ip saddr 10.0.0.0/8 accept
    tcp dport { 80, 443 } accept
  }
}

sudo nft -c -f /etc/nftables.conf     # check syntax only
sudo systemctl enable --now nftables
sudo nft list ruleset

Practical rules

  • Allow ICMP: path MTU discovery and IPv6 neighbor discovery depend on it.
  • inet tables cover IPv4 and IPv6 together.
  • Use one manager per host: raw nftables, ufw, or firewalld — not several.
  • Docker and Kubernetes add their own rules; published container ports can bypass simple INPUT rules.
  • In the cloud, security groups are the first layer; the host firewall is defense in depth.
Brute-force protection

fail2ban

fail2ban watches logs for repeated failures and temporarily blocks the source address.

# /etc/fail2ban/jail.local
[DEFAULT]
bantime  = 1h
findtime = 10m
maxretry = 5
ignoreip = 127.0.0.1/8 10.0.0.0/8

[sshd]
enabled = true
backend = systemd

sudo systemctl enable --now fail2ban
sudo fail2ban-client status sshd
sudo fail2ban-client set sshd unbanip 203.0.113.50

Know its limits

  • Edit jail.local, not jail.conf, so upgrades keep your settings.
  • backend = systemd reads the journal on hosts without /var/log/auth.log.
  • It reduces log noise and slows guessing; key-only authentication is the real protection.
  • Distributed attacks from many addresses are not stopped by per-IP bans.
  • Add your bastion and monitoring ranges to ignoreip to avoid locking yourself out.
Auditing

Record security-relevant events with auditd

The Linux audit system logs file access, command execution, and identity changes in the kernel.

# /etc/audit/rules.d/50-custom.rules
-w /etc/passwd -p wa -k identity
-w /etc/shadow -p wa -k identity
-w /etc/sudoers -p wa -k sudoers
-w /etc/sudoers.d/ -p wa -k sudoers
-w /etc/ssh/sshd_config -p wa -k sshd
-a always,exit -F arch=b64 -S execve -F euid=0 -k root-exec
# /etc/audit/rules.d/99-finalize.rules (loaded last)
-e 2

sudo augenrules --load                # build and load the rules
sudo auditctl -l                      # rules in effect
sudo ausearch -k identity -i --start today
sudo aureport --auth --summary

Read the rules

  • -w path -p wa: watch writes and attribute changes on a file or directory.
  • -k tags events so ausearch -k can find them.
  • -S execve -F euid=0: log every program run as root.
  • -e 2 (last file, own line) makes rules immutable until reboot; audit rules allow no trailing comments.
  • Forward audit logs off the host; local logs can be wiped after a compromise.
  • Broad syscall rules are expensive — start narrow.
Integrity and secrets

Detect tampering and keep secrets out of sight

Know when system files change, and never leave credentials where other processes can read them.

File integrity (AIDE)

  • aide --init builds a database of hashes and attributes (Ubuntu: aideinit).
  • Keep a trusted copy of the database off the host.
  • aide --check reports added, removed, and changed files.
  • Package verification: rpm -Va or debsums -s.

Where secrets leak

  • Command-line arguments: visible to all users in ps.
  • Environment variables: readable in /proc/<pid>/environ and often dumped in crash logs.
  • Shell history: prefix a command with a space only if HISTCONTROL=ignorespace.
  • World-readable files and backups of config directories.

Better options

  • Files with mode 0600 owned by the service user.
  • systemd credentials: LoadCredential=db:/etc/myapp/db.pass → $CREDENTIALS_DIRECTORY/db.
  • A secret manager (AWS Secrets Manager, Vault) fetched at start-up.
  • Rotate secrets and audit who can read them.
Patching

Find, apply, and verify security updates

Most real-world compromises use known vulnerabilities with available fixes.

# Debian / Ubuntu
apt list --upgradable
sudo unattended-upgrade --dry-run -d
cat /var/run/reboot-required 2>/dev/null

# RHEL family / Amazon Linux 2023
dnf updateinfo list --security
sudo dnf upgrade --security
sudo needs-restarting -r            # reboot needed?
sudo needs-restarting -s            # services to restart

# containers and images
trivy image myapp:1.4.2
trivy fs --scanners vuln /srv/app

Process

  • Automate security updates for the base OS; schedule reboots.
  • Restart services after library updates: they keep old libraries mapped in memory.
  • Kernel live patching (Ubuntu Livepatch, kpatch) delays reboots for critical fixes.
  • Rebuild container images regularly, even if your code did not change.
  • Track exposure with a scanner and prioritize internet-facing hosts.
Chapter 4

Running software

Processes, services, and the packages they come from.

Chapter 4 · Part 4.1

Processes and signals

How programs run, communicate, and stop.

Processes

A process is a running program plus its state

The same program can run as many processes, each with its own identity and resources.

Identity

  • PID and parent PID (PPID).
  • Real and effective UID/GID.
  • Working directory, environment, and command line.
  • Namespaces and cgroup membership.

Resources

  • Virtual memory map: code, heap, stack, shared libraries.
  • Open file descriptors: files, sockets, pipes.
  • Resource limits (ulimit) and scheduling priority.
  • All visible under /proc/<pid>/.

Threads

  • Threads share memory and file descriptors within a process.
  • Linux schedules each thread as a task with its own TID.
  • ps -eLf or top -H shows threads.
  • CPU% above 100% means several threads are busy.
Process lifecycle

fork, exec, exit, and wait

New processes are copies of their parent that then load a new program.

bash (PID 100)
 └─ fork()  → child PID 101 (copy of bash)
      └─ execve("/usr/bin/grep", …) → now grep
           └─ exit(0)
                → zombie until bash calls wait()
                → bash reads status into $?

parent exits first?
 → child is reparented to PID 1 (or a subreaper)

What to remember

  • Copy-on-write makes fork() cheap: memory is copied only when written.
  • A zombie has exited and holds no memory; it only keeps a process-table entry for its exit status.
  • Many zombies mean a parent is not calling wait() — fix the parent.
  • In a container, your app may be PID 1 and must reap children; use tini or docker run --init.
  • PID 1 also ignores signals it has no handler for, which affects shutdown.
Process states

Read the STAT column

The state explains whether a process is working, waiting, or stuck.

StateMeaningWhat it suggests
RRunning or runnableUsing or waiting for CPU
SInterruptible sleepWaiting for an event: normal for idle daemons
DUninterruptible sleepUsually waiting on I/O (disk, NFS); cannot be killed; counts toward load
ZZombieExited, not yet reaped by its parent
TStoppedPaused by Ctrl-Z, SIGSTOP, or a debugger
IIdle kernel threadNormal; ignore
ps -eo pid,ppid,stat,etime,cmd --forest | less
ps -eo stat,pid,cmd | awk '$1 ~ /^D/'     # processes stuck in D state

Extra STAT letters: s session leader, l multithreaded, + foreground, < high priority, N low priority.

Signals

Asynchronous messages to a process

Signals stop, reload, or interrupt processes; handlers decide most of the behavior.

Common signals (x86-64 numbers)

SignalNo.Typical meaning
SIGHUP1Terminal closed; many daemons reload config
SIGINT2Ctrl-C
SIGQUIT3Ctrl-\; may dump core
SIGKILL9Kill immediately; cannot be caught
SIGTERM15Polite stop request (default for kill)
SIGCHLD17A child exited
SIGSTOP / SIGCONT19 / 18Pause / resume; STOP cannot be caught

Graceful shutdown

  • kill PID sends SIGTERM; use kill -9 only as a last resort.
  • docker stop sends SIGTERM, then SIGKILL after 10 s by default.
  • Kubernetes waits terminationGracePeriodSeconds (30 s default) before SIGKILL.
  • Shell-form CMD npm start runs under sh -c, which may not forward signals; use exec form.
  • Handle SIGTERM: stop accepting work, finish in-flight requests, then exit.
Inspecting processes

Find what is running and what it holds

Start broad, then drill into one PID.

ps aux --sort=-%mem | head          # top memory users
ps aux --sort=-%cpu | head          # top CPU users
pgrep -af nginx                     # PIDs + full command lines
top -o %MEM                         # or htop
pidstat -u -r 1 5                   # per-process CPU and memory

cat /proc/<pid>/status | grep -E 'State|VmRSS|Threads'
ls -l /proc/<pid>/fd | wc -l        # open file descriptors
sudo lsof -p <pid>                  # files and sockets
sudo ss -tnp | grep <pid>           # its TCP connections
pstree -p <pid>                     # children

Read the output

  • VSZ is virtual size (reserved); RSS is resident physical memory.
  • RSS counts shared libraries in every process that uses them; sum of RSS overstates usage.
  • %CPU in ps is a lifetime average; top and pidstat show current rates.
  • top keys: P sort by CPU, M by memory, 1 per-CPU, H threads.
  • lsof reveals which process holds a port, a deleted file, or a mount point.
Jobs and priority

Foreground, background, and scheduling weight

The scheduler shares CPUs between runnable tasks; niceness and I/O class adjust the share.

Job control

  • cmd & runs in the background; jobs lists jobs.
  • Ctrl-Z stops the foreground job; bg resumes it in the background, fg brings it back.
  • nohup cmd & ignores SIGHUP; disown detaches a job from the shell.
  • setsid cmd starts it in a new session.

CPU priority

  • Nice values range from -20 (highest priority) to 19 (lowest).
  • nice -n 10 ./batch.sh, sudo renice -n 5 -p <pid>.
  • Only root can raise priority (lower the nice value).
  • Linux 6.6 replaced the CFS scheduler with EEVDF; nice still maps to weight.

I/O priority

  • ionice -c3 cmd runs at idle I/O priority.
  • ionice -c2 -n7 is best-effort, lowest level.
  • Effect depends on the I/O scheduler and device.
  • For hard limits, use cgroups (IOReadBandwidthMax= in systemd).
/proc in practice

Ask the kernel directly

Every tool on the previous slides reads these files; you can too.

PathWhat you learn
/proc/<pid>/cmdlineFull command line (NUL-separated): tr '\0' ' ' < …
/proc/<pid>/environEnvironment at start — may contain secrets; root/owner only
/proc/<pid>/statusState, UIDs, VmRSS, threads, capabilities
/proc/<pid>/fd/Open file descriptors as symlinks
/proc/<pid>/limitsEffective resource limits of that process
/proc/<pid>/cgroupWhich cgroup (service or container) owns it
/proc/loadavg, /proc/meminfoLoad and memory as the kernel sees them
/proc/pressure/{cpu,memory,io}Pressure Stall Information (PSI)
Reading top

Every number in the top header

The header summarizes the whole system; the table ranks processes.

top - 10:42:01 up 12 days,  3:04,  2 users,  load average: 3.12, 2.80, 2.41
Tasks: 241 total,   2 running, 239 sleeping,   0 stopped,   0 zombie
%Cpu(s): 35.2 us,  4.1 sy,  0.0 ni, 58.9 id,  1.2 wa,  0.0 hi,  0.6 si,  0.0 st
MiB Mem :  15832.4 total,   1210.5 free,   4920.3 used,   9701.6 buff/cache
MiB Swap:   2048.0 total,   2048.0 free,      0.0 used.  10589.1 avail Mem

    PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND
   4121 app       20   0 8123456   3.7g  28432 S 142.0  23.9 812:33.10 java
    981 postgres  20   0  411220  98120  92340 S   6.3   0.6  40:12.55 postgres

Read the output

  • load average over 1, 5, 15 minutes; compare with CPU count.
  • avail Mem is the realistic free memory, including reclaimable cache.
  • VIRT reserved address space, RES resident memory, SHR shared part of RES.
  • %CPU 142 = about 1.4 CPUs across several threads.
  • TIME+ is total CPU time consumed since start.
  • PR/NI show scheduling priority and nice value.
Reading ps

Decode ps aux line by line

Most columns answer a specific question about the process.

$ ps aux
USER   PID %CPU %MEM     VSZ     RSS TTY STAT START   TIME COMMAND
root     1  0.0  0.1  171820   13140 ?   Ss   Oct04   1:02 /usr/lib/systemd/systemd
app   4121  4.7 23.9 8123456 3880000 ?   Sl   Oct04 812:33 java -jar api.jar
app   5010  0.0  0.0       0       0 ?   Z    10:40   0:00 [worker] <defunct>
root   612  0.0  0.0       0       0 ?   D    09:58   0:04 [kworker/u8:2+flush-259:0]
alice 7710  0.0  0.0    9012    3360 pts/0 R+ 10:43   0:00 ps aux

Read the output

  • VSZ and RSS are in KiB.
  • TTY ? = no terminal (daemon); pts/0 = an SSH or terminal session.
  • Ss = sleeping session leader; Sl = sleeping, multithreaded; R+ = running in the foreground.
  • <defunct> with Z is a zombie — look at its parent with ps -o ppid= -p 5010.
  • Names in [brackets] are kernel threads; a kworker in D points at I/O (here a flush to device 259:0).
  • %CPU is averaged over the whole lifetime, unlike top.
Inter-process communication

How processes talk to each other

Choosing and inspecting the channel helps when one side stops responding.

Pipes and FIFOs

  • cmd1 | cmd2: an anonymous pipe with a kernel buffer (64 KiB by default).
  • A full pipe blocks the writer; a closed reader causes SIGPIPE.
  • mkfifo creates a named pipe in the filesystem.

Unix domain sockets

  • Local sockets with a path, e.g. /run/docker.sock, /run/postgresql/.s.PGSQL.5432.
  • Faster than TCP loopback and protected by file permissions.
  • ss -xlp lists listening Unix sockets and owners.

Shared memory and others

  • /dev/shm (tmpfs) and System V shared memory: ipcs -m.
  • Signals carry only a number, not data.
  • TCP on 127.0.0.1 when components may later move to other hosts.

Access to a control socket such as /run/docker.sock or /run/containerd/containerd.sock is equivalent to root on the host.

Chapter 4 · Part 4.2

systemd and services

Start, supervise, and log long-running software.

systemd concepts

Units describe everything systemd manages

systemd starts services in dependency order, supervises them, and records their logs.

Unit types

  • .service: a process to run and supervise.
  • .timer: scheduled activation (cron replacement).
  • .socket, .mount, .path: activation by socket, mount, or file change.
  • .target: a named group of units, e.g. multi-user.target.

Where unit files live

  • /usr/lib/systemd/system/: shipped by packages.
  • /etc/systemd/system/: administrator files and overrides (wins).
  • /run/systemd/system/: runtime units, lost on reboot.
  • systemctl cat name shows the effective file and drop-ins.

Dependencies

  • Wants= / Requires=: also start these units.
  • After= / Before=: ordering only.
  • Requirement and ordering are separate — you usually need both.
  • systemctl list-dependencies name shows the tree.
systemctl

The everyday commands

Starting a service and enabling it at boot are different actions.

CommandEffect
systemctl status myappState, PID, recent log lines, exit code
systemctl start | stop | restart myappChange state now
systemctl reload myappAsk the service to reload config without restarting
systemctl enable --now myappStart at boot and start now
systemctl daemon-reloadRe-read unit files after editing them
systemctl edit myappCreate a drop-in override in /etc
systemctl --failedList units that failed
systemctl mask myappPrevent it from being started at all
systemctl is-active / is-enabled myappScript-friendly checks via exit status
Writing a unit

A production-ready service file

Run as an unprivileged user, restart on failure, and harden the sandbox.

# /etc/systemd/system/myapp.service
[Unit]
Description=My API
After=network-online.target
Wants=network-online.target
StartLimitIntervalSec=300
StartLimitBurst=5

[Service]
Type=simple
User=myapp
WorkingDirectory=/opt/myapp
EnvironmentFile=/etc/myapp/env
ExecStart=/opt/myapp/bin/server --port 8080
Restart=on-failure
RestartSec=5
LimitNOFILE=65536
NoNewPrivileges=yes
ProtectSystem=strict
ReadWritePaths=/var/lib/myapp
PrivateTmp=yes

[Install]
WantedBy=multi-user.target

Key settings

  • Type=simple: the started process is the service. notify waits for readiness; oneshot for scripts; forking for legacy daemons.
  • Restart=on-failure restarts after crashes but not after a clean stop.
  • Too many restarts hit StartLimitBurst → “start request repeated too quickly”; systemctl reset-failed clears it.
  • ProtectSystem=strict makes the filesystem read-only except ReadWritePaths=.
  • systemd-analyze security myapp scores the sandbox.
journald

Query logs by unit, time, and priority

The journal stores structured log entries with metadata such as unit, PID, and boot ID.

journalctl -u myapp -f                 # follow one unit
journalctl -u myapp --since "10 min ago"
journalctl -u myapp -n 100 --no-pager
journalctl -p err -b                   # errors since this boot
journalctl -b -1 -e                    # end of previous boot
journalctl -k                          # kernel messages
journalctl _PID=1234                   # one process
journalctl -u myapp -o json-pretty | less
journalctl --disk-usage

Read the output

  • Priorities: 0 emerg … 3 err, 4 warning, 6 info, 7 debug.
  • Previous boots (-b -1) need persistent storage: /var/log/journal must exist or Storage=persistent.
  • Services write to stdout/stderr; systemd captures it into the journal.
  • Some systems also forward to rsyslog: /var/log/syslog (Debian) or /var/log/messages (RHEL).
  • Use --since/--until with exact timestamps when correlating incidents.
Scheduling

cron versus systemd timers

Both run jobs on a schedule; timers integrate with logging, dependencies, and resource limits.

# crontab -e   (min hour day-of-month month day-of-week)
*/5 * * * *  /opt/bin/sync.sh >> /var/log/sync.log 2>&1
0 3 * * 0    /opt/bin/weekly-report.sh

# /etc/systemd/system/sync.timer
[Timer]
OnCalendar=*:0/5
Persistent=true
[Install]
WantedBy=timers.target
# sync.service holds ExecStart=/opt/bin/sync.sh

systemctl enable --now sync.timer
systemctl list-timers
systemd-analyze calendar "Mon *-*-* 03:00"

Comparison

cronsystemd timer
LogsRedirect yourself or mailJournal, per unit
Missed runsSkipped (anacron helps)Persistent=true catches up
EnvironmentMinimal PATHDefined in the unit
OverlapCan start twiceNot while still running
LimitsNone built incgroup limits, sandboxing

“Works in my shell, fails in cron” is usually the minimal environment: use absolute paths and set variables explicitly.

Boot targets

Rescue modes and slow boots

Targets group units into system states; analysis tools show where boot time goes.

Targets

  • multi-user.target: normal server state.
  • graphical.target: adds a desktop.
  • rescue.target: single-user with basic services.
  • emergency.target: minimal shell, root read-only.

Inspect and change

  • systemctl get-default / set-default multi-user.target.
  • systemd-analyze blame: slowest units.
  • systemd-analyze critical-chain: what blocked boot.
  • Add systemd.unit=rescue.target to the kernel line in GRUB to boot into rescue.

Common boot failures

  • Bad /etc/fstab entry → emergency mode.
  • Full root filesystem → services fail to start.
  • Wrong kernel or missing driver in initramfs → cannot find root.
  • Cloud: use the serial console or attach the disk to another instance.
Reading systemctl status

Decode a failing service

The status output already answers most first questions.

$ systemctl status myapp
● myapp.service - My API
     Loaded: loaded (/etc/systemd/system/myapp.service; enabled; preset: disabled)
    Drop-In: /etc/systemd/system/myapp.service.d
             └─override.conf
     Active: activating (auto-restart) (Result: exit-code) since Thu 2026-10-16 10:41:55 +07; 3s ago
    Process: 9921 ExecStart=/opt/myapp/bin/server --port 8080 (code=exited, status=1/FAILURE)
   Main PID: 9921 (code=exited, status=1/FAILURE)
        CPU: 412ms

Oct 16 10:41:55 web1 server[9921]: error: open /etc/myapp/env: permission denied

Read the output

  • Loaded: the unit file path, whether it is enabled at boot, and the vendor preset.
  • Drop-In: overrides that change the unit — check them first.
  • activating (auto-restart): it failed and systemd will try again after RestartSec.
  • code=exited, status=1: the program exited with 1; code=killed, status=9/KILL means a signal.
  • The last log lines often contain the actual cause.
Drop-in overrides

Change a packaged unit without editing it

Overrides live in /etc and survive package upgrades.

$ sudo systemctl edit myapp
# opens an editor and saves to
# /etc/systemd/system/myapp.service.d/override.conf
[Service]
Environment=LOG_LEVEL=debug
MemoryMax=1G
ExecStart=
ExecStart=/opt/myapp/bin/server --port 9090

$ systemctl cat myapp           # unit + all drop-ins, in order
$ systemctl show myapp -p MemoryMax -p ExecStart
$ sudo systemctl restart myapp
$ sudo systemctl revert myapp   # remove all local overrides

Rules

  • Most settings in a drop-in replace or add to the original.
  • List settings such as ExecStart= must be cleared with an empty assignment before setting a new value.
  • systemctl edit reloads systemd for you; manual edits need daemon-reload.
  • systemctl edit --full copies the whole unit to /etc — upgrades no longer reach it.
  • systemctl show prints the effective values, including defaults.
Service sandboxing

systemd hardening directives

A few lines in a unit remove most of what a compromised service could reach.

DirectiveEffect
NoNewPrivileges=yessetuid binaries and file capabilities cannot raise privileges
ProtectSystem=strictWhole filesystem read-only except ReadWritePaths=
ProtectHome=yes/home, /root, /run/user are inaccessible
PrivateTmp=yesPrivate /tmp and /var/tmp for the service
PrivateDevices=yesOnly pseudo devices such as /dev/null are available
CapabilityBoundingSet=Limit capabilities; empty removes them all
RestrictAddressFamilies=AF_INET AF_INET6 AF_UNIXBlock other socket types
SystemCallFilter=@system-serviceAllow-list typical service syscalls (seccomp)
DynamicUser=yesRun as a temporary user created for the service
MemoryMax=, CPUQuota=, TasksMax=cgroup resource limits
Chapter 4 · Part 4.3

Packages and software

Install, update, and trust software reproducibly.

Package managers

The same tasks in three ecosystems

Learn the task, then the syntax for the distribution in front of you.

TaskDebian/UbuntuRHEL/Fedora/AL2023Alpine
Refresh metadataapt updatednf makecacheapk update
Installapt install nginxdnf install nginxapk add nginx
Searchapt search nginxdnf search nginxapk search nginx
Upgrade allapt upgradednf upgradeapk upgrade
Who owns a file?dpkg -S /usr/bin/curlrpm -qf /usr/bin/curlapk info --who-owns /usr/bin/curl
Files in a packagedpkg -L curlrpm -ql curlapk info -L curl
Which package provides?apt-file search bin/digdnf provides '*/dig'apk search cmd:dig
Hold a versionapt-mark hold pkgdnf versionlock add pkg (plugin)apk add pkg=1.2.3-r0
Repositories and updates

Trust, reproducibility, and patching

Packages are trusted because repository metadata is signed.

Repositories

  • APT: /etc/apt/sources.list.d/*.list or .sources, with signed-by= keyrings.
  • DNF: /etc/yum.repos.d/*.repo with gpgcheck=1.
  • apt-key is deprecated; store keys per repository.
  • Never disable signature checks to “fix” an install.

Patching

  • Apply security updates regularly (unattended-upgrades, dnf-automatic).
  • A kernel or glibc update needs a reboot or restart to take effect.
  • Ubuntu: /var/run/reboot-required; RHEL: needs-restarting -r.
  • Roll out to staging first; keep the previous kernel for rollback.

Reproducibility

  • Pin versions in container images and IaC.
  • Combine install and cache cleanup in one Dockerfile layer.
  • Prefer immutable images over patching servers by hand.
  • Record what changed and when; package logs are in /var/log/apt/ or dnf history.
Binaries and libraries

Why a program will not start

Most executables load shared libraries at runtime through the dynamic loader.

file /usr/bin/curl          # ELF type, architecture, dynamic/static
ldd /usr/bin/curl           # shared libraries it needs
type -a python3             # which binary runs
readlink -f "$(command -v java)"

# error: libssl.so.3: cannot open shared object file
ls /etc/ld.so.conf.d/
sudo ldconfig               # rebuild the library cache
LD_LIBRARY_PATH=/opt/lib ./app   # temporary override

Common causes

  • Missing library: install the package that provides it.
  • Wrong architecture: an x86-64 binary on an aarch64 host (e.g. Graviton).
  • glibc vs musl: a glibc binary may fail on Alpine with “not found” even though the file exists.
  • Statically linked binaries (common for Go) avoid library issues.
  • Locally built software belongs in /usr/local or /opt, not /usr.
Configuration habits

Change /etc safely

Configuration changes cause many incidents; validate before reloading.

Validate first

  • nginx -t, sshd -t, visudo -c, apachectl configtest.
  • systemd-analyze verify myapp.service.
  • named-checkconf, haproxy -c -f file.
  • Reload rather than restart when the service supports it.

Use drop-in directories

  • /etc/nginx/conf.d/, /etc/sysctl.d/, /etc/sudoers.d/.
  • Your file survives package upgrades of the main file.
  • Name files with prefixes (90-myapp.conf) to control order.
  • Upgrades may leave .dpkg-dist or .rpmnew files to review.

Track changes

  • Manage configuration with Ansible, cloud-init, or images.
  • Keep a backup before editing: cp file{,.bak}.
  • etckeeper can version /etc in Git.
  • Record who changed what, and why.
Versions and history

Which version is installed, and which will be?

Version mismatches between hosts cause “works on staging” bugs.

# Debian / Ubuntu
apt-cache policy nginx        # installed, candidate, repository priority
apt-cache madison nginx       # all available versions
sudo apt install nginx=1.24.0-2ubuntu7
grep " install " /var/log/dpkg.log | tail

# RHEL family
dnf --showduplicates list nginx
dnf info nginx
dnf history                   # transactions with IDs
dnf history info 42
sudo dnf history undo 42      # roll back a transaction
rpm -q --changelog openssl | head

Read the output

  • Installed vs Candidate in apt-cache policy shows whether an upgrade is pending.
  • Priorities explain why a package comes from one repository instead of another.
  • dnf history records who changed what and when — useful after an incident.
  • Changelogs list CVEs fixed in each package release.
  • Compare versions across hosts with your configuration management or inventory.
Chapter 5

Performance

Measure before tuning: CPU, memory, disk, and limits.

Load average

What the three numbers mean

uptime shows 1-, 5-, and 15-minute averages of tasks that are runnable or in uninterruptible sleep.

Interpretation

  • Compare load with the CPU count (nproc).
  • Load 8 on 8 CPUs ≈ fully busy; on 2 CPUs ≈ heavily queued.
  • The trend matters: 1-min ≫ 15-min means the problem just started.

Linux-specific twist

  • Linux counts D-state tasks, not only CPU demand.
  • High load with idle CPUs usually means I/O or NFS waits.
  • Check vmstat 1 columns r (runnable) and b (blocked).

Better signal: PSI

  • /proc/pressure/cpu, memory, io (kernel 4.20+).
  • some avg10=12.50 = tasks stalled 12.5% of the last 10 s.
  • PSI measures lost time directly, per resource.

Load is a symptom, not a diagnosis. Follow it with CPU breakdown, run queue, and I/O latency.

CPU breakdown

Read the %Cpu(s) line in top

Where CPU time goes points to the next tool.

FieldMeaningIf high, check
usUser-space codeWhich process: top, pidstat; profile with perf top
syKernel on behalf of processesSyscall-heavy apps, networking, context switches
niUser code at positive niceUsually batch jobs; expected
idIdle—
waIdle while waiting for I/ODisk latency: iostat -xz 1; not CPU work
hi / siHardware / software interruptsNetwork-heavy hosts; /proc/interrupts
stSteal: hypervisor gave CPU to othersNoisy neighbor, burstable instance out of credits

mpstat -P ALL 1 shows per-CPU values: one CPU at 100% with others idle suggests a single-threaded bottleneck.

Memory

Free memory is wasted memory

Linux uses spare RAM as page cache and gives it back when applications need it.

$ free -h
               total   used   free  shared  buff/cache  available
Mem:            15Gi  4.8Gi  1.2Gi   210Mi       9.0Gi      9.8Gi
Swap:          2.0Gi     0B  2.0Gi

$ vmstat 1 5        # si/so = swap in/out per second
$ grep -E 'MemAvailable|Dirty|Slab' /proc/meminfo
$ cat /proc/pressure/memory

Read the output

  • available estimates memory usable without swapping — watch this, not free.
  • buff/cache is mostly reclaimable page cache.
  • Swap that is used but idle is fine; steady si/so activity means memory pressure.
  • Per-process: RSS counts shared pages repeatedly; PSS (smem, /proc/<pid>/smaps_rollup) divides them.
  • vm.swappiness changes the balance between reclaiming cache and swapping.
OOM killer

When memory runs out

If reclaim fails, the kernel kills a process to keep the system alive.

How the victim is chosen

  • Each process has an oom_score based mainly on memory use.
  • oom_score_adj (-1000 to 1000) biases the choice.
  • -1000 exempts a process; use it sparingly.
  • systemd: OOMScoreAdjust= in the unit.

Finding evidence

  • dmesg -T | grep -i -E "killed process|out of memory".
  • journalctl -k --since "1 hour ago".
  • Exit status 137 (128 + SIGKILL).
  • Kubernetes: kubectl describe pod shows OOMKilled.

Container and cgroup OOM

  • A cgroup limit (memory.max) triggers OOM inside that cgroup, even if the host has free RAM.
  • memory.events counts oom_kill events.
  • Older runtimes may size heaps from host RAM, not the container limit.
  • Fix the leak or limit; do not just disable the OOM killer.
Disk I/O

Latency matters more than utilization

iostat shows per-device throughput, latency, and queueing.

$ iostat -xz 1
Device  r/s   w/s   rkB/s   wkB/s  r_await w_await aqu-sz  %util
nvme0n1 120  850  3840.0 54400.0    0.45   12.80  10.9   98.0

$ pidstat -d 1        # which process does the I/O
$ sudo iotop -oPa     # interactive, accumulated
$ cat /proc/pressure/io

Read the output

  • r_await/w_await: average ms per request including queue time — compare with the device’s normal latency.
  • aqu-sz: average queue length; sustained high values mean saturation.
  • %util near 100% means busy, but SSDs and NVMe serve many requests in parallel, so it is not proof of saturation.
  • Cloud volumes have IOPS and throughput caps; burst credits can run out (e.g. EBS gp2).
  • High wa in top plus high await → storage is the bottleneck.
USE method

Check every resource for Utilization, Saturation, and Errors

A checklist from Brendan Gregg that avoids guessing.

ResourceUtilizationSaturationErrors
CPUmpstat -P ALL 1, topvmstat r > CPUs, PSI cpudmesg machine-check events
Memoryfree -m (available)vmstat si/so, PSI memory, OOM killsdmesg (ECC, OOM)
Diskiostat -xz %utilaqu-sz, await, PSI iodmesg, smartctl -a
Networksar -n DEV 1 throughputDrops, ss -ti retransmits, backlog overflowsip -s link errors
File descriptors/proc/sys/fs/file-nrNear ulimit -nEMFILE / ENFILE in logs

Errors are often the quickest win: a single dmesg line about disk resets or OOM can end an investigation.

First 60 seconds

A standard performance checklist

From Brendan Gregg’s “Linux Performance Analysis in 60,000 Milliseconds” (Netflix).

uptime                 # load averages and trend
dmesg -T | tail        # OOM kills, I/O errors, hardware
vmstat 1               # r, b, si/so, us/sy/wa/st
mpstat -P ALL 1        # per-CPU balance
pidstat 1              # per-process CPU
iostat -xz 1           # disk latency and saturation
free -m                # memory and page cache
sar -n DEV 1           # network throughput per interface
sar -n TCP,ETCP 1      # connections and retransmits
top                    # overall cross-check

How to use it

  • Run the commands in order; each takes a few seconds.
  • Note anything abnormal, then drill into that resource.
  • mpstat, pidstat, iostat, and sar come from the sysstat package.
  • Enable sysstat collection to get history with sar -f.
  • Compare with a known-good baseline from the same host type.
Limits

File descriptors and other per-process limits

“Too many open files” (EMFILE) is a limit, not a disk problem.

ulimit -n                 # soft limit for this shell
ulimit -Hn                # hard limit
cat /proc/<pid>/limits    # limits of a running process
ls /proc/<pid>/fd | wc -l # current usage

# systemd service
[Service]
LimitNOFILE=65536
TasksMax=4096

# login sessions (PAM): /etc/security/limits.d/90-app.conf
app  soft  nofile  65536
app  hard  nofile  65536

cat /proc/sys/fs/file-nr  # system-wide: allocated, unused, max

Key rules

  • Soft limits can be raised by the user up to the hard limit.
  • limits.conf applies to PAM logins, not to systemd services — use LimitNOFILE= there.
  • Containers inherit limits from the runtime (--ulimit) and its configuration.
  • Process count: ulimit -u and the cgroup pids.max (TasksMax=).
  • Rising FD counts over time usually mean a leak, not a low limit.
Reading vmstat

One line per second, whole-system view

vmstat quickly shows whether the problem is CPU, memory, or I/O.

$ vmstat 1
procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
 r  b   swpd    free   buff   cache   si   so    bi    bo   in   cs us sy id wa st
 3  0      0 1210500 102400 9599200    0    0     4   220 3120 5400 30  5 64  1  0
 9  0      0 1201300 102400 9600100    0    0     0   180 4410 9800 85 12  3  0  0
 2  6      0 1190300 102400 9601100    0    0  8200  9100 2800 4100 10  5 40 45  0

Read the output

  • The first line is an average since boot; read the following lines.
  • r: runnable tasks. Above the CPU count → CPU saturation (line 2).
  • b: tasks blocked on I/O. Together with high wa → storage bottleneck (line 3).
  • si/so non-zero and sustained → memory pressure and swapping.
  • bi/bo: blocks read/written per second; in/cs: interrupts and context switches.
  • st above a few percent on a VM → the hypervisor is taking CPU away.
Historical data

Answer “what happened at 03:00?” with sar

sysstat records system metrics every 10 minutes by default once collection is enabled.

sudo systemctl enable --now sysstat
# Debian/Ubuntu may also need ENABLED="true" in /etc/default/sysstat

sar -u                                 # CPU today
sar -q                                 # run queue and load averages
sar -r                                 # memory
sar -d -p                              # disk, with device names
sar -n DEV                             # network per interface
sar -u -s 02:30:00 -e 03:30:00         # time window
sar -u -f /var/log/sa/sa15             # RHEL: day 15
sar -u -f /var/log/sysstat/sa15        # Debian/Ubuntu: day 15

When it helps

  • Incidents that ended before anyone logged in.
  • Comparing today with the same time yesterday.
  • Proving whether a nightly job saturates CPU or disk.
  • Files rotate after a retention period (often 7–28 days); export what you need.
  • Central monitoring is better for fleets, but sar works even when monitoring is broken.
Benchmarking

Measure capacity with fio, iperf3, and stress-ng

Benchmarks give a baseline; compare them with what production actually needs.

# random 4k reads, 60 s, bypassing the page cache
fio --name=randread --filename=/data/fio.test --size=2G \
    --rw=randread --bs=4k --iodepth=32 --ioengine=libaio \
    --direct=1 --runtime=60 --time_based --group_reporting

# network throughput between two hosts
iperf3 -s                              # on the server
iperf3 -c 10.0.1.5 -P 4 -t 30          # on the client, 4 streams

# CPU and memory load
stress-ng --cpu 4 --vm 2 --vm-bytes 1G --timeout 60s --metrics-brief

Use results correctly

  • --direct=1 measures the device, not the page cache.
  • Report IOPS, throughput, and latency percentiles (p99), not just averages.
  • Match block size and queue depth to the real workload.
  • Cloud results depend on instance and volume type limits.
  • Run load tests on test hosts; they disturb neighbors.

Never point a fio write test at a device or file that holds data — it overwrites it.

Chapter 6

Networking

Configure and inspect the network stack on a host.

Host network configuration

Addresses, routes, sockets, and name resolution

For protocol details, see the Computer Networking A to Z deck; these are the Linux-specific pieces.

ip -br addr                 # interfaces and addresses
ip route                    # routing table
ip route get 10.0.5.20      # actual route decision
ss -tulpn                   # listening TCP/UDP sockets + process
ss -s                       # socket summary

cat /etc/resolv.conf        # resolver config
resolvectl status           # systemd-resolved view
getent hosts api.internal   # resolve via NSS (uses /etc/hosts)
dig api.internal            # asks DNS directly (ignores /etc/hosts)

Linux-specific details

  • /etc/resolv.conf may point to 127.0.0.53, the systemd-resolved stub.
  • /etc/nsswitch.conf decides the lookup order (files, then DNS).
  • So getent and applications can disagree with dig.
  • Config tools differ: netplan (Ubuntu), NetworkManager (RHEL), systemd-networkd.
  • ip changes are temporary; persist them in the distribution’s network config.
netfilter

iptables, nftables, and connection tracking

The kernel’s packet filter is used by host firewalls, Docker, and kube-proxy.

Tables and hooks

  • filter: INPUT, FORWARD, OUTPUT chains.
  • nat: PREROUTING (DNAT), POSTROUTING (SNAT/masquerade).
  • Rules are evaluated in order; the first terminal match wins.
  • sudo iptables -L -n -v --line-numbers, sudo nft list ruleset.

nftables vs iptables

  • nftables is the modern framework.
  • Many distros ship iptables-nft, which writes nftables rules.
  • Mixing legacy and nft backends causes confusing results.
  • Frontends: ufw (Ubuntu), firewalld (RHEL).

Connection tracking

  • Stateful rules use conntrack: ESTABLISHED,RELATED.
  • conntrack -L, conntrack -S show entries and stats.
  • A full table logs “nf_conntrack: table full, dropping packet”.
  • Limit: net.netfilter.nf_conntrack_max.

Docker and Kubernetes manage their own chains; edit host rules carefully and test from outside before closing your session.

Socket tuning

Kernel settings that affect busy servers

Tune only after measurement shows the limit is being reached.

sysctl net.core.somaxconn             # accept backlog cap
sysctl net.ipv4.ip_local_port_range   # ephemeral ports
sysctl net.ipv4.tcp_tw_reuse
nstat -az | grep -i -E 'ListenOverflows|ListenDrops'
ss -ltn                                # Recv-Q on LISTEN = queued
ss -tan state time-wait | wc -l
ss -ti dst 10.0.5.20                   # rtt, cwnd, retrans per socket

Read the output

  • On a LISTEN socket, Recv-Q is the current accept queue and Send-Q is its maximum.
  • Rising ListenOverflows means the app accepts connections too slowly or the backlog is too small.
  • Many outbound connections to one destination can exhaust ephemeral ports.
  • Many TIME_WAIT sockets are normal; reuse connections (keep-alive, pooling) before tuning.
  • Change one setting at a time and record the before/after evidence.
Packet capture

tcpdump on a server

When logs disagree, packets show what actually crossed the interface.

sudo tcpdump -i any -nn port 443
sudo tcpdump -i eth0 -nn host 10.0.5.20 and tcp port 5432
sudo tcpdump -i any -nn 'tcp[tcpflags] & (tcp-syn|tcp-rst) != 0'
sudo tcpdump -i any -nn udp port 53
sudo tcpdump -i eth0 -nn -s 0 -c 10000 -w /tmp/cap.pcap
tcpdump -nn -r /tmp/cap.pcap | head     # read a capture later

# inside a container's network namespace
sudo nsenter -t "$pid" -n tcpdump -i eth0 -nn port 8080

Read the output

  • -nn disables DNS and port-name lookups, which are slow and can add traffic.
  • -i any captures on all interfaces; choose one when you need direction clarity.
  • Flags: [S] SYN, [S.] SYN-ACK, [.] ACK, [P.] data, [F.] FIN, [R] reset.
  • Write with -w and open in Wireshark for deeper analysis.
  • Limit with -c or a filter; captures may contain credentials and personal data.
Chapter 7

Containers

Containers are Linux processes with isolation and limits. Study this right before Docker.

What a container is

Processes, not virtual machines

A container is a normal process tree with kernel isolation features applied.

Building blocks

  • Namespaces: what the process can see.
  • cgroups: how much it can use.
  • Layered root filesystem (overlayfs): what files it has.
  • Capabilities, seccomp, LSM: what it may do.

Compared with a VM

  • No guest kernel: all containers share the host kernel.
  • Starts in milliseconds; much less overhead.
  • Weaker isolation boundary than a hypervisor.
  • Kernel features and bugs are shared with the host.

See it from the host

  • ps -ef on the host shows container processes.
  • docker inspect -f '{{.State.Pid}}' name gives the host PID.
  • /proc/<pid>/cgroup shows the container’s cgroup.
  • nsenter -t <pid> -n ip addr enters its network namespace.
Namespaces

What a process can see

Each namespace type isolates one kind of global resource.

NamespaceIsolatesExample effect
pidProcess IDsThe app sees itself as PID 1
netInterfaces, routes, ports, firewall rulesOwn eth0 and lo; ports do not clash
mntMount tableOwn root filesystem and mounts
utsHostname and domain nameContainer has its own hostname
ipcSystem V IPC, POSIX message queuesShared memory isolated
userUID/GID mappingsRoot inside maps to an unprivileged UID outside (rootless)
cgroupView of the cgroup treeSees its own cgroup as the root
timeMonotonic and boot clocks (5.6+)Rarely used; for checkpoint/restore

lsns lists namespaces; Kubernetes pods share the network namespace between their containers, which is why they can use localhost.

cgroups v2

How much a process can use

Control groups account for and limit CPU, memory, I/O, and process count for a group of tasks.

cat /sys/fs/cgroup/cgroup.controllers
systemd-cgls                    # cgroup tree
systemd-cgtop                   # live usage per cgroup

# inside a cgroup directory
cat cpu.max          # "200000 100000" = 2 CPUs
cat cpu.stat         # nr_throttled, throttled_usec
cat memory.max       # hard limit in bytes or "max"
cat memory.current   # current usage
cat memory.events    # high, max, oom, oom_kill
cat pids.max

Kubernetes mapping

Pod settingcgroup v2 effect
CPU requestcpu.weight (relative share)
CPU limitcpu.max quota → throttling
Memory requestScheduling only (plus memory.min with MemoryQoS)
Memory limitmemory.max → OOMKilled
PID limitpids.max

A CPU limit causes throttling even when the node is idle. Check nr_throttled before blaming the application for latency.

Container filesystems

Overlayfs and image layers

Image layers are read-only; each container adds one writable layer on top.

$ mount | grep overlay
overlay on /var/lib/docker/overlay2/<id>/merged type overlay
  (lowerdir=L1:L2:L3,       # read-only image layers
   upperdir=.../diff,       # this container's writes
   workdir=.../work)

$ docker diff mycontainer   # A added, C changed, D deleted

Behavior to expect

  • Writing a file from a lower layer copies it up first (copy-up).
  • Deleting a lower-layer file creates a “whiteout” in the upper layer; the image does not shrink.
  • Data in the writable layer is lost when the container is removed — use volumes.
  • Many writes to the container layer are slower than a volume.
  • Deleting files in a later Dockerfile layer does not reduce image size.
Container security

Reduce what a compromised container can do

Defaults are reasonable; most risk comes from options that remove isolation.

Run with less privilege

  • Run as a non-root user (USER in the Dockerfile).
  • Drop capabilities: --cap-drop ALL --cap-add NET_BIND_SERVICE.
  • Read-only root filesystem with explicit writable volumes.
  • no-new-privileges blocks setuid escalation.

Kernel filters

  • seccomp’s default profile blocks dozens of risky syscalls.
  • AppArmor or SELinux confine file and process access.
  • User namespaces (rootless mode) map container root to an unprivileged UID.
  • Keep the host kernel patched: it is shared by every container.

Dangerous options

  • --privileged: nearly all capabilities and device access.
  • Mounting /var/run/docker.sock: equivalent to root on the host.
  • --pid=host, --net=host: remove namespace isolation.
  • Host path mounts of /, /etc, or /proc.
Lab: container by hand

Build isolation with unshare and systemd-run

See that a “container” is just kernel features applied to a process.

# Terminal A: new PID, mount, UTS, and network namespaces
sudo unshare --pid --fork --mount-proc --uts --net bash
hostname lab-box && hostname    # changes only inside
ps aux                          # only bash and ps; bash is PID 1
ip link                         # only lo, and it is DOWN

# Terminal B: observe from the host
pgrep -a unshare
sudo lsns | grep bash

# Add resource limits with a transient cgroup
sudo systemd-run --scope -p MemoryMax=200M -p CPUQuota=20% bash
cat /proc/self/cgroup

What to notice

  • The process in Terminal A is visible on the host with a normal PID.
  • Hostname and network changes do not affect the host.
  • There is no new kernel and no image: only namespaces.
  • systemd-run places the shell in a cgroup with memory and CPU limits.
  • Docker and containerd automate exactly these steps, plus image layers and seccomp.
Inspect a running container

Look inside from the host

When the image has no shell or tools, use host tools against the container’s namespaces.

pid=$(docker inspect -f '{{.State.Pid}}' web)
# containerd / Kubernetes: crictl inspect <id> | jq .info.pid

sudo nsenter -t "$pid" -n ss -tlnp        # host ss, container network
sudo nsenter -t "$pid" -n ip addr
sudo ls /proc/"$pid"/root/etc/            # container filesystem
sudo cat /proc/"$pid"/cgroup              # 0::/system.slice/docker-<id>.scope
cg=$(sudo awk -F: '{print $3}' /proc/"$pid"/cgroup)
cat /sys/fs/cgroup"$cg"/memory.current /sys/fs/cgroup"$cg"/memory.max
sudo grep Cap /proc/"$pid"/status         # capability bitmasks
capsh --decode=00000000a80425fb

Why this works

  • A container process is a normal host process with a PID on the host.
  • nsenter -n enters only the network namespace, so host binaries like ss and tcpdump still work.
  • /proc/<pid>/root exposes the container’s root filesystem.
  • The cgroup path leads to its live CPU and memory accounting.
  • kubectl debug with an ephemeral container is the Kubernetes-native alternative.
Chapter 8

Troubleshooting and practice

A repeatable method, real cases, labs, and review.

Chapter 8 · Part 8.1

Troubleshooting

A repeatable method, the right tools, and real cases.

Method

From symptom to evidence

Narrow the scope before you change anything.

Scope

  • One host, one service, or everything?
  • When did it start, and what changed (deploy, package update, config, traffic)?
  • Can you reproduce it?

Walk the layers

  • Application logs and exit codes.
  • Process state, limits, and permissions.
  • Resources: CPU, memory, disk, network.
  • Kernel messages, then hardware or cloud platform.

Collect evidence

  • Record exact errors and timestamps.
  • Prefer read-only commands first.
  • Change one thing at a time, and write down what you did.

Start every incident with: systemctl status, journalctl -u … --since, dmesg -T | tail, df -h, free -m, uptime.

System call tracing

strace shows what a program asks the kernel

When logs are silent, system calls reveal missing files, denied access, and hangs.

strace -f -e trace=openat,execve ./app 2>&1 | grep -v ENOENT
strace -f -e trace=%file,%network -p <pid>
strace -c -p <pid>          # summary: counts and time per syscall
strace -tt -T -p <pid>      # timestamps + time spent in each call
strace -f -o /tmp/trace.txt systemctl start myapp

# lower-overhead alternatives (bcc / bpftrace tools)
sudo execsnoop-bpfcc        # new processes system-wide
sudo opensnoop-bpfcc -x     # failed opens

What to look for

  • openat(…) = -1 ENOENT: the file the app expected does not exist.
  • EACCES / EPERM: permissions, SELinux/AppArmor, or capabilities.
  • A call that never returns (connect, read, futex) shows where it hangs.
  • strace slows the traced process significantly; use briefly in production.
  • eBPF tools (bcc, bpftrace) trace with much lower overhead.
Error decoder

Common errno values and first checks

The same errno appears in logs, strace, and error messages.

errnoMessageFirst check
ENOENTNo such file or directoryPath, working directory, missing interpreter (#! line), missing library
EACCESPermission deniednamei -l path, mode bits, noexec mount, SELinux/AppArmor
EPERMOperation not permittedNeeds root or a capability; immutable file (lsattr)
ENOSPCNo space left on devicedf -h and df -i
EMFILEToo many open files/proc/<pid>/limits, FD leak
EADDRINUSEAddress already in usess -ltnp for the port
ECONNREFUSEDConnection refusedNothing listening, or listening on another address
EROFSRead-only file systemFilesystem errors forced remount read-only: dmesg
Case study

“Disk is full, but du finds nothing big”

df -h shows 100%, while du -sh /* adds up to much less.

Known facts

  • Writes fail with ENOSPC.
  • df reports the filesystem as full.
  • du cannot find where the space went.

Next evidence

  • sudo lsof +L1: deleted files still open, with size and PID.
  • df -i: maybe inodes, not blocks, are exhausted.
  • Hidden under a mount point: sudo mkdir -p /mnt/root && sudo mount --bind / /mnt/root, then du there.
  • Check /var/lib/docker and the journal.

Fix

  • Restart or reload the process holding the deleted file.
  • Emergency: sudo truncate -s 0 /proc/<pid>/fd/<n>.
  • Fix log rotation so the app reopens its files.
  • Add alerts on disk and inode usage.
Case study

“The service keeps restarting”

systemctl status shows activating (auto-restart) or failed.

Gather

  • systemctl status myapp — exit code or signal.
  • journalctl -u myapp -n 200 --no-pager.
  • systemctl cat myapp — the effective unit and drop-ins.
  • “start request repeated too quickly” → start limit hit.

Common causes

  • Configuration or syntax error at startup.
  • Missing environment file, secret, or permission as the service user.
  • Port already in use (EADDRINUSE).
  • OOM kill (status=9/KILL, result oom-kill) or a crash (SIGSEGV).
  • Type= mismatch: a forking daemon declared as simple.

Reproduce

  • Run ExecStart manually as the service user: sudo -u myapp ….
  • Load the same environment file.
  • Fix, daemon-reload if the unit changed, reset-failed, start.
  • Watch with journalctl -u myapp -f.
Case study

“Load is 40, but CPU is mostly idle”

top shows high load average with high wa and low us/sy.

Hypothesis

  • Load counts D-state tasks waiting on I/O.
  • Storage or a network filesystem is slow or hung.
  • The CPUs are waiting, not working.

Evidence

  • ps -eo stat,pid,wchan:32,cmd | awk '$1 ~ /^D/'.
  • iostat -xz 1: high await and aqu-sz.
  • cat /proc/pressure/io.
  • dmesg -T: I/O errors, NFS “server not responding”.

Typical causes

  • Volume hit its IOPS or throughput cap; burst credits exhausted.
  • Failing disk or controller.
  • NFS/EFS server unreachable; mounts block.
  • Heavy backup, log, or swap I/O competing with the app.
Case study

“Permission denied, but the permissions look right”

The file is 644 and owned by the right user, yet the app still gets EACCES.

Check the whole path

  • namei -l /srv/app/data/config.yml shows every directory.
  • Each parent directory needs x for the user.
  • Run the check as the real user: sudo -u app cat file.

Check beyond mode bits

  • Mount options: findmnt -T /path (noexec, ro).
  • SELinux label: ls -Z, ausearch -m avc.
  • ACLs: getfacl; immutable flag: lsattr.
  • Container volume: UID inside differs from owner UID on the host.

Fix the cause

  • Correct ownership or the directory x bit.
  • restorecon -Rv for SELinux labels.
  • Match container USER UID with volume ownership.
  • Avoid chmod 777: it hides the real problem and creates a new one.
Case study

“The container was OOMKilled”

The pod restarts with exit code 137 and reason OOMKilled.

Evidence

  • kubectl describe pod → Last State: OOMKilled.
  • Node dmesg: “Memory cgroup out of memory: Killed process …”.
  • cgroup memory.events shows oom_kill.
  • Memory graph rising to the limit before each restart.

Common causes

  • Limit too low for normal peak usage.
  • Memory leak: usage grows steadily until killed.
  • Runtime heap not sized for the container (JVM, Node.js).
  • Large tmpfs (emptyDir: medium: Memory) counts toward the limit.

Fix

  • Measure real usage, then set request and limit with headroom.
  • Configure runtime limits (-XX:MaxRAMPercentage, --max-old-space-size).
  • Fix leaks with heap profiles.
  • Alert before the limit, not after the kill.
Reading strace

Decode the trace line by line

Each line is a system call, its arguments, and its result.

$ strace -f -e trace=openat,connect ./app
openat(AT_FDCWD, "/etc/app/config.yml", O_RDONLY) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/opt/app/config.yml", O_RDONLY) = 3
connect(4, {sa_family=AF_INET, sin_port=htons(5432), sin_addr=inet_addr("10.0.5.20")}, 16) = -1 EINPROGRESS
[pid  4122] connect(5, {sa_family=AF_UNIX, sun_path="/run/redis/redis.sock"}, 110) = -1 EACCES (Permission denied)
+++ exited with 1 +++

Read the output

  • = 3 is a new file descriptor; = -1 ERRNO is a failure.
  • An ENOENT while searching several paths can be normal — look at the last attempt.
  • EINPROGRESS is normal for non-blocking connects; follow the later poll result.
  • [pid 4122] lines come from a thread or child because of -f.
  • Here the real failure is EACCES on the Redis socket: check its permissions and group.
  • +++ exited with 1 +++ gives the final status.
Case study

“SSH login suddenly fails”

The key has not changed, but the server now rejects it.

Evidence

  • Client: ssh -vvv user@host — which keys were offered and why they were refused.
  • Server: journalctl -u ssh (RHEL: sshd), /var/log/auth.log or /var/log/secure.
  • sudo sshd -T for the effective configuration.

Common causes

  • Home, ~/.ssh, or authorized_keys became group/world-writable (StrictModes).
  • User not in AllowGroups; account expired (chage -l). faillock only blocks password logins.
  • SELinux label on authorized_keys wrong after a copy (restorecon -Rv ~/.ssh).
  • Old ssh-rsa (SHA-1) signatures refused by OpenSSH 8.8+.
  • Source IP banned by fail2ban or a firewall change.

Fix and prevent

  • Use the cloud console, SSM, or another admin to reach the host.
  • Fix the specific cause; avoid enabling password logins as a workaround.
  • Manage keys and sshd settings with configuration management.
  • Monitor authentication failures.
Case study

“TLS fails with certificate not yet valid”

HTTPS calls from one host fail while the same calls work everywhere else.

Evidence

  • Error text: “certificate is not yet valid” or “certificate has expired”.
  • timedatectl shows System clock synchronized: no.
  • chronyc tracking shows a large offset.
  • Compare date -u with curl -skI https://example.com | grep -i ^date (-k: the check itself would fail).

Common causes

  • NTP traffic (UDP 123) blocked by a firewall change.
  • No time service enabled on a custom image.
  • VM restored from a snapshot or paused for a long time.
  • Two NTP clients fighting each other.

Fix

  • Enable chrony with a reachable source (cloud metadata time service where available).
  • Force a correction with sudo chronyc makestep.
  • Alert on clock offset; it also breaks Kerberos and log correlation.
  • Restart clients that cached the failure.
Chapter 8 · Part 8.2

Practice and review

Turn commands into skills you can demonstrate.

Lab 1

Explore your system

Answer each question using only the command line.

cat /etc/os-release; uname -r; nproc; free -h
lsblk -f; df -hT; findmnt /
ip -br addr; ip route; cat /etc/resolv.conf
systemctl --failed; systemctl list-timers
ps -eo pid,user,%cpu,%mem,cmd --sort=-%mem | head
journalctl -p err -b --no-pager | tail -n 20
sudo ss -tulpn

Write down

  • Distribution, kernel version, CPU count, and available memory.
  • Each disk, its filesystem, and where it is mounted.
  • Your IP address, default gateway, and DNS resolver.
  • Which services listen on which ports, and as which user.
  • The top memory consumer, and any failed units or errors since boot.
Lab 2

Permissions and processes

Create a shared directory, then observe signals and file descriptors.

sudo groupadd team
sudo mkdir /srv/shared
sudo chown root:team /srv/shared
sudo chmod 2770 /srv/shared       # setgid + rwx for group
sudo usermod -aG team "$USER"     # then log in again
touch /srv/shared/a && ls -l /srv/shared

sleep 600 &                       # start a background job
pid=$!
ls -l /proc/$pid/fd
kill -STOP $pid; ps -o pid,stat,cmd -p $pid   # T
kill -CONT $pid; kill $pid; wait $pid; echo $?  # 143

Explain

  • Why does the new file belong to group team?
  • Why can another user not delete your file if you add the sticky bit (chmod +t)?
  • Which file descriptors does sleep have open, and where do they point?
  • Why is the exit status 143?
  • What would kill -9 return instead?
Lab 3

Run your own service and timer

Turn a script into a supervised service with logs and a schedule.

sudo tee /usr/local/bin/heartbeat.sh >/dev/null <<'EOF'
#!/usr/bin/env bash
echo "heartbeat from $(hostname) at $(date -Is)"
EOF
sudo chmod 755 /usr/local/bin/heartbeat.sh

sudo tee /etc/systemd/system/heartbeat.service >/dev/null <<'EOF'
[Service]
Type=oneshot
ExecStart=/usr/local/bin/heartbeat.sh
DynamicUser=yes
EOF
sudo tee /etc/systemd/system/heartbeat.timer >/dev/null <<'EOF'
[Timer]
OnCalendar=*:0/2
[Install]
WantedBy=timers.target
EOF
sudo systemctl daemon-reload
sudo systemctl enable --now heartbeat.timer
journalctl -u heartbeat -f

Verify and extend

  • systemctl list-timers shows the next run.
  • The journal shows one line every two minutes.
  • DynamicUser=yes runs it as a temporary unprivileged user.
  • Break the script (exit 1) and see the failure in systemctl status.
  • Clean up: disable --now the timer and remove both files.
Lab 4

Watch resource limits in action

Use a transient cgroup to see throttling and an OOM kill safely.

# CPU: limit a busy loop to 20% of one CPU
sudo systemd-run --unit=burn -p CPUQuota=20% \
  bash -c 'while :; do :; done'
top -p "$(systemctl show -p MainPID --value burn)"
sudo systemctl stop burn

# Memory: allocate past a 100 MB limit
sudo systemd-run --unit=hog -p MemoryMax=100M -p MemorySwapMax=0 \
  python3 -c 'b=[]; [b.append(bytearray(10**7)) for _ in range(50)]'
systemctl status hog        # result: oom-kill
journalctl -k -n 20 | grep -i oom
sudo systemctl reset-failed hog

What to notice

  • The busy loop stays near 20% CPU even on an idle machine: that is throttling.
  • The Python process is killed by the cgroup OOM killer, not the system-wide one.
  • The host keeps running normally.
  • This is exactly what Kubernetes CPU and memory limits do.
  • Run labs on a test VM, not on a production host.
Lab 5

Practice LVM safely with loop devices

Files act as disks, so nothing on the host is at risk.

truncate -s 1G /tmp/d1.img /tmp/d2.img
L1=$(sudo losetup -fP --show /tmp/d1.img)
L2=$(sudo losetup -fP --show /tmp/d2.img)

sudo pvcreate "$L1" "$L2"
sudo vgcreate labvg "$L1" "$L2"
sudo lvcreate -n data -L 1.2G labvg      # larger than one "disk"
sudo mkfs.ext4 -q /dev/labvg/data
sudo mkdir -p /mnt/lab && sudo mount /dev/labvg/data /mnt/lab
df -h /mnt/lab

sudo lvextend -r -L +500M labvg/data     # grow online
sudo lvcreate -s -n snap -L 200M labvg/data
sudo lvs labvg; df -h /mnt/lab

# clean up
sudo umount /mnt/lab && sudo vgremove -f labvg
sudo pvremove "$L1" "$L2" && sudo losetup -d "$L1" "$L2"
rm /tmp/d1.img /tmp/d2.img

What to notice

  • The 1.2 GB volume spans two 1 GB “disks”: lvs -o +devices shows both.
  • lvextend -r grows ext4 while it is mounted and in use.
  • The snapshot’s Data% grows as you write to /mnt/lab.
  • lsblk shows loop0, loop1, and the LVs on top.
  • Repeat with mkfs.xfs and try to shrink: it is refused.
Lab 6

Run a quick hardening audit

Collect evidence first, then decide what to change.

sudo ss -tulpn                                  # exposed services
systemctl list-unit-files --state=enabled --type=service
sudo sshd -T | grep -E '^(permitrootlogin|passwordauthentication|kbdinteractiveauthentication) '
sudo find / -xdev -perm -4000 -type f 2>/dev/null   # setuid binaries
awk -F: '$3 == 0 {print $1}' /etc/passwd           # UID 0 accounts
sudo awk -F: '$2 == "" {print $1}' /etc/shadow     # empty passwords
sysctl kernel.randomize_va_space net.ipv4.tcp_syncookies
getenforce 2>/dev/null || { sudo aa-status --enabled && echo "AppArmor enabled"; }
apt list --upgradable 2>/dev/null | head           # or: dnf check-update
sudo lynis audit system --quick                    # if installed

Write down

  • Every listening port, its process, and whether it must be reachable.
  • Whether SSH allows root or password logins.
  • Any account other than root with UID 0, and any empty password.
  • Which MAC system is active and in what mode.
  • Pending security updates, and the top three Lynis suggestions.
Self-check

Can you explain these without looking?

If not, revisit the section and repeat the lab.

Foundations and files

  • What happens between typing a command and seeing its output?
  • Why can deleting a file fail to free disk space?
  • What permissions are needed to delete a file?
  • Why might cmd 2>&1 > f still print errors?

Processes and services

  • What is the difference between SIGTERM and SIGKILL?
  • Why might a container ignore docker stop until it is killed?
  • How do Wants= and After= differ?
  • Why does a job work in your shell but fail in cron?

Performance and containers

  • When is high load not a CPU problem?
  • Why is low “free” memory usually fine?
  • What do namespaces and cgroups each provide?
  • How does a Kubernetes CPU limit slow an app?
Self-check

Storage and security

Explain each answer, then prove it with a command.

Storage

  • Why is RAID not a backup?
  • What happens when a classic LVM snapshot fills up?
  • Why must fsck never run on a mounted filesystem?
  • What do hard and soft NFS mounts trade off?

Access control

  • Which wins when two sudoers rules match?
  • Why does an sshd drop-in override the main file?
  • What does required vs sufficient mean in PAM?
  • Why is access to docker.sock equivalent to root?

Detection and upkeep

  • How would you prove /etc/shadow was modified, and by whom?
  • Why must services restart after a library update?
  • Where can secrets leak on a Linux host?
  • How do you know a backup works?
Glossary

Terms to keep in your pocket

Use these as quick anchors, then explain them in your own words.

TermShort meaning
Kernel / syscallPrivileged core / request into it
InodeFile metadata and data location, without the name
File descriptorInteger handle to an open file, pipe, or socket
PID 1First process; adopts orphans and reaps zombies
SignalAsynchronous notification to a process
UnitAnything systemd manages
TermShort meaning
Page cacheFile data cached in RAM; reclaimable
Load averageRunnable + uninterruptible tasks, averaged
OOM killerKernel mechanism that kills to free memory
NamespaceIsolated view of a system resource
cgroupGroup of processes with accounting and limits
CapabilityOne slice of root’s privileges
Glossary

More terms: storage and security

Quick anchors for the storage and hardening sections.

TermShort meaning
PV / VG / LVLVM physical volume / volume group / logical volume
RAIDRedundancy across disks; not a backup
SnapshotPoint-in-time view; keeps originals of changed blocks
LUKSStandard Linux block-device encryption format
RPO / RTOAcceptable data loss / acceptable restore time
root_squashNFS maps client root to an unprivileged user
TermShort meaning
PAMPluggable authentication stack used by login services
SSH CAAuthority that signs short-lived SSH keys
auditdKernel audit event logging daemon
AIDEFile integrity checker based on a hash database
CIS BenchmarkConsensus hardening baseline per OS
Least privilegeGrant only the access a task needs
Further reading

Primary references and practice

Use the manual pages on your own system for exact behavior; use these for depth.

Documentation

  • Linux man pages — man7.org/linux/man-pages/
  • Kernel admin guide — docs.kernel.org/admin-guide/
  • cgroup v2 — docs.kernel.org/admin-guide/cgroup-v2.html
  • systemd — systemd.io and man systemd.service, systemd.exec
  • Filesystem Hierarchy Standard — refspecs.linuxfoundation.org/fhs.shtml
  • Bash reference manual — gnu.org/software/bash/manual/

Performance and practice

  • Brendan Gregg — brendangregg.com/linuxperf.html
  • “Linux Performance Analysis in 60,000 Milliseconds” (Netflix Tech Blog)
  • The Linux Command Line — William Shotts (free online)
  • ShellCheck — shellcheck.net
  • Practice on a disposable VM; break it on purpose and fix it.
1 / 1