Container engineering deck

Docker
Deep Dive

Linux internals, images, Dockerfiles, BuildKit, runtime, networking, storage, Compose, security, observability, debugging, and production — with best practices for each.

Use ← → / Space to navigate · I contents · O overview · T theme · F fullscreen

Docker Deep Dive

Contents

Click any line to jump there · press I to return to this page

Chapter 1

Foundations

What a container really is, and the Linux features underneath it.

Chapter 1 · Part 1.1

Fundamentals &
architecture

What a container really is, and what Docker adds.

Containers vs VMs

A container is an isolated process, not a small VM

Virtual machine

  • Hypervisor emulates hardware; each VM boots its own kernel
  • Strong isolation boundary (hardware virtualization)
  • GBs of disk, seconds to minutes to boot
  • Good for: different OS/kernel, hard multi-tenancy

Container

  • A normal Linux process that shares the host kernel
  • Isolation from namespaces; limits from cgroups; filesystem from image layers
  • MBs of disk, starts in milliseconds
  • Good for: packaging apps with dependencies, density, fast deploys

Key insight

Everything Docker does is built from kernel features you can use without Docker. Understanding them explains most "weird" container behavior.

Consequence

A kernel exploit affects every container on the host. For untrusted code use extra isolation (gVisor, Kata, Firecracker microVMs).

Standards

Docker, OCI, and the container ecosystem

OCI specs

  • Image spec — manifest, config, layers
  • Runtime spec — how to run a filesystem bundle
  • Distribution spec — registry HTTP API
  • Any compliant tool can build, push, or run any image

Docker components

  • docker CLI → talks to the daemon API
  • dockerd — Docker Engine (API, networking, volumes, builds)
  • containerd — container lifecycle & images
  • runc — creates the process with namespaces/cgroups
  • BuildKit — the build engine behind docker build

Alternatives

  • Podman (daemonless, rootless by default)
  • containerd + nerdctl
  • Kubernetes uses CRI runtimes (containerd, CRI-O) — dockershim was removed in 1.24, images still work
  • Desktop: Docker Desktop, Rancher Desktop, Colima, OrbStack
Architecture

What happens on docker run

docker CLI docker run nginx REST API docker.sock dockerd API · networks · volumes · BuildKit gRPC containerd container lifecycle · image store (Engine 29+) containerd-shim (per container) holds stdio · reports exit status runc namespaces · cgroups · seccomp → exec nginx process (PID 1 in its namespace) Registry Docker Hub · ECR · GHCR pull Host kernel namespaces cgroups v2 overlayfs netfilter / iptables seccomp · capabilities AppArmor / SELinux

Step by step

  • CLI sends the request to dockerd over /var/run/docker.sock
  • Image missing → pulled and unpacked (by containerd on the containerd image store, by dockerd on legacy graph drivers)
  • dockerd sets up network (veth, bridge, iptables) and mounts
  • containerd starts a shim; the shim calls runc
  • runc creates namespaces & cgroups, then execs the entrypoint and exits

Why the shim matters

With live-restore, containers keep running while dockerd restarts or upgrades.

What changed recently

Docker Engine 28–29: defaults to know

containerd image store

  • Default for fresh Engine 29 installs; upgraded hosts keep their graph driver (overlay2) until you switch; not used with userns-remap
  • Images stored as containerd content + snapshots
  • Enables local multi-platform images and keeping attestations (SBOM, provenance)
  • Disk layout and docker inspect fields differ — check with docker info

Other changes

  • Engine 29.0–29.2 required API v1.44+; 29.3+ accepts v1.40+ again — very old clients/SDKs still fail
  • Experimental nftables firewall backend ("firewall-backend": "nftables")
  • Daemon refuses unauthenticated remote TCP (Engine 27+)
  • docker-compose v1 retired — use docker compose
  • Read the release notes before upgrading production hosts
docker version --format 'client {{.Client.Version}} / server {{.Server.Version}}'
docker info --format '{{.Driver}} {{json .DriverStatus}}'      # overlay2 vs overlayfs snapshotter
Chapter 1 · Part 1.2

Linux internals

Namespaces isolate. Cgroups limit. Overlayfs layers. Capabilities & seccomp restrict.

Namespaces

What each namespace isolates

NamespaceIsolatesEffect in a containerDocker flag to share with host
pidProcess IDsYour app sees itself as PID 1--pid=host
netInterfaces, routes, ports, iptablesOwn eth0, own localhost--network=host
mntMount tableOwn root filesystem from the image(bind mounts)
utsHostname, domain nameHostname = short container ID--uts=host
ipcSysV IPC, POSIX message queues, shmOwn /dev/shm--ipc=host
userUID/GID mappingroot inside → unprivileged UID outside (if enabled)--userns=host
cgroupView of the cgroup treeSees only its own cgroup--cgroupns=host

Containers can also join each other's namespaces: --network container:web, --pid container:web — the basis of Kubernetes pods and debug sidecars.

Hands-on

Prove that a container is just a process

# Start a container
docker run -d --name web nginx

# Inside: nginx is PID 1
docker exec web cat /proc/1/cmdline; echo

# On the host (Linux): it's a normal process
PID=$(docker inspect -f '{{.State.Pid}}' web)
ps -o pid,user,cmd -p $PID
ls -l /proc/$PID/ns          # its namespaces
cat /proc/$PID/cgroup        # its cgroup

# Enter its network namespace with host tools
sudo nsenter -t $PID -n ip addr

# Build an isolated shell without Docker
sudo unshare --pid --fork --mount-proc \
  --uts --net bash
ps aux        # only bash + ps
hostname demo; hostname

What you just learned

  • The host sees the container's process with a real PID
  • /proc/<pid>/ns/* links show its namespaces
  • nsenter lets you use host tools (ip, ss, tcpdump) inside a container's namespace — great for distroless images
  • unshare creates namespaces directly: Docker automates this plus images, networking, and limits
  • On macOS/Windows, Docker runs in a Linux VM — run these inside that VM
Control groups

cgroups v2: limiting resources

Controllers

  • memory — hard limit; exceed it → kernel OOM killer → exit code 137
  • cpu — quota (--cpus) = throttling; weight (--cpu-shares) = priority under contention
  • pids — max processes; stops fork bombs
  • io — block I/O weight and bandwidth limits
  • cpuset — pin to specific CPUs / NUMA nodes
docker run -d --name api \
  --memory 512m --memory-swap 512m \
  --cpus 1.5 \
  --pids-limit 200 \
  myapp:1.4.2

# Live usage
docker stats --no-stream api

# Where the limits live (cgroup v2, systemd driver)
ID=$(docker inspect -f '{{.Id}}' api)
cat /sys/fs/cgroup/system.slice/docker-$ID.scope/memory.max
cat /sys/fs/cgroup/system.slice/docker-$ID.scope/cpu.max

# Was it OOM-killed?
docker inspect -f '{{.State.OOMKilled}}' api

Gotcha

Without limits a container can use all host memory and CPU. Some runtimes (older JVMs, Go before 1.25) size heaps/threads from host resources — use container-aware versions or set them explicitly.

Filesystem

overlay2 and copy-on-write

merged — what the container sees as / union view of all layers below upperdir — container writable layer new / changed files, whiteouts for deletes · lost on docker rm Layer 3 · COPY app/ (read-only) Layer 2 · RUN npm ci (read-only) Layer 1 · FROM node:22-slim (read-only, shared by all containers) lowerdir = layers 1–3 · shared between every container from the same image First write to a lower file → whole file copied up to upperdir (copy-on-write)

Implications

  • 100 containers from one image share its layers on disk
  • Deleting a file in a later layer does not shrink the image
  • Copy-up of large files is slow — write heavy data to volumes
  • Container layer is ephemeral: never keep state there

Inspect it

Check the backend with docker info -f '{{.DriverStatus}}'. On legacy overlay2, docker inspect -f '{{json .GraphDriver}}' web shows the dirs; with the containerd image store (overlayfs snapshotter) look at mount | grep overlay. docker diff web works on both.

Kernel security

Capabilities, seccomp, and LSMs

Capabilities

  • Root's power split into ~40 units
  • Docker keeps a small default set (e.g. CHOWN, NET_BIND_SERVICE, SETUID, NET_RAW)
  • Dropped by default: SYS_ADMIN, NET_ADMIN, SYS_PTRACE, SYS_MODULE…
  • Best: --cap-drop ALL, add back only what's needed

seccomp

  • Filters system calls the process may make
  • Default profile blocks dozens of dangerous syscalls (e.g. mount, reboot, kexec_load)
  • Never run with seccomp=unconfined in production
  • Custom profile: --security-opt seccomp=profile.json

AppArmor / SELinux

  • Mandatory access control on files, mounts, networking
  • Docker applies docker-default AppArmor profile (Ubuntu/Debian)
  • SELinux on RHEL/Fedora: label volumes with :z / :Z

--privileged turns all of this off

All capabilities, all devices, no seccomp, no AppArmor. A privileged container is effectively root on the host. Avoid it; grant the specific capability or device instead.

Chapter 2

Building images

Image layers, Dockerfiles, and fast, reproducible builds with BuildKit.

Chapter 2 · Part 2.1

Images & registries

Anatomy

What an image really is

OCI image parts

  • Index (manifest list) — one entry per platform (amd64, arm64)
  • Manifest — points to one config + ordered layers
  • Config — env, entrypoint, cmd, user, workdir, history
  • Layers — compressed tar diffs
  • Everything is content-addressed by sha256 digest

Tags vs digests

  • Tag (nginx:1.27) = mutable pointer
  • Digest (nginx@sha256:…) = immutable content
  • Deploy by digest (or immutable tags) for reproducibility
# Platforms in a multi-arch image
docker buildx imagetools inspect nginx:1.27

# Config: entrypoint, env, user, layers
docker image inspect nginx:1.27 \
  --format '{{json .Config}}'

# How each layer was created, and its size
docker history --no-trunc nginx:1.27

# Resolve a tag to its digest
docker image inspect nginx:1.27 \
  --format '{{index .RepoDigests 0}}'

# Explore layers interactively (third-party)
dive nginx:1.27

# Export an image as a tarball
docker save nginx:1.27 -o nginx.tar
tar -tf nginx.tar | head
Registries

Registries, naming, and pulling

Image reference

registry/namespace/repository:tag@digest
docker.io/library/nginx:1.27
123456789012.dkr.ecr.ap-northeast-1.amazonaws.com/api:1.4.2
ghcr.io/org/tool@sha256:…
  • No registry → docker.io; no tag → latest
  • Common: Docker Hub, Amazon ECR, GHCR, GitLab, Harbor, Artifactory
  • Login: docker login, or credential helpers (e.g. docker-credential-ecr-login)

Best practices

  • Use Docker Official Images or verified publishers as bases
  • Docker Hub rate-limits anonymous pulls — authenticate, or mirror via a pull-through cache (ECR, Harbor)
  • Enable immutable tags and scan on push in your registry
  • Lifecycle policies to delete old images
  • Push multi-arch images so arm64 (Graviton, Apple silicon) works
  • Never rely on latest in production
Chapter 2 · Part 2.2

Dockerfile in depth

Reference

Dockerfile instructions

InstructionPurposeNotes
FROMBase image / start a build stagePin version (or digest); AS name for multi-stage
RUNExecute a command → new layerChain cleanup in the same RUN
COPY / ADDAdd files from contextPrefer COPY; ADD also fetches URLs/git and auto-extracts local tars
ENTRYPOINT / CMDExecutable / default argsUse exec form ["bin","arg"]; CMD is overridable
ENV / ARGRuntime env / build-time variableBoth visible in image history — never for secrets
WORKDIRSet working directoryUse instead of RUN cd …
USERRun as this userUse a non-root UID
EXPOSEDocument the portDoesn't publish anything
HEALTHCHECKContainer health commandShown in docker ps; used by Compose/Swarm (Kubernetes ignores it)
LABEL, STOPSIGNAL, VOLUME, SHELLMetadata, stop signal, mount point, default shellAvoid VOLUME in base images — it can't be undone
PID 1 & signals

ENTRYPOINT, CMD, and graceful shutdown

Shell form traps signals

CMD node server.js runs as /bin/sh -c "node server.js". sh becomes PID 1 and doesn't forward SIGTERM → the app is killed hard after the timeout.

Rules

  • docker stop sends SIGTERM, then SIGKILL after 10 s (-t to change)
  • PID 1 gets no default signal handlers — your app must handle SIGTERM
  • PID 1 must reap zombies — use --init (tini) if the app spawns children
  • In wrapper scripts, end with exec "$@"
# BAD: shell form, sh is PID 1
CMD node server.js

# GOOD: exec form, node is PID 1
CMD ["node", "server.js"]

# ENTRYPOINT + CMD: fixed binary, default args
ENTRYPOINT ["/usr/local/bin/api"]
CMD ["--port", "8080"]
# docker run img --port 9090  → overrides CMD only

# Wrapper script that still forwards signals
# docker-entrypoint.sh
#!/bin/sh
set -e
envsubst < /tmpl/app.conf > /etc/app.conf
exec "$@"            # replace sh with the app

# Use a real init when processes fork
docker run --init myimage
Build cache

How layer caching works

Cache rules

  • Each instruction is cached by its text + its parent layer
  • COPY/ADD also checksum the copied files
  • RUN is cached by command text only — apt-get update can go stale
  • Once one step misses, all later steps rebuild
  • So: order from least to most frequently changing

Patterns

  • Copy dependency manifests first, install, then copy source
  • Use .dockerignore (.git, node_modules, secrets, build output) — smaller context, fewer cache busts
  • Combine apt-get update && install && rm -rf /var/lib/apt/lists/* in one RUN
  • --no-cache or --pull to refresh bases periodically
COPY package.json package-lock.json ./      # changes rarely → cached
RUN npm ci --omit=dev                       # reused unless the lockfile changed
COPY . .                                    # changes every commit → only this and later steps rebuild
Before / after

From a naive Dockerfile to a production one

# ❌ Naive — ~1 GB, root, slow rebuilds
FROM node:latest
COPY . /app
WORKDIR /app
RUN npm install
EXPOSE 3000
CMD npm start
# ✅ Production
# syntax=docker/dockerfile:1
FROM node:22-bookworm-slim AS deps
WORKDIR /app
COPY package.json package-lock.json ./
RUN --mount=type=cache,target=/root/.npm \
    npm ci --omit=dev

FROM node:22-bookworm-slim
ENV NODE_ENV=production
WORKDIR /app
COPY --from=deps /app/node_modules ./node_modules
COPY --chown=node:node . .
USER node
EXPOSE 3000
HEALTHCHECK --interval=30s --timeout=3s \
  CMD node healthcheck.js || exit 1
CMD ["node", "server.js"]

Changes: pinned slim base · deps layer cached by lockfile · cache mount for npm · no dev deps · non-root · exec-form CMD · healthcheck · plus a .dockerignore.

Chapter 2 · Part 2.3

BuildKit &
advanced builds

Multi-stage

Multi-stage builds: compile fat, ship thin

# syntax=docker/dockerfile:1
FROM golang:1.25 AS build
WORKDIR /src
COPY go.mod go.sum ./
RUN --mount=type=cache,target=/go/pkg/mod \
    go mod download
COPY . .
RUN --mount=type=cache,target=/go/pkg/mod \
    --mount=type=cache,target=/root/.cache/go-build \
    CGO_ENABLED=0 go build -ldflags="-s -w" -o /out/api ./cmd/api

FROM build AS test
RUN go test ./...

FROM gcr.io/distroless/static-debian12:nonroot
COPY --from=build /out/api /api
USER nonroot:nonroot
ENTRYPOINT ["/api"]

How it works

  • Each FROM starts a stage; only the last one ships
  • COPY --from=<stage> takes just the artifacts
  • BuildKit skips stages the target doesn't need
  • docker build --target test . runs tests in CI

Result

Compiler, sources, and caches stay in the build stage. The final image is a ~10 MB static binary with no shell or package manager — tiny attack surface.

BuildKit mounts

Cache, secret, and SSH mounts

# syntax=docker/dockerfile:1
# Package cache survives between builds,
# never ends up in the image
# Debian/Ubuntu images auto-clean apt; disable it
RUN rm -f /etc/apt/apt.conf.d/docker-clean
RUN --mount=type=cache,target=/var/cache/apt,sharing=locked \
    --mount=type=cache,target=/var/lib/apt,sharing=locked \
    apt-get update && \
    apt-get install -y --no-install-recommends curl

# Secret available only during this RUN
RUN --mount=type=secret,id=npmrc,target=/root/.npmrc \
    npm ci

# Forward the host SSH agent for private git
RUN mkdir -p -m 0700 ~/.ssh && \
    ssh-keyscan github.com >> ~/.ssh/known_hosts
RUN --mount=type=ssh \
    git clone git@github.com:org/private-lib.git
# Pass the secret and SSH agent at build time
docker build \
  --secret id=npmrc,src=$HOME/.npmrc \
  --ssh default \
  -t api:dev .

# ❌ Never do this — stays in history/layers
ARG NPM_TOKEN
ENV NPM_TOKEN=$NPM_TOKEN
COPY .npmrc /root/.npmrc

# Check what leaked
docker history --no-trunc api:dev

Rule: secrets go in --mount=type=secret at build time and in runtime secret stores at run time — never in ARG, ENV, COPY, or a layer you "delete" later.

buildx

Multi-platform builds and remote cache

# A builder that supports multi-platform
docker buildx create --name ci --driver docker-container --use

# Build amd64 + arm64 and push one multi-arch tag
docker buildx build \
  --platform linux/amd64,linux/arm64 \
  -t ghcr.io/org/api:1.4.2 \
  --cache-from type=registry,ref=ghcr.io/org/api:buildcache \
  --cache-to type=registry,ref=ghcr.io/org/api:buildcache,mode=max \
  --sbom=true --provenance=mode=max \
  --push .

# In GitHub Actions
#   --cache-from type=gha --cache-to type=gha,mode=max

# Many targets declared in docker-bake.hcl
docker buildx bake --push

Notes

  • Cross-arch builds use QEMU emulation (slow) or native nodes per arch (fast)
  • Go/Rust: cross-compile using $BUILDPLATFORM / $TARGETARCH args instead of emulation
  • mode=max caches intermediate stages too
  • Attestations attach an SBOM and SLSA provenance to the image
  • exec format error at runtime = wrong architecture
Base images

Choosing a base image

BaseSizeProsCons
debian / ubuntu~30–80 MBglibc, familiar tooling, broad compatibilityMore packages → more CVEs
*-slim variants~25–75 MBGood default for interpreted languagesStill has shell + package manager
alpine~5 MBTiny, fast pullsmusl libc: DNS and binary-compatibility surprises, slower Python wheels
distroless~2–25 MBNo shell or package manager, non-root variantsHarder debugging (use :debug tag or ephemeral debug)
scratch0 MBOnly your static binaryAdd CA certs, tzdata, and a user yourself
Hardened / minimal images (e.g. Chainguard, Docker Hardened Images)smallNear-zero CVEs, frequent rebuilds, SBOMsVendor ecosystem, versioning policies differ

Pin a specific version (python:3.13-slim-bookworm), rebuild regularly to pick up patches, and let Renovate/Dependabot bump base tags and digests.

Modern Dockerfile

Newer build features worth using

# syntax=docker/dockerfile:1
FROM debian:bookworm-slim

# Heredocs: readable multi-line RUN, one layer
RUN <<EOF
set -eux
apt-get update
apt-get install -y --no-install-recommends ca-certificates
rm -rf /var/lib/apt/lists/*
EOF

# Write a file inline
COPY <<EOF /etc/app/config.yaml
log_level: info
EOF

# Layer independent of the base → rebase without rebuild
COPY --link --chown=10001:10001 dist/ /app/
# Generate Dockerfile, compose.yaml, .dockerignore,
# README.Docker.md (ships with Docker Desktop)
docker init

# Lint with build checks (Buildx 0.15+)
docker build --check .

# Reproducible timestamps (file mtimes too: add
# --output type=image,rewrite-timestamp=true)
docker buildx build \
  --build-arg SOURCE_DATE_EPOCH=$(git log -1 --format=%ct) .

# .dockerignore — keep the context small and safe
.git
**/node_modules
**/*.log
.env*
!.env.example
dist/
coverage/

Keep # syntax=docker/dockerfile:1 at the top to always get the latest stable Dockerfile frontend features.

Chapter 3

Containers in practice

The runtime, networking, storage, and multi-container apps with Compose.

Chapter 3 · Part 3.1

Running containers

Lifecycle

Container states and commands

created

docker create

→

running

docker start / run

⇄

paused

pause / unpause (cgroup freezer)

→

exited

stop (TERM→KILL) / kill / process ends

→

removed

docker rm — writable layer gone

Restart policies

  • no (default)
  • on-failure[:N] — only on non-zero exit
  • always — also after daemon restart
  • unless-stopped — like always, unless you stopped it

Exit codes to know

  • 0 success · 1 app error
  • 125 docker run failed · 126 not executable · 127 not found
  • 137 = 128+9 SIGKILL (OOM or kill) · 143 = 128+15 SIGTERM
  • 139 = segfault
docker run

The flags you'll use every day

docker run -d \
  --name api \
  --restart unless-stopped \
  -p 127.0.0.1:8080:8080 \
  -e LOG_LEVEL=info \
  --env-file ./api.env \
  -v api-data:/var/lib/api \
  --network backend \
  --memory 512m --cpus 1 --pids-limit 200 \
  --read-only --tmpfs /tmp \
  --user 10001:10001 \
  --cap-drop ALL \
  --security-opt no-new-privileges \
  --init \
  --health-cmd 'wget -qO- localhost:8080/healthz || exit 1' \
  --health-interval 30s \
  --log-opt max-size=10m --log-opt max-file=3 \
  ghcr.io/org/api:1.4.2

Meaning

  • -d detached · --rm remove on exit (for one-offs)
  • -p host:container publish; bind to 127.0.0.1 unless it must be public
  • -e/--env-file config — not for secrets in production
  • -v named volume for data
  • Resource limits on every long-running container
  • Hardening: read-only root, non-root user, no caps, no privilege escalation
  • --init for signal forwarding & zombie reaping
  • Log rotation so disks don't fill
Health

Healthchecks and readiness

How HEALTHCHECK works

  • Runs inside the container every --interval (default 30 s)
  • Exit 0 = healthy, 1 = unhealthy; --retries failures (default 3) → unhealthy
  • --start-period gives slow apps time to boot (--start-interval = check rate during it)
  • Neither Docker nor Compose restarts unhealthy containers — Compose only uses health to gate depends_on; orchestrators with their own health checks (ECS, Kubernetes probes) replace them

Best practices

  • Cheap, local check (/healthz), short timeout
  • Don't check downstream dependencies in liveness
  • Distroless images: ship a tiny health binary instead of curl
HEALTHCHECK --interval=15s --timeout=3s \
  --start-period=30s --retries=3 \
  CMD ["/app/healthcheck"]

# Watch health status
docker ps --format 'table {{.Names}}\t{{.Status}}'
docker inspect -f '{{json .State.Health}}' api | jq

# Compose: start only when the DB is healthy
services:
  api:
    depends_on:
      db:
        condition: service_healthy
  db:
    image: postgres:17
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U postgres"]
      interval: 5s
      retries: 10
Chapter 3 · Part 3.2

Networking

Drivers

Network drivers

DriverHow it worksUse for
bridge (default)Linux bridge + veth pairs + NAT via iptables/nftablesSingle-host apps
user-defined bridgeSame, plus embedded DNS (127.0.0.11) and isolation per networkAlways instead of the default bridge
hostNo network namespace — shares the host's stackMax performance, many ports; no isolation
noneOnly loopbackBatch jobs that need no network
overlayVXLAN across hosts (Swarm)Multi-host Swarm services
macvlan / ipvlanContainer gets an address on the physical LANLegacy apps expecting LAN IPs

Default bridge pitfalls

No DNS by container name, all containers can talk to each other. Create your own networks.

Segment by tier

frontend and backend networks; only the API joins both; DB lives only on backend.

Under the hood

Port publishing and container DNS

Packet path for -p 8080:80

  • Client → host:8080 → DNAT rule in the DOCKER chain → container IP:80
  • Container → internet: MASQUERADE (SNAT) to the host IP
  • Between user-defined networks: blocked by default

Firewall gotcha

  • Published ports are inserted before UFW/firewalld rules → they are reachable even if your firewall "blocks" them
  • Bind to 127.0.0.1, or put rules in the DOCKER-USER chain (iptables backend; Engine 29 adds an experimental nftables backend)
# User-defined network with DNS
docker network create backend
docker run -d --name db --network backend postgres:17
docker run --rm --network backend busybox \
  nslookup db        # resolved by 127.0.0.11

# Network aliases (round-robin DNS)
docker run -d --network backend --network-alias search \
  -e discovery.type=single-node elasticsearch:8.15.0

# Inspect
docker network inspect backend
docker port api
sudo iptables -t nat -L DOCKER -n

# Debug with a toolbox in the same netns
docker run --rm -it --network container:api \
  nicolaka/netshoot
#  → curl, dig, ss, tcpdump, iperf …
Chapter 3 · Part 3.3

Storage

Mount types

Volumes, bind mounts, tmpfs

TypeLives inManaged byUse for
Named volume/var/lib/docker/volumes/…DockerDatabases, persistent app data — the production default
Bind mountAny host pathYouDev source code, config files; ties container to host layout
tmpfsHost memoryKernelScratch data, secrets at runtime, read-only root with writable /tmp
# --mount is explicit; -v is shorter
docker run -d --name db \
  --mount type=volume,src=pgdata,dst=/var/lib/postgresql/data \
  postgres:17

docker run --rm -it \
  --mount type=bind,src="$PWD",dst=/app,readonly \
  node:22-slim bash

docker run --read-only \
  --mount type=tmpfs,dst=/tmp,tmpfs-size=64m app

# Back up a volume to a tarball
docker run --rm -v pgdata:/data:ro -v "$PWD":/backup \
  busybox tar czf /backup/pgdata.tgz -C /data .

Best practices

  • Named volumes for state; never the container layer
  • Mount config read-only (:ro)
  • Match UID/GID: the container user must own the volume path
  • For databases, use the DB's own backup tool (consistent dumps)
  • docker volume prune deletes data — check first
  • Never bind-mount /, /etc, or docker.sock
Chapter 3 · Part 3.4

Docker Compose

Multi-container apps as one declarative file.

compose.yaml

A realistic Compose file

name: shop
services:
  api:
    build: { context: ., target: runtime }
    image: ghcr.io/org/api:${TAG:-dev}
    ports: ["127.0.0.1:8080:8080"]
    env_file: .env
    secrets: [db_password]
    depends_on:
      db: { condition: service_healthy }
    networks: [frontend, backend]
    deploy:
      resources:
        limits: { cpus: "1", memory: 512M }
    restart: unless-stopped
  db:
    image: postgres:17
    environment:
      POSTGRES_PASSWORD_FILE: /run/secrets/db_password
    secrets: [db_password]
    volumes: [pgdata:/var/lib/postgresql/data]
    networks: [backend]
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U postgres"]
      interval: 5s
      retries: 10

networks:
  frontend: {}
  backend: { internal: true }   # no internet
volumes:
  pgdata: {}
secrets:
  db_password: { file: ./secrets/db_password.txt }

The top-level version: key is obsolete. Use docker compose (v2 plugin), not the retired docker-compose v1.

Workflow

Compose commands and dev workflow

docker compose up -d --build     # build + start
docker compose ps
docker compose logs -f api
docker compose exec api sh
docker compose run --rm api npm test
docker compose config            # fully resolved file
docker compose down              # keep volumes
docker compose down -v           # ⚠ deletes volumes

# Layer files per environment
docker compose -f compose.yaml -f compose.prod.yaml up -d

# Optional services
docker compose --profile debug up -d

# Live sync code into containers
docker compose watch

Dev features

  • compose.override.yaml is merged automatically (unless you pass -f) — dev-only mounts and ports
  • develop.watch: sync, rebuild, sync+restart actions
  • profiles for optional tools (adminer, mailhog)
  • .env file for variable interpolation

Production caveats

Compose is single-host: no rescheduling on node failure, no rolling updates across hosts. Fine for small services and dev/CI; use ECS or Kubernetes for HA.

Chapter 4

Operating safely

Security, observability, debugging, and production practice.

Chapter 4 · Part 4.1

Security

Image, build, runtime, daemon, host.

Threat model

Where container attacks come from

Image

  • Vulnerable packages
  • Malicious base or typosquatted image
  • Secrets baked into layers

Build / CI

  • Compromised dependencies
  • Leaked registry credentials
  • Unsigned, untraceable images

Runtime

  • Root + extra caps
  • --privileged, host mounts
  • Exposed ports

Daemon / host

  • Exposed docker.sock or TCP 2375
  • Unpatched kernel
  • Shared host with untrusted workloads

docker.sock = root on the host

  • Anyone who can talk to the Docker API can run docker run -v /:/host --privileged … and own the machine
  • Membership in the docker group is equivalent to root
  • Never mount the socket into app containers; for CI use rootless builders, BuildKit, or a socket proxy with an allow-list
  • Never expose the daemon on TCP without mutual TLS — plain TCP is unauthenticated root access (recent Engines refuse to start that way)
Runtime hardening

Least privilege for containers

Checklist

  • Run as non-root: USER 10001 in the image
  • --cap-drop ALL, add back only what's needed
  • --security-opt no-new-privileges (blocks setuid escalation)
  • --read-only root filesystem + tmpfs for scratch
  • Keep default seccomp & AppArmor/SELinux
  • Resource limits (memory, CPU, pids)
  • No --privileged, no host namespaces, no sensitive host mounts
  • Untrusted code: gVisor (--runtime=runsc) or Kata
# Hardened run
docker run -d --name api \
  --user 10001:10001 \
  --cap-drop ALL \
  --security-opt no-new-privileges \
  --read-only --tmpfs /tmp:rw,noexec,nosuid,size=64m \
  --memory 512m --cpus 1 --pids-limit 200 \
  -p 127.0.0.1:8080:8080 \
  ghcr.io/org/api@sha256:<digest>

# Needs to bind :80 as non-root?
#   listen on 8080 and publish 80:8080 instead

# Verify
docker inspect api --format '{{.Config.User}}'
docker inspect api --format \
  '{{.HostConfig.CapDrop}} {{.HostConfig.ReadonlyRootfs}}'
docker exec api id
Rootless

Rootless Docker and user namespaces

Rootless mode

  • The daemon and containers run as an unprivileged user
  • A container escape lands as that user, not root
  • Setup: dockerd-rootless-setuptool.sh install
  • Limits: ports < 1024 need extra config; some network and storage features differ
  • Podman is rootless by default

userns-remap

  • Daemon still runs as root, but container UID 0 maps to a high unprivileged host UID
  • Enable in daemon.json: "userns-remap": "default"
  • Affects volume ownership — plan UID mapping
  • Per-container opt-out: --userns=host

Recommendation: non-root images everywhere; rootless Docker on developer and CI machines where possible; userns-remap or rootless on shared hosts.

Supply chain

Scan, attest, sign

# Vulnerability scanning
docker scout quickview ghcr.io/org/api:1.4.2
docker scout cves --only-severity critical,high \
  ghcr.io/org/api:1.4.2
trivy image --severity HIGH,CRITICAL --exit-code 1 \
  ghcr.io/org/api:1.4.2

# SBOM
docker buildx imagetools inspect ghcr.io/org/api:1.4.2 \
  --format '{{json .SBOM}}'
syft ghcr.io/org/api:1.4.2 -o spdx-json > sbom.json

# Sign and verify (keyless, Sigstore)
cosign sign ghcr.io/org/api@sha256:<digest>
cosign verify ghcr.io/org/api@sha256:<digest> \
  --certificate-identity-regexp 'https://github.com/org/.*' \
  --certificate-oidc-issuer \
    https://token.actions.githubusercontent.com

Best practices

  • Scan in CI and fail on fixable critical/high CVEs
  • Rescan stored images continuously — new CVEs appear daily
  • Generate SBOM + provenance at build time
  • Sign images and verify at deploy (Kyverno, admission controllers)
  • Pin base images by digest; rebuild on base updates
  • Run docker-bench-security / CIS Docker Benchmark on hosts
Secrets & daemon

Runtime secrets and daemon hardening

Runtime secrets

  • Environment variables leak via docker inspect, /proc, crash dumps, child processes
  • Prefer files: Compose/Swarm secrets mounted at /run/secrets/…
  • Or fetch at startup from Secrets Manager / Vault with an IAM role
  • Support *_FILE variables in your app (many official images do)
  • Scan repos and images for leaked secrets (gitleaks, trufflehog)
# /etc/docker/daemon.json — hardened baseline
{
  "live-restore": true,
  "no-new-privileges": true,
  "icc": false,              # default bridge only
  "userland-proxy": false,
  "log-driver": "local",
  "log-opts": { "max-size": "10m", "max-file": "3" },
  "default-ulimits": {
    "nofile": { "Name": "nofile", "Soft": 65536, "Hard": 65536 }
  }
}
# Never plain TCP 2375 (Engine 27+ refuses it without TLS)
# Remote access → SSH (DOCKER_HOST=ssh://…) or mTLS
Chapter 4 · Part 4.2

Observability &
operations

Logs, metrics, events, disk, daemon.

Logging

Logging drivers and rotation

DriverNotes
json-fileDefault; no rotation unless configured → full disks
localCompressed, rotated by default — recommended for single hosts
journald, syslogInto the host's log system
awslogs, gcplogsStraight to cloud logging
fluentd, gelf, splunkCentral pipelines

Best practices

  • Apps log to stdout/stderr, structured JSON
  • Set rotation globally in daemon.json
  • Remote drivers can block or drop logs — use mode=non-blocking with a buffer
# Read logs
docker logs -f --since 10m --timestamps api
docker logs --tail 100 api 2>&1 | grep -i error

# Where json-file logs live
docker inspect -f '{{.LogPath}}' api

# Per-container driver
docker run -d \
  --log-driver awslogs \
  --log-opt awslogs-region=ap-northeast-1 \
  --log-opt awslogs-group=/app/api \
  --log-opt mode=non-blocking \
  --log-opt max-buffer-size=4m \
  api:1.4.2

# Change the default (restart dockerd;
# affects only new containers)
{ "log-driver": "local",
  "log-opts": { "max-size": "10m", "max-file": "5" } }
Metrics

Metrics and events

# Live resource usage
docker stats
docker stats --no-stream --format \
  'table {{.Name}}\t{{.CPUPerc}}\t{{.MemUsage}}\t{{.PIDs}}'

# Real-time daemon events: start, die, oom, health
docker events --filter type=container \
  --filter event=die --filter event=oom

# Engine metrics for Prometheus (daemon.json)
{ "metrics-addr": "127.0.0.1:9323" }
curl -s localhost:9323/metrics | head

# Per-container metrics: cAdvisor — a trusted monitoring
# agent, the deliberate exception to "no host mounts"
docker run -d --name cadvisor -p 127.0.0.1:8081:8080 \
  --privileged --device=/dev/kmsg \
  -v /:/rootfs:ro -v /var/run:/var/run:ro \
  -v /sys:/sys:ro -v /var/lib/docker:/var/lib/docker:ro \
  ghcr.io/google/cadvisor:<version>

What to watch

  • Container restarts, OOM kills, unhealthy status
  • CPU throttling and memory near limit
  • Host disk usage of /var/lib/docker
  • Daemon health, image pull failures
  • App RED metrics (rate, errors, duration)

Typical stack

cAdvisor + node-exporter → Prometheus → Grafana + Alertmanager, or CloudWatch agent / Datadog / OpenTelemetry Collector on the host.

Housekeeping

Disk usage and cleanup

# What uses space?
docker system df
docker system df -v

# Safe-ish cleanup
docker container prune          # stopped containers
docker image prune              # dangling images
docker builder prune --keep-storage 20gb

# Aggressive — read before running
docker image prune -a --filter "until=168h"
docker system prune -a          # all unused images too
docker volume prune             # ⚠ unused volumes = data

# Find big log files
sudo sh -c 'du -sh /var/lib/docker/containers/*/*-json.log' \
  | sort -h | tail

"No space left on device"

  • Usual causes: unrotated logs, build cache, old images, data written into the container layer
  • Also check inodes: df -i

Best practices

  • Put /var/lib/docker on its own volume (data-root)
  • Global log rotation
  • Scheduled prune on CI runners
  • Alert at 80% disk
Chapter 4 · Part 4.3

Debugging &
troubleshooting

Cheat sheet

Debugging commands

# Why did it stop?
docker ps -a --filter name=api
docker inspect api --format \
  '{{.State.Status}} exit={{.State.ExitCode}}'
docker inspect api --format \
  'oom={{.State.OOMKilled}} err={{.State.Error}}'
docker logs --tail 200 api

# Get inside
docker exec -it api sh
docker exec -u 0 -it api sh      # as root

# Crashes on start? Override the entrypoint (needs a shell)
docker run --rm -it --entrypoint sh api:1.4.2

# Distroless: share namespaces with a toolbox
docker run --rm -it --pid container:api \
  --network container:api --cap-add SYS_PTRACE \
  nicolaka/netshoot
ls /proc/1/root/app              # target's filesystem
# What changed in the filesystem?
docker diff api

# Copy files in and out
docker cp api:/app/logs/error.log .

# Full config: mounts, env, network, limits
docker inspect api | jq '.[0].HostConfig'

# Processes inside, seen from the host
docker top api

# Debug a build step by step
docker build --progress=plain --no-cache .
docker build --target deps -t dbg . && \
  docker run --rm -it dbg sh

# Daemon side
journalctl -u docker --since "1 hour ago"
docker info
Troubleshooting

Common problems & fixes

SymptomLikely causeFix
Exit 137OOM kill or SIGKILL after stop timeoutCheck OOMKilled; raise limit or fix leak; handle SIGTERM
exec format errorImage architecture ≠ host (arm64 vs amd64)Build multi-arch; --platform
permission denied on volumeContainer UID ≠ volume ownerchown in image, matching UID, :z on SELinux
Container exits immediatelyMain process ended (daemonized app, wrong CMD)Run in foreground; check docker logs
Can't reach app on published portApp listens on 127.0.0.1 inside containerListen on 0.0.0.0
Containers can't resolve each otherUsing the default bridgeUser-defined network
port is already allocatedHost port in usess -ltnp; pick another port
Slow stop (10 s)Shell-form CMD, SIGTERM ignoredExec form, --init, handle signals
Slow buildsCache busted early, large contextReorder, .dockerignore, cache mounts
toomanyrequests on pullDocker Hub rate limitdocker login, mirror / pull-through cache
Chapter 4 · Part 4.4

CI/CD & production

Pipeline

A container CI pipeline

Lint

hadolint on Dockerfile

→

Build

buildx, registry/gha cache, multi-arch

→

Test

--target test, Compose integration tests

→

Scan

Trivy / Scout, fail on criticals

→

Push & sign

Immutable tag, SBOM, provenance, cosign

→

Deploy

By digest to ECS / Kubernetes

Tagging strategy

  • Immutable: api:1.4.2 and api:git-3f2c1ab
  • Promote the same digest dev → staging → prod; never rebuild per env
  • Config via env/secrets at runtime, not baked per environment
  • OCI labels: org.opencontainers.image.source, .revision, .version

CI security

  • Auth to registries with OIDC, not long-lived passwords
  • No privileged Docker-in-Docker on shared runners — use BuildKit rootless or remote builders
  • Ephemeral runners; prune caches regularly
Beyond one host

From Docker to orchestrators

Docker conceptKubernetesAmazon ECS
docker run / containerPod (1+ containers)Task
Compose serviceDeployment + ServiceService
-p published portService / IngressALB target group
Named volumePersistentVolumeClaimEBS / EFS volume
--memory, --cpusresources.requests / limitsTask / container CPU & memory
HEALTHCHECKliveness / readiness / startup probesContainer health check
Compose secretsSecret / External SecretsSecrets Manager / SSM in task definition
--restartControllers recreate podsService scheduler

Kubernetes doesn't use Docker Engine as its runtime anymore (dockershim removed in 1.24), but it runs the same OCI images you build with Docker.

More tools

Features worth knowing

# Change limits on a running container
docker update --memory 1g --memory-swap 1g \
  --cpus 2 api

# Talk to a remote engine over SSH
docker context create prod \
  --docker host=ssh://ops@10.0.1.20
docker --context prod ps

# GPUs (needs NVIDIA Container Toolkit on the host)
docker run --rm --gpus all \
  nvidia/cuda:12.6.0-base-ubuntu24.04 nvidia-smi

# Compose: wait until healthy, reuse other files
docker compose up -d --wait
#   compose.yaml →  include: [../shared/observability.yaml]

# Swarm: built-in multi-host orchestration
docker swarm init
docker service create --name web --replicas 3 \
  --update-parallelism 1 --update-delay 10s nginx:1.27

Notes

  • docker update applies cgroup limits live — no restart
  • Contexts switch between local, remote, and Desktop engines
  • GPU access is exposed via the NVIDIA runtime / CDI devices
  • --wait makes CI scripts block until services are healthy
  • Swarm adds services, rolling updates, secrets, and overlay networks — simpler than Kubernetes, far smaller ecosystem
  • Docker Desktop's docker debug attaches a toolbox shell to slim containers (free for all users since Desktop 4.49)
Tooling notes

Docker Desktop, licensing, and alternatives

Docker Desktop

  • Runs Docker Engine inside a Linux VM on macOS/Windows
  • Bind-mount I/O crosses the VM boundary — use VirtioFS / synchronized file shares, or keep heavy deps in volumes
  • Paid subscription required for larger companies (more than 250 employees or more than $10M revenue); free for personal use, education, small businesses, non-commercial open source

Alternatives

  • Docker Engine on Linux — free (Apache 2.0)
  • Podman / Podman Desktop — daemonless, rootless
  • Colima, Rancher Desktop, OrbStack on macOS
  • nerdctl + containerd — Docker-compatible CLI
  • Check your company policy before installing
Chapter 5

Hands-on labs

stub

Lab setup

Before you start

A Linux Docker host, a scratch directory, and the habit of cleaning up after every lab.

Tools

  • Docker Engine 28+ with the Buildx and Compose v2 plugins.
  • docker version and docker info should both answer.
  • Optional: trivy, jq, hadolint.
  • A free registry account (Docker Hub or GHCR) for Lab 12.

macOS and Windows

  • Docker Desktop runs the engine in a Linux VM.
  • Host-level steps (/var/lib/docker, nsenter, iptables) must run inside that VM or on a Linux VM.
  • Easiest: a small Ubuntu VM (Multipass, Lima, UTM, or a cloud instance).

How to use the labs

  • Type the commands; do not paste blindly.
  • Predict the output before you press Enter.
  • Do every "Break it" step: that is where debugging skill comes from.
  • Finish each lab with its cleanup line.

Work in ~/docker-labs/<lab>. Check what is left with docker ps -a, docker volume ls, and docker system df.

Lab 1

Look inside an image

Goal: see that an image is a config plus ordered layers, and find them on disk.

docker pull nginx:1.27
docker image inspect nginx:1.27 --format '{{.Architecture}} {{.Os}}'
docker image inspect nginx:1.27 --format '{{json .Config.Cmd}}'
docker image inspect nginx:1.27 \
  --format '{{json .RootFS.Layers}}' | jq length
docker history --no-trunc nginx:1.27 | head    # one row per instruction

# Index vs manifest: one tag, several platforms
docker buildx imagetools inspect nginx:1.27

# Pull by digest: immutable reference
DIGEST=$(docker buildx imagetools inspect nginx:1.27 \
  --format '{{json .Manifest}}' | jq -r .digest)
docker pull nginx@$DIGEST

# overlay2 hosts only (Linux): the container's merged view
docker run -d --name l1 nginx:1.27
docker inspect l1 --format '{{json .GraphDriver.Data}}' | jq

Verify / Break it

  • imagetools inspect lists amd64, arm64, and more under one index digest.
  • history rows with 0B are metadata-only instructions (ENV, CMD).
  • Create a file with docker exec l1 touch /x; find it under UpperDir.
  • On the containerd image store this output differs: docker info shows which store you use.
  • Cleanup: docker rm -f l1
Lab 2

Make a Dockerfile fast and small

Goal: measure the effect of instruction order, .dockerignore, and a slim base.

mkdir l2 && cd l2 && npm init -y >/dev/null && npm i express
echo "require('express')().get('/',(q,r)=>r.send('ok'))
  .listen(3000)" > server.js

cat > Dockerfile.naive <<'EOF'
FROM node:22
COPY . /app
WORKDIR /app
RUN npm install
CMD npm start
EOF
docker build -f Dockerfile.naive -t l2:naive .
echo // >> server.js
time docker build -f Dockerfile.naive -t l2:naive .

printf 'node_modules\n.git\nDockerfile*\n' > .dockerignore
# Now write Dockerfile.good: node:22-bookworm-slim, COPY package*.json,
# RUN npm ci --omit=dev, COPY . ., USER node, CMD ["node","server.js"]
docker build -f Dockerfile.good -t l2:good .
docker images l2

Verify / Break it

  • After editing server.js (BuildKit ignores mtime, so touch is not enough), the naive build re-runs npm install; the good one reuses the cached npm ci layer.
  • Compare sizes: the slim image should be several times smaller.
  • docker run --rm l2:good id prints a non-root user.
  • Break it: move COPY . . above npm ci and watch the cache miss.
  • Cleanup: docker rmi l2:naive l2:good
Lab 3

Multi-stage build to a minimal runtime

Goal: compile in a fat stage and ship only the binary.

mkdir l3 && cd l3
cat > main.go <<'EOF'
package main
import ("fmt"; "net/http")
func main() {
  http.HandleFunc("/", func(w http.ResponseWriter,
    r *http.Request) { fmt.Fprintln(w, "hi") })
  http.ListenAndServe(":8080", nil)
}
EOF
cat > Dockerfile <<'EOF'
# syntax=docker/dockerfile:1
FROM golang:1.25 AS build
WORKDIR /src
COPY main.go .
RUN go mod init l3 && CGO_ENABLED=0 go build -o /app .
FROM gcr.io/distroless/static-debian12:nonroot
COPY --from=build /app /app
ENTRYPOINT ["/app"]
EOF
docker build -t l3 . && docker images l3

Verify / Break it

  • Image is a few MB; golang:1.25 is hundreds of MB.
  • docker run --rm -p 8080:8080 l3 then curl localhost:8080.
  • docker run --rm --entrypoint sh l3 fails: there is no shell. That is the point.
  • Build only the first stage with --target build and compare.
  • Break it: drop CGO_ENABLED=0 and see the binary fail on a static base.
Lab 4

BuildKit: cache mounts, secrets, and attestations

Goal: speed up dependency installs, use a secret without leaking it, and attach an SBOM.

# syntax=docker/dockerfile:1
FROM python:3.13-slim
RUN --mount=type=cache,target=/root/.cache/pip \
    pip install requests
RUN --mount=type=secret,id=token \
    test -s /run/secrets/token && echo "secret present"

# build with the secret from a file (never --build-arg)
echo "s3cr3t" > token.txt
docker build --secret id=token,src=token.txt -t l4 .
docker history l4 | grep -c s3cr3t      # 0

# SBOM + provenance: docker-container driver (or containerd store)
docker buildx create --name lab --driver docker-container --use
docker buildx build --sbom=true --provenance=mode=max \
  -t <registry>/l4:1 --push .
docker buildx imagetools inspect <registry>/l4:1 \
  --format '{{json .SBOM}}' | jq '.SPDX.packages | length'

Verify / Break it

  • Add a package to the pip install line and rebuild: cached wheels come from the cache mount.
  • The secret never appears in history or in any layer.
  • Break it: pass the token with --build-arg and find it in docker history.
  • Multi-arch: add --platform linux/amd64,linux/arm64; .SBOM output is then keyed by platform.
  • Cleanup: docker buildx rm lab, rm token.txt
Lab 5

Runtime: signals, PID 1, restarts, and limits

Goal: see why exec form and --init matter, and what an OOM kill looks like.

# No init: PID 1 has no TERM handler, so TERM is ignored → 10 s, then KILL
docker run -d --name l5a alpine sh -c 'sleep 600'
time docker stop l5a                     # ~10 s
docker run -d --name l5b --init alpine sleep 600
time docker stop l5b                     # ~instant: tini forwards TERM

# Restart policy and restart count
docker run -d --name l5c --restart on-failure:3 alpine sh -c 'exit 1'
sleep 20; docker inspect l5c \
  --format '{{.RestartCount}} {{.State.Status}}'

# Memory limit → OOM kill → exit 137
docker run --name l5d --memory 64m --memory-swap 64m alpine \
  sh -c 'tail /dev/zero'
docker inspect l5d \
  --format 'exit={{.State.ExitCode}} oom={{.State.OOMKilled}}'

docker stats --no-stream
docker rm -f l5a l5b l5c l5d

Verify / Break it

  • l5a takes the full stop timeout; l5b stops at once.
  • l5c gives up after 3 restarts and stays exited.
  • l5d shows exit=137 oom=true.
  • Try --pids-limit 20 with a fork loop and watch it hit the limit.
  • Change limits live with docker update --cpus 0.5 <name>.
Lab 6

Networking: DNS, isolation, and the packet path

Goal: prove why user-defined bridges beat the default bridge, and see the NAT rules.

docker network create app
docker run -d --name db --network app redis:7
docker run --rm --network app alpine nslookup db  # via 127.0.0.11
docker run --rm alpine ping -c1 db   # fails: default bridge

# Isolation between networks
docker network create other
docker run --rm --network other alpine ping -c1 -W1 db  # fails

# Publish safely and inspect the NAT rule (Linux host)
docker run -d --name web --network app -p 127.0.0.1:8080:80 nginx:1.27
sudo iptables -t nat -L DOCKER -n | grep 8080
sudo iptables -L DOCKER-USER -n      # your own rules go here

# none and host
docker run --rm --network none alpine ip addr       # only lo
docker run --rm --network host alpine ip addr       # host interfaces

Verify / Break it

  • nslookup db returns the container IP on app.
  • The DNAT rule targets the container IP on port 80.
  • Publish with -p 8080:80 (no IP) and note it is reachable from other machines, bypassing typical host firewall rules.
  • On iptables-nft hosts the rules also show in sudo nft list ruleset; a native nftables backend is experimental from Engine 29.
  • Cleanup: docker rm -f db web && docker network rm app other
Lab 7

Storage: volumes, bind mounts, tmpfs, and backups

Goal: choose the right mount, back up a volume, and fix a permission error.

docker volume create pgdata
docker run -d --name pg -e POSTGRES_PASSWORD=pw \
  -v pgdata:/var/lib/postgresql/data postgres:17
sleep 5; docker exec pg psql -U postgres \
  -c 'create table t(x int); insert into t values (1);'
docker rm -f pg   # data survives in the volume

# Back up and restore a volume with a throwaway container
docker run --rm -v pgdata:/data:ro -v "$PWD":/backup alpine \
  tar czf /backup/pgdata.tgz -C /data .
docker volume create pgdata2
docker run --rm -v pgdata2:/data -v "$PWD":/backup alpine \
  tar xzf /backup/pgdata.tgz -C /data

# Bind mount + non-root user → permission problem
mkdir out
docker run --rm -u 10001 -v "$PWD/out":/out alpine touch /out/f
# tmpfs: in-memory scratch, gone on stop
docker run --rm --tmpfs /scratch:size=16m alpine df -h /scratch

Verify / Break it

  • Start a new postgres:17 on pgdata2 and select * from t returns 1.
  • The touch fails: out/ is owned by your host UID, not 10001.
  • Fix it with sudo chown 10001 out or by running with -u $(id -u).
  • Mount a config file read-only with :ro and try to edit it inside.
  • Cleanup: docker volume rm pgdata pgdata2
Lab 8a

Compose: a three-service app with health-gated startup

Goal: run web + api + db where the api waits for a healthy database.

# compose.yaml
services:
  db:
    image: postgres:17
    environment: { POSTGRES_PASSWORD: pw }
    volumes: [dbdata:/var/lib/postgresql/data]
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U postgres"]
      interval: 5s
      retries: 10
  api:
    build: ./api
    depends_on: { db: { condition: service_healthy } }
    environment: { DATABASE_URL: postgres://postgres:pw@db:5432/postgres }
  web:
    image: nginx:1.27
    ports: ["127.0.0.1:8080:80"]
    depends_on: [api]
volumes: { dbdata: {} }

Run it

  • docker compose up -d --wait: returns when services are healthy or running.
  • docker compose ps shows healthy for db.
  • docker compose logs -f api to follow one service.
  • docker compose config prints the fully resolved file.
  • Services reach each other by name on the project network.
Lab 8b

Compose: profiles, watch mode, and teardown

Goal: a fast inner dev loop and optional tooling that only starts on demand.

# add to compose.yaml
  adminer:
    image: adminer
    profiles: [debug]
    ports: ["127.0.0.1:8081:8080"]
  api:
    develop:
      watch:
        - action: sync
          path: ./api/src
          target: /app/src
        - action: rebuild
          path: ./api/package.json

docker compose --profile debug up -d   # adminer starts only now
docker compose up --watch              # edit api/src and see it sync
docker compose down                    # keeps the dbdata volume
docker compose down -v                 # also removes volumes

Verify / Break it

  • Without --profile debug, adminer never starts.
  • Editing a file under api/src syncs without a rebuild; editing package.json rebuilds.
  • Break it: point the healthcheck at a wrong port (pg_isready -p 5433); db turns unhealthy and api never starts.
  • Run tests in a one-off container: docker compose run --rm api npm test.
Lab 9

Security: harden a container, then scan its image

Goal: apply least privilege at runtime and find vulnerabilities before shipping.

# Default vs hardened
docker run --rm alpine sh -c 'id; grep Cap /proc/self/status'
docker run --rm -u 10001 --cap-drop ALL \
  --security-opt no-new-privileges --read-only \
  --tmpfs /tmp --pids-limit 100 alpine \
  sh -c 'id; grep CapEff /proc/self/status; touch /x'

# Which capability does the app really need?
docker run --rm --cap-drop ALL nginx:1.27            # fails to start
docker run --rm --cap-drop ALL --cap-add CHOWN --cap-add SETUID \
  --cap-add SETGID --cap-add NET_BIND_SERVICE nginx:1.27

# Scan
docker scout quickview nginx:1.27            # needs docker login
docker scout cves --only-severity critical,high nginx:1.27
trivy image --severity HIGH,CRITICAL --exit-code 1 nginx:1.27

Verify / Break it

  • Hardened run: non-root UID, CapEff: 0000000000000000, and touch /x fails (read-only).
  • nginx as root needs CHOWN, SETUID, SETGID back; NET_BIND_SERVICE only matters with --network host (Docker sets ip_unprivileged_port_start=0 otherwise). nginxinc/nginx-unprivileged needs none.
  • Compare CVE counts for nginx:1.27 vs an -alpine tag.
  • Read docker info --format "{{.SecurityOptions}}": seccomp and AppArmor/SELinux status.
  • Stretch: install rootless Docker in a VM and repeat Lab 5.
Lab 10

Observability: logs, rotation, events, and metrics

Goal: keep logs from filling the disk and watch the engine report what happens.

# Unrotated json-file grows forever; cap it per container
docker run -d --name chatty --log-driver json-file \
  --log-opt max-size=1m --log-opt max-file=3 \
  alpine sh -c 'while true; do date; done'
ID=$(docker inspect -f '{{.Id}}' chatty)
sleep 20; sudo ls -lh /var/lib/docker/containers/$ID/

# Engine-wide default: /etc/docker/daemon.json
#   { "log-driver": "local",
#     "log-opts": { "max-size": "10m", "max-file": "3" },
#     "metrics-addr": "127.0.0.1:9323" }
sudo systemctl restart docker
curl -s localhost:9323/metrics | grep engine_daemon_container_states

# Events stream in another terminal
docker events --filter type=container \
  --filter event=die --filter event=oom
docker system df -v

Verify / Break it

  • Log files stay at ≤1 MB × 3 for chatty.
  • Daemon-level log options apply only to containers created after the restart.
  • Re-run Lab 5's OOM step and see oom then die in the events stream.
  • The metrics endpoint is Prometheus format: scrape it, never bind it to 0.0.0.0.
  • Cleanup: docker rm -f chatty
Lab 11

Debugging: a container that keeps crashing

Goal: find the cause from state, logs, and the process namespace, even without a shell.

docker run -d --name bad --restart on-failure:5 alpine \
  sh -c 'echo "FATAL: DATABASE_URL not set" >&2; exit 3'
docker ps -a --filter name=bad
docker inspect bad --format \
  '{{.State.ExitCode}} oom={{.State.OOMKilled}} n={{.RestartCount}}'
docker logs --tail 50 bad

# Distroless or no shell: run tools from outside
docker run -d --name ok -p 8081:8080 l3
PID=$(docker inspect -f '{{.State.Pid}}' ok)
sudo nsenter -t $PID -n ss -lntp        # listening sockets
docker run --rm --network container:ok nicolaka/netshoot \
  curl -s localhost:8080
docker cp ok:/app ./app-copy       # copy files out

# Wrong architecture?
docker image inspect l3 --format '{{.Architecture}}'

Verify / Break it

  • Exit code tells the story: 1 app error, 126/127 bad entrypoint, 137 kill/OOM, 139 segfault.
  • --network container:ok shares the network namespace with a toolbox image.
  • Docker Desktop also offers docker debug <container> for a toolbox shell (free for all users since Desktop 4.49).
  • Break it: set ENTRYPOINT ["/app2"] and read the error in docker inspect .State.Error.
  • Cleanup: docker rm -f bad ok
Lab 12

Production: tag, push, pin by digest, and lint

Goal: produce an image you can trace, reproduce, and roll back.

hadolint Dockerfile                     # lint before building

docker buildx create --use --driver docker-container
REG=ghcr.io/<you>; SHA=$(git rev-parse --short HEAD)
docker buildx build --platform linux/amd64,linux/arm64 \
  --label org.opencontainers.image.revision=$SHA \
  --sbom=true --provenance=mode=max \
  -t $REG/api:1.0.0 -t $REG/api:sha-$SHA --push .

# Deploy by digest, not by tag
docker buildx imagetools inspect $REG/api:1.0.0 | grep Digest
docker run -d $REG/api@sha256:<digest>

# Optional: sign with cosign (keyless in CI)
cosign sign $REG/api@sha256:<digest>
cosign verify $REG/api@sha256:<digest> \
  --certificate-identity-regexp '.*' \
  --certificate-oidc-issuer-regexp '.*'

Verify / Break it

  • Both tags point to the same digest.
  • Retag 1.0.0 to a new build and note that the digest-pinned container is unaffected.
  • docker buildx imagetools inspect --format "{{json .Provenance}}" shows the build source.
  • In real pipelines, pin the cosign identity and issuer; the regex wildcards above are only for the lab.
  • Fix every hadolint warning before you ship.
Capstone

Put it all together

One app, every practice from the deck. Treat it as a portfolio piece.

Build

  • Multi-stage, slim or distroless runtime, non-root USER.
  • BuildKit cache mounts; secrets via --secret only.
  • .dockerignore, pinned base images, OCI labels.
  • Multi-arch with SBOM and provenance; hadolint clean.

Run

  • Compose file with healthchecks and service_healthy gating.
  • Named volumes for data; a tested backup and restore script.
  • --read-only, --cap-drop ALL, no-new-privileges, memory, CPU and PID limits.
  • Ports published on 127.0.0.1 behind a reverse proxy.

Operate

  • local log driver with rotation; metrics scraped by Prometheus.
  • An alert on oom events and restart loops.
  • Trivy or Scout gate that fails on critical CVEs.
  • A one-page runbook: deploy by digest, roll back, debug a crash.
Checklist

After these labs you can…

Tick each item only when you can do it without notes.

Images and builds

  • Explain index, manifest, config, and layers for any image.
  • Order a Dockerfile for cache hits and measure the difference.
  • Write multi-stage builds that ship only runtime files.
  • Use cache and secret mounts, and attach SBOM and provenance.

Runtime

  • Predict docker stop behavior from the PID 1 process.
  • Diagnose exit 137 as OOM vs a stop timeout.
  • Wire services with user-defined networks and read the NAT rules.
  • Back up and restore a named volume; fix UID permission errors.

Operations

  • Run a health-gated Compose stack with profiles and watch.
  • Harden a container to non-root, no caps, read-only.
  • Keep logs bounded and watch engine events and metrics.
  • Debug a distroless container and ship by digest.
Checklist

Production readiness checklist

Image

  • Pinned, minimal base; rebuilt regularly
  • Multi-stage; no build tools shipped
  • .dockerignore; cache-friendly order
  • Non-root USER; exec-form CMD
  • No secrets in layers, ARG, or ENV
  • Multi-arch; OCI labels; SBOM + provenance

Runtime

  • Memory, CPU, pids limits
  • Cap-drop ALL, no-new-privileges, read-only
  • Never --privileged or docker.sock
  • User-defined networks; ports bound narrowly
  • Named volumes for state, backed up
  • Healthcheck + graceful SIGTERM

Operations

  • Log rotation (local driver)
  • Metrics, OOM and restart alerts
  • Disk monitoring and prune policy
  • Scan in CI + continuous rescans
  • Signed images deployed by digest
  • Hardened daemon.json, patched host
Glossary

Terms you will meet

OCIOpen Container Initiative — image/runtime/distribution specsNamespaceKernel feature isolating a view of a resource
cgroupKernel feature limiting and accounting resourcesoverlay2 / snapshotterLegacy storage driver / containerd image store backend (Engine 29+ default)
LayerRead-only filesystem diff in an imageDigestsha256 content hash — immutable reference
Manifest / indexImage description / multi-platform listBuildKitBuild engine behind docker build
buildxCLI plugin for BuildKit featuresBakeDeclarative multi-target builds
containerd / runcContainer manager / low-level runtimeShimPer-container process between containerd and the app
CapabilityA slice of root privilegesseccompSystem-call filter
RootlessDaemon + containers as non-root userSBOMSoftware bill of materials
ProvenanceAttestation of how/where an image was builtDistrolessImages without shell or package manager
Further reading

References

  • Docker docs — docs.docker.com
  • Dockerfile reference & build best practices — docs.docker.com/build/
  • Engine security — docs.docker.com/engine/security/
  • Compose specification — compose-spec.io
  • OCI specs — opencontainers.org
  • CIS Docker Benchmark — cisecurity.org
  • Linux man pages: namespaces(7), cgroups(7), capabilities(7)
  • Tools: hadolint, dive, Trivy, Syft, cosign, netshoot

Tool versions, defaults, and licensing terms change — verify against current docs before relying on them.

1 / 1