Linux internals, images, Dockerfiles, BuildKit, runtime, networking, storage, Compose, security, observability, debugging, and production — with best practices for each.
Use ← → / Space to navigate · I contents · O overview · T theme · F fullscreen
Click any line to jump there · press I to return to this page
What a container really is, and the Linux features underneath it.
What a container really is, and what Docker adds.
Everything Docker does is built from kernel features you can use without Docker. Understanding them explains most "weird" container behavior.
A kernel exploit affects every container on the host. For untrusted code use extra isolation (gVisor, Kata, Firecracker microVMs).
docker CLI → talks to the daemon APIdockerd — Docker Engine (API, networking, volumes, builds)containerd — container lifecycle & imagesrunc — creates the process with namespaces/cgroupsdocker buildnerdctldocker rundockerd over /var/run/docker.sockWith live-restore, containers keep running while dockerd restarts or upgrades.
userns-remapdocker inspect fields differ — check with docker info"firewall-backend": "nftables")docker-compose v1 retired — use docker composedocker version --format 'client {{.Client.Version}} / server {{.Server.Version}}'
docker info --format '{{.Driver}} {{json .DriverStatus}}' # overlay2 vs overlayfs snapshotter
Namespaces isolate. Cgroups limit. Overlayfs layers. Capabilities & seccomp restrict.
| Namespace | Isolates | Effect in a container | Docker flag to share with host |
|---|---|---|---|
| pid | Process IDs | Your app sees itself as PID 1 | --pid=host |
| net | Interfaces, routes, ports, iptables | Own eth0, own localhost | --network=host |
| mnt | Mount table | Own root filesystem from the image | (bind mounts) |
| uts | Hostname, domain name | Hostname = short container ID | --uts=host |
| ipc | SysV IPC, POSIX message queues, shm | Own /dev/shm | --ipc=host |
| user | UID/GID mapping | root inside → unprivileged UID outside (if enabled) | --userns=host |
| cgroup | View of the cgroup tree | Sees only its own cgroup | --cgroupns=host |
Containers can also join each other's namespaces: --network container:web, --pid container:web — the basis of Kubernetes pods and debug sidecars.
# Start a container
docker run -d --name web nginx
# Inside: nginx is PID 1
docker exec web cat /proc/1/cmdline; echo
# On the host (Linux): it's a normal process
PID=$(docker inspect -f '{{.State.Pid}}' web)
ps -o pid,user,cmd -p $PID
ls -l /proc/$PID/ns # its namespaces
cat /proc/$PID/cgroup # its cgroup
# Enter its network namespace with host tools
sudo nsenter -t $PID -n ip addr
# Build an isolated shell without Docker
sudo unshare --pid --fork --mount-proc \
--uts --net bash
ps aux # only bash + ps
hostname demo; hostname
/proc/<pid>/ns/* links show its namespacesnsenter lets you use host tools (ip, ss, tcpdump) inside a container's namespace — great for distroless imagesunshare creates namespaces directly: Docker automates this plus images, networking, and limits137--cpus) = throttling; weight (--cpu-shares) = priority under contentiondocker run -d --name api \
--memory 512m --memory-swap 512m \
--cpus 1.5 \
--pids-limit 200 \
myapp:1.4.2
# Live usage
docker stats --no-stream api
# Where the limits live (cgroup v2, systemd driver)
ID=$(docker inspect -f '{{.Id}}' api)
cat /sys/fs/cgroup/system.slice/docker-$ID.scope/memory.max
cat /sys/fs/cgroup/system.slice/docker-$ID.scope/cpu.max
# Was it OOM-killed?
docker inspect -f '{{.State.OOMKilled}}' api
Without limits a container can use all host memory and CPU. Some runtimes (older JVMs, Go before 1.25) size heaps/threads from host resources — use container-aware versions or set them explicitly.
Check the backend with docker info -f '{{.DriverStatus}}'. On legacy overlay2, docker inspect -f '{{json .GraphDriver}}' web shows the dirs; with the containerd image store (overlayfs snapshotter) look at mount | grep overlay. docker diff web works on both.
CHOWN, NET_BIND_SERVICE, SETUID, NET_RAW)SYS_ADMIN, NET_ADMIN, SYS_PTRACE, SYS_MODULE…--cap-drop ALL, add back only what's neededmount, reboot, kexec_load)seccomp=unconfined in production--security-opt seccomp=profile.jsondocker-default AppArmor profile (Ubuntu/Debian):z / :Z--privileged turns all of this offAll capabilities, all devices, no seccomp, no AppArmor. A privileged container is effectively root on the host. Avoid it; grant the specific capability or device instead.
Image layers, Dockerfiles, and fast, reproducible builds with BuildKit.
sha256 digestnginx:1.27) = mutable pointernginx@sha256:…) = immutable content# Platforms in a multi-arch image
docker buildx imagetools inspect nginx:1.27
# Config: entrypoint, env, user, layers
docker image inspect nginx:1.27 \
--format '{{json .Config}}'
# How each layer was created, and its size
docker history --no-trunc nginx:1.27
# Resolve a tag to its digest
docker image inspect nginx:1.27 \
--format '{{index .RepoDigests 0}}'
# Explore layers interactively (third-party)
dive nginx:1.27
# Export an image as a tarball
docker save nginx:1.27 -o nginx.tar
tar -tf nginx.tar | head
registry/namespace/repository:tag@digest
docker.io/library/nginx:1.27
123456789012.dkr.ecr.ap-northeast-1.amazonaws.com/api:1.4.2
ghcr.io/org/tool@sha256:…
docker.io; no tag → latestdocker login, or credential helpers (e.g. docker-credential-ecr-login)latest in production| Instruction | Purpose | Notes |
|---|---|---|
FROM | Base image / start a build stage | Pin version (or digest); AS name for multi-stage |
RUN | Execute a command → new layer | Chain cleanup in the same RUN |
COPY / ADD | Add files from context | Prefer COPY; ADD also fetches URLs/git and auto-extracts local tars |
ENTRYPOINT / CMD | Executable / default args | Use exec form ["bin","arg"]; CMD is overridable |
ENV / ARG | Runtime env / build-time variable | Both visible in image history — never for secrets |
WORKDIR | Set working directory | Use instead of RUN cd … |
USER | Run as this user | Use a non-root UID |
EXPOSE | Document the port | Doesn't publish anything |
HEALTHCHECK | Container health command | Shown in docker ps; used by Compose/Swarm (Kubernetes ignores it) |
LABEL, STOPSIGNAL, VOLUME, SHELL | Metadata, stop signal, mount point, default shell | Avoid VOLUME in base images — it can't be undone |
CMD node server.js runs as /bin/sh -c "node server.js". sh becomes PID 1 and doesn't forward SIGTERM → the app is killed hard after the timeout.
docker stop sends SIGTERM, then SIGKILL after 10 s (-t to change)SIGTERM--init (tini) if the app spawns childrenexec "$@"# BAD: shell form, sh is PID 1
CMD node server.js
# GOOD: exec form, node is PID 1
CMD ["node", "server.js"]
# ENTRYPOINT + CMD: fixed binary, default args
ENTRYPOINT ["/usr/local/bin/api"]
CMD ["--port", "8080"]
# docker run img --port 9090 → overrides CMD only
# Wrapper script that still forwards signals
# docker-entrypoint.sh
#!/bin/sh
set -e
envsubst < /tmpl/app.conf > /etc/app.conf
exec "$@" # replace sh with the app
# Use a real init when processes fork
docker run --init myimage
COPY/ADD also checksum the copied filesRUN is cached by command text only — apt-get update can go stale.dockerignore (.git, node_modules, secrets, build output) — smaller context, fewer cache bustsapt-get update && install && rm -rf /var/lib/apt/lists/* in one RUN--no-cache or --pull to refresh bases periodicallyCOPY package.json package-lock.json ./ # changes rarely → cached
RUN npm ci --omit=dev # reused unless the lockfile changed
COPY . . # changes every commit → only this and later steps rebuild
# ❌ Naive — ~1 GB, root, slow rebuilds
FROM node:latest
COPY . /app
WORKDIR /app
RUN npm install
EXPOSE 3000
CMD npm start
# ✅ Production
# syntax=docker/dockerfile:1
FROM node:22-bookworm-slim AS deps
WORKDIR /app
COPY package.json package-lock.json ./
RUN --mount=type=cache,target=/root/.npm \
npm ci --omit=dev
FROM node:22-bookworm-slim
ENV NODE_ENV=production
WORKDIR /app
COPY --from=deps /app/node_modules ./node_modules
COPY --chown=node:node . .
USER node
EXPOSE 3000
HEALTHCHECK --interval=30s --timeout=3s \
CMD node healthcheck.js || exit 1
CMD ["node", "server.js"]
Changes: pinned slim base · deps layer cached by lockfile · cache mount for npm · no dev deps · non-root · exec-form CMD · healthcheck · plus a .dockerignore.
# syntax=docker/dockerfile:1
FROM golang:1.25 AS build
WORKDIR /src
COPY go.mod go.sum ./
RUN --mount=type=cache,target=/go/pkg/mod \
go mod download
COPY . .
RUN --mount=type=cache,target=/go/pkg/mod \
--mount=type=cache,target=/root/.cache/go-build \
CGO_ENABLED=0 go build -ldflags="-s -w" -o /out/api ./cmd/api
FROM build AS test
RUN go test ./...
FROM gcr.io/distroless/static-debian12:nonroot
COPY --from=build /out/api /api
USER nonroot:nonroot
ENTRYPOINT ["/api"]
FROM starts a stage; only the last one shipsCOPY --from=<stage> takes just the artifactsdocker build --target test . runs tests in CICompiler, sources, and caches stay in the build stage. The final image is a ~10 MB static binary with no shell or package manager — tiny attack surface.
# syntax=docker/dockerfile:1
# Package cache survives between builds,
# never ends up in the image
# Debian/Ubuntu images auto-clean apt; disable it
RUN rm -f /etc/apt/apt.conf.d/docker-clean
RUN --mount=type=cache,target=/var/cache/apt,sharing=locked \
--mount=type=cache,target=/var/lib/apt,sharing=locked \
apt-get update && \
apt-get install -y --no-install-recommends curl
# Secret available only during this RUN
RUN --mount=type=secret,id=npmrc,target=/root/.npmrc \
npm ci
# Forward the host SSH agent for private git
RUN mkdir -p -m 0700 ~/.ssh && \
ssh-keyscan github.com >> ~/.ssh/known_hosts
RUN --mount=type=ssh \
git clone git@github.com:org/private-lib.git
# Pass the secret and SSH agent at build time
docker build \
--secret id=npmrc,src=$HOME/.npmrc \
--ssh default \
-t api:dev .
# ❌ Never do this — stays in history/layers
ARG NPM_TOKEN
ENV NPM_TOKEN=$NPM_TOKEN
COPY .npmrc /root/.npmrc
# Check what leaked
docker history --no-trunc api:dev
Rule: secrets go in --mount=type=secret at build time and in runtime secret stores at run time — never in ARG, ENV, COPY, or a layer you "delete" later.
# A builder that supports multi-platform
docker buildx create --name ci --driver docker-container --use
# Build amd64 + arm64 and push one multi-arch tag
docker buildx build \
--platform linux/amd64,linux/arm64 \
-t ghcr.io/org/api:1.4.2 \
--cache-from type=registry,ref=ghcr.io/org/api:buildcache \
--cache-to type=registry,ref=ghcr.io/org/api:buildcache,mode=max \
--sbom=true --provenance=mode=max \
--push .
# In GitHub Actions
# --cache-from type=gha --cache-to type=gha,mode=max
# Many targets declared in docker-bake.hcl
docker buildx bake --push
$BUILDPLATFORM / $TARGETARCH args instead of emulationmode=max caches intermediate stages tooexec format error at runtime = wrong architecture| Base | Size | Pros | Cons |
|---|---|---|---|
| debian / ubuntu | ~30–80 MB | glibc, familiar tooling, broad compatibility | More packages → more CVEs |
| *-slim variants | ~25–75 MB | Good default for interpreted languages | Still has shell + package manager |
| alpine | ~5 MB | Tiny, fast pulls | musl libc: DNS and binary-compatibility surprises, slower Python wheels |
| distroless | ~2–25 MB | No shell or package manager, non-root variants | Harder debugging (use :debug tag or ephemeral debug) |
| scratch | 0 MB | Only your static binary | Add CA certs, tzdata, and a user yourself |
| Hardened / minimal images (e.g. Chainguard, Docker Hardened Images) | small | Near-zero CVEs, frequent rebuilds, SBOMs | Vendor ecosystem, versioning policies differ |
Pin a specific version (python:3.13-slim-bookworm), rebuild regularly to pick up patches, and let Renovate/Dependabot bump base tags and digests.
# syntax=docker/dockerfile:1
FROM debian:bookworm-slim
# Heredocs: readable multi-line RUN, one layer
RUN <<EOF
set -eux
apt-get update
apt-get install -y --no-install-recommends ca-certificates
rm -rf /var/lib/apt/lists/*
EOF
# Write a file inline
COPY <<EOF /etc/app/config.yaml
log_level: info
EOF
# Layer independent of the base → rebase without rebuild
COPY --link --chown=10001:10001 dist/ /app/
# Generate Dockerfile, compose.yaml, .dockerignore,
# README.Docker.md (ships with Docker Desktop)
docker init
# Lint with build checks (Buildx 0.15+)
docker build --check .
# Reproducible timestamps (file mtimes too: add
# --output type=image,rewrite-timestamp=true)
docker buildx build \
--build-arg SOURCE_DATE_EPOCH=$(git log -1 --format=%ct) .
# .dockerignore — keep the context small and safe
.git
**/node_modules
**/*.log
.env*
!.env.example
dist/
coverage/
Keep # syntax=docker/dockerfile:1 at the top to always get the latest stable Dockerfile frontend features.
The runtime, networking, storage, and multi-container apps with Compose.
docker create
docker start / run
pause / unpause (cgroup freezer)
stop (TERM→KILL) / kill / process ends
docker rm — writable layer gone
no (default)on-failure[:N] — only on non-zero exitalways — also after daemon restartunless-stopped — like always, unless you stopped it0 success · 1 app error125 docker run failed · 126 not executable · 127 not found137 = 128+9 SIGKILL (OOM or kill) · 143 = 128+15 SIGTERM139 = segfaultdocker run -d \
--name api \
--restart unless-stopped \
-p 127.0.0.1:8080:8080 \
-e LOG_LEVEL=info \
--env-file ./api.env \
-v api-data:/var/lib/api \
--network backend \
--memory 512m --cpus 1 --pids-limit 200 \
--read-only --tmpfs /tmp \
--user 10001:10001 \
--cap-drop ALL \
--security-opt no-new-privileges \
--init \
--health-cmd 'wget -qO- localhost:8080/healthz || exit 1' \
--health-interval 30s \
--log-opt max-size=10m --log-opt max-file=3 \
ghcr.io/org/api:1.4.2
-d detached · --rm remove on exit (for one-offs)-p host:container publish; bind to 127.0.0.1 unless it must be public-e/--env-file config — not for secrets in production-v named volume for data--init for signal forwarding & zombie reaping--interval (default 30 s)--retries failures (default 3) → unhealthy--start-period gives slow apps time to boot (--start-interval = check rate during it)depends_on; orchestrators with their own health checks (ECS, Kubernetes probes) replace them/healthz), short timeoutHEALTHCHECK --interval=15s --timeout=3s \
--start-period=30s --retries=3 \
CMD ["/app/healthcheck"]
# Watch health status
docker ps --format 'table {{.Names}}\t{{.Status}}'
docker inspect -f '{{json .State.Health}}' api | jq
# Compose: start only when the DB is healthy
services:
api:
depends_on:
db:
condition: service_healthy
db:
image: postgres:17
healthcheck:
test: ["CMD-SHELL", "pg_isready -U postgres"]
interval: 5s
retries: 10
| Driver | How it works | Use for |
|---|---|---|
| bridge (default) | Linux bridge + veth pairs + NAT via iptables/nftables | Single-host apps |
| user-defined bridge | Same, plus embedded DNS (127.0.0.11) and isolation per network | Always instead of the default bridge |
| host | No network namespace — shares the host's stack | Max performance, many ports; no isolation |
| none | Only loopback | Batch jobs that need no network |
| overlay | VXLAN across hosts (Swarm) | Multi-host Swarm services |
| macvlan / ipvlan | Container gets an address on the physical LAN | Legacy apps expecting LAN IPs |
No DNS by container name, all containers can talk to each other. Create your own networks.
frontend and backend networks; only the API joins both; DB lives only on backend.
-p 8080:80DOCKER chain → container IP:80127.0.0.1, or put rules in the DOCKER-USER chain (iptables backend; Engine 29 adds an experimental nftables backend)# User-defined network with DNS
docker network create backend
docker run -d --name db --network backend postgres:17
docker run --rm --network backend busybox \
nslookup db # resolved by 127.0.0.11
# Network aliases (round-robin DNS)
docker run -d --network backend --network-alias search \
-e discovery.type=single-node elasticsearch:8.15.0
# Inspect
docker network inspect backend
docker port api
sudo iptables -t nat -L DOCKER -n
# Debug with a toolbox in the same netns
docker run --rm -it --network container:api \
nicolaka/netshoot
# → curl, dig, ss, tcpdump, iperf …
| Type | Lives in | Managed by | Use for |
|---|---|---|---|
| Named volume | /var/lib/docker/volumes/… | Docker | Databases, persistent app data — the production default |
| Bind mount | Any host path | You | Dev source code, config files; ties container to host layout |
| tmpfs | Host memory | Kernel | Scratch data, secrets at runtime, read-only root with writable /tmp |
# --mount is explicit; -v is shorter
docker run -d --name db \
--mount type=volume,src=pgdata,dst=/var/lib/postgresql/data \
postgres:17
docker run --rm -it \
--mount type=bind,src="$PWD",dst=/app,readonly \
node:22-slim bash
docker run --read-only \
--mount type=tmpfs,dst=/tmp,tmpfs-size=64m app
# Back up a volume to a tarball
docker run --rm -v pgdata:/data:ro -v "$PWD":/backup \
busybox tar czf /backup/pgdata.tgz -C /data .
:ro)docker volume prune deletes data — check first/, /etc, or docker.sockMulti-container apps as one declarative file.
name: shop
services:
api:
build: { context: ., target: runtime }
image: ghcr.io/org/api:${TAG:-dev}
ports: ["127.0.0.1:8080:8080"]
env_file: .env
secrets: [db_password]
depends_on:
db: { condition: service_healthy }
networks: [frontend, backend]
deploy:
resources:
limits: { cpus: "1", memory: 512M }
restart: unless-stopped
db:
image: postgres:17
environment:
POSTGRES_PASSWORD_FILE: /run/secrets/db_password
secrets: [db_password]
volumes: [pgdata:/var/lib/postgresql/data]
networks: [backend]
healthcheck:
test: ["CMD-SHELL", "pg_isready -U postgres"]
interval: 5s
retries: 10
networks:
frontend: {}
backend: { internal: true } # no internet
volumes:
pgdata: {}
secrets:
db_password: { file: ./secrets/db_password.txt }
The top-level version: key is obsolete. Use docker compose (v2 plugin), not the retired docker-compose v1.
docker compose up -d --build # build + start
docker compose ps
docker compose logs -f api
docker compose exec api sh
docker compose run --rm api npm test
docker compose config # fully resolved file
docker compose down # keep volumes
docker compose down -v # ⚠ deletes volumes
# Layer files per environment
docker compose -f compose.yaml -f compose.prod.yaml up -d
# Optional services
docker compose --profile debug up -d
# Live sync code into containers
docker compose watch
compose.override.yaml is merged automatically (unless you pass -f) — dev-only mounts and portsdevelop.watch: sync, rebuild, sync+restart actionsprofiles for optional tools (adminer, mailhog).env file for variable interpolationCompose is single-host: no rescheduling on node failure, no rolling updates across hosts. Fine for small services and dev/CI; use ECS or Kubernetes for HA.
Security, observability, debugging, and production practice.
Image, build, runtime, daemon, host.
--privileged, host mountsdocker.sock or TCP 2375docker.sock = root on the hostdocker run -v /:/host --privileged … and own the machinedocker group is equivalent to rootUSER 10001 in the image--cap-drop ALL, add back only what's needed--security-opt no-new-privileges (blocks setuid escalation)--read-only root filesystem + tmpfs for scratch--privileged, no host namespaces, no sensitive host mounts--runtime=runsc) or Kata# Hardened run
docker run -d --name api \
--user 10001:10001 \
--cap-drop ALL \
--security-opt no-new-privileges \
--read-only --tmpfs /tmp:rw,noexec,nosuid,size=64m \
--memory 512m --cpus 1 --pids-limit 200 \
-p 127.0.0.1:8080:8080 \
ghcr.io/org/api@sha256:<digest>
# Needs to bind :80 as non-root?
# listen on 8080 and publish 80:8080 instead
# Verify
docker inspect api --format '{{.Config.User}}'
docker inspect api --format \
'{{.HostConfig.CapDrop}} {{.HostConfig.ReadonlyRootfs}}'
docker exec api id
dockerd-rootless-setuptool.sh installdaemon.json: "userns-remap": "default"--userns=hostRecommendation: non-root images everywhere; rootless Docker on developer and CI machines where possible; userns-remap or rootless on shared hosts.
# Vulnerability scanning
docker scout quickview ghcr.io/org/api:1.4.2
docker scout cves --only-severity critical,high \
ghcr.io/org/api:1.4.2
trivy image --severity HIGH,CRITICAL --exit-code 1 \
ghcr.io/org/api:1.4.2
# SBOM
docker buildx imagetools inspect ghcr.io/org/api:1.4.2 \
--format '{{json .SBOM}}'
syft ghcr.io/org/api:1.4.2 -o spdx-json > sbom.json
# Sign and verify (keyless, Sigstore)
cosign sign ghcr.io/org/api@sha256:<digest>
cosign verify ghcr.io/org/api@sha256:<digest> \
--certificate-identity-regexp 'https://github.com/org/.*' \
--certificate-oidc-issuer \
https://token.actions.githubusercontent.com
docker-bench-security / CIS Docker Benchmark on hostsdocker inspect, /proc, crash dumps, child processessecrets mounted at /run/secrets/…*_FILE variables in your app (many official images do)# /etc/docker/daemon.json — hardened baseline
{
"live-restore": true,
"no-new-privileges": true,
"icc": false, # default bridge only
"userland-proxy": false,
"log-driver": "local",
"log-opts": { "max-size": "10m", "max-file": "3" },
"default-ulimits": {
"nofile": { "Name": "nofile", "Soft": 65536, "Hard": 65536 }
}
}
# Never plain TCP 2375 (Engine 27+ refuses it without TLS)
# Remote access → SSH (DOCKER_HOST=ssh://…) or mTLS
Logs, metrics, events, disk, daemon.
| Driver | Notes |
|---|---|
json-file | Default; no rotation unless configured → full disks |
local | Compressed, rotated by default — recommended for single hosts |
journald, syslog | Into the host's log system |
awslogs, gcplogs | Straight to cloud logging |
fluentd, gelf, splunk | Central pipelines |
daemon.jsonmode=non-blocking with a buffer# Read logs
docker logs -f --since 10m --timestamps api
docker logs --tail 100 api 2>&1 | grep -i error
# Where json-file logs live
docker inspect -f '{{.LogPath}}' api
# Per-container driver
docker run -d \
--log-driver awslogs \
--log-opt awslogs-region=ap-northeast-1 \
--log-opt awslogs-group=/app/api \
--log-opt mode=non-blocking \
--log-opt max-buffer-size=4m \
api:1.4.2
# Change the default (restart dockerd;
# affects only new containers)
{ "log-driver": "local",
"log-opts": { "max-size": "10m", "max-file": "5" } }
# Live resource usage
docker stats
docker stats --no-stream --format \
'table {{.Name}}\t{{.CPUPerc}}\t{{.MemUsage}}\t{{.PIDs}}'
# Real-time daemon events: start, die, oom, health
docker events --filter type=container \
--filter event=die --filter event=oom
# Engine metrics for Prometheus (daemon.json)
{ "metrics-addr": "127.0.0.1:9323" }
curl -s localhost:9323/metrics | head
# Per-container metrics: cAdvisor — a trusted monitoring
# agent, the deliberate exception to "no host mounts"
docker run -d --name cadvisor -p 127.0.0.1:8081:8080 \
--privileged --device=/dev/kmsg \
-v /:/rootfs:ro -v /var/run:/var/run:ro \
-v /sys:/sys:ro -v /var/lib/docker:/var/lib/docker:ro \
ghcr.io/google/cadvisor:<version>
/var/lib/dockercAdvisor + node-exporter → Prometheus → Grafana + Alertmanager, or CloudWatch agent / Datadog / OpenTelemetry Collector on the host.
# What uses space?
docker system df
docker system df -v
# Safe-ish cleanup
docker container prune # stopped containers
docker image prune # dangling images
docker builder prune --keep-storage 20gb
# Aggressive — read before running
docker image prune -a --filter "until=168h"
docker system prune -a # all unused images too
docker volume prune # ⚠ unused volumes = data
# Find big log files
sudo sh -c 'du -sh /var/lib/docker/containers/*/*-json.log' \
| sort -h | tail
df -i/var/lib/docker on its own volume (data-root)# Why did it stop?
docker ps -a --filter name=api
docker inspect api --format \
'{{.State.Status}} exit={{.State.ExitCode}}'
docker inspect api --format \
'oom={{.State.OOMKilled}} err={{.State.Error}}'
docker logs --tail 200 api
# Get inside
docker exec -it api sh
docker exec -u 0 -it api sh # as root
# Crashes on start? Override the entrypoint (needs a shell)
docker run --rm -it --entrypoint sh api:1.4.2
# Distroless: share namespaces with a toolbox
docker run --rm -it --pid container:api \
--network container:api --cap-add SYS_PTRACE \
nicolaka/netshoot
ls /proc/1/root/app # target's filesystem
# What changed in the filesystem?
docker diff api
# Copy files in and out
docker cp api:/app/logs/error.log .
# Full config: mounts, env, network, limits
docker inspect api | jq '.[0].HostConfig'
# Processes inside, seen from the host
docker top api
# Debug a build step by step
docker build --progress=plain --no-cache .
docker build --target deps -t dbg . && \
docker run --rm -it dbg sh
# Daemon side
journalctl -u docker --since "1 hour ago"
docker info
| Symptom | Likely cause | Fix |
|---|---|---|
Exit 137 | OOM kill or SIGKILL after stop timeout | Check OOMKilled; raise limit or fix leak; handle SIGTERM |
exec format error | Image architecture ≠ host (arm64 vs amd64) | Build multi-arch; --platform |
permission denied on volume | Container UID ≠ volume owner | chown in image, matching UID, :z on SELinux |
| Container exits immediately | Main process ended (daemonized app, wrong CMD) | Run in foreground; check docker logs |
| Can't reach app on published port | App listens on 127.0.0.1 inside container | Listen on 0.0.0.0 |
| Containers can't resolve each other | Using the default bridge | User-defined network |
port is already allocated | Host port in use | ss -ltnp; pick another port |
| Slow stop (10 s) | Shell-form CMD, SIGTERM ignored | Exec form, --init, handle signals |
| Slow builds | Cache busted early, large context | Reorder, .dockerignore, cache mounts |
toomanyrequests on pull | Docker Hub rate limit | docker login, mirror / pull-through cache |
hadolint on Dockerfile
buildx, registry/gha cache, multi-arch
--target test, Compose integration tests
Trivy / Scout, fail on criticals
Immutable tag, SBOM, provenance, cosign
By digest to ECS / Kubernetes
api:1.4.2 and api:git-3f2c1aborg.opencontainers.image.source, .revision, .version| Docker concept | Kubernetes | Amazon ECS |
|---|---|---|
docker run / container | Pod (1+ containers) | Task |
| Compose service | Deployment + Service | Service |
-p published port | Service / Ingress | ALB target group |
| Named volume | PersistentVolumeClaim | EBS / EFS volume |
--memory, --cpus | resources.requests / limits | Task / container CPU & memory |
HEALTHCHECK | liveness / readiness / startup probes | Container health check |
| Compose secrets | Secret / External Secrets | Secrets Manager / SSM in task definition |
--restart | Controllers recreate pods | Service scheduler |
Kubernetes doesn't use Docker Engine as its runtime anymore (dockershim removed in 1.24), but it runs the same OCI images you build with Docker.
# Change limits on a running container
docker update --memory 1g --memory-swap 1g \
--cpus 2 api
# Talk to a remote engine over SSH
docker context create prod \
--docker host=ssh://ops@10.0.1.20
docker --context prod ps
# GPUs (needs NVIDIA Container Toolkit on the host)
docker run --rm --gpus all \
nvidia/cuda:12.6.0-base-ubuntu24.04 nvidia-smi
# Compose: wait until healthy, reuse other files
docker compose up -d --wait
# compose.yaml → include: [../shared/observability.yaml]
# Swarm: built-in multi-host orchestration
docker swarm init
docker service create --name web --replicas 3 \
--update-parallelism 1 --update-delay 10s nginx:1.27
docker update applies cgroup limits live — no restart--wait makes CI scripts block until services are healthydocker debug attaches a toolbox shell to slim containers (free for all users since Desktop 4.49)stub
A Linux Docker host, a scratch directory, and the habit of cleaning up after every lab.
docker version and docker info should both answer.trivy, jq, hadolint./var/lib/docker, nsenter, iptables) must run inside that VM or on a Linux VM.Work in ~/docker-labs/<lab>. Check what is left with docker ps -a, docker volume ls, and docker system df.
Goal: see that an image is a config plus ordered layers, and find them on disk.
docker pull nginx:1.27
docker image inspect nginx:1.27 --format '{{.Architecture}} {{.Os}}'
docker image inspect nginx:1.27 --format '{{json .Config.Cmd}}'
docker image inspect nginx:1.27 \
--format '{{json .RootFS.Layers}}' | jq length
docker history --no-trunc nginx:1.27 | head # one row per instruction
# Index vs manifest: one tag, several platforms
docker buildx imagetools inspect nginx:1.27
# Pull by digest: immutable reference
DIGEST=$(docker buildx imagetools inspect nginx:1.27 \
--format '{{json .Manifest}}' | jq -r .digest)
docker pull nginx@$DIGEST
# overlay2 hosts only (Linux): the container's merged view
docker run -d --name l1 nginx:1.27
docker inspect l1 --format '{{json .GraphDriver.Data}}' | jqimagetools inspect lists amd64, arm64, and more under one index digest.history rows with 0B are metadata-only instructions (ENV, CMD).docker exec l1 touch /x; find it under UpperDir.docker info shows which store you use.docker rm -f l1Goal: measure the effect of instruction order, .dockerignore, and a slim base.
mkdir l2 && cd l2 && npm init -y >/dev/null && npm i express
echo "require('express')().get('/',(q,r)=>r.send('ok'))
.listen(3000)" > server.js
cat > Dockerfile.naive <<'EOF'
FROM node:22
COPY . /app
WORKDIR /app
RUN npm install
CMD npm start
EOF
docker build -f Dockerfile.naive -t l2:naive .
echo // >> server.js
time docker build -f Dockerfile.naive -t l2:naive .
printf 'node_modules\n.git\nDockerfile*\n' > .dockerignore
# Now write Dockerfile.good: node:22-bookworm-slim, COPY package*.json,
# RUN npm ci --omit=dev, COPY . ., USER node, CMD ["node","server.js"]
docker build -f Dockerfile.good -t l2:good .
docker images l2server.js (BuildKit ignores mtime, so touch is not enough), the naive build re-runs npm install; the good one reuses the cached npm ci layer.docker run --rm l2:good id prints a non-root user.COPY . . above npm ci and watch the cache miss.docker rmi l2:naive l2:goodGoal: compile in a fat stage and ship only the binary.
mkdir l3 && cd l3
cat > main.go <<'EOF'
package main
import ("fmt"; "net/http")
func main() {
http.HandleFunc("/", func(w http.ResponseWriter,
r *http.Request) { fmt.Fprintln(w, "hi") })
http.ListenAndServe(":8080", nil)
}
EOF
cat > Dockerfile <<'EOF'
# syntax=docker/dockerfile:1
FROM golang:1.25 AS build
WORKDIR /src
COPY main.go .
RUN go mod init l3 && CGO_ENABLED=0 go build -o /app .
FROM gcr.io/distroless/static-debian12:nonroot
COPY --from=build /app /app
ENTRYPOINT ["/app"]
EOF
docker build -t l3 . && docker images l3golang:1.25 is hundreds of MB.docker run --rm -p 8080:8080 l3 then curl localhost:8080.docker run --rm --entrypoint sh l3 fails: there is no shell. That is the point.--target build and compare.CGO_ENABLED=0 and see the binary fail on a static base.Goal: speed up dependency installs, use a secret without leaking it, and attach an SBOM.
# syntax=docker/dockerfile:1
FROM python:3.13-slim
RUN --mount=type=cache,target=/root/.cache/pip \
pip install requests
RUN --mount=type=secret,id=token \
test -s /run/secrets/token && echo "secret present"
# build with the secret from a file (never --build-arg)
echo "s3cr3t" > token.txt
docker build --secret id=token,src=token.txt -t l4 .
docker history l4 | grep -c s3cr3t # 0
# SBOM + provenance: docker-container driver (or containerd store)
docker buildx create --name lab --driver docker-container --use
docker buildx build --sbom=true --provenance=mode=max \
-t <registry>/l4:1 --push .
docker buildx imagetools inspect <registry>/l4:1 \
--format '{{json .SBOM}}' | jq '.SPDX.packages | length'pip install line and rebuild: cached wheels come from the cache mount.history or in any layer.--build-arg and find it in docker history.--platform linux/amd64,linux/arm64; .SBOM output is then keyed by platform.docker buildx rm lab, rm token.txtGoal: see why exec form and --init matter, and what an OOM kill looks like.
# No init: PID 1 has no TERM handler, so TERM is ignored → 10 s, then KILL
docker run -d --name l5a alpine sh -c 'sleep 600'
time docker stop l5a # ~10 s
docker run -d --name l5b --init alpine sleep 600
time docker stop l5b # ~instant: tini forwards TERM
# Restart policy and restart count
docker run -d --name l5c --restart on-failure:3 alpine sh -c 'exit 1'
sleep 20; docker inspect l5c \
--format '{{.RestartCount}} {{.State.Status}}'
# Memory limit → OOM kill → exit 137
docker run --name l5d --memory 64m --memory-swap 64m alpine \
sh -c 'tail /dev/zero'
docker inspect l5d \
--format 'exit={{.State.ExitCode}} oom={{.State.OOMKilled}}'
docker stats --no-stream
docker rm -f l5a l5b l5c l5dl5a takes the full stop timeout; l5b stops at once.l5c gives up after 3 restarts and stays exited.l5d shows exit=137 oom=true.--pids-limit 20 with a fork loop and watch it hit the limit.docker update --cpus 0.5 <name>.Goal: prove why user-defined bridges beat the default bridge, and see the NAT rules.
docker network create app
docker run -d --name db --network app redis:7
docker run --rm --network app alpine nslookup db # via 127.0.0.11
docker run --rm alpine ping -c1 db # fails: default bridge
# Isolation between networks
docker network create other
docker run --rm --network other alpine ping -c1 -W1 db # fails
# Publish safely and inspect the NAT rule (Linux host)
docker run -d --name web --network app -p 127.0.0.1:8080:80 nginx:1.27
sudo iptables -t nat -L DOCKER -n | grep 8080
sudo iptables -L DOCKER-USER -n # your own rules go here
# none and host
docker run --rm --network none alpine ip addr # only lo
docker run --rm --network host alpine ip addr # host interfacesnslookup db returns the container IP on app.-p 8080:80 (no IP) and note it is reachable from other machines, bypassing typical host firewall rules.sudo nft list ruleset; a native nftables backend is experimental from Engine 29.docker rm -f db web && docker network rm app otherGoal: choose the right mount, back up a volume, and fix a permission error.
docker volume create pgdata
docker run -d --name pg -e POSTGRES_PASSWORD=pw \
-v pgdata:/var/lib/postgresql/data postgres:17
sleep 5; docker exec pg psql -U postgres \
-c 'create table t(x int); insert into t values (1);'
docker rm -f pg # data survives in the volume
# Back up and restore a volume with a throwaway container
docker run --rm -v pgdata:/data:ro -v "$PWD":/backup alpine \
tar czf /backup/pgdata.tgz -C /data .
docker volume create pgdata2
docker run --rm -v pgdata2:/data -v "$PWD":/backup alpine \
tar xzf /backup/pgdata.tgz -C /data
# Bind mount + non-root user → permission problem
mkdir out
docker run --rm -u 10001 -v "$PWD/out":/out alpine touch /out/f
# tmpfs: in-memory scratch, gone on stop
docker run --rm --tmpfs /scratch:size=16m alpine df -h /scratchpostgres:17 on pgdata2 and select * from t returns 1.touch fails: out/ is owned by your host UID, not 10001.sudo chown 10001 out or by running with -u $(id -u).:ro and try to edit it inside.docker volume rm pgdata pgdata2Goal: run web + api + db where the api waits for a healthy database.
# compose.yaml
services:
db:
image: postgres:17
environment: { POSTGRES_PASSWORD: pw }
volumes: [dbdata:/var/lib/postgresql/data]
healthcheck:
test: ["CMD-SHELL", "pg_isready -U postgres"]
interval: 5s
retries: 10
api:
build: ./api
depends_on: { db: { condition: service_healthy } }
environment: { DATABASE_URL: postgres://postgres:pw@db:5432/postgres }
web:
image: nginx:1.27
ports: ["127.0.0.1:8080:80"]
depends_on: [api]
volumes: { dbdata: {} }docker compose up -d --wait: returns when services are healthy or running.docker compose ps shows healthy for db.docker compose logs -f api to follow one service.docker compose config prints the fully resolved file.Goal: a fast inner dev loop and optional tooling that only starts on demand.
# add to compose.yaml
adminer:
image: adminer
profiles: [debug]
ports: ["127.0.0.1:8081:8080"]
api:
develop:
watch:
- action: sync
path: ./api/src
target: /app/src
- action: rebuild
path: ./api/package.json
docker compose --profile debug up -d # adminer starts only now
docker compose up --watch # edit api/src and see it sync
docker compose down # keeps the dbdata volume
docker compose down -v # also removes volumes--profile debug, adminer never starts.api/src syncs without a rebuild; editing package.json rebuilds.pg_isready -p 5433); db turns unhealthy and api never starts.docker compose run --rm api npm test.Goal: apply least privilege at runtime and find vulnerabilities before shipping.
# Default vs hardened
docker run --rm alpine sh -c 'id; grep Cap /proc/self/status'
docker run --rm -u 10001 --cap-drop ALL \
--security-opt no-new-privileges --read-only \
--tmpfs /tmp --pids-limit 100 alpine \
sh -c 'id; grep CapEff /proc/self/status; touch /x'
# Which capability does the app really need?
docker run --rm --cap-drop ALL nginx:1.27 # fails to start
docker run --rm --cap-drop ALL --cap-add CHOWN --cap-add SETUID \
--cap-add SETGID --cap-add NET_BIND_SERVICE nginx:1.27
# Scan
docker scout quickview nginx:1.27 # needs docker login
docker scout cves --only-severity critical,high nginx:1.27
trivy image --severity HIGH,CRITICAL --exit-code 1 nginx:1.27CapEff: 0000000000000000, and touch /x fails (read-only).--network host (Docker sets ip_unprivileged_port_start=0 otherwise). nginxinc/nginx-unprivileged needs none.nginx:1.27 vs an -alpine tag.docker info --format "{{.SecurityOptions}}": seccomp and AppArmor/SELinux status.Goal: keep logs from filling the disk and watch the engine report what happens.
# Unrotated json-file grows forever; cap it per container
docker run -d --name chatty --log-driver json-file \
--log-opt max-size=1m --log-opt max-file=3 \
alpine sh -c 'while true; do date; done'
ID=$(docker inspect -f '{{.Id}}' chatty)
sleep 20; sudo ls -lh /var/lib/docker/containers/$ID/
# Engine-wide default: /etc/docker/daemon.json
# { "log-driver": "local",
# "log-opts": { "max-size": "10m", "max-file": "3" },
# "metrics-addr": "127.0.0.1:9323" }
sudo systemctl restart docker
curl -s localhost:9323/metrics | grep engine_daemon_container_states
# Events stream in another terminal
docker events --filter type=container \
--filter event=die --filter event=oom
docker system df -vchatty.oom then die in the events stream.docker rm -f chattyGoal: find the cause from state, logs, and the process namespace, even without a shell.
docker run -d --name bad --restart on-failure:5 alpine \
sh -c 'echo "FATAL: DATABASE_URL not set" >&2; exit 3'
docker ps -a --filter name=bad
docker inspect bad --format \
'{{.State.ExitCode}} oom={{.State.OOMKilled}} n={{.RestartCount}}'
docker logs --tail 50 bad
# Distroless or no shell: run tools from outside
docker run -d --name ok -p 8081:8080 l3
PID=$(docker inspect -f '{{.State.Pid}}' ok)
sudo nsenter -t $PID -n ss -lntp # listening sockets
docker run --rm --network container:ok nicolaka/netshoot \
curl -s localhost:8080
docker cp ok:/app ./app-copy # copy files out
# Wrong architecture?
docker image inspect l3 --format '{{.Architecture}}'--network container:ok shares the network namespace with a toolbox image.docker debug <container> for a toolbox shell (free for all users since Desktop 4.49).ENTRYPOINT ["/app2"] and read the error in docker inspect .State.Error.docker rm -f bad okGoal: produce an image you can trace, reproduce, and roll back.
hadolint Dockerfile # lint before building
docker buildx create --use --driver docker-container
REG=ghcr.io/<you>; SHA=$(git rev-parse --short HEAD)
docker buildx build --platform linux/amd64,linux/arm64 \
--label org.opencontainers.image.revision=$SHA \
--sbom=true --provenance=mode=max \
-t $REG/api:1.0.0 -t $REG/api:sha-$SHA --push .
# Deploy by digest, not by tag
docker buildx imagetools inspect $REG/api:1.0.0 | grep Digest
docker run -d $REG/api@sha256:<digest>
# Optional: sign with cosign (keyless in CI)
cosign sign $REG/api@sha256:<digest>
cosign verify $REG/api@sha256:<digest> \
--certificate-identity-regexp '.*' \
--certificate-oidc-issuer-regexp '.*'1.0.0 to a new build and note that the digest-pinned container is unaffected.docker buildx imagetools inspect --format "{{json .Provenance}}" shows the build source.hadolint warning before you ship.One app, every practice from the deck. Treat it as a portfolio piece.
USER.--secret only..dockerignore, pinned base images, OCI labels.service_healthy gating.--read-only, --cap-drop ALL, no-new-privileges, memory, CPU and PID limits.local log driver with rotation; metrics scraped by Prometheus.oom events and restart loops.Tick each item only when you can do it without notes.
docker stop behavior from the PID 1 process..dockerignore; cache-friendly orderUSER; exec-form CMD--privileged or docker.socklocal driver)daemon.json, patched host| OCI | Open Container Initiative — image/runtime/distribution specs | Namespace | Kernel feature isolating a view of a resource |
| cgroup | Kernel feature limiting and accounting resources | overlay2 / snapshotter | Legacy storage driver / containerd image store backend (Engine 29+ default) |
| Layer | Read-only filesystem diff in an image | Digest | sha256 content hash — immutable reference |
| Manifest / index | Image description / multi-platform list | BuildKit | Build engine behind docker build |
| buildx | CLI plugin for BuildKit features | Bake | Declarative multi-target builds |
| containerd / runc | Container manager / low-level runtime | Shim | Per-container process between containerd and the app |
| Capability | A slice of root privileges | seccomp | System-call filter |
| Rootless | Daemon + containers as non-root user | SBOM | Software bill of materials |
| Provenance | Attestation of how/where an image was built | Distroless | Images without shell or package manager |
docs.docker.comdocs.docker.com/build/docs.docker.com/engine/security/compose-spec.ioopencontainers.orgcisecurity.orgnamespaces(7), cgroups(7), capabilities(7)Tool versions, defaults, and licensing terms change — verify against current docs before relying on them.