Cloud-native engineering deck

Kubernetes
from Zero to Hero

Architecture, workloads, configuration, networking, storage, scheduling, scaling, security, observability, delivery, extending, operations, and troubleshooting — with best practices and runnable examples.

Use ← → / Space to navigate · I contents · O overview · T theme · F fullscreen

Learning path

Zero → hero in four levels

Level 1 · Zero

  • Containers & Docker basics
  • Linux, networking, YAML
  • Local cluster (kind / minikube)
  • Pods, Deployments, Services
  • kubectl fluency

Level 2 · Practitioner

  • ConfigMaps, Secrets, probes
  • Requests/limits, QoS
  • Ingress / Gateway API
  • PV/PVC, StatefulSets
  • Helm, Kustomize
  • Target: CKAD

Level 3 · Operator

  • Architecture internals
  • Scheduling, autoscaling
  • RBAC, NetworkPolicy, PSA
  • Upgrades, etcd backup
  • Monitoring & troubleshooting
  • Target: CKA

Level 4 · Hero

  • Operators & CRDs
  • GitOps at scale
  • Supply-chain & runtime security
  • Multi-cluster, platform engineering
  • SLOs, cost, reliability
  • Target: CKS
Kubernetes · Zero to Hero

Contents

Click any line to jump there · press I to return to this page

Chapter 1

Foundations

Why Kubernetes exists, how it is built, and your first cluster.

Chapter 1 · Part 1.1

Why Kubernetes

The problem it solves and the idea that makes it work.

The problem

Running containers at scale is hard

Without an orchestrator, you must…

  • Pick a host for every container and track free capacity
  • Restart crashed containers and replace dead hosts
  • Wire service discovery and load balancing by hand
  • Roll out new versions without downtime, roll back on failure
  • Scale up and down with traffic
  • Distribute config and secrets safely

Kubernetes gives you

  • Scheduling onto a pool of machines
  • Self-healing: restart, reschedule, replace
  • Service discovery, DNS, load balancing
  • Rolling updates and rollbacks
  • Horizontal autoscaling
  • Config, secrets, storage orchestration
  • One extensible API for everything

What it is not

Not a PaaS: it doesn't build your code, provide databases, or pick your logging/monitoring stack. It's a platform for building platforms — and it adds real operational complexity. A few containers may not need it.

The core idea

Declarative desired state + control loops

You declare

"I want 3 replicas of web:1.4" — YAML sent to the API server

→

Stored

Object saved in etcd: spec = desired state

→

Observe

Controllers watch objects and the real world

→

Diff & act

Create / delete / update to close the gap

→

Report

Write actual state to status; repeat forever

Why this matters

  • Self-healing is the default: a deleted pod is simply recreated
  • Same file applied twice = same result (idempotent)
  • Everything can live in Git → reviewable, reproducible

Mental model

  • Don't think "run this command"
  • Think "this is how the world should look"
  • When something is wrong, ask: which controller should fix this, and why isn't it?
Chapter 1 · Part 1.2

Architecture

Components

Control plane and nodes

Control plane kube-apiserver the only door to etcd etcd key-value state scheduler picks nodes controller-manager Deployment, Node, Job… cloud-controller-mgr LBs, routes, nodes watch / update over HTTPS Worker node 1 kubelet runs pods kube-proxy Service rules runtime (containerd / CRI-O) · CNI · CSI Pod Pod Pod Worker node 2 kubelet runs pods kube-proxy Service rules container runtime · CNI · CSI Pod Pod

Control plane

  • apiserver: REST API, authN/authZ, admission, validation
  • etcd: consistent store for all objects (Raft, odd member count)
  • scheduler: assigns pods to nodes
  • controller-manager: built-in control loops
  • cloud-controller-manager: cloud integrations

Every node

  • kubelet: makes pod specs real via CRI
  • kube-proxy: Service virtual IPs (or replaced by eBPF CNI)
  • runtime, CNI (pod network), CSI (storage)
Request flow

From kubectl apply to a running pod

1 · API server

  • Authentication — who are you? (certs, tokens, OIDC)
  • Authorization — may you? (RBAC)
  • Admission — mutate, then validate (webhooks, policies, PSA)
  • Persist to etcd

2 · Controllers

  • Deployment controller → ReplicaSet
  • ReplicaSet controller → Pods (no node yet)
  • All via watches on the API server

3 · Scheduler

  • Filter: resources, taints, affinity, volumes, ports
  • Score: spread, balance, preferences
  • Bind pod → node (or stays Pending)

4 · kubelet

  • Sees a pod bound to its node
  • CRI: create sandbox; CNI: assign pod IP
  • Pull images, mount volumes, start containers

5 · Health

  • Startup probe, then readiness + liveness run in parallel
  • Status written back to the API
  • Pod Ready when containers + readiness gates pass

6 · Traffic

  • EndpointSlice controller adds the pod IP
  • kube-proxy / CNI updates Service routing
  • Ingress / Gateway controllers update LBs
Distributions

Ways to run Kubernetes

OptionExamplesUse for
Local / learningkind, minikube, k3d, Docker Desktop, Rancher DesktopLabs, dev loops, CI tests
ManagedAmazon EKS, Google GKE, Azure AKS, DigitalOcean, OKEProduction without running the control plane
Self-managedkubeadm, Cluster API, Talos, RKE2On-prem, full control, learning internals (CKA)
Lightweightk3s, k0s, MicroK8sEdge, IoT, small footprints
Enterprise platformsOpenShift, Rancher, TanzuOpinionated stacks with support

Release cycle

About three minor releases per year; each minor gets roughly 14 months of patches upstream. Managed providers publish their own support calendars.

Recommendation

Learn on kind/minikube, build one cluster with kubeadm to understand internals, run production on a managed service.

Chapter 1 · Part 1.3

Getting started
& kubectl

Hands-on lab

A local cluster in five minutes

# kind = Kubernetes IN Docker
cat > kind.yaml <<'EOF'
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
- role: control-plane
- role: worker
- role: worker
EOF
kind create cluster --name lab --config kind.yaml

kubectl cluster-info --context kind-lab
kubectl get nodes -o wide
kubectl get pods -n kube-system

# Deploy and reach an app
kubectl create deployment web --image=nginx:1.27 --replicas=3
kubectl expose deployment web --port=80
kubectl port-forward svc/web 8080:80
# → http://localhost:8080

kind delete cluster --name lab

Tools to install

  • kubectl — the CLI (within ±1 minor of the cluster)
  • kind or minikube — local clusters
  • helm, k9s (terminal UI), kubectx/kubens
  • Shell completion: source <(kubectl completion zsh)

Try this

  • kubectl delete pod one web pod — watch it come back
  • Scale to 6: kubectl scale deploy web --replicas=6
  • kubectl get pods -o wide — see them spread across workers
Objects

Anatomy of a Kubernetes object

apiVersion: apps/v1          # group/version
kind: Deployment             # object type
metadata:
  name: web
  namespace: shop
  labels:                    # identifying, selectable
    app.kubernetes.io/name: web
    app.kubernetes.io/part-of: shop
  annotations:               # non-identifying metadata
    owner: team-checkout
spec:                        # desired state (you write)
  replicas: 3
  selector:
    matchLabels: { app.kubernetes.io/name: web }
  template:
    metadata:
      labels: { app.kubernetes.io/name: web }
    spec:
      containers:
      - name: web
        image: nginx:1.27
status:                      # actual state (controllers write)
  readyReplicas: 3

Key ideas

  • Labels + selectors glue objects together (Service → Pods, Deployment → Pods)
  • Namespaces scope names, RBAC, quotas, policies
  • Some objects are cluster-scoped (Node, PV, ClusterRole, Namespace)
  • Use recommended labels app.kubernetes.io/*

Discover the API

  • kubectl api-resources — every type
  • kubectl explain deploy.spec.strategy — field docs
  • kubectl get deploy web -o yaml — the full object
kubectl

kubectl essentials

# Read
kubectl get pods -A -o wide
kubectl get deploy,svc,ing -n shop
kubectl get pods -l app.kubernetes.io/name=web
kubectl get pod web-abc -o jsonpath='{.status.podIP}'
kubectl describe pod web-abc
kubectl logs -f deploy/web --all-containers
kubectl get events --sort-by=.lastTimestamp

# Write (declarative)
kubectl apply -f k8s/            # create or update
kubectl diff -f k8s/             # preview changes
kubectl delete -f k8s/

# Generate YAML instead of writing it from scratch
kubectl create deploy api --image=api:1.0 \
  --dry-run=client -o yaml > api.yaml
# Interact
kubectl exec -it web-abc -- sh
kubectl port-forward svc/web 8080:80
kubectl cp web-abc:/var/log/app.log ./app.log

# Rollouts
kubectl rollout status deploy/web
kubectl rollout history deploy/web
kubectl rollout undo deploy/web
kubectl rollout restart deploy/web

# Contexts (kubeconfig)
kubectl config get-contexts
kubectl config use-context prod
kubectl config set-context --current --namespace=shop

# Watch & resources
kubectl get pods -w
kubectl top pods --sort-by=memory

Imperative vs declarative: use kubectl create/run/expose to learn and generate YAML; manage real environments with files in Git + kubectl apply (or GitOps).

Chapter 2

Running applications

Workloads, configuration, networking, and storage.

Chapter 2 · Part 2.1

Workloads

Pods and the controllers that manage them.

Pods

The Pod: smallest deployable unit

What a pod is

  • 1+ containers sharing network namespace (one IP, localhost) and volumes
  • Scheduled together onto one node; ephemeral — never "fixed", only replaced
  • Init containers run to completion, in order, before app containers
  • Sidecars (init containers with restartPolicy: Always) start first and run alongside the app

Lifecycle

  • Phases: Pending → Running → Succeeded / Failed (or Unknown)
  • Container states: Waiting, Running, Terminated
  • Deletion: grace period starts (default 30 s) → preStop hook → SIGTERM → SIGKILL when the period ends (preStop time counts against it; a one-off 2 s extension if preStop overruns)
apiVersion: v1
kind: Pod
metadata:
  name: app
  labels: { app: demo }
spec:
  initContainers:
  - name: log-shipper           # native sidecar, starts first
    image: fluent/fluent-bit:3.1
    restartPolicy: Always
    volumeMounts: [{ name: logs, mountPath: /logs }]
  - name: wait-db               # runs to completion
    image: busybox:1.36
    command: ["sh", "-c",
      "until nc -z db 5432; do sleep 2; done"]
  containers:
  - name: app
    image: ghcr.io/org/app:1.4.2
    ports: [{ containerPort: 8080 }]
    volumeMounts: [{ name: logs, mountPath: /var/log/app }]
  volumes:
  - name: logs
    emptyDir: {}
Deployments

Deployment → ReplicaSet → Pods

apiVersion: apps/v1
kind: Deployment
metadata: { name: web }
spec:
  replicas: 3
  revisionHistoryLimit: 5
  selector:
    matchLabels: { app: web }
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 25%          # extra pods during rollout
      maxUnavailable: 0      # never drop below 3 ready
  minReadySeconds: 10
  template:
    metadata:
      labels: { app: web }
    spec:
      containers:
      - name: web
        image: ghcr.io/org/web:1.4.2
        readinessProbe:
          httpGet: { path: /ready, port: 8080 }

How rollouts work

  • Changing the pod template creates a new ReplicaSet
  • New RS scales up, old RS scales down, gated by readiness
  • Old ReplicaSets kept for rollout undo
  • Stuck rollout → progressDeadlineSeconds (600 s) marks it failed — no automatic rollback
  • Recreate strategy: kill all, then start new (downtime)

Best practices

  • Never manage ReplicaSets or bare Pods directly
  • Immutable image tags/digests, not latest
  • Readiness probe + maxUnavailable: 0 for zero downtime
Stateful & per-node

StatefulSet and DaemonSet

StatefulSet

  • Stable names: db-0, db-1, db-2
  • Stable DNS via a headless Service: db-0.db.ns.svc.cluster.local
  • Own PVC per pod via volumeClaimTemplates — survives rescheduling
  • Ordered create/update/delete (or podManagementPolicy: Parallel)
  • Use for: databases, Kafka, ZooKeeper, anything with identity
  • PVCs aren't deleted on scale-down by default

DaemonSet

  • One pod per (matching) node — new nodes get one automatically
  • Use for: log shippers, node exporters, CNI/CSI agents, security agents
  • Target subsets with nodeSelector / tolerations
  • Rolling updates with maxUnavailable
  • Set requests carefully — it runs on every node
volumeClaimTemplates:
- metadata: { name: data }
  spec: { accessModes: ["ReadWriteOnce"], storageClassName: fast-ssd, resources: { requests: { storage: 50Gi } } }
Batch

Jobs and CronJobs

apiVersion: batch/v1
kind: CronJob
metadata: { name: nightly-report }
spec:
  schedule: "0 2 * * *"
  timeZone: Asia/Ho_Chi_Minh
  concurrencyPolicy: Forbid       # Allow | Forbid | Replace
  startingDeadlineSeconds: 300
  successfulJobsHistoryLimit: 3
  failedJobsHistoryLimit: 3
  jobTemplate:
    spec:
      backoffLimit: 3             # retries (default 6)
      activeDeadlineSeconds: 3600
      ttlSecondsAfterFinished: 86400
      template:
        spec:
          restartPolicy: OnFailure
          containers:
          - name: report
            image: ghcr.io/org/report:2.1.0
            args: ["--date", "yesterday"]

Job patterns

  • completions + parallelism for fan-out work
  • completionMode: Indexed — each pod gets an index
  • restartPolicy must be OnFailure or Never
  • podFailurePolicy to fail fast on non-retriable exit codes

Best practices

  • Make jobs idempotent — they can run twice
  • Set deadlines and history limits
  • ttlSecondsAfterFinished to clean up
  • Alert on failed jobs, not only on success
Cheat sheet

Which workload type?

NeedUseWhy
Stateless web/API serviceDeploymentInterchangeable replicas, rolling updates
Database, broker, clustered appStatefulSet (or an operator)Stable identity and storage per replica
Agent on every nodeDaemonSetFollows nodes automatically
Run to completion onceJobRetries, parallelism, completion tracking
Run on a scheduleCronJobCreates Jobs on a cron schedule
Helper next to the appSidecar containerProxies, log shipping, config reloaders
One-off debuggingkubectl run / kubectl debugTemporary, not managed

Bare Pods are not recreated if their node dies. Always use a controller in real environments.

Chapter 2 · Part 2.2

Configuration
& resources

ConfigMaps & Secrets

Injecting configuration

apiVersion: v1
kind: ConfigMap
metadata: { name: web-config }
data:
  LOG_LEVEL: info
  app.yaml: |
    cache: { ttl: 60 }
---
apiVersion: v1
kind: Secret
metadata: { name: web-secrets }
type: Opaque
stringData:                 # stored base64 — NOT encrypted
  DB_PASSWORD: change-me
---
# in the container spec
envFrom:
- configMapRef: { name: web-config }
env:
- name: DB_PASSWORD
  valueFrom:
    secretKeyRef: { name: web-secrets, key: DB_PASSWORD }
volumeMounts:
- { name: cfg, mountPath: /etc/web, readOnly: true }
# in the pod spec
volumes:
- { name: cfg, configMap: { name: web-config } }

Env vars vs files

  • Env vars are read once at start — changes need a restart
  • Mounted files update in place (not with subPath)
  • immutable: true protects and speeds up large configs
  • Downward API exposes pod name, namespace, labels, limits

Secrets are not secret by default

  • Base64 is encoding, not encryption
  • Enable encryption at rest (KMS); restrict RBAC on secrets
  • Prefer External Secrets / CSI from a vault; never commit plaintext
Resources

Requests, limits, and QoS classes

Semantics

  • Request — reserved; used by the scheduler
  • Limit — enforced ceiling by the kernel (cgroups)
  • CPU over limit → throttled; memory over limit → OOMKilled
  • Units: 500m = 0.5 CPU; 256Mi = 256 MiB
QoSRuleEvicted
GuaranteedEvery container: CPU & memory request = limitLast
BurstableAt least one request or limit setMiddle
BestEffortNothing setFirst
resources:
  requests: { cpu: 250m, memory: 256Mi }
  limits:   { memory: 256Mi }   # no CPU limit

Best practices

  • Always set requests — the scheduler is blind without them
  • Memory: limit = request (predictable, no overcommit surprise)
  • CPU: often no limit for latency-sensitive apps; watch throttling
  • Right-size from real usage (VPA recommendations, metrics)
  • Enforce defaults with LimitRange, totals with ResourceQuota
Health

Probes and graceful shutdown

# container spec
startupProbe:                     # slow boot protection
  httpGet: { path: /healthz, port: 8080 }
  periodSeconds: 5
  failureThreshold: 30            # up to 150 s
readinessProbe:                   # gets traffic?
  httpGet: { path: /ready, port: 8080 }
  periodSeconds: 5
  failureThreshold: 3
livenessProbe:                    # restart if stuck
  httpGet: { path: /healthz, port: 8080 }
  periodSeconds: 10
  failureThreshold: 3
lifecycle:
  preStop:
    sleep: { seconds: 10 }        # let LBs drain
# pod spec
terminationGracePeriodSeconds: 45

Three probes, three jobs

  • startup: disables the others until the app has booted
  • readiness: fail → removed from Service endpoints (no restart)
  • liveness: fail → container restarted
  • Types: httpGet, tcpSocket, grpc, exec

Common mistakes

  • Liveness checks the database → one DB blip restarts every pod
  • Same endpoint and timing for liveness and readiness
  • No SIGTERM handling → dropped requests on every deploy
Chapter 2 · Part 2.3

Networking

Pod network, Services, DNS, Ingress, Gateway API, NetworkPolicy.

The model

Kubernetes networking rules

The rules

  • Every pod gets its own IP
  • Pods talk to all pods without NAT
  • Nodes talk to all pods without NAT
  • Containers in a pod share localhost
  • Implemented by a CNI plugin

CNI plugins

  • Calico — routing/BGP, rich policy
  • Cilium — eBPF, policy, observability, kube-proxy replacement
  • Flannel — simple overlay
  • Cloud-native: AWS VPC CNI, Azure CNI, GKE Dataplane V2

Three IP ranges

  • Node IPs — from your network
  • Pod CIDR — per cluster, split per node
  • Service CIDR — virtual IPs, never on an interface
  • Plan them so they don't overlap with VPNs/peered networks
Services

Services: stable names for moving pods

TypeReachable from
ClusterIP (default)Inside the cluster only
NodePortEvery node's IP on a port (30000–32767)
LoadBalancerCloud LB → NodePort/pods
ExternalNameDNS CNAME to an outside name
Headless (clusterIP: None)DNS returns pod IPs directly

How it works

  • Selector → EndpointSlices list ready pod IPs
  • kube-proxy programs iptables (default) or nftables rules — IPVS mode is deprecated since v1.35 — or eBPF via Cilium
  • port = Service port; targetPort = container port
apiVersion: v1
kind: Service
metadata: { name: web }
spec:
  type: ClusterIP
  selector: { app: web }
  ports:
  - name: http
    port: 80
    targetPort: 8080
---
# Debug: does the Service have endpoints?
kubectl get endpointslices -l kubernetes.io/service-name=web
kubectl get pods -l app=web --show-labels

# Test from inside the cluster
kubectl run tmp --rm -it --image=busybox:1.36 -- \
  wget -qO- http://web.default.svc.cluster.local
DNS

Service discovery with CoreDNS

Names

  • Service: <svc>.<ns>.svc.cluster.local
  • Same namespace: just <svc>
  • Other namespace: <svc>.<ns>
  • StatefulSet pod: <pod>.<headless-svc>.<ns>.svc.cluster.local
  • SRV records for named ports

Gotchas

  • Default ndots:5 → external names are tried with every search suffix first → extra queries
  • Use FQDNs with a trailing dot, or lower ndots via dnsConfig
  • Scale CoreDNS with the cluster; consider NodeLocal DNSCache
  • DNS failures look like random timeouts — check CoreDNS logs and metrics
kubectl run dns --rm -it --image=busybox:1.36 -- sh -c 'cat /etc/resolv.conf; nslookup web.shop; nslookup kubernetes.default'
North-south traffic

Ingress and Gateway API

Ingress

  • L7 HTTP(S) routing by host/path; needs an ingress controller
  • Advanced features live in controller-specific annotations
  • Stable but frozen — no new features
  • The community ingress-nginx controller has been retired — plan a migration

Gateway API (the successor)

  • Role-oriented: GatewayClass (infra) → Gateway (platform) → HTTPRoute (app teams)
  • Portable features: header matching, traffic splitting, timeouts, TLS
  • Also GRPCRoute; cross-namespace refs via ReferenceGrant
  • Implemented by Envoy Gateway, Istio, Cilium, NGINX, cloud LBs…
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata: { name: public, namespace: infra }
spec:
  gatewayClassName: envoy
  listeners:
  - name: https
    protocol: HTTPS
    port: 443
    tls: { certificateRefs: [{ name: wildcard-tls }] }
    allowedRoutes: { namespaces: { from: All } }
---
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata: { name: shop, namespace: shop }
spec:
  parentRefs: [{ name: public, namespace: infra }]
  hostnames: ["shop.example.com"]
  rules:
  - matches: [{ path: { type: PathPrefix, value: /api } }]
    backendRefs:
    - { name: api-v1, port: 80, weight: 90 }
    - { name: api-v2, port: 80, weight: 10 }   # canary
Segmentation

NetworkPolicy

# 1. Deny everything in the namespace
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata: { name: default-deny, namespace: shop }
spec:
  podSelector: {}
  policyTypes: [Ingress, Egress]
---
# 2. Allow api ← web on 8080, and api → DNS
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata: { name: api, namespace: shop }
spec:
  podSelector: { matchLabels: { app: api } }
  policyTypes: [Ingress, Egress]
  ingress:
  - from: [{ podSelector: { matchLabels: { app: web } } }]
    ports: [{ port: 8080 }]
  egress:
  - to: [{ namespaceSelector: {} }]
    ports:
    - { port: 53, protocol: UDP }
    - { port: 53, protocol: TCP }

Rules of thumb

  • Only works if the CNI enforces it (Calico, Cilium… — not plain Flannel)
  • No policy selecting a pod → everything allowed
  • Policies are additive allow-lists; there is no "deny" rule
  • Selecting pods for egress blocks DNS unless you allow it
  • Selectors match pods/namespaces/IP blocks — not DNS names

Practice

Default-deny every app namespace, allow explicitly, and test with a throwaway pod. Use the CNI's tooling (Cilium Hubble, Calico) to see dropped flows.

Chapter 2 · Part 2.4

Storage

Volumes

Volume types and persistent storage

Ephemeral volumes

  • emptyDir — scratch space shared by containers; gone with the pod (medium: Memory for tmpfs)
  • configMap, secret, downwardAPI, projected — config as files
  • hostPath — node filesystem; avoid in apps (security, portability)
  • Generic ephemeral volumes — a PVC that lives and dies with the pod

Persistent storage

  • PersistentVolume (PV) — a piece of storage in the cluster
  • PersistentVolumeClaim (PVC) — a request for storage by a pod
  • StorageClass — how to provision PVs dynamically (CSI driver + parameters)
  • CSI — the plugin interface (EBS, EFS, Ceph, Longhorn, Portworx…)
Access modeMeaningReclaim policyMeaning
ReadWriteOnceRead-write by one nodeDeleteDelete the disk with the PVC (dynamic default)
ReadWriteOncePodRead-write by one podRetainKeep the disk; manual cleanup
ReadOnlyMany / ReadWriteManyMany nodes (needs a shared filesystem)
Dynamic provisioning

StorageClass → PVC → Pod

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata: { name: fast-ssd }
provisioner: ebs.csi.aws.com        # your CSI driver
parameters: { type: gp3 }
reclaimPolicy: Retain
volumeBindingMode: WaitForFirstConsumer
allowVolumeExpansion: true
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata: { name: data }
spec:
  accessModes: [ReadWriteOnce]
  storageClassName: fast-ssd
  resources: { requests: { storage: 20Gi } }
---
# Pod spec
volumes:
- name: data
  persistentVolumeClaim: { claimName: data }

Lifecycle

  • PVC created → provisioner creates a PV → PVC Bound
  • WaitForFirstConsumer provisions in the zone where the pod lands
  • Expand by editing the PVC size (if allowed)
  • VolumeSnapshots for backups/clones (CSI feature)

Best practices

  • Retain for important data
  • Zonal disks + zonal pods: mind topology
  • Back up with Velero or the storage system
  • Prefer managed databases unless you have strong operators and skills
Chapter 3

Scheduling and scaling

Where pods land and how capacity follows demand.

Chapter 3 · Part 3.1

Scheduling

Controlling where pods run.

Placement tools

nodeSelector, affinity, taints & tolerations

spec:
  nodeSelector:                       # simple, hard
    kubernetes.io/arch: arm64
  affinity:
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
        - matchExpressions:
          - { key: node.kubernetes.io/instance-type,
              operator: In, values: [m7g.large, m7g.xlarge] }
    podAntiAffinity:                  # spread replicas
      preferredDuringSchedulingIgnoredDuringExecution:
      - weight: 100
        podAffinityTerm:
          topologyKey: kubernetes.io/hostname
          labelSelector: { matchLabels: { app: web } }
  tolerations:                        # allowed on GPU nodes
  - { key: nvidia.com/gpu, operator: Exists, effect: NoSchedule }

Who decides what

  • Pod says where it wants to go: nodeSelector, node affinity, pod (anti-)affinity
  • Node says who may come: taints; pods need matching tolerations
  • required… = hard; preferred… = soft (weighted)

Taint effects

  • NoSchedule — no new pods without a toleration
  • PreferNoSchedule — avoid if possible
  • NoExecute — also evicts running pods
  • kubectl taint nodes n1 dedicated=gpu:NoSchedule
Resilience

Topology spread, priority, and disruption budgets

Topology spread

  • Even spread across zones/nodes
  • maxSkew, topologyKey, whenUnsatisfiable
  • Simpler and more precise than anti-affinity for spreading

Priority & preemption

  • PriorityClass ranks pods
  • High-priority pending pods can evict lower ones
  • Built-in: system-cluster-critical, system-node-critical

PodDisruptionBudget

  • Limits voluntary evictions (drain, autoscaler)
  • minAvailable or maxUnavailable
  • Doesn't protect against node crashes
topologySpreadConstraints:
- { maxSkew: 1, topologyKey: topology.kubernetes.io/zone, whenUnsatisfiable: DoNotSchedule, labelSelector: { matchLabels: { app: web } } }
---
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata: { name: web }
spec: { maxUnavailable: 1, selector: { matchLabels: { app: web } } }
Chapter 3 · Part 3.2

Autoscaling

Pods & nodes

HPA, VPA, KEDA, and node autoscaling

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata: { name: web }
spec:
  scaleTargetRef:
    { apiVersion: apps/v1, kind: Deployment, name: web }
  minReplicas: 3
  maxReplicas: 30
  metrics:
  - type: Resource
    resource:
      name: cpu
      target: { type: Utilization, averageUtilization: 70 }
  behavior:
    scaleDown:
      stabilizationWindowSeconds: 300
      policies: [{ type: Percent, value: 50, periodSeconds: 60 }]

The layers

  • HPA — replicas from CPU/memory (metrics-server) or custom/external metrics
    desired = ceil(current × metric / target)
  • VPA — right-sizes requests; don't let it fight HPA on the same metric
  • KEDA — event-driven (queues, Kafka, cron), scale to zero
  • Cluster Autoscaler / Karpenter — add nodes for pending pods, remove idle ones

Best practices

Requests must be set (utilization = usage / request) · scale on the real bottleneck · minReplicas ≥ 2 · slow scale-down · load-test the scaling path.

Chapter 4

Security and observability

Protect the cluster and see what it is doing.

Chapter 4 · Part 4.1

Security

AuthN, RBAC, service accounts, admission, pod hardening, secrets, audit.

Access control

Authentication and RBAC

Authentication

  • Kubernetes has no user database
  • Humans: OIDC (SSO), cloud IAM, client certificates
  • Workloads: ServiceAccount tokens
  • Tokens are short-lived, audience-bound, auto-rotated (projected volumes); no long-lived token Secret is auto-created since v1.24

RBAC building blocks

  • Role / ClusterRole — verbs on resources
  • RoleBinding / ClusterRoleBinding — grant to users, groups, SAs
  • Additive only; no deny rules
  • Check: kubectl auth can-i create deploy -n shop --as jane
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata: { name: app-reader, namespace: shop }
rules:
- apiGroups: [""]
  resources: ["configmaps"]
  verbs: ["get", "list", "watch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata: { name: api-reads-config, namespace: shop }
subjects:
- kind: ServiceAccount
  name: api
  namespace: shop
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: app-reader
---
# In the pod spec
serviceAccountName: api
automountServiceAccountToken: false  # unless it calls the API
Admission

Pod Security Admission and policy engines

PSS levelSummary
privilegedUnrestricted — system components only
baselineBlocks privileged, host namespaces, hostPath, dangerous caps
restricted+ non-root, drop ALL caps, no privilege escalation, seccomp
kubectl label ns shop \
  pod-security.kubernetes.io/enforce=restricted \
  pod-security.kubernetes.io/warn=restricted

Beyond PSA

  • ValidatingAdmissionPolicy — built-in CEL rules (GA 1.30)
  • Kyverno — YAML policies: validate, mutate, generate, verify images
  • OPA Gatekeeper — Rego policies
  • Admission webhooks — your own logic

Typical guardrails

Allowed registries · no latest · required labels and requests · signed images · no LoadBalancer Services without approval. Start in audit mode, then enforce.

Hardening

A hardened workload

spec:
  serviceAccountName: api
  automountServiceAccountToken: false
  securityContext:
    runAsNonRoot: true
    runAsUser: 10001
    fsGroup: 10001
    seccompProfile: { type: RuntimeDefault }
  containers:
  - name: api
    image: ghcr.io/org/api@sha256:<digest>
    securityContext:
      allowPrivilegeEscalation: false
      readOnlyRootFilesystem: true
      capabilities: { drop: ["ALL"] }
    resources:
      requests: { cpu: 200m, memory: 256Mi }
      limits: { memory: 256Mi }
    volumeMounts: [{ name: tmp, mountPath: /tmp }]
  volumes:
  - { name: tmp, emptyDir: {} }

Cluster security checklist

  • Private API endpoint, SSO + RBAC least privilege, no shared admin
  • Encryption at rest for Secrets (KMS provider)
  • Audit logging enabled and shipped off-cluster
  • PSA restricted + policy engine
  • Default-deny NetworkPolicies
  • Image scanning, signing, admission verification
  • Runtime detection (Falco, Tetragon, cloud tools)
  • Keep Kubernetes and nodes patched
  • Benchmarks: CIS (kube-bench), kubescape
Chapter 4 · Part 4.2

Observability

Signals

Metrics, logs, events, traces

Metrics

  • metrics-server — live CPU/memory for kubectl top and HPA (not for history)
  • Prometheus + kube-state-metrics + node-exporter + cAdvisor (kubelet)
  • kube-prometheus-stack Helm chart bundles it with Grafana & Alertmanager
  • Control plane: apiserver, etcd, scheduler /metrics

Logs, events, traces

  • Apps log to stdout/stderr; kubelet keeps per-container files on the node
  • Node agent (Fluent Bit, Vector, OTel Collector) ships logs to Loki / Elasticsearch / cloud
  • Events expire (≈1 h by default) — export them if you need history
  • Traces: OpenTelemetry SDK/operator → collector → Tempo / Jaeger / vendor

Alert on

Crash-looping pods · pods pending too long · deployments with unavailable replicas · nodes NotReady / under pressure · PVCs filling up · CPU throttling · API server errors & latency · certificate expiry · SLO burn rate for user-facing services.

PromQL starter pack

Queries every operator should know

QuestionPromQL
Which pods restart?increase(kube_pod_container_status_restarts_total[1h]) > 0
Pending pods per namespacesum by (namespace) (kube_pod_status_phase{phase="Pending"})
Deployments missing replicaskube_deployment_status_replicas_available < kube_deployment_spec_replicas
Memory used vs limitcontainer_memory_working_set_bytes{container!=""} / on(namespace,pod,container) kube_pod_container_resource_limits{resource="memory"}
CPU throttling ratiorate(container_cpu_cfs_throttled_periods_total[5m]) / rate(container_cpu_cfs_periods_total[5m])
Nodes not readykube_node_status_condition{condition="Ready",status="true"} == 0
PVC usagekubelet_volume_stats_used_bytes / kubelet_volume_stats_capacity_bytes
API server error ratesum(rate(apiserver_request_total{code=~"5.."}[5m])) / sum(rate(apiserver_request_total[5m]))
Chapter 5

Delivery and extensibility

Ship changes safely and extend the API.

Chapter 5 · Part 5.1

Packaging & delivery

Helm, Kustomize, GitOps, deployment strategies.

Packaging

Helm vs Kustomize

Helm — templating + releases

  • Chart = templates + values.yaml; versioned, shareable (OCI registries)
  • Tracks releases; upgrade and rollback
  • Best for third-party software and reusable app charts
helm repo add prometheus-community \
  https://prometheus-community.github.io/helm-charts
helm upgrade --install mon \
  prometheus-community/kube-prometheus-stack \
  -n monitoring --create-namespace -f values.yaml
helm list -A
helm template ./chart -f values-prod.yaml   # render
helm rollback mon 2

Kustomize — patch plain YAML

  • No templates: a base/ plus per-env overlays/
  • Built into kubectl: kubectl apply -k overlays/prod
  • Best for your own manifests across environments
# overlays/prod/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
namespace: shop-prod
resources: [../../base]
images:
- { name: ghcr.io/org/web, newTag: "1.4.2" }
patches:
- target: { kind: Deployment, name: web }
  patch: |-
    - op: replace
      path: /spec/replicas
      value: 6
GitOps

GitOps with Argo CD or Flux

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata: { name: shop-prod, namespace: argocd }
spec:
  project: default
  source:
    repoURL: https://github.com/org/deploy.git
    targetRevision: main
    path: shop/overlays/prod
  destination:
    server: https://kubernetes.default.svc
    namespace: shop-prod
  syncPolicy:
    automated: { prune: true, selfHeal: true }
    syncOptions: [CreateNamespace=true]

Principles

  • Git is the single source of truth for desired state
  • An in-cluster agent pulls and reconciles continuously
  • Drift is detected and reverted (selfHeal)
  • Deploy = merge a PR; rollback = revert a commit

Repository layout

App code repo builds and pushes images (CI) → updates the image tag in a deploy repo (PR or image updater) → Argo CD / Flux syncs. CI never needs cluster credentials.

Release strategies

Rolling, blue/green, canary

Rolling update

  • Built into Deployments
  • Gradual replacement gated by readiness
  • Both versions serve traffic during rollout — keep them compatible

Blue/green

  • Run new version alongside, switch the Service selector or route at once
  • Instant rollback by switching back
  • Costs double capacity during the switch

Canary

  • Send 1% → 10% → 50% → 100% of traffic
  • Promote or abort automatically on metrics
  • Tools: Argo Rollouts, Flagger, Gateway API weights, service mesh

Make any strategy safe

Backward-compatible database migrations (expand → migrate → contract) · feature flags · readiness probes that reflect real readiness · automated analysis of error rate and latency before promotion.

Chapter 5 · Part 5.2

Extending Kubernetes

CRDs, controllers, operators.

Custom resources

CRDs and the operator pattern

apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata: { name: databases.example.com }
spec:
  group: example.com
  scope: Namespaced
  names: { kind: Database, plural: databases, singular: database }
  versions:
  - name: v1
    served: true
    storage: true
    schema:
      openAPIV3Schema:
        type: object
        properties:
          spec:
            type: object
            properties:
              engine: { type: string, enum: [postgres, mysql] }
              storageGi: { type: integer, minimum: 1 }
---
apiVersion: example.com/v1
kind: Database
metadata: { name: orders }
spec: { engine: postgres, storageGi: 20 }

How it fits together

  • CRD adds a new API type — stored and served like built-ins
  • A controller watches it and reconciles the real world
  • Operator = CRD + controller encoding human ops knowledge (backup, failover, upgrade)
  • Build with Kubebuilder / Operator SDK (Go), or kopf (Python)

Mechanics to know

  • ownerReferences → garbage collection of children
  • finalizers block deletion until cleanup is done
  • status subresource, conditions, observedGeneration
  • Reconcile must be idempotent and level-based
Chapter 6

Operations

Day-2 work, troubleshooting, and the road to hero.

Chapter 6 · Part 6.1

Cluster operations

Upgrades, node maintenance, etcd, multi-tenancy.

Upgrades

Upgrading a cluster (kubeadm example)

# 0. Release notes, removed APIs (pluto/kubent); point the
#    pkgs.k8s.io apt repo at the new minor (v1.37)
# 1. First control-plane node
apt-mark unhold kubeadm && apt-get update
apt-get install -y kubeadm='1.37.x-*' && apt-mark hold kubeadm
kubeadm upgrade plan
kubeadm upgrade apply v1.37.x
# 2. Other control planes AND workers: kubeadm pkg first
#    (the kubeadm install lines above), then
kubeadm upgrade node
# 3. Every node: drain → kubelet/kubectl → uncordon
kubectl drain node-1 --ignore-daemonsets --delete-emptydir-data
apt-mark unhold kubelet kubectl
apt-get install -y kubelet='1.37.x-*' kubectl='1.37.x-*'
apt-mark hold kubelet kubectl
systemctl daemon-reload && systemctl restart kubelet
kubectl uncordon node-1

Rules

  • One minor version at a time (1.36 → 1.37)
  • Control plane first, then nodes
  • kubelet may lag the API server by up to 3 minors, never lead
  • kubectl within ±1 minor of the API server
  • Upgrade add-ons (CNI, CSI, ingress, operators) to compatible versions

Make it routine

Upgrade non-prod first, keep PDBs on every service, back up etcd before starting, and upgrade regularly — skipping versions makes each upgrade riskier.

Note: the official Linux worker-node guide (updated 2026-08-29) upgrades kubeadm and kubectl together before kubeadm upgrade node, upgrades only kubelet after the drain, and runs drain/uncordon from a control-plane node. Both work — follow the docs for your exact version.

Maintenance & backup

Nodes, etcd, and disaster recovery

# Node maintenance
kubectl cordon node-2          # no new pods
kubectl drain node-2 --ignore-daemonsets \
  --delete-emptydir-data       # evicts, honoring PDBs
kubectl uncordon node-2

# etcd snapshot (self-managed control plane)
ETCDCTL_API=3 etcdctl snapshot save /backup/etcd.db \
  --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key
etcdutl snapshot status /backup/etcd.db --write-out=table

# Restore into a new data dir, then point etcd at it
etcdutl snapshot restore /backup/etcd.db \
  --data-dir /var/lib/etcd-restore

# kubeadm certificates (renewed on upgrade)
kubeadm certs check-expiration

DR layers

  • Cluster state: Git (GitOps) + IaC — rebuild instead of restore
  • etcd: snapshots for self-managed clusters (managed services handle it)
  • App data: Velero (resources + volume snapshots), database-native backups
  • Test restores regularly — an untested backup is a hope

Managed clusters

You can't snapshot etcd yourself — rely on GitOps, Velero, and the provider's control-plane resilience.

Multi-tenancy

Sharing a cluster safely

Per-namespace guardrails

  • RBAC: teams get edit in their namespaces only
  • ResourceQuota: total CPU, memory, pods, PVCs, LoadBalancers
  • LimitRange: default and max requests/limits per container
  • Default-deny NetworkPolicy
  • PSA restricted
  • Dedicated nodes via taints for sensitive tenants
apiVersion: v1
kind: ResourceQuota
metadata: { name: team-a, namespace: team-a }
spec:
  hard:
    requests.cpu: "20"
    requests.memory: 64Gi
    limits.memory: 64Gi
    pods: "200"
    services.loadbalancers: "2"
---
apiVersion: v1
kind: LimitRange
metadata: { name: defaults, namespace: team-a }
spec:
  limits:
  - type: Container
    defaultRequest: { cpu: 100m, memory: 128Mi }
    default: { memory: 256Mi }
    max: { cpu: "4", memory: 8Gi }

Namespaces give soft isolation. For hostile tenants use separate clusters (or virtual clusters such as vCluster) and separate cloud accounts.

Chapter 6 · Part 6.2

Troubleshooting

Method

A troubleshooting flow that always works

1 · Status

kubectl get — what state is it in?

→

2 · Events

describe — what did controllers say?

→

3 · Logs

logs, --previous — what did the app say?

→

4 · Inside

exec / debug — test from the pod

→

5 · Below

Node, CNI, DNS, storage, control plane

kubectl get pods -n shop -o wide
kubectl describe pod web-abc -n shop | tail -20
kubectl logs web-abc -n shop -c web --previous
kubectl get events -n shop --sort-by=.lastTimestamp

# Ephemeral debug container (works on distroless)
kubectl debug -it web-abc -n shop \
  --image=nicolaka/netshoot --target=web

# Copy of the pod with a different command
kubectl debug web-abc -n shop -it \
  --copy-to=web-dbg --container=web -- sh

# Shell on a node
kubectl debug node/node-1 -it --image=ubuntu

Ask in order

  • Was it scheduled? (Pending → scheduler events)
  • Did it start? (image, config, volumes)
  • Does it stay up? (logs, OOM, probes)
  • Is it ready? (readiness, endpoints)
  • Can traffic reach it? (Service, DNS, policy, ingress)
Common failures

Symptoms, causes, fixes

SymptomTypical causeWhere to look
PendingNot enough CPU/memory, taints, affinity, unbound PVCScheduler events in describe
ImagePullBackOffWrong name/tag, private registry auth, rate limitsEvents; imagePullSecrets
CrashLoopBackOffApp exits: bad config, missing dependency, failing livenesslogs --previous, exit code
OOMKilled (exit 137)Memory limit too low or leakdescribe last state; memory metrics
CreateContainerConfigErrorMissing ConfigMap/Secret or keyEvents
Running but not ReadyReadiness probe failingProbe config, app health endpoint
Service unreachableSelector mismatch → no endpoints; wrong targetPortget endpointslices, labels
Intermittent DNS timeoutsCoreDNS overloaded, ndots, conntrackCoreDNS logs/metrics
EvictedNode pressure (disk, memory)Node conditions, kubelet logs
Stuck TerminatingFinalizers, unresponsive nodemetadata.finalizers, node status
Chapter 6 · Part 6.3

The road to hero

Going deeper

Advanced platform topics

API & control plane

  • Server-side apply (kubectl apply --server-side): the API server tracks which manager owns each field and reports conflicts
  • Feature stages: alpha (off by default) → beta → GA; check the feature state for your version
  • API deprecation: removed beta APIs break manifests on upgrade — scan before upgrading
  • HA control plane: 3 or 5 etcd members (stacked or external), API servers behind a load balancer
  • Leases: scheduler and controller-manager elect a leader; node heartbeats live in kube-node-lease

Workload features

  • In-place pod resize: change CPU/memory of a running pod via the resize subresource
  • Dynamic Resource Allocation (DRA): request GPUs and other devices with claims instead of plain counts
  • Pod topology & scheduling gates: hold pods until external conditions are met
  • Windows nodes: separate node pools, kubernetes.io/os selectors
kubectl apply --server-side --field-manager=platform -f deploy.yaml
kubectl get leases -n kube-system
kubectl patch pod web-abc --subresource resize --patch '{"spec":{"containers":[{"name":"web","resources":{"requests":{"cpu":"500m"}}}]}}'
Ecosystem

Add-ons most production clusters run

NeedCommon choiceWhat it does
TLS certificatescert-managerIssues and renews certificates (e.g. Let's Encrypt) as Secrets
DNS recordsExternalDNSCreates DNS records for Services/Ingresses/Gateways
Secrets from a vaultExternal Secrets Operator, Sealed SecretsSync secrets from AWS/GCP/Vault; encrypted secrets safe for Git
Service-to-service securityIstio, Linkerd, CiliummTLS, retries, traffic splitting, telemetry
PolicyKyverno, GatekeeperAdmission guardrails, image verification
Supply chaincosign (Sigstore), Trivy, SyftSign, scan, SBOM
BackupVeleroBack up resources and volumes
CostOpenCost / KubecostCost per namespace, workload, label
Many clustersCluster API, Argo CD ApplicationSetsDeclarative cluster lifecycle; fleet deployments
Practice

Projects and certifications

Build these, in order

  • Deploy a 3-tier app (web, API, Postgres StatefulSet) with probes, quotas, NetworkPolicies
  • Package it with Helm, then manage environments with Kustomize overlays
  • Install kube-prometheus-stack; write 5 alerts and a dashboard
  • Set up Argo CD; deploy only through Git
  • Do a canary release with Gateway API weights or Argo Rollouts
  • Build a cluster with kubeadm; upgrade it; back up and restore etcd
  • Write a small operator with Kubebuilder
  • Break things on purpose and fix them (chaos practice)

CNCF certifications

  • KCNA — cloud-native fundamentals (multiple choice)
  • KCSA — security fundamentals (multiple choice)
  • CKAD — application developer (hands-on)
  • CKA — administrator (hands-on)
  • CKS — security specialist (hands-on, requires CKA)
  • Hands-on exams reward kubectl speed: aliases, --dry-run=client -o yaml, kubectl explain
Chapter 7

Hands-on labs

Twenty labs on one kind cluster: every chapter, with a break-it-and-fix-it step in each.

Lab 0 · Setup

One cluster for every lab

Recreate the three-node kind cluster from Getting started, plus metrics-server for top and HPA.

# kind v0.33 defaults to Kubernetes 1.37 (kindest/node:v1.37.0)
cat > kind.yaml <<'EOF'
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
- role: control-plane
- role: worker
- role: worker
EOF
kind create cluster --name lab --config kind.yaml

# metrics-server (kubelet certs on kind are self-signed)
MS=https://github.com/kubernetes-sigs/metrics-server/releases/latest/download
kubectl apply -f $MS/components.yaml
kubectl -n kube-system patch deploy metrics-server --type=json -p \
  '[{"op":"add","path":"/spec/template/spec/containers/0/args/-",
    "value":"--kubelet-insecure-tls"}]'
kubectl top nodes        # works after ~1 minute

Before you start

  • Tools: kind, kubectl (±1 minor of the cluster), helm, Docker or Podman.
  • Give Docker at least 4 CPUs and 8 GB RAM; labs 10–12 fill nodes on purpose.
  • Work in a namespace per lab: kubectl create ns lab3 && kubectl config set-context --current --namespace lab3.
  • kind ≥ 0.24 enforces NetworkPolicy out of the box (kube-network-policies) and ships the standard local-path StorageClass.
  • Reset any time: kind delete cluster --name lab and recreate in about a minute.
Lab 1 · Architecture

The control plane up close

Goal: see that the control plane is ordinary pods, that state lives in etcd, and how the scheduler decides.

# static pod manifests, read by the kubelet on the node
docker exec lab-control-plane ls /etc/kubernetes/manifests
kubectl -n kube-system get pods -o wide | grep control-plane

# peek at the raw keys in etcd
kubectl create deployment web --image=nginx:1.27 --replicas=3
kubectl -n kube-system exec etcd-lab-control-plane -- etcdctl \
  --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key \
  get /registry/deployments/default/web --keys-only

# watch controllers and the scheduler react
kubectl get events -w --field-selector involvedObject.kind=Pod &
kubectl scale deploy web --replicas=5

Verify / Break it

  • Manifests: etcd, kube-apiserver, kube-controller-manager, kube-scheduler.
  • Events show Scheduled (scheduler), then Pulled, Created, Started (kubelet).
  • Break it: docker exec lab-control-plane mv /etc/kubernetes/manifests/kube-scheduler.yaml /tmp/ then scale up: new pods stay Pending with no node.
  • Fix: move the file back; the kubelet restarts the scheduler and pods are placed.
  • Explain why deleting the scheduler pod with kubectl does not help.
Lab 2 · Workloads

Rollouts, rollbacks, and probes

Goal: control how a Deployment replaces pods, and see a bad readiness probe stop a rollout.

kubectl create deployment web --image=nginx:1.27 --replicas=4
kubectl patch deploy web -p '{"spec":{"strategy":{"rollingUpdate":
  {"maxSurge":1,"maxUnavailable":0}}}}'

kubectl set image deploy/web nginx=nginx:1.28
kubectl annotate deploy/web kubernetes.io/change-cause="nginx 1.28"
kubectl rollout status deploy/web
kubectl rollout history deploy/web

# add a readiness probe that can never pass
kubectl patch deploy web --type=json -p '[{"op":"add",
  "path":"/spec/template/spec/containers/0/readinessProbe",
  "value":{"httpGet":{"path":"/nope","port":80},"periodSeconds":2}}]'
kubectl get pods -w            # new pod never becomes Ready
kubectl rollout undo deploy/web

Verify / Break it

  • With maxUnavailable: 0 the old ReplicaSet keeps all 4 pods serving while the bad pod is stuck 0/1 Running.
  • kubectl get rs shows two ReplicaSets during a rollout.
  • kubectl describe pod shows Readiness probe failed: HTTP probe failed with statuscode: 404.
  • Try a liveness probe on /nope instead: the container restarts repeatedly. Compare the two failure modes.
  • After progressDeadlineSeconds (600 s) the Progressing condition turns False with ProgressDeadlineExceeded and rollout status exits non-zero.
Lab 3 · Workloads

Jobs, CronJobs, StatefulSets, and DaemonSets

Goal: run the other controllers and see how each one names, orders, and places pods.

kubectl create job pi --image=perl:5.40 -- \
  perl -Mbignum=bpi -wle 'print bpi(200)'
kubectl wait --for=condition=complete job/pi --timeout=3m && kubectl logs job/pi
kubectl create cronjob tick --image=busybox:1.37 \
  --schedule="*/1 * * * *" -- date
kubectl get jobs -w                      # a new Job every minute

# StatefulSet + the headless Service it requires
kubectl create service clusterip db --clusterip=None --tcp=80:80
cat <<'EOF' | kubectl apply -f -
apiVersion: apps/v1
kind: StatefulSet
metadata: {name: db}
spec:
  serviceName: db
  replicas: 3
  selector: {matchLabels: {app: db}}
  template:
    metadata: {labels: {app: db}}
    spec: {containers: [{name: nginx, image: "nginx:1.27"}]}
EOF
kubectl get pods -w -l app=db            # db-0, then db-1, then db-2

Verify / Break it

  • kubectl create service labels its selector app=db, matching the pods.
  • Delete db-1: it comes back with the same name; nslookup db-1.db resolves from another pod.
  • Scale down to 1: pods go away in reverse order (db-2 first).
  • DaemonSet: write one with a busybox sleeper; it lands on both workers but not the control plane until you add a toleration for node-role.kubernetes.io/control-plane:NoSchedule.
  • Set backoffLimit: 2 on a Job whose command exits 1 and watch it fail.
Lab 4 · Configuration

ConfigMaps and Secrets: env vs volume

Goal: learn which consumption method picks up changes, and how Secrets are really stored.

kubectl create configmap app --from-literal=COLOR=blue
kubectl create secret generic db --from-literal=PASSWORD=s3cr3t
cat <<'EOF' | kubectl apply -f -
apiVersion: v1
kind: Pod
metadata: {name: cfg}
spec:
  containers:
  - name: c
    image: busybox:1.37
    command: ["sh","-c","while true; do echo $COLOR $(cat /cfg/COLOR);
      sleep 5; done"]
    envFrom: [{configMapRef: {name: app}}]
    volumeMounts: [{name: cfg, mountPath: /cfg}]
  volumes: [{name: cfg, configMap: {name: app}}]
EOF
kubectl create configmap app --from-literal=COLOR=green \
  -o yaml --dry-run=client | kubectl apply -f -
kubectl logs -f cfg      # file changes after ~1 min; env stays blue

Verify / Break it

  • Env vars are read once at container start; the mounted file is refreshed by the kubelet (eventually consistent, typically under a minute).
  • A subPath mount never updates. Test it.
  • kubectl get secret db -o jsonpath={.data.PASSWORD} | base64 -d: Secrets are base64-encoded, not encrypted.
  • Mark the ConfigMap immutable: true and try to edit it.
  • Production answer: hash the config into a pod annotation so changes trigger a rollout (Helm and Kustomize do this).
Lab 5 · Configuration

Requests, limits, QoS, and OOMKilled

Goal: see how requests and limits set a pod’s QoS class and what happens at the memory limit.

cat <<'EOF' | kubectl apply -f -
apiVersion: v1
kind: Pod
metadata: {name: hog}
spec:
  containers:
  - name: hog
    image: busybox:1.37
    command: ["sh","-c","tail /dev/zero"]     # grows memory forever
    resources:
      requests: {cpu: 100m, memory: 64Mi}
      limits:   {cpu: 200m, memory: 64Mi}
EOF
kubectl get pod hog -w
kubectl describe pod hog | grep -A3 'Last State'
kubectl get pod hog -o jsonpath='{.status.qosClass}'; echo

Verify / Break it

  • Status cycles OOMKilled → CrashLoopBackOff; Last State shows Reason: OOMKilled, exit code 137.
  • QoS: requests = limits for every resource → Guaranteed; only requests → Burstable; nothing → BestEffort.
  • CPU over the limit is throttled, never killed: replace the command with a busy loop and watch kubectl top pod.
  • Request 100 CPUs: the pod stays Pending with Insufficient cpu.
  • Add a LimitRange to the namespace and create a pod with no resources: defaults are injected.
Lab 6 · Networking

Services, DNS, and EndpointSlices

Goal: trace a request from a Service name to pod IPs, and see readiness control membership.

kubectl create deployment web --image=nginx:1.27 --replicas=3
kubectl expose deploy web --port=80
kubectl get endpointslices -l kubernetes.io/service-name=web

kubectl run dnsutils \
  --image=registry.k8s.io/e2e-test-images/jessie-dnsutils:1.3 \
  --restart=Never -- sleep infinity
kubectl exec dnsutils -- nslookup web
kubectl exec dnsutils -- nslookup web.default.svc.cluster.local
kubectl exec dnsutils -- cat /etc/resolv.conf     # search domains, ndots:5
kubectl run tmp --rm -it --image=busybox:1.37 --restart=Never -- wget -qO- web

# NodePort: reach it through any node's IP
kubectl patch svc web -p '{"spec":{"type":"NodePort"}}'
kubectl get svc web          # note the 30000-32767 port
docker exec lab-worker curl -s localhost:<nodePort> | head -4

Verify / Break it

  • The EndpointSlice lists the three pod IPs; kubectl get pods -o wide matches.
  • Make one pod unready (exec in and delete /usr/share/nginx/html/index.html, with a readiness probe on /): its IP leaves the slice.
  • Break the selector: kubectl patch svc web -p '{"spec":{"selector":{"app":"wbe"}}}' → empty slice, requests get Connection refused. Fix it.
  • type: LoadBalancer stays <pending> on kind unless you run cloud-provider-kind.
Lab 7 · Networking

North-south traffic with the Gateway API

Goal: route HTTP by host and path with a Gateway API implementation (Envoy Gateway).

helm install eg oci://docker.io/envoyproxy/gateway-helm --version v1.9.2 \
  -n envoy-gateway-system --create-namespace
kubectl wait --timeout=5m -n envoy-gateway-system \
  deployment/envoy-gateway --for=condition=Available
# GatewayClass "eg", Gateway "eg", and a sample backend
EG=https://github.com/envoyproxy/gateway/releases/download/v1.9.2
kubectl apply -f $EG/quickstart.yaml -n default

cat <<'EOF' | kubectl apply -f -
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata: {name: web}
spec:
  parentRefs: [{name: eg}]
  hostnames: ["web.example.com"]
  rules: [{backendRefs: [{name: web, port: 80}]}]
EOF

Verify / Break it

  • Find the Envoy Service: kubectl get svc -n envoy-gateway-system -l gateway.envoyproxy.io/owning-gateway-name=eg, then kubectl -n envoy-gateway-system port-forward svc/<name> 8888:80.
  • curl -H "Host: web.example.com" localhost:8888 returns the nginx page; -H "Host: www.example.com" hits the sample backend.
  • kubectl get httproute web -o yaml: Accepted and ResolvedRefs conditions are True.
  • Break it: point backendRefs at port: 81 → ResolvedRefs false, HTTP 500. Fix and re-check.
  • Why not Ingress NGINX? It was retired in March 2026; new work should use Gateway API.
Lab 8 · Networking

NetworkPolicy: default deny, then allow

Goal: lock a namespace down and open exactly the paths you need, including DNS.

kubectl create ns np && kubectl -n np create deployment api --image=nginx:1.27
kubectl -n np expose deploy api --port=80
kubectl -n np run client --image=busybox:1.37 --labels=role=client -- sleep 1d
kubectl -n np exec client -- wget -qO- -T 3 api | head -2        # works

cat <<'EOF' | kubectl -n np apply -f -
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata: {name: default-deny}
spec: {podSelector: {}, policyTypes: [Ingress, Egress]}
EOF
kubectl -n np exec client -- wget -qO- -T 3 api  # DNS blocked too

Now allow, step by step

  • Allow egress to DNS: to: [{namespaceSelector: {matchLabels: {kubernetes.io/metadata.name: kube-system}}, podSelector: {matchLabels: {k8s-app: kube-dns}}}] on UDP and TCP 53.
  • Allow egress from role=client to app=api on TCP 80.
  • Allow ingress to app=api from role=client only.
  • Verify: client works; a pod without the label times out.
  • Rules are additive allows; there is no deny rule in the core API. Order does not matter.
Lab 9 · Storage

PVCs, StatefulSet volumes, and reclaim policy

Goal: provision storage dynamically and prove data survives pod deletion.

kubectl get storageclass        # standard (default), WaitForFirstConsumer
cat <<'EOF' | kubectl apply -f -
apiVersion: v1
kind: PersistentVolumeClaim
metadata: {name: data}
spec:
  accessModes: [ReadWriteOnce]
  resources: {requests: {storage: 1Gi}}
EOF
kubectl get pvc data            # Pending until a pod uses it
kubectl run writer --image=busybox:1.37 --overrides='{"spec":{
  "volumes":[{"name":"d","persistentVolumeClaim":{"claimName":"data"}}],
  "containers":[{"name":"writer","image":"busybox:1.37",
    "command":["sh","-c","date >> /d/log; sleep 1d"],
    "volumeMounts":[{"name":"d","mountPath":"/d"}]}]}}'
kubectl get pvc,pv              # Bound; PV created on the pod's node
kubectl delete pod writer       # then rerun the same kubectl run

Verify / Break it

  • /d/log gains a second line: the data outlived the pod.
  • local-path volumes are tied to one node: cordon that node, recreate the pod, and see it stay Pending (volume node affinity conflict).
  • Add volumeClaimTemplates to the Lab 3 StatefulSet: each replica gets data-db-0, data-db-1… and keeps it across restarts and scale-down.
  • Delete the PVC: the PV disappears because the class uses reclaimPolicy: Delete. Patch a PV to Retain first and compare.
Lab 10 · Scheduling

Labels, affinity, taints, and spread

Goal: steer pods onto nodes, keep them off others, and spread replicas evenly.

kubectl label node lab-worker disk=ssd
kubectl taint node lab-worker2 dedicated=batch:NoSchedule

kubectl create deployment fast --image=nginx:1.27 --replicas=4
kubectl patch deploy fast -p '{"spec":{"template":{"spec":
  {"nodeSelector":{"disk":"ssd"}}}}}'
kubectl get pods -o wide -l app=fast      # all on lab-worker

# spread evenly across nodes instead
kubectl patch deploy fast --type=json -p '[
 {"op":"remove","path":"/spec/template/spec/nodeSelector"},
 {"op":"add","path":"/spec/template/spec/topologySpreadConstraints",
  "value":[{"maxSkew":1,"topologyKey":"kubernetes.io/hostname",
   "whenUnsatisfiable":"ScheduleAnyway",
   "labelSelector":{"matchLabels":{"app":"fast"}}}]}]'
kubectl get pods -o wide -l app=fast   # lab-worker2: still tainted

Verify / Break it

  • Add a toleration for dedicated=batch:NoSchedule: pods may now land on lab-worker2, but a toleration never attracts them; use affinity for that.
  • Set whenUnsatisfiable: DoNotSchedule with the taint still in place and scale to 6: some pods go Pending. Read the scheduler message in kubectl describe.
  • Replace nodeSelector with requiredDuringScheduling… node affinity using operator: In.
  • Clean up: kubectl taint node lab-worker2 dedicated- and kubectl label node lab-worker disk-.
Lab 11 · Scheduling

PriorityClass and preemption

Goal: watch the scheduler evict lower-priority pods to make room for a critical one.

kubectl create priorityclass critical --value=1000000 \
  --description="lab: preempts others"
kubectl create priorityclass low --value=100

# fill the workers with low-priority pods that reserve CPU
kubectl create deployment filler --image=nginx:1.27 --replicas=6
kubectl set resources deploy filler --requests=cpu=500m
kubectl patch deploy filler -p '{"spec":{"template":{"spec":
  {"priorityClassName":"low"}}}}'
kubectl get pods -l app=filler           # some Pending: nodes are full

kubectl create deployment vip --image=nginx:1.27
kubectl set resources deploy vip --requests=cpu=1
kubectl patch deploy vip -p '{"spec":{"template":{"spec":
  {"priorityClassName":"critical"}}}}'
kubectl get events --field-selector reason=Preempted

Verify / Break it

  • Adjust replicas and requests so the workers are full on your Docker CPU allocation.
  • vip becomes Running; some filler pods are Preempted (deleted) and their replacements wait in Pending.
  • kubectl get pod <vip> -o jsonpath={.spec.priority} shows 1000000.
  • preemptionPolicy is immutable: create a new class with preemptionPolicy: Never, use it for vip, and it now waits in the queue instead.
  • Real clusters: system-cluster-critical and system-node-critical protect add-ons; never give apps those.
Lab 12 · Autoscaling

Horizontal Pod Autoscaler under load

Goal: scale a CPU-bound app on demand and understand the HPA math.

kubectl create deployment php-apache --image=registry.k8s.io/hpa-example
kubectl set resources deploy php-apache --requests=cpu=200m --limits=cpu=500m
kubectl expose deploy php-apache --port=80
cat <<'EOF' | kubectl apply -f -
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata: {name: php-apache}
spec:
  scaleTargetRef: {apiVersion: apps/v1, kind: Deployment, name: php-apache}
  minReplicas: 1
  maxReplicas: 10
  metrics:
  - type: Resource
    resource: {name: cpu, target: {type: Utilization, averageUtilization: 50}}
EOF
kubectl run -it --rm load --image=busybox:1.37 --restart=Never -- \
  sh -c "while sleep 0.01; do wget -q -O- http://php-apache; done"
kubectl get hpa php-apache -w            # in a second terminal

Verify / Break it

  • TARGETS climbs above 50%, replicas rise; stop the load and replicas fall after the 5-minute scale-down stabilization window.
  • Formula: desired = ceil(current × currentUtilization / target).
  • Remove the CPU request and limit (a limit alone defaults the request): TARGETS shows <unknown> and the HPA cannot act. kubectl describe hpa explains why.
  • Delete metrics-server: same symptom, different message. Reinstall it from Lab 0.
  • Add behavior.scaleDown.stabilizationWindowSeconds: 30 and compare.
Lab 13 · Security

RBAC and ServiceAccount tokens

Goal: give a workload identity exactly the API access it needs, and prove it.

kubectl create ns dev
kubectl -n dev create serviceaccount app
kubectl -n dev create role pod-reader --verb=get,list,watch --resource=pods
kubectl -n dev create rolebinding app-reads-pods --role=pod-reader \
  --serviceaccount=dev:app

SA=system:serviceaccount:dev:app
kubectl auth can-i list pods -n dev --as=$SA        # yes
kubectl auth can-i delete pods -n dev --as=$SA      # no
kubectl auth can-i list pods -n default --as=$SA    # no

# short-lived, audience-bound token (TokenRequest API)
TOKEN=$(kubectl -n dev create token app --duration=10m)
kubectl --token="$TOKEN" -n dev get pods     # allowed
kubectl --token="$TOKEN" get nodes           # Forbidden

Verify / Break it

  • Decode the token payload (second JWT segment): it has exp, aud, and the SA name.
  • A pod using serviceAccountName: app gets a projected token at /var/run/secrets/kubernetes.io/serviceaccount/token that the kubelet rotates.
  • Turn the Role into a ClusterRole + ClusterRoleBinding and re-run the third can-i: now yes. That is the blast-radius difference.
  • kubectl auth can-i --list --as=system:serviceaccount:dev:app -n dev shows every allowed verb.
  • Set automountServiceAccountToken: false on pods that never call the API.
Lab 14 · Security

Pod Security Admission and securityContext

Goal: enforce the restricted profile on a namespace and make a pod compliant.

kubectl create ns secure
kubectl label ns secure pod-security.kubernetes.io/enforce=restricted \
  pod-security.kubernetes.io/warn=restricted
kubectl -n secure run bad --image=nginx:1.27      # rejected, lists violations

cat <<'EOF' | kubectl -n secure apply -f -
apiVersion: v1
kind: Pod
metadata: {name: good}
spec:
  securityContext:
    runAsNonRoot: true
    seccompProfile: {type: RuntimeDefault}
  containers:
  - name: web
    image: nginxinc/nginx-unprivileged:1.27     # listens on 8080 as UID 101
    securityContext:
      allowPrivilegeEscalation: false
      capabilities: {drop: [ALL]}
EOF

Verify / Break it

  • The rejection message names each missing field: allowPrivilegeEscalation, capabilities, runAsNonRoot, seccompProfile.
  • Add readOnlyRootFilesystem: true: nginx fails to start until you mount emptyDir volumes for /tmp and its cache path.
  • Test before enforcing: kubectl label --dry-run=server --overwrite ns default pod-security.kubernetes.io/enforce=restricted prints warnings for existing pods.
  • PSA only checks pod specs; it is not a replacement for RBAC or NetworkPolicy.
Lab 15 · Observability

Logs, events, metrics, and a monitoring stack

Goal: build the habit of reading the cluster’s own signals before reaching for tools.

kubectl logs deploy/web --all-containers --since=10m
kubectl logs <pod> --previous           # the crashed container's logs
kubectl get events --sort-by=.lastTimestamp | tail -20
kubectl top pods --sort-by=memory -A | head
kubectl get --raw '/readyz?verbose' | tail -5

# optional: Prometheus + Grafana + alerting in one chart
helm repo add prometheus-community \
  https://prometheus-community.github.io/helm-charts
helm install mon prometheus-community/kube-prometheus-stack \
  -n monitoring --create-namespace
kubectl -n monitoring get pods
kubectl -n monitoring port-forward svc/mon-grafana 3000:80
# admin password: kubectl -n monitoring get secret mon-grafana \
#   -o jsonpath='{.data.admin-password}' | base64 -d

Verify / Break it

  • Open the Kubernetes / Compute Resources / Namespace (Pods) dashboard and find the hog pod from Lab 5.
  • In Prometheus, graph CPU per pod: sum by (pod) (rate(container_cpu_usage_seconds_total[5m])).
  • Recreate the Lab 5 OOM pod and find it three ways: events, kubectl describe, and the KubePodCrashLooping alert.
  • Events are kept for about one hour by default: export them if you need history.
  • Uninstall with helm uninstall mon -n monitoring; it uses a lot of memory on kind.
Lab 16 · Delivery

Helm releases and Kustomize overlays

Goal: package an app, upgrade and roll it back with Helm, and vary it per environment with Kustomize.

helm create webapp                  # scaffold a chart
helm lint webapp
helm template webapp ./webapp | head -30  # render without installing
helm install shop ./webapp -n shop --create-namespace
helm upgrade shop ./webapp -n shop --set replicaCount=3
helm history shop -n shop
helm rollback shop 1 -n shop

# Kustomize: base + overlay, built into kubectl
mkdir -p kz/base kz/overlays/prod
kubectl create deployment api --image=nginx:1.27 \
  --dry-run=client -o yaml > kz/base/deploy.yaml
printf 'resources:\n- deploy.yaml\n' > kz/base/kustomization.yaml
K=kz/overlays/prod/kustomization.yaml
printf 'resources:\n- ../../base\nnamePrefix: prod-\n' > $K
printf 'replicas:\n- name: api\n  count: 3\n' >> $K
kubectl kustomize kz/overlays/prod   # then: kubectl apply -k kz/overlays/prod

Verify / Break it

  • helm history shows revisions 1–3; rollback creates a new revision rather than deleting one.
  • Break the chart (bad indentation in templates/deployment.yaml): helm lint and helm template catch it before the cluster does.
  • helm upgrade --install --rollback-on-failure (Helm 4; --atomic in Helm 3) rolls back automatically if pods never become ready.
  • Add an images: transformer to the prod overlay to pin nginx:1.28 and diff with kubectl diff -k.
  • Rule of thumb: Helm to package and share; Kustomize to patch per environment.
Lab 17 · Extending

Your own API with a CustomResourceDefinition

Goal: extend the API server with a typed, validated resource that kubectl understands.

cat <<'EOF' | kubectl apply -f -
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata: {name: backups.lab.example.com}
spec:
  group: lab.example.com
  scope: Namespaced
  names: {plural: backups, singular: backup, kind: Backup, shortNames: [bk]}
  versions:
  - name: v1
    served: true
    storage: true
    schema:
      openAPIV3Schema:
        type: object
        properties:
          spec:
            type: object
            required: [schedule]
            properties:
              schedule: {type: string}
              retentionDays: {type: integer, minimum: 1, maximum: 90}
EOF

Verify / Break it

  • kubectl explain backup.spec shows your fields; kubectl api-resources | grep lab.example.com lists the new type.
  • Create a Backup with schedule: "0 2 * * *" and get it with kubectl get bk.
  • Break it: retentionDays: 365 or omit schedule → the API server rejects it with a validation error.
  • Nothing happens on create: a CRD is only storage. A controller (operator) watches it and acts; scaffold one with Kubebuilder as a stretch goal.
  • Delete the CRD and every Backup object goes with it.
Lab 18 · Operations

Drain, PodDisruptionBudgets, etcd snapshots, upgrades

Goal: do node maintenance without an outage, and rehearse the control plane chores.

kubectl create deployment web --image=nginx:1.27 --replicas=3
kubectl create poddisruptionbudget web --selector=app=web --min-available=3
kubectl drain lab-worker --ignore-daemonsets --delete-emptydir-data
# bare pods from earlier labs block drain: delete them or add --force
# retries: "Cannot evict pod as it would violate the pod's disruption budget"
kubectl patch pdb web -p '{"spec":{"minAvailable":2}}'   # drain now proceeds
kubectl uncordon lab-worker

# etcd snapshot (kubeadm layout; /var/lib/etcd is a hostPath)
kubectl -n kube-system exec etcd-lab-control-plane -- etcdctl \
  --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key snapshot save /var/lib/etcd/lab.db
docker exec lab-control-plane ls -lh /var/lib/etcd/lab.db

Upgrades and restore

  • kind cannot upgrade in place: create a new cluster with --image kindest/node:<newer> and redeploy. That is a blue/green cluster upgrade.
  • kubeadm clusters: upgrade one minor at a time — kubeadm upgrade plan, kubeadm upgrade apply, then each node: drain, upgrade kubelet, uncordon.
  • Restore is done offline with etcdutl snapshot restore into a new data dir, then point the etcd static pod at it.
  • Managed services (EKS, GKE, AKS) handle etcd for you; you still own drains and PDBs.
Lab 19 · Troubleshooting

Break-fix drills

Have a partner (or a script) break one thing; diagnose with the commands, not by reading the YAML first.

Break itSymptomDiagnose withFix
kubectl set image deploy/web nginx=nginx:9.9ImagePullBackOffdescribe pod → Events: manifest unknownCorrect tag; rollout undo
Command sh -c "exit 1"CrashLoopBackOfflogs --previous, exit code in describeFix command or config
Request cpu: 50Pendingdescribe pod: Insufficient cpuLower request or add nodes
Taint every workerPendingdescribe pod: untolerated taintRemove taint or tolerate
Service selector typoConnection refused, pods healthyget endpointslices is emptyMatch pod labels
kubectl -n kube-system scale deploy coredns --replicas=0Names fail, IPs worknslookup from dnsutils; get pods -n kube-systemScale CoreDNS back to 2
Default-deny NetworkPolicy without DNS egressBad address / timeoutsget netpol; test by IPAllow UDP/TCP 53 to kube-dns

Universal order: kubectl get (what state?) → describe (events, conditions) → logs / logs --previous → exec / debug → node level (docker exec lab-worker crictl ps, journalctl -u kubelet).

Lab 20 · Capstone

Ship a two-tier app like production

Combine every lab into one namespace you could defend in a review.

Build

  • Namespace with PSA restricted and a ResourceQuota.
  • web Deployment (unprivileged nginx) and a Redis StatefulSet with a PVC.
  • ConfigMap + Secret; checksum annotation for rollouts.
  • Probes, requests and limits on every container.
  • Packaged as a Helm chart or Kustomize base + overlays.

Operate

  • HTTPRoute on the Lab 7 Gateway.
  • HPA on web; PDB on both tiers.
  • Default-deny NetworkPolicy plus explicit allows, DNS included.
  • Dedicated ServiceAccounts, tokens off where unused.
  • A Grafana dashboard showing request rate and memory.

Prove it

  • Drain a worker during a load test: no failed requests.
  • Roll out a bad image: the rollout stops, users unaffected.
  • Kill the Redis pod: data survives.
  • kubectl auth can-i --list for each SA is minimal.
  • Delete the namespace and redeploy from Git in one command.
Checklist

Lab completion checklist

Tick a lab only when you can do it again without notes and explain what happened.

Core (CKAD level)

  • ☐ 2 Rollout, rollback, probes
  • ☐ 3 Jobs, CronJobs, StatefulSets, DaemonSets
  • ☐ 4 ConfigMap and Secret update behavior
  • ☐ 5 Requests, limits, QoS, OOMKilled
  • ☐ 6 Services, DNS, EndpointSlices
  • ☐ 9 PVCs and StatefulSet volumes
  • ☐ 16 Helm and Kustomize

Operator (CKA level)

  • ☐ 1 Control plane and etcd
  • ☐ 7 Gateway API routing
  • ☐ 10, 11 Scheduling and preemption
  • ☐ 12 HPA under load
  • ☐ 13 RBAC and tokens
  • ☐ 18 Drain, PDB, etcd snapshot
  • ☐ 19 All seven break-fix drills

Hero (CKS level and beyond)

  • ☐ 8 NetworkPolicy default deny
  • ☐ 14 PSA restricted + securityContext
  • ☐ 15 Monitoring stack and alerts
  • ☐ 17 CRD with validation
  • ☐ 20 Capstone passes every proof
  • Then: write a controller, try GitOps with Argo CD, move to EKS.

Clean up when finished: kind delete cluster --name lab.

Checklist

Production readiness checklist

Workloads

  • Deployment/StatefulSet, never bare pods
  • Requests (and memory limits) on everything
  • Startup/readiness/liveness probes
  • ≥ 2 replicas, PDB, topology spread
  • Graceful shutdown (SIGTERM, preStop)
  • Immutable image tags or digests

Security

  • SSO + least-privilege RBAC
  • Dedicated ServiceAccounts, token automount off
  • PSA restricted + policy engine
  • Default-deny NetworkPolicies
  • Secrets from a vault, encrypted at rest
  • Scanned, signed images; audit logs

Operations

  • Everything in Git (GitOps)
  • Metrics, logs, traces, SLO alerts
  • Autoscaling for pods and nodes
  • Backups tested (Velero, etcd)
  • Regular, rehearsed upgrades
  • Quotas and cost visibility per team
Glossary

Terms you will meet

PodOne or more containers scheduled togetherControllerLoop that drives actual state to desired state
ReplicaSetKeeps N identical pods runningOperatorCRD + controller with operational knowledge
ServiceStable virtual IP/DNS for a set of podsCRDCustom resource definition — new API type
EndpointSliceList of pod IPs behind a ServiceCNI / CSI / CRINetwork / storage / runtime plugin interfaces
Ingress / GatewayL7 entry points into the clusterkubeletNode agent that runs pods
PV / PVCStorage / a claim on storageetcdConsistent key-value store for cluster state
Taint / tolerationNode repels pods / pod accepts taintQoS classGuaranteed, Burstable, BestEffort
PDBLimit on voluntary disruptionsPSA / PSSPod Security Admission / Standards
HPA / VPAHorizontal / vertical pod autoscalingGitOpsGit as source of truth, pulled by an agent
Further reading

References

  • Kubernetes docs & tutorials — kubernetes.io/docs
  • API reference — kubernetes.io/docs/reference
  • Gateway API — gateway-api.sigs.k8s.io
  • Kubernetes the Hard Way — github.com/kelseyhightower/kubernetes-the-hard-way
  • CNCF certifications — training.linuxfoundation.org
  • Helm — helm.sh · Kustomize — kustomize.io
  • Argo CD — argo-cd.readthedocs.io · Flux — fluxcd.io
  • Kubebuilder book — book.kubebuilder.io

Kubernetes ships ~3 releases a year — check feature states and versions in the docs for your cluster version.

1 / 1