Architecture, workloads, configuration, networking, storage, scheduling, scaling, security, observability, delivery, extending, operations, and troubleshooting — with best practices and runnable examples.
Use ← → / Space to navigate · I contents · O overview · T theme · F fullscreen
Click any line to jump there · press I to return to this page
Why Kubernetes exists, how it is built, and your first cluster.
The problem it solves and the idea that makes it work.
Not a PaaS: it doesn't build your code, provide databases, or pick your logging/monitoring stack. It's a platform for building platforms — and it adds real operational complexity. A few containers may not need it.
"I want 3 replicas of web:1.4" — YAML sent to the API server
Object saved in etcd: spec = desired state
Controllers watch objects and the real world
Create / delete / update to close the gap
Write actual state to status; repeat forever
kubectl apply to a running podPending)Ready when containers + readiness gates pass| Option | Examples | Use for |
|---|---|---|
| Local / learning | kind, minikube, k3d, Docker Desktop, Rancher Desktop | Labs, dev loops, CI tests |
| Managed | Amazon EKS, Google GKE, Azure AKS, DigitalOcean, OKE | Production without running the control plane |
| Self-managed | kubeadm, Cluster API, Talos, RKE2 | On-prem, full control, learning internals (CKA) |
| Lightweight | k3s, k0s, MicroK8s | Edge, IoT, small footprints |
| Enterprise platforms | OpenShift, Rancher, Tanzu | Opinionated stacks with support |
About three minor releases per year; each minor gets roughly 14 months of patches upstream. Managed providers publish their own support calendars.
Learn on kind/minikube, build one cluster with kubeadm to understand internals, run production on a managed service.
# kind = Kubernetes IN Docker
cat > kind.yaml <<'EOF'
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
- role: control-plane
- role: worker
- role: worker
EOF
kind create cluster --name lab --config kind.yaml
kubectl cluster-info --context kind-lab
kubectl get nodes -o wide
kubectl get pods -n kube-system
# Deploy and reach an app
kubectl create deployment web --image=nginx:1.27 --replicas=3
kubectl expose deployment web --port=80
kubectl port-forward svc/web 8080:80
# → http://localhost:8080
kind delete cluster --name lab
kubectl — the CLI (within ±1 minor of the cluster)kind or minikube — local clustershelm, k9s (terminal UI), kubectx/kubenssource <(kubectl completion zsh)kubectl delete pod one web pod — watch it come backkubectl scale deploy web --replicas=6kubectl get pods -o wide — see them spread across workersapiVersion: apps/v1 # group/version
kind: Deployment # object type
metadata:
name: web
namespace: shop
labels: # identifying, selectable
app.kubernetes.io/name: web
app.kubernetes.io/part-of: shop
annotations: # non-identifying metadata
owner: team-checkout
spec: # desired state (you write)
replicas: 3
selector:
matchLabels: { app.kubernetes.io/name: web }
template:
metadata:
labels: { app.kubernetes.io/name: web }
spec:
containers:
- name: web
image: nginx:1.27
status: # actual state (controllers write)
readyReplicas: 3
app.kubernetes.io/*kubectl api-resources — every typekubectl explain deploy.spec.strategy — field docskubectl get deploy web -o yaml — the full object# Read
kubectl get pods -A -o wide
kubectl get deploy,svc,ing -n shop
kubectl get pods -l app.kubernetes.io/name=web
kubectl get pod web-abc -o jsonpath='{.status.podIP}'
kubectl describe pod web-abc
kubectl logs -f deploy/web --all-containers
kubectl get events --sort-by=.lastTimestamp
# Write (declarative)
kubectl apply -f k8s/ # create or update
kubectl diff -f k8s/ # preview changes
kubectl delete -f k8s/
# Generate YAML instead of writing it from scratch
kubectl create deploy api --image=api:1.0 \
--dry-run=client -o yaml > api.yaml
# Interact
kubectl exec -it web-abc -- sh
kubectl port-forward svc/web 8080:80
kubectl cp web-abc:/var/log/app.log ./app.log
# Rollouts
kubectl rollout status deploy/web
kubectl rollout history deploy/web
kubectl rollout undo deploy/web
kubectl rollout restart deploy/web
# Contexts (kubeconfig)
kubectl config get-contexts
kubectl config use-context prod
kubectl config set-context --current --namespace=shop
# Watch & resources
kubectl get pods -w
kubectl top pods --sort-by=memory
Imperative vs declarative: use kubectl create/run/expose to learn and generate YAML; manage real environments with files in Git + kubectl apply (or GitOps).
Workloads, configuration, networking, and storage.
Pods and the controllers that manage them.
localhost) and volumesrestartPolicy: Always) start first and run alongside the appPending → Running → Succeeded / Failed (or Unknown)Waiting, Running, TerminatedpreStop hook → SIGTERM → SIGKILL when the period ends (preStop time counts against it; a one-off 2 s extension if preStop overruns)apiVersion: v1
kind: Pod
metadata:
name: app
labels: { app: demo }
spec:
initContainers:
- name: log-shipper # native sidecar, starts first
image: fluent/fluent-bit:3.1
restartPolicy: Always
volumeMounts: [{ name: logs, mountPath: /logs }]
- name: wait-db # runs to completion
image: busybox:1.36
command: ["sh", "-c",
"until nc -z db 5432; do sleep 2; done"]
containers:
- name: app
image: ghcr.io/org/app:1.4.2
ports: [{ containerPort: 8080 }]
volumeMounts: [{ name: logs, mountPath: /var/log/app }]
volumes:
- name: logs
emptyDir: {}
apiVersion: apps/v1
kind: Deployment
metadata: { name: web }
spec:
replicas: 3
revisionHistoryLimit: 5
selector:
matchLabels: { app: web }
strategy:
type: RollingUpdate
rollingUpdate:
maxSurge: 25% # extra pods during rollout
maxUnavailable: 0 # never drop below 3 ready
minReadySeconds: 10
template:
metadata:
labels: { app: web }
spec:
containers:
- name: web
image: ghcr.io/org/web:1.4.2
readinessProbe:
httpGet: { path: /ready, port: 8080 }
template creates a new ReplicaSetrollout undoprogressDeadlineSeconds (600 s) marks it failed — no automatic rollbackRecreate strategy: kill all, then start new (downtime)latestmaxUnavailable: 0 for zero downtimedb-0, db-1, db-2db-0.db.ns.svc.cluster.localvolumeClaimTemplates — survives reschedulingpodManagementPolicy: Parallel)nodeSelector / tolerationsmaxUnavailablevolumeClaimTemplates:
- metadata: { name: data }
spec: { accessModes: ["ReadWriteOnce"], storageClassName: fast-ssd, resources: { requests: { storage: 50Gi } } }
apiVersion: batch/v1
kind: CronJob
metadata: { name: nightly-report }
spec:
schedule: "0 2 * * *"
timeZone: Asia/Ho_Chi_Minh
concurrencyPolicy: Forbid # Allow | Forbid | Replace
startingDeadlineSeconds: 300
successfulJobsHistoryLimit: 3
failedJobsHistoryLimit: 3
jobTemplate:
spec:
backoffLimit: 3 # retries (default 6)
activeDeadlineSeconds: 3600
ttlSecondsAfterFinished: 86400
template:
spec:
restartPolicy: OnFailure
containers:
- name: report
image: ghcr.io/org/report:2.1.0
args: ["--date", "yesterday"]
completions + parallelism for fan-out workcompletionMode: Indexed — each pod gets an indexrestartPolicy must be OnFailure or NeverpodFailurePolicy to fail fast on non-retriable exit codesttlSecondsAfterFinished to clean up| Need | Use | Why |
|---|---|---|
| Stateless web/API service | Deployment | Interchangeable replicas, rolling updates |
| Database, broker, clustered app | StatefulSet (or an operator) | Stable identity and storage per replica |
| Agent on every node | DaemonSet | Follows nodes automatically |
| Run to completion once | Job | Retries, parallelism, completion tracking |
| Run on a schedule | CronJob | Creates Jobs on a cron schedule |
| Helper next to the app | Sidecar container | Proxies, log shipping, config reloaders |
| One-off debugging | kubectl run / kubectl debug | Temporary, not managed |
Bare Pods are not recreated if their node dies. Always use a controller in real environments.
apiVersion: v1
kind: ConfigMap
metadata: { name: web-config }
data:
LOG_LEVEL: info
app.yaml: |
cache: { ttl: 60 }
---
apiVersion: v1
kind: Secret
metadata: { name: web-secrets }
type: Opaque
stringData: # stored base64 — NOT encrypted
DB_PASSWORD: change-me
---
# in the container spec
envFrom:
- configMapRef: { name: web-config }
env:
- name: DB_PASSWORD
valueFrom:
secretKeyRef: { name: web-secrets, key: DB_PASSWORD }
volumeMounts:
- { name: cfg, mountPath: /etc/web, readOnly: true }
# in the pod spec
volumes:
- { name: cfg, configMap: { name: web-config } }
subPath)immutable: true protects and speeds up large configssecrets500m = 0.5 CPU; 256Mi = 256 MiB| QoS | Rule | Evicted |
|---|---|---|
| Guaranteed | Every container: CPU & memory request = limit | Last |
| Burstable | At least one request or limit set | Middle |
| BestEffort | Nothing set | First |
resources:
requests: { cpu: 250m, memory: 256Mi }
limits: { memory: 256Mi } # no CPU limit
LimitRange, totals with ResourceQuota# container spec
startupProbe: # slow boot protection
httpGet: { path: /healthz, port: 8080 }
periodSeconds: 5
failureThreshold: 30 # up to 150 s
readinessProbe: # gets traffic?
httpGet: { path: /ready, port: 8080 }
periodSeconds: 5
failureThreshold: 3
livenessProbe: # restart if stuck
httpGet: { path: /healthz, port: 8080 }
periodSeconds: 10
failureThreshold: 3
lifecycle:
preStop:
sleep: { seconds: 10 } # let LBs drain
# pod spec
terminationGracePeriodSeconds: 45
httpGet, tcpSocket, grpc, execSIGTERM handling → dropped requests on every deployPod network, Services, DNS, Ingress, Gateway API, NetworkPolicy.
localhost| Type | Reachable from |
|---|---|
ClusterIP (default) | Inside the cluster only |
NodePort | Every node's IP on a port (30000–32767) |
LoadBalancer | Cloud LB → NodePort/pods |
ExternalName | DNS CNAME to an outside name |
Headless (clusterIP: None) | DNS returns pod IPs directly |
port = Service port; targetPort = container portapiVersion: v1
kind: Service
metadata: { name: web }
spec:
type: ClusterIP
selector: { app: web }
ports:
- name: http
port: 80
targetPort: 8080
---
# Debug: does the Service have endpoints?
kubectl get endpointslices -l kubernetes.io/service-name=web
kubectl get pods -l app=web --show-labels
# Test from inside the cluster
kubectl run tmp --rm -it --image=busybox:1.36 -- \
wget -qO- http://web.default.svc.cluster.local
<svc>.<ns>.svc.cluster.local<svc><svc>.<ns><pod>.<headless-svc>.<ns>.svc.cluster.localndots:5 → external names are tried with every search suffix first → extra queriesndots via dnsConfigkubectl run dns --rm -it --image=busybox:1.36 -- sh -c 'cat /etc/resolv.conf; nslookup web.shop; nslookup kubernetes.default'
GatewayClass (infra) → Gateway (platform) → HTTPRoute (app teams)GRPCRoute; cross-namespace refs via ReferenceGrantapiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata: { name: public, namespace: infra }
spec:
gatewayClassName: envoy
listeners:
- name: https
protocol: HTTPS
port: 443
tls: { certificateRefs: [{ name: wildcard-tls }] }
allowedRoutes: { namespaces: { from: All } }
---
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata: { name: shop, namespace: shop }
spec:
parentRefs: [{ name: public, namespace: infra }]
hostnames: ["shop.example.com"]
rules:
- matches: [{ path: { type: PathPrefix, value: /api } }]
backendRefs:
- { name: api-v1, port: 80, weight: 90 }
- { name: api-v2, port: 80, weight: 10 } # canary
# 1. Deny everything in the namespace
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata: { name: default-deny, namespace: shop }
spec:
podSelector: {}
policyTypes: [Ingress, Egress]
---
# 2. Allow api ← web on 8080, and api → DNS
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata: { name: api, namespace: shop }
spec:
podSelector: { matchLabels: { app: api } }
policyTypes: [Ingress, Egress]
ingress:
- from: [{ podSelector: { matchLabels: { app: web } } }]
ports: [{ port: 8080 }]
egress:
- to: [{ namespaceSelector: {} }]
ports:
- { port: 53, protocol: UDP }
- { port: 53, protocol: TCP }
Default-deny every app namespace, allow explicitly, and test with a throwaway pod. Use the CNI's tooling (Cilium Hubble, Calico) to see dropped flows.
emptyDir — scratch space shared by containers; gone with the pod (medium: Memory for tmpfs)configMap, secret, downwardAPI, projected — config as fileshostPath — node filesystem; avoid in apps (security, portability)| Access mode | Meaning | Reclaim policy | Meaning |
|---|---|---|---|
ReadWriteOnce | Read-write by one node | Delete | Delete the disk with the PVC (dynamic default) |
ReadWriteOncePod | Read-write by one pod | Retain | Keep the disk; manual cleanup |
ReadOnlyMany / ReadWriteMany | Many nodes (needs a shared filesystem) |
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata: { name: fast-ssd }
provisioner: ebs.csi.aws.com # your CSI driver
parameters: { type: gp3 }
reclaimPolicy: Retain
volumeBindingMode: WaitForFirstConsumer
allowVolumeExpansion: true
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata: { name: data }
spec:
accessModes: [ReadWriteOnce]
storageClassName: fast-ssd
resources: { requests: { storage: 20Gi } }
---
# Pod spec
volumes:
- name: data
persistentVolumeClaim: { claimName: data }
BoundWaitForFirstConsumer provisions in the zone where the pod landsRetain for important dataWhere pods land and how capacity follows demand.
Controlling where pods run.
spec:
nodeSelector: # simple, hard
kubernetes.io/arch: arm64
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- { key: node.kubernetes.io/instance-type,
operator: In, values: [m7g.large, m7g.xlarge] }
podAntiAffinity: # spread replicas
preferredDuringSchedulingIgnoredDuringExecution:
- weight: 100
podAffinityTerm:
topologyKey: kubernetes.io/hostname
labelSelector: { matchLabels: { app: web } }
tolerations: # allowed on GPU nodes
- { key: nvidia.com/gpu, operator: Exists, effect: NoSchedule }
required… = hard; preferred… = soft (weighted)NoSchedule — no new pods without a tolerationPreferNoSchedule — avoid if possibleNoExecute — also evicts running podskubectl taint nodes n1 dedicated=gpu:NoSchedulemaxSkew, topologyKey, whenUnsatisfiablePriorityClass ranks podssystem-cluster-critical, system-node-criticalminAvailable or maxUnavailabletopologySpreadConstraints:
- { maxSkew: 1, topologyKey: topology.kubernetes.io/zone, whenUnsatisfiable: DoNotSchedule, labelSelector: { matchLabels: { app: web } } }
---
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata: { name: web }
spec: { maxUnavailable: 1, selector: { matchLabels: { app: web } } }
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata: { name: web }
spec:
scaleTargetRef:
{ apiVersion: apps/v1, kind: Deployment, name: web }
minReplicas: 3
maxReplicas: 30
metrics:
- type: Resource
resource:
name: cpu
target: { type: Utilization, averageUtilization: 70 }
behavior:
scaleDown:
stabilizationWindowSeconds: 300
policies: [{ type: Percent, value: 50, periodSeconds: 60 }]
desired = ceil(current × metric / target)Requests must be set (utilization = usage / request) · scale on the real bottleneck · minReplicas ≥ 2 · slow scale-down · load-test the scaling path.
Protect the cluster and see what it is doing.
AuthN, RBAC, service accounts, admission, pod hardening, secrets, audit.
Role / ClusterRole — verbs on resourcesRoleBinding / ClusterRoleBinding — grant to users, groups, SAskubectl auth can-i create deploy -n shop --as janeapiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata: { name: app-reader, namespace: shop }
rules:
- apiGroups: [""]
resources: ["configmaps"]
verbs: ["get", "list", "watch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata: { name: api-reads-config, namespace: shop }
subjects:
- kind: ServiceAccount
name: api
namespace: shop
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: Role
name: app-reader
---
# In the pod spec
serviceAccountName: api
automountServiceAccountToken: false # unless it calls the API
| PSS level | Summary |
|---|---|
| privileged | Unrestricted — system components only |
| baseline | Blocks privileged, host namespaces, hostPath, dangerous caps |
| restricted | + non-root, drop ALL caps, no privilege escalation, seccomp |
kubectl label ns shop \
pod-security.kubernetes.io/enforce=restricted \
pod-security.kubernetes.io/warn=restricted
Allowed registries · no latest · required labels and requests · signed images · no LoadBalancer Services without approval. Start in audit mode, then enforce.
spec:
serviceAccountName: api
automountServiceAccountToken: false
securityContext:
runAsNonRoot: true
runAsUser: 10001
fsGroup: 10001
seccompProfile: { type: RuntimeDefault }
containers:
- name: api
image: ghcr.io/org/api@sha256:<digest>
securityContext:
allowPrivilegeEscalation: false
readOnlyRootFilesystem: true
capabilities: { drop: ["ALL"] }
resources:
requests: { cpu: 200m, memory: 256Mi }
limits: { memory: 256Mi }
volumeMounts: [{ name: tmp, mountPath: /tmp }]
volumes:
- { name: tmp, emptyDir: {} }
restricted + policy enginekubectl top and HPA (not for history)kube-prometheus-stack Helm chart bundles it with Grafana & Alertmanager/metricsCrash-looping pods · pods pending too long · deployments with unavailable replicas · nodes NotReady / under pressure · PVCs filling up · CPU throttling · API server errors & latency · certificate expiry · SLO burn rate for user-facing services.
| Question | PromQL |
|---|---|
| Which pods restart? | increase(kube_pod_container_status_restarts_total[1h]) > 0 |
| Pending pods per namespace | sum by (namespace) (kube_pod_status_phase{phase="Pending"}) |
| Deployments missing replicas | kube_deployment_status_replicas_available < kube_deployment_spec_replicas |
| Memory used vs limit | container_memory_working_set_bytes{container!=""} / on(namespace,pod,container) kube_pod_container_resource_limits{resource="memory"} |
| CPU throttling ratio | rate(container_cpu_cfs_throttled_periods_total[5m]) / rate(container_cpu_cfs_periods_total[5m]) |
| Nodes not ready | kube_node_status_condition{condition="Ready",status="true"} == 0 |
| PVC usage | kubelet_volume_stats_used_bytes / kubelet_volume_stats_capacity_bytes |
| API server error rate | sum(rate(apiserver_request_total{code=~"5.."}[5m])) / sum(rate(apiserver_request_total[5m])) |
Ship changes safely and extend the API.
Helm, Kustomize, GitOps, deployment strategies.
values.yaml; versioned, shareable (OCI registries)helm repo add prometheus-community \
https://prometheus-community.github.io/helm-charts
helm upgrade --install mon \
prometheus-community/kube-prometheus-stack \
-n monitoring --create-namespace -f values.yaml
helm list -A
helm template ./chart -f values-prod.yaml # render
helm rollback mon 2
base/ plus per-env overlays/kubectl apply -k overlays/prod# overlays/prod/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
namespace: shop-prod
resources: [../../base]
images:
- { name: ghcr.io/org/web, newTag: "1.4.2" }
patches:
- target: { kind: Deployment, name: web }
patch: |-
- op: replace
path: /spec/replicas
value: 6
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata: { name: shop-prod, namespace: argocd }
spec:
project: default
source:
repoURL: https://github.com/org/deploy.git
targetRevision: main
path: shop/overlays/prod
destination:
server: https://kubernetes.default.svc
namespace: shop-prod
syncPolicy:
automated: { prune: true, selfHeal: true }
syncOptions: [CreateNamespace=true]
selfHeal)App code repo builds and pushes images (CI) → updates the image tag in a deploy repo (PR or image updater) → Argo CD / Flux syncs. CI never needs cluster credentials.
Backward-compatible database migrations (expand → migrate → contract) · feature flags · readiness probes that reflect real readiness · automated analysis of error rate and latency before promotion.
CRDs, controllers, operators.
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata: { name: databases.example.com }
spec:
group: example.com
scope: Namespaced
names: { kind: Database, plural: databases, singular: database }
versions:
- name: v1
served: true
storage: true
schema:
openAPIV3Schema:
type: object
properties:
spec:
type: object
properties:
engine: { type: string, enum: [postgres, mysql] }
storageGi: { type: integer, minimum: 1 }
---
apiVersion: example.com/v1
kind: Database
metadata: { name: orders }
spec: { engine: postgres, storageGi: 20 }
status subresource, conditions, observedGenerationDay-2 work, troubleshooting, and the road to hero.
Upgrades, node maintenance, etcd, multi-tenancy.
# 0. Release notes, removed APIs (pluto/kubent); point the
# pkgs.k8s.io apt repo at the new minor (v1.37)
# 1. First control-plane node
apt-mark unhold kubeadm && apt-get update
apt-get install -y kubeadm='1.37.x-*' && apt-mark hold kubeadm
kubeadm upgrade plan
kubeadm upgrade apply v1.37.x
# 2. Other control planes AND workers: kubeadm pkg first
# (the kubeadm install lines above), then
kubeadm upgrade node
# 3. Every node: drain → kubelet/kubectl → uncordon
kubectl drain node-1 --ignore-daemonsets --delete-emptydir-data
apt-mark unhold kubelet kubectl
apt-get install -y kubelet='1.37.x-*' kubectl='1.37.x-*'
apt-mark hold kubelet kubectl
systemctl daemon-reload && systemctl restart kubelet
kubectl uncordon node-1
Upgrade non-prod first, keep PDBs on every service, back up etcd before starting, and upgrade regularly — skipping versions makes each upgrade riskier.
Note: the official Linux worker-node guide (updated 2026-08-29) upgrades kubeadm and kubectl together before kubeadm upgrade node, upgrades only kubelet after the drain, and runs drain/uncordon from a control-plane node. Both work — follow the docs for your exact version.
# Node maintenance
kubectl cordon node-2 # no new pods
kubectl drain node-2 --ignore-daemonsets \
--delete-emptydir-data # evicts, honoring PDBs
kubectl uncordon node-2
# etcd snapshot (self-managed control plane)
ETCDCTL_API=3 etcdctl snapshot save /backup/etcd.db \
--endpoints=https://127.0.0.1:2379 \
--cacert=/etc/kubernetes/pki/etcd/ca.crt \
--cert=/etc/kubernetes/pki/etcd/server.crt \
--key=/etc/kubernetes/pki/etcd/server.key
etcdutl snapshot status /backup/etcd.db --write-out=table
# Restore into a new data dir, then point etcd at it
etcdutl snapshot restore /backup/etcd.db \
--data-dir /var/lib/etcd-restore
# kubeadm certificates (renewed on upgrade)
kubeadm certs check-expiration
You can't snapshot etcd yourself — rely on GitOps, Velero, and the provider's control-plane resilience.
edit in their namespaces onlyResourceQuota: total CPU, memory, pods, PVCs, LoadBalancersLimitRange: default and max requests/limits per containerrestrictedapiVersion: v1
kind: ResourceQuota
metadata: { name: team-a, namespace: team-a }
spec:
hard:
requests.cpu: "20"
requests.memory: 64Gi
limits.memory: 64Gi
pods: "200"
services.loadbalancers: "2"
---
apiVersion: v1
kind: LimitRange
metadata: { name: defaults, namespace: team-a }
spec:
limits:
- type: Container
defaultRequest: { cpu: 100m, memory: 128Mi }
default: { memory: 256Mi }
max: { cpu: "4", memory: 8Gi }
Namespaces give soft isolation. For hostile tenants use separate clusters (or virtual clusters such as vCluster) and separate cloud accounts.
kubectl get — what state is it in?
describe — what did controllers say?
logs, --previous — what did the app say?
exec / debug — test from the pod
Node, CNI, DNS, storage, control plane
kubectl get pods -n shop -o wide
kubectl describe pod web-abc -n shop | tail -20
kubectl logs web-abc -n shop -c web --previous
kubectl get events -n shop --sort-by=.lastTimestamp
# Ephemeral debug container (works on distroless)
kubectl debug -it web-abc -n shop \
--image=nicolaka/netshoot --target=web
# Copy of the pod with a different command
kubectl debug web-abc -n shop -it \
--copy-to=web-dbg --container=web -- sh
# Shell on a node
kubectl debug node/node-1 -it --image=ubuntu
Pending → scheduler events)| Symptom | Typical cause | Where to look |
|---|---|---|
Pending | Not enough CPU/memory, taints, affinity, unbound PVC | Scheduler events in describe |
ImagePullBackOff | Wrong name/tag, private registry auth, rate limits | Events; imagePullSecrets |
CrashLoopBackOff | App exits: bad config, missing dependency, failing liveness | logs --previous, exit code |
OOMKilled (exit 137) | Memory limit too low or leak | describe last state; memory metrics |
CreateContainerConfigError | Missing ConfigMap/Secret or key | Events |
| Running but not Ready | Readiness probe failing | Probe config, app health endpoint |
| Service unreachable | Selector mismatch → no endpoints; wrong targetPort | get endpointslices, labels |
| Intermittent DNS timeouts | CoreDNS overloaded, ndots, conntrack | CoreDNS logs/metrics |
Evicted | Node pressure (disk, memory) | Node conditions, kubelet logs |
Stuck Terminating | Finalizers, unresponsive node | metadata.finalizers, node status |
kubectl apply --server-side): the API server tracks which manager owns each field and reports conflictskube-node-leaseresize subresourcekubernetes.io/os selectorskubectl apply --server-side --field-manager=platform -f deploy.yaml
kubectl get leases -n kube-system
kubectl patch pod web-abc --subresource resize --patch '{"spec":{"containers":[{"name":"web","resources":{"requests":{"cpu":"500m"}}}]}}'
| Need | Common choice | What it does |
|---|---|---|
| TLS certificates | cert-manager | Issues and renews certificates (e.g. Let's Encrypt) as Secrets |
| DNS records | ExternalDNS | Creates DNS records for Services/Ingresses/Gateways |
| Secrets from a vault | External Secrets Operator, Sealed Secrets | Sync secrets from AWS/GCP/Vault; encrypted secrets safe for Git |
| Service-to-service security | Istio, Linkerd, Cilium | mTLS, retries, traffic splitting, telemetry |
| Policy | Kyverno, Gatekeeper | Admission guardrails, image verification |
| Supply chain | cosign (Sigstore), Trivy, Syft | Sign, scan, SBOM |
| Backup | Velero | Back up resources and volumes |
| Cost | OpenCost / Kubecost | Cost per namespace, workload, label |
| Many clusters | Cluster API, Argo CD ApplicationSets | Declarative cluster lifecycle; fleet deployments |
--dry-run=client -o yaml, kubectl explainTwenty labs on one kind cluster: every chapter, with a break-it-and-fix-it step in each.
Recreate the three-node kind cluster from Getting started, plus metrics-server for top and HPA.
# kind v0.33 defaults to Kubernetes 1.37 (kindest/node:v1.37.0)
cat > kind.yaml <<'EOF'
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
- role: control-plane
- role: worker
- role: worker
EOF
kind create cluster --name lab --config kind.yaml
# metrics-server (kubelet certs on kind are self-signed)
MS=https://github.com/kubernetes-sigs/metrics-server/releases/latest/download
kubectl apply -f $MS/components.yaml
kubectl -n kube-system patch deploy metrics-server --type=json -p \
'[{"op":"add","path":"/spec/template/spec/containers/0/args/-",
"value":"--kubelet-insecure-tls"}]'
kubectl top nodes # works after ~1 minutekind, kubectl (±1 minor of the cluster), helm, Docker or Podman.kubectl create ns lab3 && kubectl config set-context --current --namespace lab3.standard local-path StorageClass.kind delete cluster --name lab and recreate in about a minute.Goal: see that the control plane is ordinary pods, that state lives in etcd, and how the scheduler decides.
# static pod manifests, read by the kubelet on the node
docker exec lab-control-plane ls /etc/kubernetes/manifests
kubectl -n kube-system get pods -o wide | grep control-plane
# peek at the raw keys in etcd
kubectl create deployment web --image=nginx:1.27 --replicas=3
kubectl -n kube-system exec etcd-lab-control-plane -- etcdctl \
--endpoints=https://127.0.0.1:2379 \
--cacert=/etc/kubernetes/pki/etcd/ca.crt \
--cert=/etc/kubernetes/pki/etcd/server.crt \
--key=/etc/kubernetes/pki/etcd/server.key \
get /registry/deployments/default/web --keys-only
# watch controllers and the scheduler react
kubectl get events -w --field-selector involvedObject.kind=Pod &
kubectl scale deploy web --replicas=5etcd, kube-apiserver, kube-controller-manager, kube-scheduler.Scheduled (scheduler), then Pulled, Created, Started (kubelet).docker exec lab-control-plane mv /etc/kubernetes/manifests/kube-scheduler.yaml /tmp/ then scale up: new pods stay Pending with no node.Goal: control how a Deployment replaces pods, and see a bad readiness probe stop a rollout.
kubectl create deployment web --image=nginx:1.27 --replicas=4
kubectl patch deploy web -p '{"spec":{"strategy":{"rollingUpdate":
{"maxSurge":1,"maxUnavailable":0}}}}'
kubectl set image deploy/web nginx=nginx:1.28
kubectl annotate deploy/web kubernetes.io/change-cause="nginx 1.28"
kubectl rollout status deploy/web
kubectl rollout history deploy/web
# add a readiness probe that can never pass
kubectl patch deploy web --type=json -p '[{"op":"add",
"path":"/spec/template/spec/containers/0/readinessProbe",
"value":{"httpGet":{"path":"/nope","port":80},"periodSeconds":2}}]'
kubectl get pods -w # new pod never becomes Ready
kubectl rollout undo deploy/webmaxUnavailable: 0 the old ReplicaSet keeps all 4 pods serving while the bad pod is stuck 0/1 Running.kubectl get rs shows two ReplicaSets during a rollout.kubectl describe pod shows Readiness probe failed: HTTP probe failed with statuscode: 404./nope instead: the container restarts repeatedly. Compare the two failure modes.progressDeadlineSeconds (600 s) the Progressing condition turns False with ProgressDeadlineExceeded and rollout status exits non-zero.Goal: run the other controllers and see how each one names, orders, and places pods.
kubectl create job pi --image=perl:5.40 -- \
perl -Mbignum=bpi -wle 'print bpi(200)'
kubectl wait --for=condition=complete job/pi --timeout=3m && kubectl logs job/pi
kubectl create cronjob tick --image=busybox:1.37 \
--schedule="*/1 * * * *" -- date
kubectl get jobs -w # a new Job every minute
# StatefulSet + the headless Service it requires
kubectl create service clusterip db --clusterip=None --tcp=80:80
cat <<'EOF' | kubectl apply -f -
apiVersion: apps/v1
kind: StatefulSet
metadata: {name: db}
spec:
serviceName: db
replicas: 3
selector: {matchLabels: {app: db}}
template:
metadata: {labels: {app: db}}
spec: {containers: [{name: nginx, image: "nginx:1.27"}]}
EOF
kubectl get pods -w -l app=db # db-0, then db-1, then db-2kubectl create service labels its selector app=db, matching the pods.db-1: it comes back with the same name; nslookup db-1.db resolves from another pod.db-2 first).busybox sleeper; it lands on both workers but not the control plane until you add a toleration for node-role.kubernetes.io/control-plane:NoSchedule.backoffLimit: 2 on a Job whose command exits 1 and watch it fail.Goal: learn which consumption method picks up changes, and how Secrets are really stored.
kubectl create configmap app --from-literal=COLOR=blue
kubectl create secret generic db --from-literal=PASSWORD=s3cr3t
cat <<'EOF' | kubectl apply -f -
apiVersion: v1
kind: Pod
metadata: {name: cfg}
spec:
containers:
- name: c
image: busybox:1.37
command: ["sh","-c","while true; do echo $COLOR $(cat /cfg/COLOR);
sleep 5; done"]
envFrom: [{configMapRef: {name: app}}]
volumeMounts: [{name: cfg, mountPath: /cfg}]
volumes: [{name: cfg, configMap: {name: app}}]
EOF
kubectl create configmap app --from-literal=COLOR=green \
-o yaml --dry-run=client | kubectl apply -f -
kubectl logs -f cfg # file changes after ~1 min; env stays bluesubPath mount never updates. Test it.kubectl get secret db -o jsonpath={.data.PASSWORD} | base64 -d: Secrets are base64-encoded, not encrypted.immutable: true and try to edit it.Goal: see how requests and limits set a pod’s QoS class and what happens at the memory limit.
cat <<'EOF' | kubectl apply -f -
apiVersion: v1
kind: Pod
metadata: {name: hog}
spec:
containers:
- name: hog
image: busybox:1.37
command: ["sh","-c","tail /dev/zero"] # grows memory forever
resources:
requests: {cpu: 100m, memory: 64Mi}
limits: {cpu: 200m, memory: 64Mi}
EOF
kubectl get pod hog -w
kubectl describe pod hog | grep -A3 'Last State'
kubectl get pod hog -o jsonpath='{.status.qosClass}'; echoOOMKilled → CrashLoopBackOff; Last State shows Reason: OOMKilled, exit code 137.Guaranteed; only requests → Burstable; nothing → BestEffort.kubectl top pod.Pending with Insufficient cpu.Goal: trace a request from a Service name to pod IPs, and see readiness control membership.
kubectl create deployment web --image=nginx:1.27 --replicas=3
kubectl expose deploy web --port=80
kubectl get endpointslices -l kubernetes.io/service-name=web
kubectl run dnsutils \
--image=registry.k8s.io/e2e-test-images/jessie-dnsutils:1.3 \
--restart=Never -- sleep infinity
kubectl exec dnsutils -- nslookup web
kubectl exec dnsutils -- nslookup web.default.svc.cluster.local
kubectl exec dnsutils -- cat /etc/resolv.conf # search domains, ndots:5
kubectl run tmp --rm -it --image=busybox:1.37 --restart=Never -- wget -qO- web
# NodePort: reach it through any node's IP
kubectl patch svc web -p '{"spec":{"type":"NodePort"}}'
kubectl get svc web # note the 30000-32767 port
docker exec lab-worker curl -s localhost:<nodePort> | head -4kubectl get pods -o wide matches./usr/share/nginx/html/index.html, with a readiness probe on /): its IP leaves the slice.kubectl patch svc web -p '{"spec":{"selector":{"app":"wbe"}}}' → empty slice, requests get Connection refused. Fix it.type: LoadBalancer stays <pending> on kind unless you run cloud-provider-kind.Goal: route HTTP by host and path with a Gateway API implementation (Envoy Gateway).
helm install eg oci://docker.io/envoyproxy/gateway-helm --version v1.9.2 \
-n envoy-gateway-system --create-namespace
kubectl wait --timeout=5m -n envoy-gateway-system \
deployment/envoy-gateway --for=condition=Available
# GatewayClass "eg", Gateway "eg", and a sample backend
EG=https://github.com/envoyproxy/gateway/releases/download/v1.9.2
kubectl apply -f $EG/quickstart.yaml -n default
cat <<'EOF' | kubectl apply -f -
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata: {name: web}
spec:
parentRefs: [{name: eg}]
hostnames: ["web.example.com"]
rules: [{backendRefs: [{name: web, port: 80}]}]
EOFkubectl get svc -n envoy-gateway-system -l gateway.envoyproxy.io/owning-gateway-name=eg, then kubectl -n envoy-gateway-system port-forward svc/<name> 8888:80.curl -H "Host: web.example.com" localhost:8888 returns the nginx page; -H "Host: www.example.com" hits the sample backend.kubectl get httproute web -o yaml: Accepted and ResolvedRefs conditions are True.backendRefs at port: 81 → ResolvedRefs false, HTTP 500. Fix and re-check.Goal: lock a namespace down and open exactly the paths you need, including DNS.
kubectl create ns np && kubectl -n np create deployment api --image=nginx:1.27
kubectl -n np expose deploy api --port=80
kubectl -n np run client --image=busybox:1.37 --labels=role=client -- sleep 1d
kubectl -n np exec client -- wget -qO- -T 3 api | head -2 # works
cat <<'EOF' | kubectl -n np apply -f -
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata: {name: default-deny}
spec: {podSelector: {}, policyTypes: [Ingress, Egress]}
EOF
kubectl -n np exec client -- wget -qO- -T 3 api # DNS blocked tooto: [{namespaceSelector: {matchLabels: {kubernetes.io/metadata.name: kube-system}}, podSelector: {matchLabels: {k8s-app: kube-dns}}}] on UDP and TCP 53.role=client to app=api on TCP 80.app=api from role=client only.Goal: provision storage dynamically and prove data survives pod deletion.
kubectl get storageclass # standard (default), WaitForFirstConsumer
cat <<'EOF' | kubectl apply -f -
apiVersion: v1
kind: PersistentVolumeClaim
metadata: {name: data}
spec:
accessModes: [ReadWriteOnce]
resources: {requests: {storage: 1Gi}}
EOF
kubectl get pvc data # Pending until a pod uses it
kubectl run writer --image=busybox:1.37 --overrides='{"spec":{
"volumes":[{"name":"d","persistentVolumeClaim":{"claimName":"data"}}],
"containers":[{"name":"writer","image":"busybox:1.37",
"command":["sh","-c","date >> /d/log; sleep 1d"],
"volumeMounts":[{"name":"d","mountPath":"/d"}]}]}}'
kubectl get pvc,pv # Bound; PV created on the pod's node
kubectl delete pod writer # then rerun the same kubectl run/d/log gains a second line: the data outlived the pod.Pending (volume node affinity conflict).volumeClaimTemplates to the Lab 3 StatefulSet: each replica gets data-db-0, data-db-1… and keeps it across restarts and scale-down.reclaimPolicy: Delete. Patch a PV to Retain first and compare.Goal: steer pods onto nodes, keep them off others, and spread replicas evenly.
kubectl label node lab-worker disk=ssd
kubectl taint node lab-worker2 dedicated=batch:NoSchedule
kubectl create deployment fast --image=nginx:1.27 --replicas=4
kubectl patch deploy fast -p '{"spec":{"template":{"spec":
{"nodeSelector":{"disk":"ssd"}}}}}'
kubectl get pods -o wide -l app=fast # all on lab-worker
# spread evenly across nodes instead
kubectl patch deploy fast --type=json -p '[
{"op":"remove","path":"/spec/template/spec/nodeSelector"},
{"op":"add","path":"/spec/template/spec/topologySpreadConstraints",
"value":[{"maxSkew":1,"topologyKey":"kubernetes.io/hostname",
"whenUnsatisfiable":"ScheduleAnyway",
"labelSelector":{"matchLabels":{"app":"fast"}}}]}]'
kubectl get pods -o wide -l app=fast # lab-worker2: still tainteddedicated=batch:NoSchedule: pods may now land on lab-worker2, but a toleration never attracts them; use affinity for that.whenUnsatisfiable: DoNotSchedule with the taint still in place and scale to 6: some pods go Pending. Read the scheduler message in kubectl describe.nodeSelector with requiredDuringScheduling… node affinity using operator: In.kubectl taint node lab-worker2 dedicated- and kubectl label node lab-worker disk-.Goal: watch the scheduler evict lower-priority pods to make room for a critical one.
kubectl create priorityclass critical --value=1000000 \
--description="lab: preempts others"
kubectl create priorityclass low --value=100
# fill the workers with low-priority pods that reserve CPU
kubectl create deployment filler --image=nginx:1.27 --replicas=6
kubectl set resources deploy filler --requests=cpu=500m
kubectl patch deploy filler -p '{"spec":{"template":{"spec":
{"priorityClassName":"low"}}}}'
kubectl get pods -l app=filler # some Pending: nodes are full
kubectl create deployment vip --image=nginx:1.27
kubectl set resources deploy vip --requests=cpu=1
kubectl patch deploy vip -p '{"spec":{"template":{"spec":
{"priorityClassName":"critical"}}}}'
kubectl get events --field-selector reason=Preemptedvip becomes Running; some filler pods are Preempted (deleted) and their replacements wait in Pending.kubectl get pod <vip> -o jsonpath={.spec.priority} shows 1000000.preemptionPolicy is immutable: create a new class with preemptionPolicy: Never, use it for vip, and it now waits in the queue instead.system-cluster-critical and system-node-critical protect add-ons; never give apps those.Goal: scale a CPU-bound app on demand and understand the HPA math.
kubectl create deployment php-apache --image=registry.k8s.io/hpa-example
kubectl set resources deploy php-apache --requests=cpu=200m --limits=cpu=500m
kubectl expose deploy php-apache --port=80
cat <<'EOF' | kubectl apply -f -
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata: {name: php-apache}
spec:
scaleTargetRef: {apiVersion: apps/v1, kind: Deployment, name: php-apache}
minReplicas: 1
maxReplicas: 10
metrics:
- type: Resource
resource: {name: cpu, target: {type: Utilization, averageUtilization: 50}}
EOF
kubectl run -it --rm load --image=busybox:1.37 --restart=Never -- \
sh -c "while sleep 0.01; do wget -q -O- http://php-apache; done"
kubectl get hpa php-apache -w # in a second terminaldesired = ceil(current × currentUtilization / target).<unknown> and the HPA cannot act. kubectl describe hpa explains why.behavior.scaleDown.stabilizationWindowSeconds: 30 and compare.Goal: give a workload identity exactly the API access it needs, and prove it.
kubectl create ns dev
kubectl -n dev create serviceaccount app
kubectl -n dev create role pod-reader --verb=get,list,watch --resource=pods
kubectl -n dev create rolebinding app-reads-pods --role=pod-reader \
--serviceaccount=dev:app
SA=system:serviceaccount:dev:app
kubectl auth can-i list pods -n dev --as=$SA # yes
kubectl auth can-i delete pods -n dev --as=$SA # no
kubectl auth can-i list pods -n default --as=$SA # no
# short-lived, audience-bound token (TokenRequest API)
TOKEN=$(kubectl -n dev create token app --duration=10m)
kubectl --token="$TOKEN" -n dev get pods # allowed
kubectl --token="$TOKEN" get nodes # Forbiddenexp, aud, and the SA name.serviceAccountName: app gets a projected token at /var/run/secrets/kubernetes.io/serviceaccount/token that the kubelet rotates.can-i: now yes. That is the blast-radius difference.kubectl auth can-i --list --as=system:serviceaccount:dev:app -n dev shows every allowed verb.automountServiceAccountToken: false on pods that never call the API.Goal: enforce the restricted profile on a namespace and make a pod compliant.
kubectl create ns secure
kubectl label ns secure pod-security.kubernetes.io/enforce=restricted \
pod-security.kubernetes.io/warn=restricted
kubectl -n secure run bad --image=nginx:1.27 # rejected, lists violations
cat <<'EOF' | kubectl -n secure apply -f -
apiVersion: v1
kind: Pod
metadata: {name: good}
spec:
securityContext:
runAsNonRoot: true
seccompProfile: {type: RuntimeDefault}
containers:
- name: web
image: nginxinc/nginx-unprivileged:1.27 # listens on 8080 as UID 101
securityContext:
allowPrivilegeEscalation: false
capabilities: {drop: [ALL]}
EOFallowPrivilegeEscalation, capabilities, runAsNonRoot, seccompProfile.readOnlyRootFilesystem: true: nginx fails to start until you mount emptyDir volumes for /tmp and its cache path.kubectl label --dry-run=server --overwrite ns default pod-security.kubernetes.io/enforce=restricted prints warnings for existing pods.Goal: build the habit of reading the cluster’s own signals before reaching for tools.
kubectl logs deploy/web --all-containers --since=10m
kubectl logs <pod> --previous # the crashed container's logs
kubectl get events --sort-by=.lastTimestamp | tail -20
kubectl top pods --sort-by=memory -A | head
kubectl get --raw '/readyz?verbose' | tail -5
# optional: Prometheus + Grafana + alerting in one chart
helm repo add prometheus-community \
https://prometheus-community.github.io/helm-charts
helm install mon prometheus-community/kube-prometheus-stack \
-n monitoring --create-namespace
kubectl -n monitoring get pods
kubectl -n monitoring port-forward svc/mon-grafana 3000:80
# admin password: kubectl -n monitoring get secret mon-grafana \
# -o jsonpath='{.data.admin-password}' | base64 -dhog pod from Lab 5.sum by (pod) (rate(container_cpu_usage_seconds_total[5m])).kubectl describe, and the KubePodCrashLooping alert.helm uninstall mon -n monitoring; it uses a lot of memory on kind.Goal: package an app, upgrade and roll it back with Helm, and vary it per environment with Kustomize.
helm create webapp # scaffold a chart
helm lint webapp
helm template webapp ./webapp | head -30 # render without installing
helm install shop ./webapp -n shop --create-namespace
helm upgrade shop ./webapp -n shop --set replicaCount=3
helm history shop -n shop
helm rollback shop 1 -n shop
# Kustomize: base + overlay, built into kubectl
mkdir -p kz/base kz/overlays/prod
kubectl create deployment api --image=nginx:1.27 \
--dry-run=client -o yaml > kz/base/deploy.yaml
printf 'resources:\n- deploy.yaml\n' > kz/base/kustomization.yaml
K=kz/overlays/prod/kustomization.yaml
printf 'resources:\n- ../../base\nnamePrefix: prod-\n' > $K
printf 'replicas:\n- name: api\n count: 3\n' >> $K
kubectl kustomize kz/overlays/prod # then: kubectl apply -k kz/overlays/prodhelm history shows revisions 1–3; rollback creates a new revision rather than deleting one.templates/deployment.yaml): helm lint and helm template catch it before the cluster does.helm upgrade --install --rollback-on-failure (Helm 4; --atomic in Helm 3) rolls back automatically if pods never become ready.images: transformer to the prod overlay to pin nginx:1.28 and diff with kubectl diff -k.Goal: extend the API server with a typed, validated resource that kubectl understands.
cat <<'EOF' | kubectl apply -f -
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata: {name: backups.lab.example.com}
spec:
group: lab.example.com
scope: Namespaced
names: {plural: backups, singular: backup, kind: Backup, shortNames: [bk]}
versions:
- name: v1
served: true
storage: true
schema:
openAPIV3Schema:
type: object
properties:
spec:
type: object
required: [schedule]
properties:
schedule: {type: string}
retentionDays: {type: integer, minimum: 1, maximum: 90}
EOFkubectl explain backup.spec shows your fields; kubectl api-resources | grep lab.example.com lists the new type.Backup with schedule: "0 2 * * *" and get it with kubectl get bk.retentionDays: 365 or omit schedule → the API server rejects it with a validation error.Backup object goes with it.Goal: do node maintenance without an outage, and rehearse the control plane chores.
kubectl create deployment web --image=nginx:1.27 --replicas=3
kubectl create poddisruptionbudget web --selector=app=web --min-available=3
kubectl drain lab-worker --ignore-daemonsets --delete-emptydir-data
# bare pods from earlier labs block drain: delete them or add --force
# retries: "Cannot evict pod as it would violate the pod's disruption budget"
kubectl patch pdb web -p '{"spec":{"minAvailable":2}}' # drain now proceeds
kubectl uncordon lab-worker
# etcd snapshot (kubeadm layout; /var/lib/etcd is a hostPath)
kubectl -n kube-system exec etcd-lab-control-plane -- etcdctl \
--endpoints=https://127.0.0.1:2379 \
--cacert=/etc/kubernetes/pki/etcd/ca.crt \
--cert=/etc/kubernetes/pki/etcd/server.crt \
--key=/etc/kubernetes/pki/etcd/server.key snapshot save /var/lib/etcd/lab.db
docker exec lab-control-plane ls -lh /var/lib/etcd/lab.db--image kindest/node:<newer> and redeploy. That is a blue/green cluster upgrade.kubeadm upgrade plan, kubeadm upgrade apply, then each node: drain, upgrade kubelet, uncordon.etcdutl snapshot restore into a new data dir, then point the etcd static pod at it.Have a partner (or a script) break one thing; diagnose with the commands, not by reading the YAML first.
| Break it | Symptom | Diagnose with | Fix |
|---|---|---|---|
kubectl set image deploy/web nginx=nginx:9.9 | ImagePullBackOff | describe pod → Events: manifest unknown | Correct tag; rollout undo |
Command sh -c "exit 1" | CrashLoopBackOff | logs --previous, exit code in describe | Fix command or config |
Request cpu: 50 | Pending | describe pod: Insufficient cpu | Lower request or add nodes |
| Taint every worker | Pending | describe pod: untolerated taint | Remove taint or tolerate |
| Service selector typo | Connection refused, pods healthy | get endpointslices is empty | Match pod labels |
kubectl -n kube-system scale deploy coredns --replicas=0 | Names fail, IPs work | nslookup from dnsutils; get pods -n kube-system | Scale CoreDNS back to 2 |
| Default-deny NetworkPolicy without DNS egress | Bad address / timeouts | get netpol; test by IP | Allow UDP/TCP 53 to kube-dns |
Universal order: kubectl get (what state?) → describe (events, conditions) → logs / logs --previous → exec / debug → node level (docker exec lab-worker crictl ps, journalctl -u kubelet).
Combine every lab into one namespace you could defend in a review.
restricted and a ResourceQuota.web Deployment (unprivileged nginx) and a Redis StatefulSet with a PVC.web; PDB on both tiers.kubectl auth can-i --list for each SA is minimal.Tick a lab only when you can do it again without notes and explain what happened.
Clean up when finished: kind delete cluster --name lab.
| Pod | One or more containers scheduled together | Controller | Loop that drives actual state to desired state |
| ReplicaSet | Keeps N identical pods running | Operator | CRD + controller with operational knowledge |
| Service | Stable virtual IP/DNS for a set of pods | CRD | Custom resource definition — new API type |
| EndpointSlice | List of pod IPs behind a Service | CNI / CSI / CRI | Network / storage / runtime plugin interfaces |
| Ingress / Gateway | L7 entry points into the cluster | kubelet | Node agent that runs pods |
| PV / PVC | Storage / a claim on storage | etcd | Consistent key-value store for cluster state |
| Taint / toleration | Node repels pods / pod accepts taint | QoS class | Guaranteed, Burstable, BestEffort |
| PDB | Limit on voluntary disruptions | PSA / PSS | Pod Security Admission / Standards |
| HPA / VPA | Horizontal / vertical pod autoscaling | GitOps | Git as source of truth, pulled by an agent |
kubernetes.io/docskubernetes.io/docs/referencegateway-api.sigs.k8s.iogithub.com/kelseyhightower/kubernetes-the-hard-waytraining.linuxfoundation.orghelm.sh · Kustomize — kustomize.ioargo-cd.readthedocs.io · Flux — fluxcd.iobook.kubebuilder.ioKubernetes ships ~3 releases a year — check feature states and versions in the docs for your cluster version.