Platform engineering deck

Amazon EKS
from A to Z

Architecture, compute, networking, identity, security, storage, scaling, observability, upgrades, cost — and the best practices for each.

Use ← → / Space to navigate · I contents · O overview · T theme · F fullscreen

Amazon EKS · A to Z

Contents

Click any line to jump there · press I to return to this page

Chapter 1

Foundations

What EKS adds to Kubernetes, how it is built, and how to create a cluster.

Chapter 1 · Part 1.1

Fundamentals

Kubernetes in one slide, and what EKS adds on top.

Kubernetes recap

The core objects you deploy every day

Kubernetes is a declarative control loop: you describe desired state, controllers reconcile it.

Workloads

  • Pod — smallest unit, 1+ containers
  • Deployment → ReplicaSet — stateless
  • StatefulSet — stable identity + volumes
  • DaemonSet — one per node
  • Job / CronJob — batch

Networking

  • Service — ClusterIP / NodePort / LoadBalancer
  • Ingress / Gateway API — L7 routing
  • NetworkPolicy — pod firewall
  • CoreDNS — svc.ns.svc.cluster.local

Config & control

  • ConfigMap, Secret
  • PersistentVolumeClaim, StorageClass
  • ServiceAccount, Role, RoleBinding
  • Namespace, ResourceQuota, LimitRange
  • CRDs + controllers/operators
Why EKS

What Amazon EKS gives you

Upstream, CNCF-conformant Kubernetes with a managed control plane and deep AWS integration.

Managed for you

  • API server + etcd run across 3 AZs in an AWS-owned account
  • Control plane scaling, patching, backups of etcd
  • Optional management of nodes (Auto Mode), add-ons, and capabilities
  • SLA-backed regional endpoint

AWS integrations

  • IAM for cluster auth (access entries) and pod auth (IRSA / Pod Identity)
  • VPC-native pod IPs (VPC CNI), ALB/NLB, Route 53
  • EBS, EFS, FSx, S3 via CSI drivers
  • CloudWatch, Prometheus (AMP), GuardDuty, ECR, KMS

Pricing basics: $0.10 per cluster-hour in standard support, $0.60 per cluster-hour in extended support — plus the compute, storage, and data transfer you use. Auto Mode adds a per-instance management fee; Provisioned Control Plane tiers (for very large clusters) and EKS Capabilities are billed on top.

How it works

From kubectl apply to a running pod

Know this path by heart — almost every troubleshooting session maps to one of these steps.

1 · API server

  • AuthN: IAM token from aws eks get-token
  • AuthZ: RBAC / EKS access policies
  • Admission: mutating → validating webhooks, Pod Security Admission
  • Object persisted to etcd

2 · Controllers

  • Deployment controller creates a ReplicaSet
  • ReplicaSet controller creates Pods with no node yet
  • Every controller is a loop: watch → diff → act

3 · Scheduler

  • Filters nodes (resources, taints, affinity, topology, volumes)
  • Scores and binds the pod to the best node
  • No fit → Pending → Karpenter / Auto Mode launches a node

4 · kubelet

  • Sees the pod bound to its node
  • Creates the pod sandbox — VPC CNI assigns a VPC IP
  • Mounts volumes (CSI), injects env/secrets

5 · Containers

  • Pulls the image (ECR), containerd starts containers
  • Startup → readiness → liveness probes run
  • Pod becomes Ready

6 · Traffic

  • EndpointSlice updated with the pod IP
  • kube-proxy programs Service rules
  • LB Controller registers the pod IP in the ALB/NLB target group
Chapter 1 · Part 1.2

Architecture &
shared responsibility

Architecture

Control plane vs data plane

AWS-managed VPC — EKS control plane kube-apiserver behind NLB, multi-AZ public/private endpoint etcd 3 AZs, encrypted scheduler controller-manager cloud controller cross-account ENIs in your subnets Your VPC — data plane AZ a · private subnet EC2 node kubelet · VPC CNI Pods VPC IPs AZ b · private subnet EC2 / Auto Mode Karpenter nodes Fargate pod microVM per pod Public subnets ALB / NLB via LB Controller NAT Gateway egress (or endpoints)

Control plane

  • Single-tenant, runs in an AWS-owned VPC
  • At least 2 API server instances and 3 etcd members across 3 AZs
  • EKS places ENIs in your subnets so the API server can reach kubelets (logs, exec, webhooks)

Data plane

  • Your nodes in your VPC: EC2 (MNG, self-managed, Karpenter, Auto Mode), Fargate, or Hybrid Nodes on-prem
  • Nodes register via the API endpoint and authenticate with IAM
Shared responsibility

Who owns what

The more managed the compute option, the more AWS takes on.

AreaSelf-managed nodesManaged node groupsFargateAuto Mode
Control plane, etcdAWSAWSAWSAWS
Node AMI / OS patchingYouYou trigger, AWS provides AMIAWSAWS (nodes recycled, max 21-day lifetime)
Node provisioning & scalingYouYou (ASG + autoscaler)AWS (per pod)AWS (Karpenter-based)
Core add-ons (CNI, CoreDNS, kube-proxy, EBS CSI, LB)YouYouPartially AWSAWS (built in)
Version upgradesYouYouYou (control plane)You (control plane); nodes follow
IAM, RBAC, network policy, app security, dataYouYouYouYou
Chapter 1 · Part 1.3

Creating clusters

Clusters are cattle too — always build them from code.

Provisioning

Tools & a minimal Terraform example

Options

  • Terraform + terraform-aws-modules/eks — most common in production
  • eksctl — fast, opinionated; great for labs
  • CDK / CloudFormation, Pulumi
  • Console — learning only

Best practices

  • Separate repos/stacks: VPC → cluster → platform add-ons → apps
  • Pin the Kubernetes version explicitly
  • Use access entries, not the aws-auth ConfigMap
  • Enable control plane logs from day one
module "eks" {
  source  = "terraform-aws-modules/eks/aws"
  version = "~> 21.0"

  name               = "prod-apne1"
  kubernetes_version = "1.36"

  vpc_id     = module.vpc.vpc_id
  subnet_ids = module.vpc.private_subnets

  endpoint_public_access = false
  authentication_mode = "API"
  enable_cluster_creator_admin_permissions = false

  enabled_log_types = ["api", "audit", "authenticator"]

  addons = {
    vpc-cni                = { before_compute = true }
    coredns                = {}
    kube-proxy             = {}
    eks-pod-identity-agent = {}
  }
}
Versions

Kubernetes version lifecycle on EKS

Standard support

14 months from the version's EKS release. $0.10/cluster-hour.

Extended support

Additional 12 months. $0.60/cluster-hour. Enabled by default — set the upgrade policy to STANDARD to opt out.

End of support

After extended support ends, EKS auto-upgrades the control plane to the oldest supported version.

Do

  • Plan ~3 minor upgrades per year to stay in standard support
  • Track the EKS version calendar and AMI release notes

Avoid

  • Running in extended support unintentionally — 6× control plane cost
  • Skipping versions: the control plane upgrades one minor at a time
Hands-on lab

Your first learning cluster

# 0. Tools: AWS CLI v2, kubectl, eksctl, helm
aws sts get-caller-identity

# 1. Create a cluster with Auto Mode (~15 min)
eksctl create cluster --name lab \
  --region ap-northeast-1 --enable-auto-mode

# 2. Connect and deploy something
aws eks update-kubeconfig --name lab --region ap-northeast-1
kubectl get nodes        # empty until pods need capacity
kubectl create deployment web \
  --image=public.ecr.aws/nginx/nginx --replicas=2
kubectl get pods -o wide -w   # watch a node appear
kubectl get nodeclaims

# 3. Expose it
kubectl expose deployment web --port=80 --type=ClusterIP
kubectl port-forward svc/web 8080:80

# 4. ALWAYS clean up — clusters bill every hour
eksctl delete cluster --name lab --region ap-northeast-1

Exercises

  • Scale to 20 replicas and watch new nodes launch
  • Add an HPA and generate load
  • Break the readiness probe and see the rollout stop
  • Drain a node and observe PDB behavior
  • Create an access entry for a second IAM role
  • Give a pod S3 access with Pod Identity
  • Install the CloudWatch Observability add-on and read the dashboards
  • Apply a default-deny NetworkPolicy and fix DNS
Chapter 2

Compute and networking

Where pods run and how they get IP addresses and traffic.

Chapter 2 · Part 2.1

Compute options

Data plane

Choosing compute

OptionWhat it isUse whenWatch out
EKS Auto ModeAWS manages nodes (Karpenter-based), CNI, CoreDNS, kube-proxy, EBS CSI, LB controller; Bottlerocket-based, immutable nodesNew clusters, small platform teams, want least opsPer-instance fee; less node-level customization (no SSH/SSM)
Karpenter (self-run)Open-source node autoscaler; provisions right-sized EC2 directlyDiverse workloads, Spot, cost focus, full controlYou run and upgrade Karpenter itself (on MNG or Fargate)
Managed node groupsEKS-managed ASGs with EKS-optimized AMIs, managed rolling updatesStable, homogeneous capacity; system/add-on poolsScaling via Cluster Autoscaler is slower and less bin-packing
Self-managed nodesYour own ASG + launch templateCustom OS / AMI requirementsEverything is on you
FargateServerless pod — one microVM per podIsolation, bursty or low-ops jobsNo DaemonSets, no GPUs, no EBS volumes, no privileged pods; slower start
Hybrid NodesOn-prem/edge machines joined to an EKS control planeData residency, existing hardwareNeeds private connectivity (VPN / Direct Connect)
Nodes

AMIs & node best practices

AMI families

  • Amazon Linux 2023 — default EKS-optimized AMI (AL2 AMIs ended with K8s 1.32)
  • Bottlerocket — minimal, container-only, immutable, atomic updates
  • Accelerated variants for NVIDIA / Neuron GPUs
  • Windows Server AMIs for Windows pods

Best practices

  • Prefer Bottlerocket or AL2023; don't SSH — use SSM if needed
  • Enforce IMDSv2, hop limit 1 unless pods truly need IMDS
  • Use a small dedicated MNG (or Fargate) for critical add-ons like Karpenter
  • Diversify instance types; prefer Graviton (arm64) where images support it
  • Treat nodes as immutable — replace, don't patch in place
  • Set kube-reserved/system-reserved (EKS AMIs set sane defaults)
EKS Auto Mode

Auto Mode in practice

What you get

  • Built-in NodePools: general-purpose and system
  • Custom NodePool + NodeClass (eks.amazonaws.com) for GPU, Spot, or isolated pools
  • Compute, block storage, load balancing, pod networking and DNS managed by AWS
  • Nodes expire after 14 days by default (max 21) — patched by replacement
  • Enable on new or existing clusters; migrate from MNG/Karpenter gradually

Best practices

  • Use it as the default for new clusters unless you need custom AMIs or node access
  • Define PDBs — node replacement respects them
  • Use Auto Mode's own StorageClass (ebs.csi.eks.amazonaws.com) and IngressClass
  • Don't also install self-managed Karpenter / LB controller for the same resources
  • Compare the per-instance fee against the ops time you save
Chapter 2 · Part 2.2

Networking

VPC CNI, IP planning, load balancing, DNS.

Amazon VPC CNI

Every pod gets a real VPC IP

How it works

  • aws-node DaemonSet attaches ENIs and secondary IPs to each node
  • Pods are directly routable in the VPC — no overlay
  • Max pods per node limited by ENIs × IPs per ENI of the instance type
  • Keeps a warm pool of IPs (WARM_ENI_TARGET, WARM_IP_TARGET)

Scaling IP capacity

  • Prefix delegation — assigns /28 prefixes per slot → many more pods per node
  • Custom networking — pods use a secondary CIDR (e.g. 100.64.0.0/10) via ENIConfig
  • IPv6 clusters — removes pod IP exhaustion entirely
  • Security groups for pods — per-pod SGs via branch ENIs
kubectl set env daemonset aws-node -n kube-system ENABLE_PREFIX_DELEGATION=true WARM_PREFIX_TARGET=1
# Prefer configuring this through the managed add-on configuration values instead of kubectl in production
Under the hood

IP math and how traffic flows

Max pods per node

  • Secondary-IP mode:
    ENIs × (IPs per ENI − 1) + 2
  • m5.large: 3 × (10 − 1) + 2 = 29 pods
  • Prefix delegation: each slot holds a /28 (16 IPs); EKS recommends capping at 110 (< 30 vCPU) or 250
  • Each node holds warm IPs too — size subnets for nodes × pods + warm pool

Pod-to-pod

  • Same node: veth pair + host routes
  • Across nodes: plain VPC routing — no overlay, no encapsulation
  • To the internet: SNAT to the node's primary IP by default (or external SNAT via NAT GW)
  • Node security groups apply unless you use Security Groups for Pods

Services

  • ClusterIP: virtual IP; kube-proxy (iptables/IPVS/nftables) DNATs to pod IPs
  • NodePort: port on every node
  • LoadBalancer: NLB via LB Controller; IP mode targets pods directly
  • externalTrafficPolicy: Local keeps the client IP and avoids an extra hop
VPC design

Network best practices

Plan IPs generously

  • Use large private subnets (/19 or bigger) for nodes and pods
  • Add a secondary CIDR (100.64.0.0/10) when RFC1918 space is scarce
  • Give the cluster dedicated subnets for control plane ENIs (small, /28+)
  • Tag subnets: kubernetes.io/role/elb and kubernetes.io/role/internal-elb

Connectivity & cost

  • Spread nodes across ≥ 3 AZs
  • Use VPC endpoints (ECR api/dkr, S3 gateway, STS, EC2, ELB, EKS, EKS Auth, CloudWatch Logs) to cut NAT cost and enable private clusters
  • Enable VPC CNI native network policies (or Calico/Cilium)
  • Watch cross-AZ traffic — use topology-aware routing where it helps
Load balancing

Exposing services: AWS Load Balancer Controller

Mapping

  • Ingress → ALB (L7, path/host routing, WAF, Cognito/OIDC)
  • Service type: LoadBalancer → NLB (L4, static IPs, PrivateLink)
  • IngressGroup shares one ALB across many Ingresses
  • Gateway API supported for ALB/NLB, and VPC Lattice via its own controller

Best practices

  • Use target-type: ip — traffic goes straight to pods
  • Add a preStop sleep + tune deregistration delay to avoid 502s on rollout
  • Enable pod readiness gates for zero-downtime deploys
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: web
  annotations:
    alb.ingress.kubernetes.io/scheme: internet-facing
    alb.ingress.kubernetes.io/target-type: ip
    alb.ingress.kubernetes.io/group.name: public
    alb.ingress.kubernetes.io/listen-ports: '[{"HTTPS":443}]'
    alb.ingress.kubernetes.io/certificate-arn: arn:aws:acm:...
spec:
  ingressClassName: alb
  rules:
  - host: app.example.com
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service: { name: web, port: { number: 80 } }
DNS & endpoints

DNS and API endpoint access

DNS

  • CoreDNS — in-cluster resolution; run ≥ 2 replicas across AZs, consider autoscaling
  • NodeLocal DNSCache — reduces latency and VPC resolver throttling (1024 pps per ENI)
  • ExternalDNS — syncs Ingress/Service hostnames to Route 53
  • Lower ndots or use FQDNs with trailing dot for heavy external lookups

Cluster endpoint

  • Public — reachable from internet (IAM-authenticated)
  • Public + private — nodes use private path; restrict public with CIDR allow-list
  • Private only — reach via VPN / Direct Connect / bastion / SSM
  • ✓ Best practice: private (or public with tight CIDRs) for production
Chapter 3

Identity and security

Who can reach the cluster, what pods may call in AWS, and how to harden it.

Chapter 3 · Part 3.1

Identity & access

Humans → cluster, and pods → AWS.

Cluster access

Access entries replace aws-auth

How auth works

  • AuthN: IAM principal → token via aws eks get-token
  • AuthZ: Kubernetes RBAC and/or EKS access policies
  • Auth modes: CONFIG_MAP → API_AND_CONFIG_MAP → API (one-way migration)
  • Access policies: AmazonEKSClusterAdminPolicy (cluster scope), AmazonEKSAdminPolicy, AmazonEKSEditPolicy, AmazonEKSViewPolicy

Best practices

  • Use API mode; manage entries in IaC
  • Map IAM Identity Center roles, not IAM users
  • Disable creator admin; grant break-glass role explicitly
  • Least-privilege, namespace-scoped access for teams
aws eks create-access-entry \
  --cluster-name prod \
  --principal-arn arn:aws:iam::111122223333:role/TeamA

POLICY=arn:aws:eks::aws:cluster-access-policy
aws eks associate-access-policy \
  --cluster-name prod \
  --principal-arn arn:aws:iam::111122223333:role/TeamA \
  --policy-arn $POLICY/AmazonEKSEditPolicy \
  --access-scope \
    type=namespace,namespaces=team-a

# Or bind to RBAC groups instead of access policies:
aws eks create-access-entry ... \
  --kubernetes-groups team-a-devs
Pod IAM

IRSA vs EKS Pod Identity

IRSA (IAM Roles for Service Accounts)EKS Pod Identity
MechanismOIDC provider + projected SA token → AssumeRoleWithWebIdentityeks-pod-identity-agent DaemonSet + association via EKS API
Trust policyPer cluster OIDC issuer, per SA — must be edited for each clusterOne trust to pods.eks.amazonaws.com — reusable across clusters
SetupCreate OIDC provider, annotate SA with role ARNaws eks create-pod-identity-association
Session tags / ABACNoYes (cluster, namespace, SA tags)
Works onEC2, Fargate, any cluster incl. self-managed K8sLinux EC2 nodes incl. Auto Mode (not Fargate, not Windows)

Best practice

Default to Pod Identity for new EKS workloads; one role per app; never use node instance role permissions for apps.

Block IMDS

Set IMDSv2 hop limit to 1 so pods can't steal the node role credentials.

Kubernetes RBAC

Roles, bindings, and least privilege

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata: { name: deployer, namespace: team-a }
rules:
- apiGroups: ["apps"]
  resources: ["deployments"]
  verbs: ["get", "list", "watch", "create", "update", "patch"]
- apiGroups: [""]
  resources: ["pods", "pods/log", "services", "configmaps"]
  verbs: ["get", "list", "watch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata: { name: team-a-deployers, namespace: team-a }
subjects:
- kind: Group
  name: team-a-devs   # from access entry --kubernetes-groups
  apiGroup: rbac.authorization.k8s.io
roleRef:
  kind: Role
  name: deployer
  apiGroup: rbac.authorization.k8s.io

Concepts

  • Role = namespaced; ClusterRole = cluster-wide or reusable
  • Subjects: User, Group, ServiceAccount
  • RBAC is additive only — there is no "deny"

Best practices

  • No day-to-day cluster-admin for humans
  • Avoid * verbs/resources
  • Treat secrets get/list, pods/exec, escalate, bind, impersonate as privileged
  • Audit: kubectl auth can-i --list --as=...
Pod Identity step by step

Give a pod access to AWS — no keys

# 1. IAM role trusted by EKS Pod Identity
cat > trust.json <<'EOF'
{ "Version": "2012-10-17",
  "Statement": [{ "Effect": "Allow",
    "Principal": { "Service": "pods.eks.amazonaws.com" },
    "Action": ["sts:AssumeRole", "sts:TagSession"] }] }
EOF
aws iam create-role --role-name orders-app \
  --assume-role-policy-document file://trust.json
aws iam put-role-policy --role-name orders-app \
  --policy-name s3-read --policy-document file://s3-read.json

# 2. Agent (built into Auto Mode)
aws eks create-addon --cluster-name prod \
  --addon-name eks-pod-identity-agent

# 3. Bind role ↔ namespace + service account
aws eks create-pod-identity-association \
  --cluster-name prod --namespace orders \
  --service-account orders \
  --role-arn arn:aws:iam::111122223333:role/orders-app

# 4. Verify from inside the pod
kubectl exec -n orders deploy/orders -- \
  aws sts get-caller-identity

Notes

  • Credentials are injected only into pods created after the association — restart existing pods
  • Requires a recent AWS SDK that supports the container credential provider
  • Session tags enable ABAC: eks-cluster-name, kubernetes-namespace, kubernetes-service-account
  • One role per app; never reuse the node role
  • Same pattern for add-ons: EBS CSI, LB Controller, CloudWatch agent, ESO
Chapter 3 · Part 3.2

Security hardening

Threat model

Security layers (the 4C model)

Cloud

  • Account & IAM
  • VPC, SGs, endpoints
  • KMS, CloudTrail
  • GuardDuty, Security Hub

Cluster

  • API endpoint access
  • Access entries + RBAC
  • Admission & policy
  • Audit logs

Container

  • Image provenance
  • Pod Security Standards
  • securityContext
  • Runtime detection

Code

  • Dependencies (SCA)
  • Secrets handling
  • SAST / reviews
  • TLS between services

Common attack paths on EKS

  • Public API endpoint + leaked kubeconfig/IAM keys → full cluster access
  • Pod reaches IMDS → steals the node role credentials
  • Over-privileged ServiceAccount token → reads secrets or creates privileged pods
  • Privileged / hostPath pod → escapes to the node, then to other pods
  • Vulnerable or malicious image → crypto mining, reverse shell, lateral movement
Defense in depth

Security best practices

Control plane & data

  • Private API endpoint / CIDR allow-list
  • Envelope encryption of K8s API data is on by default (K8s 1.28+); use your own KMS key if required
  • Enable audit & authenticator logs
  • External secrets: Secrets Manager via External Secrets Operator or Secrets Store CSI

Workloads

  • Pod Security Admission: restricted by default, baseline where needed
  • runAsNonRoot, readOnlyRootFilesystem, drop ALL caps, no privileged
  • Policy-as-code: Kyverno or OPA Gatekeeper
  • Default-deny NetworkPolicy per namespace

Supply chain & runtime

  • ECR with scan on push (Inspector enhanced scanning)
  • Immutable tags, pin digests, sign images (Notation / cosign)
  • GuardDuty EKS Protection + Runtime Monitoring
  • Security Hub / CIS EKS benchmark checks
kubectl label ns payments pod-security.kubernetes.io/enforce=restricted pod-security.kubernetes.io/warn=restricted
Example

A hardened pod spec

spec:
  serviceAccountName: orders
  automountServiceAccountToken: false
  securityContext:
    runAsNonRoot: true
    runAsUser: 10001
    fsGroup: 10001
    seccompProfile: { type: RuntimeDefault }
  containers:
  - name: app
    image: <ecr-repo>/orders@sha256:<digest>
    securityContext:
      allowPrivilegeEscalation: false
      readOnlyRootFilesystem: true
      capabilities: { drop: ["ALL"] }
    resources:
      requests: { cpu: 250m, memory: 256Mi }
      limits:   { memory: 256Mi }
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata: { name: default-deny, namespace: orders }
spec:
  podSelector: {}
  policyTypes: [Ingress, Egress]
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata: { name: allow-dns, namespace: orders }
spec:
  podSelector: {}
  policyTypes: [Egress]
  egress:
  - to:
    - namespaceSelector:
        matchLabels:
          kubernetes.io/metadata.name: kube-system
    ports:
    - { protocol: UDP, port: 53 }
    - { protocol: TCP, port: 53 }

Set automountServiceAccountToken: true only for pods that call the Kubernetes API (IRSA / Pod Identity inject their own token volumes). With default-deny egress, also allow the Pod Identity agent at 169.254.170.23.

Pod Security Standards

Pod Security Admission (PSA)

LevelWhat it allows / blocksUse for
privilegedNo restrictionsSystem namespaces only (CNI, CSI, monitoring agents)
baselineBlocks privileged containers, hostNetwork/hostPID/hostIPC, hostPath, hostPorts, dangerous capabilitiesLegacy apps during migration
restrictedBaseline + must run as non-root, drop ALL capabilities, no privilege escalation, seccomp RuntimeDefault, limited volume typesDefault for all application namespaces
apiVersion: v1
kind: Namespace
metadata:
  name: orders
  labels:
    pod-security.kubernetes.io/enforce: restricted
    pod-security.kubernetes.io/enforce-version: latest
    pod-security.kubernetes.io/warn: restricted
    pod-security.kubernetes.io/audit: restricted

Modes & rollout

  • enforce rejects pods · warn shows client warnings · audit annotates audit logs
  • Roll out: warn + audit first, fix workloads, then enforce
  • PSA checks pods; Deployments only get warnings — check ReplicaSet events
  • PodSecurityPolicy was removed in 1.25 — PSA replaces it
Policy as code

Guardrails with Kyverno, Gatekeeper, or CEL

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata: { name: disallow-latest-tag }
spec:
  background: true
  rules:
  - name: require-pinned-image
    match:
      any:
      - resources: { kinds: [Pod] }
    validate:
      failureAction: Enforce
      message: "Pin an image tag or digest, not :latest"
      pattern:
        spec:
          containers:
          - image: "!*:latest"

What to enforce

  • Images only from your ECR registries
  • No :latest; signed images (verifyImages)
  • Resource requests and required labels (team, cost-center)
  • No hostPath, no LoadBalancer without approval

Tips

  • Kyverno = YAML policies; Gatekeeper = Rego
  • ValidatingAdmissionPolicy (CEL, GA in 1.30) for simple rules, no extra controller
  • Start in Audit, read policy reports, then Enforce
  • Run the engine HA; exclude system namespaces deliberately
Secrets management

Where secrets should live

ApproachHowTrade-off
Kubernetes SecretBase64 in etcd, envelope-encrypted with KMSSimple; anyone with get secrets in the namespace can read it
External Secrets OperatorSyncs Secrets Manager / Parameter Store into K8s SecretsSource of truth in AWS, rotation via refresh; still a K8s Secret
Secrets Store CSI + AWS providerMounts secrets as files at pod startNo K8s Secret unless you enable sync
SDK call with Pod IdentityApp reads Secrets Manager directlyNothing in the cluster; app code depends on AWS
apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata: { name: db, namespace: orders }
spec:
  refreshInterval: 1h
  secretStoreRef: { kind: ClusterSecretStore, name: aws-sm }
  target: { name: db-credentials }
  data:
  - secretKey: password
    remoteRef: { key: prod/orders/db, property: password }

Best practices

  • Never commit plaintext secrets (use SOPS / Sealed Secrets if they must live in Git)
  • Restrict RBAC on secrets; mount as files rather than env vars
  • Enable rotation; separate secrets per environment
Network security

Layers of network control

Controls

  • Cluster security group — created by EKS for control plane ↔ node traffic
  • Security Groups for Pods — per-pod SG, e.g. only orders pods reach RDS (needs ENABLE_POD_ENI=true on the VPC CNI)
  • NetworkPolicy — L3/L4 rules between pods (VPC CNI eBPF agent)
  • WAF / Shield on ALBs; NACLs as a coarse backstop
  • Egress control — NAT + Network Firewall or proxy allow-list

Best practices

  • Default-deny + explicit allows (DNS, Pod Identity agent)
  • Nodes in private subnets only; TLS everywhere, mTLS via mesh if needed
# Turn on native network policy in the VPC CNI
aws eks update-addon --cluster-name prod \
  --addon-name vpc-cni \
  --configuration-values '{"enableNetworkPolicy":"true"}'
apiVersion: vpcresources.k8s.aws/v1beta1
kind: SecurityGroupPolicy
metadata: { name: orders-db, namespace: orders }
spec:
  podSelector: { matchLabels: { app: orders } }
  securityGroups:
    groupIds: [sg-0123456789abcdef0]
Supply chain

Secure the image pipeline

Build

  • Minimal / distroless base
  • Multi-stage, non-root USER
  • Pin base by digest
→

Scan

  • Trivy / Inspector in CI
  • Fail on critical CVEs
  • SBOM (syft)
→

Sign

  • cosign or Notation + AWS Signer
  • Provenance attestations
→

Store

  • ECR, immutable tags
  • Scan on push
  • Lifecycle policies
→

Admit

  • Verify signatures
  • Allowed registries
  • Deploy by digest

Keep it fresh

Inspector re-scans images continuously when new CVEs appear; Renovate/Dependabot bump dependencies; rebuild images regularly, not only on code changes.

CI identity

CI authenticates to AWS with OIDC (no static keys), can push only to its own repos, and cannot touch the cluster directly when GitOps does the deploy.

Node security

Hardening the worker nodes

Host

  • Bottlerocket: read-only root fs, SELinux enforcing, no shell by default
  • IMDSv2 required, hop limit 1
  • No SSH keys — SSM Session Manager with session logging
  • Encrypted EBS root volumes
  • Patch by replacement (expireAfter, AMI updates)
  • Check against CIS benchmark (kube-bench)

Node IAM role — keep it minimal

  • AmazonEKSWorkerNodePolicy
  • AmazonEC2ContainerRegistryPullOnly
  • Move AmazonEKS_CNI_Policy to the aws-node service account (Pod Identity / IRSA)
  • AmazonSSMManagedInstanceCore only if you use SSM
  • Nothing app-specific — apps get their own roles
Detection & audit

Know what is happening in the cluster

Services

  • GuardDuty EKS Protection — analyzes audit logs: anonymous access, exposed dashboards, privileged pods, suspicious API calls
  • GuardDuty Runtime Monitoring — agent add-on: crypto mining, reverse shells, malicious files
  • CloudTrail — EKS API calls (create access entry, update cluster)
  • Security Hub — aggregated findings & controls
  • Open-source alternative: Falco
# Logs Insights on /aws/eks/<cluster>/cluster
# Who deleted what?
fields @timestamp, user.username, objectRef.resource,
       objectRef.namespace, objectRef.name
| filter @logStream like /kube-apiserver-audit/
| filter verb = "delete"
| sort @timestamp desc | limit 50

# Who reads secrets?
fields @timestamp, user.username, objectRef.name
| filter @logStream like /kube-apiserver-audit/
| filter objectRef.resource = "secrets"
    and verb in ["get", "list"]
| stats count(*) by user.username

# RBAC denials
fields user.username, verb, objectRef.resource
| filter @logStream like /kube-apiserver-audit/
| filter responseStatus.code = 403
| stats count(*) as denials by user.username, verb
| sort denials desc
Incident response

A compromised pod — first 30 minutes

Steps

  • Identify: GuardDuty finding, audit logs — which pod, node, SA, role
  • Contain: isolate the pod with a deny-all NetworkPolicy; cordon the node
  • Revoke: remove the Pod Identity association / deny old role sessions
  • Preserve: don't delete first — snapshot EBS, save logs, describe output
  • Eradicate: redeploy from a clean image, rotate secrets
  • Learn: postmortem, add a policy so it can't recur
# Detach from Service & ReplicaSet (a clean replica starts)
kubectl label pod web-7d9f -n orders app=quarantine --overwrite

# Cut all traffic to/from it
kubectl apply -f - <<'EOF'
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata: { name: quarantine, namespace: orders }
spec:
  podSelector: { matchLabels: { app: quarantine } }
  policyTypes: [Ingress, Egress]
EOF

# Stop new pods landing on the node
kubectl cordon ip-10-0-12-34.ap-northeast-1.compute.internal

# Preserve evidence
aws ec2 create-snapshot --volume-id vol-0abc123 \
  --description "IR case 123 - node root volume"
Chapter 4

Storage, scaling, and add-ons

Persistent data, capacity that follows demand, and the platform add-ons around them.

Chapter 4 · Part 4.1

Storage

CSI drivers

Persistent storage options

DriverAccessBest forNotes
EBS CSIRWO, single AZDatabases, StatefulSetsUse gp3; volume is AZ-bound → use WaitForFirstConsumer
EFS CSIRWX, multi-AZShared files, CMS, ML datasetsDynamic provisioning via access points; Fargate supports static provisioning only
FSx (Lustre/ONTAP/OpenZFS)RWXHPC/ML throughput, enterprise NASHigher performance, higher cost
Mountpoint for S3 CSIRWX (read-heavy)Large datasets, ML trainingNot POSIX-complete; sequential writes only
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata: { name: gp3 }
provisioner: ebs.csi.aws.com
volumeBindingMode: WaitForFirstConsumer
allowVolumeExpansion: true
parameters: { type: gp3, encrypted: "true" }

Best practices

  • Install CSI drivers as EKS add-ons with Pod Identity
  • Encrypt by default; back up with AWS Backup or Velero
  • Prefer managed databases (RDS/Aurora) over DBs on EKS unless you need it
Chapter 4 · Part 4.2

Autoscaling

Pods scale first, nodes follow.

Layers

The autoscaling stack

HPA

Scales replicas on CPU/memory or custom/external metrics. Needs metrics-server.

+

KEDA

Event-driven scaling (SQS, Kafka, Prometheus, cron), incl. scale-to-zero.

+

VPA

Recommends/sets requests. Use in recommendation mode alongside HPA on CPU.

→

Node scaling

Karpenter (or Auto Mode) provisions nodes for pending pods; Cluster Autoscaler scales ASGs.

Cluster AutoscalerKarpenter
UnitNode groups (ASGs) with fixed instance typesIndividual instances, chosen per pending pod batch
SpeedMinutes (ASG round-trip)Faster — calls EC2 Fleet directly
Cost featuresLimitedConsolidation, Spot/On-Demand mix, Graviton, drift replacement
Karpenter

NodePool & EC2NodeClass

apiVersion: karpenter.sh/v1
kind: NodePool
metadata: { name: default }
spec:
  template:
    spec:
      nodeClassRef: { group: karpenter.k8s.aws,
        kind: EC2NodeClass, name: default }
      requirements:
      - key: karpenter.sh/capacity-type
        operator: In
        values: ["spot", "on-demand"]
      - key: kubernetes.io/arch
        operator: In
        values: ["arm64", "amd64"]
      - key: karpenter.k8s.aws/instance-category
        operator: In
        values: ["c", "m", "r"]
      expireAfter: 720h
  limits: { cpu: "1000" }
  disruption:
    consolidationPolicy: WhenEmptyOrUnderutilized
    consolidateAfter: 1m
    budgets: [{ nodes: "10%" }]

Best practices

  • Run Karpenter on a small MNG or Fargate — never on nodes it manages
  • Allow broad instance families for better Spot availability
  • Set limits to cap runaway spend
  • Use disruption budgets + PDBs to control churn
  • expireAfter forces regular node refresh (patching)
  • Separate NodePools for GPU, system, or tainted workloads
  • Handle Spot interruptions via the SQS interruption queue
Chapter 4 · Part 4.3

Add-ons

The platform components every cluster needs.

Add-ons

The platform layer

EKS managed add-ons

  • Core: vpc-cni, coredns, kube-proxy
  • eks-pod-identity-agent, aws-ebs-csi-driver, aws-efs-csi-driver, aws-mountpoint-s3-csi-driver
  • amazon-cloudwatch-observability, adot, aws-guardduty-agent
  • Community add-ons (e.g. metrics-server) & Marketplace add-ons
  • Check compatibility: aws eks describe-addon-versions

Common Helm-installed tools

  • AWS Load Balancer Controller, Karpenter
  • ExternalDNS, cert-manager
  • External Secrets Operator
  • Kyverno / Gatekeeper, KEDA
  • Argo CD / Flux

Best practices

Prefer managed add-ons where available · pin versions in IaC · upgrade add-ons with every cluster upgrade · avoid manual kubectl edit on managed add-ons (use configuration values) · give each add-on its own Pod Identity role.

Chapter 5

Observability and reliability

See what the cluster is doing and keep workloads available.

Chapter 5 · Part 5.1

Observability &
monitoring

Control plane, nodes, workloads, logs, traces, alerts, SLOs.

Observability overview

Logs, metrics, traces

Control plane

  • Log types: api, audit, authenticator, controllerManager, scheduler → CloudWatch Logs
  • Control plane metrics exposed via Prometheus endpoints
  • Cluster insights: upgrade readiness & config checks

Metrics

  • CloudWatch Container Insights (enhanced observability) via the add-on
  • Or Prometheus → Amazon Managed Prometheus + Amazon Managed Grafana
  • kube-state-metrics, node-exporter, DCGM for GPUs

Logs & traces

  • Fluent Bit DaemonSet → CloudWatch / OpenSearch / S3
  • ADOT / OpenTelemetry Collector → X-Ray or other backends
  • Set log retention policies — never "Never expire"

Alert on what matters

Pending pods & unschedulable reasons · node NotReady · pod restarts/OOMKilled · API server 5xx & latency · CoreDNS errors · free IPs per subnet · PVC usage · certificate expiry · SLO-based app alerts.

What to monitor

Signals per layer

LayerKey signalsSource
Control planeAPI latency & 5xx, request rate, throttling (APF), etcd size, webhook latency, pending podsCloudWatch AWS/EKS, API /metrics
NodesCPU, memory, disk, network, NotReady, pressure conditions, pods per nodenode-exporter, Container Insights
K8s objectsAvailable vs desired replicas, restarts, OOMKills, failed Jobs, HPA at max, PVC usagekube-state-metrics, kubelet
ApplicationsRED: rate, errors, duration; saturation; business KPIsApp metrics, OpenTelemetry, ALB metrics
Add-onsCoreDNS errors & latency, CNI IP allocation, Karpenter launch errors, LB controller errorsAdd-on /metrics endpoints
Network & costFree IPs per subnet, NAT bytes, cross-AZ traffic, requests vs usageCloudWatch, VPC Flow Logs, Kubecost

Methods: USE (utilization, saturation, errors) for resources · RED (rate, errors, duration) for services · Golden signals (latency, traffic, errors, saturation) for user-facing SLOs.

Control plane monitoring

Watching the API server, scheduler, and etcd

Built-in sources

  • CloudWatch AWS/EKS metrics (new clusters on K8s 1.28+, no extra cost): API request totals & 5xx, storage size, scheduler attempts, pending pods
  • EKS console Observability tab with control plane dashboards
  • Prometheus endpoints scrapeable from inside the cluster
  • Control plane logs → /aws/eks/<cluster>/cluster

Watch for

  • apiserver_request_duration_seconds p99 rising
  • apiserver_flowcontrol_rejected_requests_total > 0 (throttling)
  • etcd_mvcc_db_total_size_in_use_in_bytes near the 8 GiB etcd limit
  • Slow admission webhooks blocking every request
# API server metrics
kubectl get --raw /metrics | grep apiserver_request_total

# Scheduler metrics (EKS-proxied endpoint)
kubectl get --raw \
  "/apis/metrics.eks.amazonaws.com/v1/ksh/container/metrics"

# Controller manager metrics
kubectl get --raw \
  "/apis/metrics.eks.amazonaws.com/v1/kcm/container/metrics"

# Enable control plane logs
aws eks update-cluster-config --name prod \
  --logging '{"clusterLogging":[{"types":
    ["api","audit","authenticator",
     "controllerManager","scheduler"],
    "enabled":true}]}'

# Audit logs can be large: set retention
aws logs put-retention-policy \
  --log-group-name /aws/eks/prod/cluster \
  --retention-in-days 90
AWS-native path

CloudWatch Container Insights & Application Signals

# IAM for the CloudWatch agent via Pod Identity
# (role with CloudWatchAgentServerPolicy)
aws eks create-pod-identity-association \
  --cluster-name prod \
  --namespace amazon-cloudwatch \
  --service-account cloudwatch-agent \
  --role-arn arn:aws:iam::111122223333:role/cw-agent

# Install the add-on: CloudWatch agent + Fluent Bit
aws eks create-addon --cluster-name prod \
  --addon-name amazon-cloudwatch-observability

# Log groups created
/aws/containerinsights/prod/application
/aws/containerinsights/prod/host
/aws/containerinsights/prod/dataplane
/aws/containerinsights/prod/performance

What you get

  • Cluster → namespace → workload → pod → container metrics and curated dashboards
  • Enhanced observability: kube-state and control plane detail, GPU / Neuron metrics
  • Container logs shipped by Fluent Bit
  • Application Signals: auto-instrumented APM (Java, Python, .NET, Node.js), service map, SLOs
  • Billing per observation — exclude noisy namespaces and set log retention
Open-source path

Prometheus & Grafana

Options

  • Amazon Managed Prometheus + AWS managed collector (agentless scraper) — no Prometheus to run
  • AMP + ADOT / Prometheus agent with remote_write
  • Self-managed kube-prometheus-stack: Prometheus Operator, Alertmanager, Grafana, node-exporter, kube-state-metrics
  • Amazon Managed Grafana with AMP, CloudWatch, X-Ray data sources and SSO

Best practices

  • Control label cardinality — no user IDs or URLs with IDs
  • Recording rules for expensive queries
  • Dashboards and rules as code, in Git
  • Start from community mixins (kubernetes-mixin)
# Tell Prometheus Operator to scrape your app
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: orders
  namespace: orders
  # must match Prometheus serviceMonitorSelector
  labels: { release: kube-prometheus-stack }
spec:
  selector:
    matchLabels: { app: orders }
  endpoints:
  - port: metrics      # named Service port
    path: /metrics
    interval: 30s
Alerting

Starter alert rules (PromQL)

AlertExpression
Pod crash-loopingincrease(kube_pod_container_status_restarts_total[15m]) > 3
Pods pending (with for: 10m)sum by (namespace) (kube_pod_status_phase{phase="Pending"}) > 0
Node not readykube_node_status_condition{condition="Ready",status="true"} == 0
Deployment degradedkube_deployment_status_replicas_available < kube_deployment_spec_replicas
OOMKilledkube_pod_container_status_last_terminated_reason{reason="OOMKilled"} == 1
CPU throttlingrate(container_cpu_cfs_throttled_periods_total[5m]) / rate(container_cpu_cfs_periods_total[5m]) > 0.25
PVC almost fullkubelet_volume_stats_used_bytes / kubelet_volume_stats_capacity_bytes > 0.85
API server 5xxsum(rate(apiserver_request_total{code=~"5.."}[5m])) / sum(rate(apiserver_request_total[5m])) > 0.01
HPA maxed outkube_horizontalpodautoscaler_status_current_replicas == kube_horizontalpodautoscaler_spec_max_replicas
Alerting

Writing alerts people trust

apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata: { name: workloads, namespace: monitoring }
spec:
  groups:
  - name: workloads
    rules:
    - alert: PodCrashLooping
      expr: |
        increase(kube_pod_container_status_restarts_total[15m]) > 3
      for: 5m
      labels:
        severity: warning
        team: "{{ $labels.namespace }}"
      annotations:
        summary: "{{ $labels.pod }} is restarting"
        runbook_url: https://wiki.example.com/runbooks/crashloop

Principles

  • Page on symptoms users feel (SLO burn, 5xx, latency); ticket on causes (disk 80%)
  • Every alert has an owner, a severity, and a runbook link
  • Use for: to avoid flapping
  • Route by team label in Alertmanager / SNS → Slack, PagerDuty
  • Review noisy alerts monthly — delete or fix them
  • Alert on the monitoring itself (scrape failures, agent down)
Logging

Log pipeline design

App

Writes structured JSON to stdout/stderr

→

Node

containerd writes to /var/log/containers

→

Fluent Bit

DaemonSet tails files, adds pod / namespace / labels metadata

→

Destination

CloudWatch Logs, OpenSearch, or S3 via Firehose

Best practices

  • JSON logs with level, timestamp, request and trace IDs
  • Never log secrets, tokens, or personal data
  • Retention per log group; Infrequent Access class for rarely queried logs
  • Archive to S3 for long-term / compliance
  • Fargate: built-in Fluent Bit via the aws-observability namespace ConfigMap
# Logs Insights: top error sources in the last hour
fields @timestamp, kubernetes.namespace_name,
       kubernetes.pod_name, log
| filter log like /(?i)error|exception/
| stats count(*) as errors
    by kubernetes.namespace_name,
       kubernetes.container_name
| sort errors desc
| limit 20
Tracing

Distributed tracing with OpenTelemetry

Building blocks

  • OpenTelemetry SDKs or auto-instrumentation
  • OpenTelemetry Operator injects agents via annotations
  • Collector (ADOT) as DaemonSet or gateway Deployment
  • Backends: X-Ray / CloudWatch, Tempo, Jaeger, vendors

Best practices

  • Propagate W3C traceparent across all services
  • Put the trace ID in every log line to jump log ↔ trace
  • Sample by tail (keep errors and slow requests)
  • Name spans and services consistently
apiVersion: opentelemetry.io/v1alpha1
kind: Instrumentation
metadata: { name: default, namespace: orders }
spec:
  exporter:
    endpoint: http://otel-collector.observability:4318  # OTLP/HTTP
  propagators: [tracecontext, baggage]
  sampler:
    type: parentbased_traceidratio
    argument: "0.1"
---
# Opt a workload in (pod template annotation)
metadata:
  annotations:
    instrumentation.opentelemetry.io/inject-java: "true"
SLOs

SLIs, SLOs, and error budgets

SLI

  • Availability = good requests / all requests (from ALB or app metrics)
  • Latency = share of requests faster than 300 ms
  • Measure as close to the user as possible

SLO & budget

  • 99.9% over 30 days → error budget = 0.1% ≈ 43.2 minutes
  • Budget left → ship faster
  • Budget burnt → focus on reliability

Burn-rate alerts

  • Fast burn: 14.4× over 1h (and 5m) → page
  • Slow burn: 6× over 6h → page; 1× over 3d → ticket
  • Fewer, better pages than threshold alerts

Dashboard hierarchy

Fleet overview → cluster health → namespace / team → workload detail → pod. Each level links to the next, and every alert links to the dashboard that explains it.

Chapter 5 · Part 5.2

Workload reliability

Resilient workloads

Reliability best practices

Scheduling & resources

  • Always set requests; set memory limit = request
  • Avoid CPU limits for latency-sensitive apps (throttling)
  • topologySpreadConstraints across zones and nodes
  • PodDisruptionBudget for every multi-replica service
  • PriorityClass for critical workloads
  • ≥ 2 replicas (3 across AZs for critical)

Lifecycle

  • Separate readiness, liveness, and startup probes
  • Liveness must not depend on downstream services
  • Handle SIGTERM; preStop sleep for LB deregistration
  • Tune terminationGracePeriodSeconds
  • Rolling update: maxUnavailable: 0, maxSurge: 25%
  • Design for node loss — Spot and consolidation will happen
topologySpreadConstraints:
- { maxSkew: 1, topologyKey: topology.kubernetes.io/zone, whenUnsatisfiable: ScheduleAnyway,
    labelSelector: { matchLabels: { app: web } } }
Chapter 6

Day-2 operations

Upgrades, cost control, and running EKS over time.

Chapter 6 · Part 6.1

Upgrades & lifecycle

Upgrade runbook

Upgrading an EKS cluster safely

1 · Prepare

  • Read K8s & EKS release notes
  • Check upgrade insights
  • Scan deprecated APIs (pluto, kubent)
  • Upgrade in non-prod first
→

2 · Control plane

  • One minor version at a time
  • In-place, no downtime for API
  • Rollback to N-1 possible within 7 days (check rollback readiness insights)
→

3 · Add-ons

  • vpc-cni, coredns, kube-proxy
  • CSI drivers, LB controller, Karpenter
  • Match compatibility matrix
→

4 · Data plane

  • MNG: update version (rolling)
  • Karpenter / Auto Mode: drift replaces nodes
  • PDBs keep apps up

Version skew

kubelet may be up to 3 minor versions older than the API server (K8s 1.28+), but never newer.

Blue/green clusters

For big jumps or risky changes: build a new cluster, shift traffic via DNS/ALB weights, then retire the old one. Requires stateless clusters + GitOps.

Chapter 6 · Part 6.2

Cost optimization

FinOps on EKS

Where the money goes — and how to cut it

Compute

  • Right-size requests (VPA recommendations, Goldilocks)
  • Karpenter consolidation for bin-packing
  • Spot for stateless/fault-tolerant workloads
  • Graviton: typically better price-performance
  • Compute Savings Plans for the baseline
  • Scale non-prod to zero off-hours

Network & storage

  • VPC endpoints to reduce NAT data processing
  • Minimize cross-AZ chatter
  • Share ALBs with IngressGroups
  • gp3 over gp2; delete orphaned PVs/EBS
  • Set CloudWatch log retention

Visibility & control

  • Split cost allocation data for EKS in CUR
  • Kubecost / OpenCost per namespace/team
  • Consistent labels → cost tags
  • ResourceQuotas per team
  • Stay off extended support ($0.60/h)
  • Consolidate small clusters
Chapter 6 · Part 6.3

GitOps, multi-tenancy,
DR & troubleshooting

Delivery & tenancy

GitOps and multi-tenancy

GitOps

  • Git is the source of truth; Argo CD or Flux reconciles
  • EKS Capabilities can run Argo CD, ACK, and kro as AWS-managed services
  • Helm / Kustomize for packaging; app-of-apps or ApplicationSets for fleets
  • CI builds & pushes images (OIDC to AWS, no static keys); CD only deploys
  • Drift detection + auto-sync with self-heal

Multi-tenancy (soft)

  • Namespace per team/app + RBAC
  • ResourceQuota & LimitRange per namespace
  • Default-deny NetworkPolicy
  • Pod Identity per workload; PSA restricted
  • Dedicated NodePools with taints for sensitive tenants
  • Hard isolation → separate clusters / accounts

Cluster strategy: separate prod and non-prod clusters (ideally separate AWS accounts); prefer fewer, larger shared clusters over many small ones unless isolation or blast radius requires it.

Beyond the basics

DR, special workloads & scale

Backup & DR

  • Cluster state = Git + IaC; rebuild rather than restore
  • Velero or AWS Backup for namespaces & PVs
  • ARC zonal shift / autoshift to move traffic away from an impaired AZ
  • Multi-region: active/passive clusters + data replication; test failover

Special workloads

  • GPU/ML: accelerated AMIs, NVIDIA device plugin or GPU Operator, Neuron for Trainium/Inferentia
  • Dedicated tainted NodePools for GPUs
  • Windows nodes: separate node group, VPC CNI Windows support, node selectors
  • Service mesh (Istio) or VPC Lattice for cross-cluster traffic

Scale & images

  • Provisioned Control Plane tiers for very large clusters
  • Watch API server request rates; avoid chatty controllers
  • ECR pull-through cache for public images
  • Small images, SOCI / pre-pulled images for faster start
  • Spot on MNG: interruption handling is built in (Capacity Rebalancing); Node Termination Handler only for self-managed nodes
Troubleshooting

Common problems & first checks

SymptomLikely causeCheck
Pods PendingNo capacity, taints, affinity, PVC AZ mismatchkubectl describe pod events; Karpenter logs
Pods stuck ContainerCreatingSubnet out of IPs, CNI errors, volume attachSubnet free IPs; aws-node logs; ipamd
ImagePullBackOffWrong tag, no ECR perms, no NAT/endpointsNode role ECR policy; VPC endpoints
CrashLoopBackOff / OOMKilledApp error, low memory limit, bad probeskubectl logs --previous; describe
Node NotReadykubelet/CNI failure, disk/memory pressureNode conditions; SSM to node; log collector script
Unauthorized / ForbiddenMissing access entry or RBACaws sts get-caller-identity; kubectl auth can-i
ALB 502/504 on deployNo preStop delay, readiness, SG rulesTarget health; LB controller logs
DNS timeoutsCoreDNS overload, VPC resolver limitsCoreDNS metrics; NodeLocal DNSCache
Cheat sheet

Debugging commands

# What just happened?
kubectl get events -A --sort-by=.lastTimestamp | tail -30
kubectl describe pod <pod> -n <ns>
kubectl logs <pod> -n <ns> -c <container> --previous

# Resources
kubectl top nodes
kubectl top pods -A --sort-by=memory
kubectl describe node <node> | grep -A8 Allocated

# Get a shell next to a broken container
kubectl debug -it pod/<pod> -n <ns> \
  --image=busybox --target=<container>
kubectl debug node/<node> -it --image=ubuntu

# Networking
kubectl get endpointslices -n <ns>
kubectl run tmp --rm -it --image=busybox -- \
  nslookup web.orders.svc.cluster.local
# Access
aws sts get-caller-identity
kubectl auth can-i --list -n <ns>
aws eks list-access-entries --cluster-name prod

# Cluster health & upgrades
aws eks describe-cluster --name prod \
  --query 'cluster.{status:status,version:version}'
aws eks list-insights --cluster-name prod
aws eks describe-addon --cluster-name prod \
  --addon-name vpc-cni --query 'addon.health'

# Karpenter / Auto Mode
kubectl get nodepools,nodeclaims
kubectl describe nodeclaim <name>   # events, conditions
# self-managed Karpenter only (namespace may differ):
kubectl logs -n kube-system -l app.kubernetes.io/name=karpenter

# CNI / IPs
kubectl logs -n kube-system -l k8s-app=aws-node -c aws-node
aws ec2 describe-subnets --subnet-ids subnet-0abc \
  --query 'Subnets[].AvailableIpAddressCount'
Chapter 7

Hands-on labs

Build, secure, scale, observe, upgrade, break, and tear down real EKS clusters — one lab per part of this deck.

Lab setup

Before you start: tools, permissions, and cost

Use a sandbox account. Every lab assumes Region ap-northeast-1 and a cluster named lab.

Tools

  • AWS CLI v2, kubectl (±1 minor of the cluster), eksctl, helm.
  • Optional: k9s, jq, hey or busybox for load.
  • An IAM principal with admin rights in the sandbox (eksctl creates IAM roles, VPC, and CloudFormation stacks).
  • aws sts get-caller-identity before anything else.

What bills while the lab runs

ItemPrice driver
Control planeUS$0.10 per cluster-hour (standard support); US$0.60 in extended support
Auto ModePer-instance management fee on top of EC2
NodesEC2 instance-hours (Spot is cheaper)
NAT gatewayHourly + per-GB processed
Load balancersHourly + LCU for each ALB/NLB
EBS, logsGB-month; CloudWatch Logs ingestion

Rules

  • Create an AWS Budgets alert first (e.g. US$20).
  • Pin the version: build at 1.35 so Lab 14 can upgrade to 1.36.
  • Never leave a cluster overnight.
  • Run the cleanup lab at the end — load balancers and volumes outlive kubectl sessions.
Lab 1

Cluster as code with an eksctl config file

Goal: replace the one-line eksctl create cluster with a reviewable file you can rebuild from.

# cluster.yaml
apiVersion: eksctl.io/v1alpha5
kind: ClusterConfig
metadata: { name: lab, region: ap-northeast-1, version: "1.35" }
autoModeConfig: { enabled: true }      # general-purpose + system pools
accessConfig: { authenticationMode: API }
vpc:
  clusterEndpoints: { publicAccess: true, privateAccess: true }
  publicAccessCIDRs: ["203.0.113.10/32"] # your IP only
cloudWatch:
  clusterLogging: { enableTypes: ["api", "audit", "authenticator"] }

eksctl create cluster -f cluster.yaml              # ~15 min
aws eks describe-cluster --name lab --query \
  'cluster.{v:version,auth:accessConfig,ep:resourcesVpcConfig}'

Verify / Break it

  • kubectl get nodes is empty until a pod needs capacity — that is Auto Mode working.
  • Remove your IP from publicAccessCIDRs (eksctl utils update-cluster-vpc-config) and watch kubectl time out; restore it.
  • aws logs describe-log-groups --log-group-name-prefix /aws/eks/lab shows the control-plane log group.
  • Production equivalent: terraform-aws-modules/eks — same settings, one PR per change.
Lab 2

Access entries for a second IAM role

Goal: give a read-only team role access to one namespace — no aws-auth ConfigMap.

ACCT=$(aws sts get-caller-identity --query Account --output text)
ROLE=arn:aws:iam::$ACCT:role/eks-viewer   # role you can assume
kubectl create namespace demo

aws eks create-access-entry --cluster-name lab \
  --principal-arn $ROLE --type STANDARD
aws eks associate-access-policy --cluster-name lab \
  --principal-arn $ROLE \
  --policy-arn arn:aws:eks::aws:cluster-access-policy/AmazonEKSViewPolicy \
  --access-scope type=namespace,namespaces=demo

aws eks update-kubeconfig --name lab --region ap-northeast-1 \
  --role-arn $ROLE --alias lab-viewer
kubectl --context lab-viewer get pods -n demo        # allowed
kubectl --context lab-viewer get pods -n kube-system # Forbidden
kubectl --context lab-viewer auth can-i delete pods -n demo  # no

Verify / Break it

  • aws eks list-access-entries --cluster-name lab shows your creator entry, the insights entry, and the new one.
  • Swap to AmazonEKSEditPolicy and re-run auth can-i.
  • Delete the entry: the role gets Unauthorized (authentication), not Forbidden (authorization). Know the difference.
  • Find the CreateAccessEntry event in CloudTrail.
Lab 3

Pod Identity: S3 access without keys

Goal: one pod can list S3; every other pod cannot. Auto Mode has the Pod Identity agent built in.

cat > trust.json <<'EOF'
{"Version":"2012-10-17","Statement":[{"Effect":"Allow",
 "Principal":{"Service":"pods.eks.amazonaws.com"},
 "Action":["sts:AssumeRole","sts:TagSession"]}]}
EOF
aws iam create-role --role-name lab-s3-reader \
  --assume-role-policy-document file://trust.json
aws iam attach-role-policy --role-name lab-s3-reader \
  --policy-arn arn:aws:iam::aws:policy/AmazonS3ReadOnlyAccess
kubectl create sa s3-reader -n demo
aws eks create-pod-identity-association --cluster-name lab \
  --namespace demo --service-account s3-reader \
  --role-arn arn:aws:iam::$ACCT:role/lab-s3-reader
kubectl run cli -n demo --image=public.ecr.aws/aws-cli/aws-cli \
  --overrides='{"spec":{"serviceAccountName":"s3-reader"}}' \
  --command -- sleep 3600
kubectl exec -n demo cli -- aws s3 ls

Verify / Break it

  • kubectl exec … -- env | grep AWS_CONTAINER shows the agent endpoint, not keys.
  • Run the same pod with the default SA: aws s3 ls fails with no credentials.
  • Delete the association, restart the pod → AccessDenied / no credentials.
  • Compare IRSA: needs an OIDC provider and a per-cluster trust policy; Pod Identity reuses one trust across clusters.
  • On MNG clusters install the eks-pod-identity-agent add-on first.
Lab 4

Expose apps: NLB for a Service, ALB for an Ingress

Goal: see which Kubernetes object creates which AWS load balancer. Auto Mode runs the controller for you.

# Service → NLB (default loadBalancerClass: eks.amazonaws.com/nlb)
apiVersion: v1
kind: Service
metadata:
  name: web-nlb
  annotations:
    service.beta.kubernetes.io/aws-load-balancer-scheme: internet-facing
spec:
  type: LoadBalancer
  selector: { app: web }
  ports: [{ port: 80, targetPort: 80 }]
---
apiVersion: eks.amazonaws.com/v1
kind: IngressClassParams
metadata: { name: alb }
spec: { scheme: internet-facing }
---
apiVersion: networking.k8s.io/v1
kind: IngressClass
metadata: { name: alb }
spec:
  controller: eks.amazonaws.com/alb
  parameters:
    { apiGroup: eks.amazonaws.com, kind: IngressClassParams, name: alb }

Then

  • Create an Ingress with ingressClassName: alb, path /, backend web port 80 (needs a ClusterIP or NodePort Service).
  • kubectl get svc,ingress -w until an address appears (2–4 min); curl both.
  • In the EC2 console: one NLB, one ALB, target type ip — targets are pod IPs.
  • Break it: remove the kubernetes.io/role/elb tag from public subnets in a test VPC — the load balancer stays pending; kubectl describe shows why.
  • Self-managed clusters: install the AWS Load Balancer Controller with Helm and use its elbv2.k8s.aws APIs instead.
Lab 5

NetworkPolicy: default-deny, then restore DNS

Goal: prove isolation works and learn why DNS is the first thing a deny-all breaks.

# Auto Mode: turn on the network policy controller once
kubectl apply -f - <<'EOF'
apiVersion: v1
kind: ConfigMap
metadata: { name: amazon-vpc-cni, namespace: kube-system }
data: { enable-network-policy-controller: "true" }
EOF
# deny all ingress + egress in demo
kubectl apply -n demo -f - <<'EOF'
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata: { name: default-deny }
spec: { podSelector: {}, policyTypes: [Ingress, Egress] }
EOF
kubectl exec -n demo cli -- aws s3 ls     # now times out
aws eks describe-cluster --name lab --query \
  cluster.kubernetesNetworkConfig.serviceIpv4Cidr   # e.g. 10.100.0.0/16

Fix it step by step

  • Allow egress UDP/TCP 53 to the CoreDNS IP. In Auto Mode CoreDNS runs on each node at the .10 address of the service CIDR (e.g. 10.100.0.10/32), so use an ipBlock, not a pod selector.
  • DNS now resolves but S3 still fails: add egress TCP 443.
  • Allow ingress to web only from pods labelled role=frontend; test with two curl pods.
  • On MNG clusters enable it on the vpc-cni add-on (enableNetworkPolicy: "true") and select CoreDNS with k8s-app: kube-dns.
Lab 6

Workload security: PSA, image scanning, detection

Goal: block unsafe pods at admission and see vulnerabilities before deploy.

# 1. Pod Security Admission: enforce restricted on demo
kubectl label ns demo pod-security.kubernetes.io/enforce=restricted
kubectl run bad -n demo --image=nginx
#  → Error: violates PodSecurity "restricted:latest" (runAsNonRoot …)
kubectl label ns demo pod-security.kubernetes.io/warn=restricted \
  pod-security.kubernetes.io/enforce=baseline --overwrite

# 2. ECR: scan on push, immutable tags
aws ecr create-repository --repository-name lab/web \
  --image-scanning-configuration scanOnPush=true \
  --image-tag-mutability IMMUTABLE
# push an image, then:
aws ecr describe-image-scan-findings --repository-name lab/web \
  --image-id imageTag=1.0 \
  --query 'imageScanFindings.findingSeverityCounts'

Verify / Break it

  • Rewrite the bad pod with the hardened spec from the Security part (non-root, drop ALL, seccomp RuntimeDefault) until it is admitted.
  • Push the same tag twice: IMMUTABLE rejects the second push.
  • All Kubernetes API data, Secrets included, is envelope-encrypted by default on 1.28+ (AWS owned key); describe-cluster --query cluster.encryptionConfig shows your own KMS key only if you set one.
  • Optional: enable GuardDuty EKS Protection in the sandbox and look for findings after running kubectl run with a privileged pod in a non-restricted namespace.
Lab 7

Persistent storage: gp3 volumes and a StatefulSet

Goal: watch an EBS volume follow a pod, and learn why volumes are AZ-bound.

kubectl apply -f - <<'EOF'
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata: { name: auto-ebs-sc }
provisioner: ebs.csi.eks.amazonaws.com   # Auto Mode block storage
volumeBindingMode: WaitForFirstConsumer
allowedTopologies:
- matchLabelExpressions:
  - { key: eks.amazonaws.com/compute-type, values: [auto] }
parameters: { type: gp3, encrypted: "true" }
EOF
# StatefulSet "db" with volumeClaimTemplates → storageClassName: auto-ebs-sc
kubectl get pvc,pv -n demo
kubectl exec -n demo db-0 -- sh -c 'date >> /data/log; cat /data/log'
kubectl delete pod -n demo db-0          # comes back, same data

Verify / Break it

  • aws ec2 describe-volumes --filters Name=tag-key,Values=kubernetes.io/created-for/pvc/name shows the gp3 volume and its AZ.
  • Scale the StatefulSet to 0 and back: the PVC and data survive.
  • Delete the PVC: with reclaimPolicy: Delete the EBS volume goes too.
  • Snapshots: install the CSI Snapshot Controller add-on, create a VolumeSnapshotClass and a VolumeSnapshot, then restore into a new PVC.
  • MNG clusters use the aws-ebs-csi-driver add-on and provisioner ebs.csi.aws.com.
Lab 8

Autoscaling: HPA scales pods, Auto Mode adds nodes

Goal: drive load, watch replicas then nodes grow, and see consolidation shrink them.

aws eks create-addon --cluster-name lab \
  --addon-name metrics-server
kubectl set resources deploy web -n demo \
  --requests=cpu=200m,memory=128Mi
kubectl apply -n demo -f - <<'EOF'
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata: { name: web }
spec:
  scaleTargetRef: { apiVersion: apps/v1, kind: Deployment, name: web }
  minReplicas: 2
  maxReplicas: 30
  metrics:
  - type: Resource
    resource:
      name: cpu
      target: { type: Utilization, averageUtilization: 50 }
EOF
kubectl run load -n demo --image=busybox:1.36 --restart=Never -- \
  sh -c 'while true; do wget -q -O- http://web; done'
kubectl get hpa,pods -n demo -w ; kubectl get nodeclaims -w

Verify / Break it

  • kubectl top pods -n demo works only after metrics-server is ready.
  • Replicas rise first; Pending pods trigger new NodeClaims within about a minute.
  • Delete the load pod: HPA scales down after its stabilization window, then empty or underused nodes are consolidated.
  • Remove the CPU request: HPA reports <unknown> — utilization needs requests.
  • If demo still enforces restricted PSA from Lab 6, the busybox load pod is rejected — use baseline or harden it.
Lab 9

Cost: a Spot NodePool with limits and consolidation

Goal: run stateless work on Spot, cap spend, and see how a workload chooses a pool.

kubectl apply -f - <<'EOF'
apiVersion: karpenter.sh/v1
kind: NodePool
metadata: { name: spot }
spec:
  template:
    spec:
      nodeClassRef:
        { group: eks.amazonaws.com, kind: NodeClass, name: default }
      requirements:
      - key: karpenter.sh/capacity-type
        operator: In
        values: [spot]
      - key: eks.amazonaws.com/instance-category
        operator: In
        values: [c, m, r]
  limits: { cpu: "32" }
  disruption:
    consolidationPolicy: WhenEmptyOrUnderutilized
    consolidateAfter: 1m
EOF
kubectl patch deploy web -n demo --type=json -p '[{"op":"add",
  "path":"/spec/template/spec/nodeSelector",
  "value":{"karpenter.sh/capacity-type":"spot"}}]'
kubectl get nodes -L karpenter.sh/capacity-type

Verify / Break it

  • New pods land on Spot nodes; compare instance types chosen.
  • Set limits.cpu: "2" and scale up: pods stay Pending once the cap is reached — a budget guardrail.
  • Rightsize: compare kubectl top pods with requests; lower requests and watch consolidation pack nodes.
  • Keep stateful and singleton pods on on-demand; add a PDB before trusting Spot interruptions.
Lab 10

Self-run compute: managed node group + Karpenter

Goal: build the non-Auto Mode path once, so you know what Auto Mode hides. Use a second cluster, lab-k.

# Follow karpenter.sh Getting Started (current release)
export CLUSTER_NAME=lab-k KARPENTER_NAMESPACE=kube-system
export KARPENTER_VERSION="<current, e.g. 1.15.0>"
# 1. IAM via the release's CloudFormation template
aws cloudformation deploy --stack-name "Karpenter-${CLUSTER_NAME}" \
  --template-file cloudformation.yaml \
  --capabilities CAPABILITY_NAMED_IAM \
  --parameter-overrides "ClusterName=${CLUSTER_NAME}"
# 2. eksctl config: managedNodeGroups (2 × m5.large AL2023),
#    podIdentityAssociations for the karpenter SA, eks-pod-identity-agent
eksctl create cluster -f lab-k.yaml
# 3. Controller
helm upgrade --install karpenter \
  oci://public.ecr.aws/karpenter/karpenter \
  --version "${KARPENTER_VERSION}" --namespace "${KARPENTER_NAMESPACE}" \
  --set settings.clusterName=${CLUSTER_NAME} \
  --set settings.interruptionQueue=${CLUSTER_NAME} --wait
# 4. NodePool + EC2NodeClass (karpenter.k8s.aws/v1,
#    amiSelectorTerms: alias al2023@latest)

Which one when?

NeedPick
Least ops, small platform teamAuto Mode
Custom AMIs, DaemonSet-heavy, SSH/SSM to nodesMNG + Karpenter
Steady baseline, simple scalingMNG (+ Cluster Autoscaler)
Karpenter controller placementOn the MNG (or Fargate), never on nodes it manages
Core add-onsAuto Mode: built in · MNG: you upgrade them
Lab 11

Add-ons: versions, updates, and conflicts

Goal: manage the platform layer on lab-k the way you would in production.

aws eks list-addons --cluster-name lab-k
aws eks describe-addon-versions --addon-name vpc-cni \
  --kubernetes-version 1.35 \
  --query 'addons[].addonVersions[].addonVersion'

aws eks update-addon --cluster-name lab-k --addon-name vpc-cni \
  --addon-version <version> --resolve-conflicts PRESERVE
aws eks describe-addon --cluster-name lab-k --addon-name vpc-cni \
  --query 'addon.{status:status,v:addonVersion,health:health}'

# configuration values (JSON schema per add-on)
aws eks describe-addon-configuration --addon-name vpc-cni \
  --addon-version <version> --query configurationSchema \
  --output text | jq .

Verify / Break it

  • Edit the aws-node DaemonSet by hand, then update with --resolve-conflicts NONE: the update fails on the conflict. OVERWRITE wins; PRESERVE keeps your change.
  • Enable prefix delegation through configuration values (ENABLE_PREFIX_DELEGATION) and compare max pods on a new node.
  • Rule: upgrade add-ons right after each control-plane minor upgrade, one step at a time.
  • On Auto Mode (lab) the CNI, CoreDNS, kube-proxy, and EBS CSI do not appear as add-ons — AWS runs them.
Lab 12

Observability: control-plane logs, metrics, and events

Goal: answer "who deleted my deployment?" and "why is this pod restarting?" from data, not guesses.

# Logs Insights on /aws/eks/lab/cluster
fields @timestamp, user.username, verb, objectRef.name
| filter @logStream like /kube-apiserver-audit/
| filter verb = "delete" and objectRef.resource = "deployments"
| sort @timestamp desc | limit 20

# From the shell
kubectl delete deploy web -n demo     # create the evidence first
kubectl get events -n demo --sort-by=.lastTimestamp | tail
kubectl top nodes ; kubectl top pods -A --sort-by=memory | head
aws cloudwatch list-metrics --namespace AWS/EKS \
  --dimensions Name=ClusterName,Value=lab --query 'Metrics[].MetricName'

Verify / Break it

  • The audit query names your IAM identity — access entries make this traceable.
  • Make a pod OOM (--limits=memory=32Mi on a memory-hungry image) and find OOMKilled in describe and events.
  • Optional: install the amazon-cloudwatch-observability add-on (Pod Identity for its agent; check describe-addon-versions for your cluster) and open Container Insights.
  • Turn off log types you do not need afterwards: audit logs are the most voluminous.
Lab 13

Reliability: PDBs, drains, spread, and a broken rollout

Goal: keep a service up through node maintenance and a bad deploy.

kubectl create deploy web -n demo --replicas=4 \
  --image=public.ecr.aws/nginx/nginx
kubectl create pdb web -n demo --selector=app=web --min-available=3
# spread across zones (add to the pod template)
#   topologySpreadConstraints:
#   - maxSkew: 1
#     topologyKey: topology.kubernetes.io/zone
#     whenUnsatisfiable: ScheduleAnyway
#     labelSelector: { matchLabels: { app: web } }
kubectl get pods -n demo -o wide -L topology.kubernetes.io/zone

NODE=$(kubectl get pods -n demo -l app=web \
  -o jsonpath='{.items[0].spec.nodeName}')
kubectl drain $NODE --ignore-daemonsets --delete-emptydir-data
#  → evictions wait whenever they would break minAvailable

# broken rollout: an image tag that does not exist
kubectl set image deploy/web -n demo \
  nginx=public.ecr.aws/nginx/nginx:does-not-exist
kubectl rollout status deploy/web -n demo --timeout=60s
kubectl rollout undo deploy/web -n demo

Verify / Break it

  • During the drain kubectl get pdb -n demo shows ALLOWED DISRUPTIONS hitting 0.
  • Set --min-available=4 with 4 replicas: the drain blocks forever — a classic upgrade blocker.
  • The bad image leaves old pods serving (maxUnavailable) while new ones sit in ImagePullBackOff.
  • Add a readiness probe on /healthz that returns 404: pods run but never become Ready, and the rollout stalls the same way.
Lab 14

Upgrade the control plane — and roll it back

Goal: run the upgrade runbook from the Upgrades part on lab (built at 1.35).

aws eks start-insights-refresh --cluster-name lab
aws eks list-insights --cluster-name lab \
  --filter '{"categories":["UPGRADE_READINESS"],
            "statuses":["ERROR","WARNING"]}'

aws eks update-cluster-version --name lab --kubernetes-version 1.36
#   or: eksctl upgrade cluster --name lab --version 1.36 --approve
aws eks describe-update --name lab --update-id <id>  # → Successful
kubectl version ; kubectl get nodes -o wide   # kubelet versions
# Auto Mode replaces nodes; MNG: update-nodegroup-version

# within 7 days you may roll back one minor version
aws eks list-insights --cluster-name lab \
  --filter '{"categories":["ROLLBACK_READINESS"]}'
aws eks update-cluster-version --name lab --kubernetes-version 1.35

Order on non-Auto clusters

  • 1 · Insights clean, deprecated APIs migrated (pluto, kubent).
  • 2 · Control plane, one minor at a time.
  • 3 · Add-ons: vpc-cni, coredns, kube-proxy, CSI, LB controller, Karpenter.
  • 4 · Nodes: MNG rolling update or Karpenter drift.
  • Break it: keep the 4-of-4 PDB from Lab 13 — node replacement stalls until you fix it.
Lab 15

Break-fix drills

Have a partner cause one of these without telling you which; fix it using only the cluster and AWS CLI.

InjectSymptomWhere to lookFix
NodePool limits.cpu too lowPods Pending, no new nodesdescribe pod events, kubectl get nodepools, NodeClaim eventsRaise limits or rightsize requests
Small subnet / no free IPs (lab-k)Pods stuck ContainerCreatingaws-node / ipamd logs, subnet free IP countPrefix delegation, bigger or secondary CIDR
Pod Identity association removedApp gets AccessDenied / no credentialslist-pod-identity-associations, CloudTrail AssumeRoleForPodIdentityRecreate association; check trust and SA name
Public subnets missing ELB tagService EXTERNAL-IP stays <pending>kubectl describe svc eventsTag subnets kubernetes.io/role/elb=1
Wrong image tagImagePullBackOffdescribe pod; ECR repo; node pull permissionsFix tag / digest; ECR pull policy
Default-deny without DNS allowTimeouts, no such hostkubectl get networkpolicy; test with nslookupAllow port 53 to CoreDNS
Lab 16

Cleanup: release AWS resources first, then the clusters

Goal: zero orphaned load balancers, volumes, or IAM roles after the labs.

# 1. Delete what creates AWS resources, while controllers still run
kubectl delete ingress,svc --all -n demo
kubectl delete statefulset,pvc --all -n demo
kubectl delete nodepool spot
# 2. Clusters (and Karpenter's CloudFormation stack)
eksctl delete cluster --name lab --region ap-northeast-1 --wait
eksctl delete cluster --name lab-k --region ap-northeast-1 --wait
aws cloudformation delete-stack --stack-name Karpenter-lab-k
# 3. Prove nothing is left
aws elbv2 describe-load-balancers --query 'LoadBalancers[].LoadBalancerName'
aws ec2 describe-volumes --filters Name=status,Values=available \
  --query 'Volumes[].[VolumeId,Size,CreateTime]'
aws iam delete-role --role-name lab-s3-reader   # after detaching its policy
aws ecr delete-repository --repository-name lab/web --force

Why this order

  • Deleting a cluster first kills the controllers that would remove load balancers and volumes — they are then orphaned and keep billing.
  • Check the NAT gateway and Elastic IPs are gone with the eksctl VPC stack.
  • Delete the control-plane log group if you do not want to keep it.
  • Look at Cost Explorer the next day to confirm.
Lab checklist

You have finished the EKS labs when you can…

Tick each one without looking at the slides.

Build and access

  • Rebuild lab from cluster.yaml alone.
  • Grant a role namespace-scoped read access with access entries.
  • Explain Unauthorized vs Forbidden.
  • Give one pod S3 access with Pod Identity, and explain IRSA.

Run and protect

  • Expose a service through an NLB and an ALB.
  • Write a default-deny policy that keeps DNS working.
  • Get an unsafe pod rejected by PSA.
  • Keep data across pod restarts with a gp3 StatefulSet.

Scale and operate

  • Show HPA → NodeClaim → consolidation end to end.
  • Find who deleted a deployment in the audit log.
  • Upgrade one minor version and roll it back.
  • Clean up with zero orphaned resources.
Checklist

Production readiness checklist

Foundation

  • Cluster built from IaC, version pinned
  • ≥ 3 AZs, large subnets, VPC endpoints
  • Private / restricted API endpoint
  • API auth mode + access entries
  • Control plane logs enabled
  • Upgrade cadence within standard support

Security

  • Pod Identity per app, IMDS blocked
  • PSA restricted, policy engine
  • Default-deny NetworkPolicy
  • Secrets in Secrets Manager
  • Image scanning & signing
  • GuardDuty EKS + Runtime Monitoring
  • Audit-log queries & IR runbook ready

Operations

  • Karpenter or Auto Mode, Spot + Graviton
  • Requests, probes, PDBs, topology spread
  • HPA/KEDA on real signals
  • Control plane + node + app metrics, logs, traces
  • SLO burn-rate alerts with runbooks
  • GitOps delivery, backups tested
  • Cost visibility per team
Glossary

Terms you will meet

MNGManaged node group — EKS-managed Auto Scaling groupIRSAIAM Roles for Service Accounts (OIDC-based)
Auto ModeAWS-managed nodes and core add-onsPod IdentityEKS-native pod → IAM role mapping
KarpenterNode autoscaler that launches right-sized EC2Access entryIAM principal → cluster access mapping
NodePool / NodeClaimKarpenter's node template / a requested nodePSA / PSSPod Security Admission / Standards
VPC CNIPlugin giving pods VPC IPsPDBPodDisruptionBudget — limits voluntary evictions
ENI / prefix delegationNetwork interface / /28 blocks per slotHPA / VPA / KEDAHorizontal, vertical, event-driven pod autoscaling
CSIContainer Storage Interface driversAMP / AMGAmazon Managed Prometheus / Grafana
Add-onEKS-managed cluster componentADOTAWS Distro for OpenTelemetry
Extended supportExtra 12 months per version at $0.60/hSLOService level objective with an error budget
Further reading

References

  • EKS Best Practices Guide — docs.aws.amazon.com/eks/latest/best-practices/
  • EKS User Guide — docs.aws.amazon.com/eks/latest/userguide/
  • EKS Kubernetes version calendar — User Guide › Kubernetes versions
  • EKS Workshop — eksworkshop.com
  • Karpenter — karpenter.sh
  • AWS Load Balancer Controller — kubernetes-sigs.github.io/aws-load-balancer-controller
  • terraform-aws-modules/eks — registry.terraform.io
  • Kubernetes docs — kubernetes.io/docs

Prices and version support windows change — verify against the AWS pages before making decisions.

1 / 1