Platform engineering deck

Amazon EKS
from A to Z

Architecture, compute, networking, identity, security, storage, scaling, observability, upgrades, cost — and the best practices for each.

Use ← → / Space to navigate · O for overview · T toggles theme · F fullscreen

Agenda

What we cover

Each section ends with concrete best practices.

  1. Fundamentals
  2. Architecture & shared responsibility
  3. Creating clusters & versions
  4. Compute: MNG, Fargate, Karpenter, Auto Mode
  5. Networking, load balancing, DNS
  6. Identity: access entries & Pod IAM
  7. Security hardening
  8. Storage
  9. Autoscaling
  10. Add-ons
  11. Observability & monitoring
  12. Workload reliability
  13. Upgrades & lifecycle
  14. Cost optimization
  15. GitOps, multi-tenancy, DR, troubleshooting
Section 01

Fundamentals

Kubernetes in one slide, and what EKS adds on top.

Kubernetes recap

The core objects you deploy every day

Kubernetes is a declarative control loop: you describe desired state, controllers reconcile it.

Workloads

  • Pod — smallest unit, 1+ containers
  • Deployment → ReplicaSet — stateless
  • StatefulSet — stable identity + volumes
  • DaemonSet — one per node
  • Job / CronJob — batch

Networking

  • Service — ClusterIP / NodePort / LoadBalancer
  • Ingress / Gateway API — L7 routing
  • NetworkPolicy — pod firewall
  • CoreDNS — svc.ns.svc.cluster.local

Config & control

  • ConfigMap, Secret
  • PersistentVolumeClaim, StorageClass
  • ServiceAccount, Role, RoleBinding
  • Namespace, ResourceQuota, LimitRange
  • CRDs + controllers/operators
Why EKS

What Amazon EKS gives you

Upstream, CNCF-conformant Kubernetes with a managed control plane and deep AWS integration.

Managed for you

  • API server + etcd run across 3 AZs in an AWS-owned account
  • Control plane scaling, patching, backups of etcd
  • Optional management of nodes (Auto Mode), add-ons, and capabilities
  • SLA-backed regional endpoint

AWS integrations

  • IAM for cluster auth (access entries) and pod auth (IRSA / Pod Identity)
  • VPC-native pod IPs (VPC CNI), ALB/NLB, Route 53
  • EBS, EFS, FSx, S3 via CSI drivers
  • CloudWatch, Prometheus (AMP), GuardDuty, ECR, KMS

Pricing basics: $0.10 per cluster-hour in standard support, $0.60 per cluster-hour in extended support — plus the compute, storage, and data transfer you use. Auto Mode adds a per-instance management fee; Provisioned Control Plane tiers (for very large clusters) and EKS Capabilities are billed on top.

How it works

From kubectl apply to a running pod

Know this path by heart — almost every troubleshooting session maps to one of these steps.

1 · API server

  • AuthN: IAM token from aws eks get-token
  • AuthZ: RBAC / EKS access policies
  • Admission: mutating → validating webhooks, Pod Security Admission
  • Object persisted to etcd

2 · Controllers

  • Deployment controller creates a ReplicaSet
  • ReplicaSet controller creates Pods with no node yet
  • Every controller is a loop: watch → diff → act

3 · Scheduler

  • Filters nodes (resources, taints, affinity, topology, volumes)
  • Scores and binds the pod to the best node
  • No fit → Pending → Karpenter / Auto Mode launches a node

4 · kubelet

  • Sees the pod bound to its node
  • Creates the pod sandbox — VPC CNI assigns a VPC IP
  • Mounts volumes (CSI), injects env/secrets

5 · Containers

  • Pulls the image (ECR), containerd starts containers
  • Startup → readiness → liveness probes run
  • Pod becomes Ready

6 · Traffic

  • EndpointSlice updated with the pod IP
  • kube-proxy programs Service rules
  • LB Controller registers the pod IP in the ALB/NLB target group
Section 02

Architecture &
shared responsibility

Architecture

Control plane vs data plane

AWS-managed VPC — EKS control plane kube-apiserver behind NLB, multi-AZ public/private endpoint etcd 3 AZs, encrypted scheduler controller-manager cloud controller cross-account ENIs in your subnets Your VPC — data plane AZ a · private subnet EC2 node kubelet · VPC CNI Pods VPC IPs AZ b · private subnet EC2 / Auto Mode Karpenter nodes Fargate pod microVM per pod Public subnets ALB / NLB via LB Controller NAT Gateway egress (or endpoints)

Control plane

  • Single-tenant, runs in an AWS-owned VPC
  • At least 2 API server instances and 3 etcd members across 3 AZs
  • EKS places ENIs in your subnets so the API server can reach kubelets (logs, exec, webhooks)

Data plane

  • Your nodes in your VPC: EC2 (MNG, self-managed, Karpenter, Auto Mode), Fargate, or Hybrid Nodes on-prem
  • Nodes register via the API endpoint and authenticate with IAM
Shared responsibility

Who owns what

The more managed the compute option, the more AWS takes on.

AreaSelf-managed nodesManaged node groupsFargateAuto Mode
Control plane, etcdAWSAWSAWSAWS
Node AMI / OS patchingYouYou trigger, AWS provides AMIAWSAWS (nodes recycled, max 21-day lifetime)
Node provisioning & scalingYouYou (ASG + autoscaler)AWS (per pod)AWS (Karpenter-based)
Core add-ons (CNI, CoreDNS, kube-proxy, EBS CSI, LB)YouYouPartially AWSAWS (built in)
Version upgradesYouYouYou (control plane)You (control plane); nodes follow
IAM, RBAC, network policy, app security, dataYouYouYouYou
Section 03

Creating clusters

Clusters are cattle too — always build them from code.

Provisioning

Tools & a minimal Terraform example

Standard EKS control plane; add compute and an access entry on the next slide.

Options

  • Terraform + terraform-aws-modules/eks — most common in production
  • eksctl — fast, opinionated; great for labs
  • CDK / CloudFormation, Pulumi
  • Console — learning only

Best practices

  • Separate repos/stacks: VPC → cluster → platform add-ons → apps
  • Pin the Kubernetes version explicitly
  • Use access entries, not the aws-auth ConfigMap
  • Enable control plane logs from day one
module "eks" {
  source  = "terraform-aws-modules/eks/aws"
  version = "~> 21.0"

  name               = "prod-apne1"
  kubernetes_version = "1.37"  # available Oct 2026
  upgrade_policy    = { support_type = "STANDARD" }

  vpc_id     = module.vpc.vpc_id
  subnet_ids = module.vpc.private_subnets

  endpoint_public_access = false
  authentication_mode = "API"
  enable_cluster_creator_admin_permissions = false

  enabled_log_types = ["api", "audit", "authenticator"]

  addons = {
    vpc-cni                = { before_compute = true }
    coredns                = {}
    kube-proxy             = {}
    eks-pod-identity-agent = { before_compute = true }
  }
}
Terraform completion

Give the cluster nodes and an operator

Add these blocks inside the preceding module. Supply an existing VPC and a predeclared platform_admin_role_arn variable.

# Managed EC2 capacity
eks_managed_node_groups = {
  system = {
    instance_types = ["m7i.large"]
    min_size     = 2
    max_size     = 4
    desired_size = 2
  }
}
# Explicit break-glass administrator
access_entries = {
  admin = {
    principal_arn = var.platform_admin_role_arn
    policy_associations = {
      admin = {
        policy_arn = join("", [
          "arn:aws:eks::aws:cluster-access-policy/",
          "AmazonEKSClusterAdminPolicy"
        ])
        access_scope = { type = "cluster" }
      }
    }
  }
}

The API endpoint is private, so run Terraform and kubectl from a network with VPC access. For Auto Mode, use compute_config instead of this managed node group and omit the standard CNI, CoreDNS, kube-proxy, and Pod Identity Agent add-ons.

Versions

Kubernetes version lifecycle on EKS

Standard support

14 months from the version's EKS release. $0.10/cluster-hour.

Extended support

Additional 12 months. $0.60/cluster-hour. Enabled by default — set the upgrade policy to STANDARD to opt out.

End of support

After extended support ends, EKS auto-upgrades the control plane to the oldest supported version.

Do

  • Plan ~3 minor upgrades per year to stay in standard support
  • Track the EKS version calendar and AMI release notes

Avoid

  • Running in extended support unintentionally — 6× control plane cost
  • Skipping versions: the control plane upgrades one minor at a time
Hands-on lab

Your first learning cluster

# 0. Tools: AWS CLI v2, kubectl, eksctl, helm
aws sts get-caller-identity

# 1. Create a cluster with Auto Mode (~15 min)
eksctl create cluster --name lab \
  --region ap-northeast-1 --enable-auto-mode

# 2. Connect and deploy something
aws eks update-kubeconfig --name lab --region ap-northeast-1
kubectl get nodes        # empty until pods need capacity
kubectl create deployment web \
  --image=public.ecr.aws/nginx/nginx --replicas=2
kubectl get pods -o wide -w   # watch a node appear
kubectl get nodeclaims

# 3. Expose it
kubectl expose deployment web --port=80 --type=ClusterIP
kubectl port-forward svc/web 8080:80

# 4. AFTER the exercises, clean up — clusters bill every hour
eksctl delete cluster --name lab --region ap-northeast-1

Exercises

  • Scale to 20 replicas and watch new nodes launch
  • Add an HPA and generate load
  • Break the readiness probe and see the rollout stop
  • Drain a node and observe PDB behavior
  • Create an access entry for a second IAM role
  • Give a pod S3 access with Pod Identity
  • Install the CloudWatch Observability add-on and read the dashboards
  • Apply a default-deny NetworkPolicy and fix DNS
Auto Mode lab prerequisites

Prepare the follow-up exercises

Run after creating web. Auto Mode supplies compute and Pod Identity; the exercises still need workload resources and policy setup.

# HPA needs resource requests and metrics
kubectl top nodes  # check whether metrics-server exists
# Only if the metrics API is unavailable:
aws eks create-addon --cluster-name lab \
  --region ap-northeast-1 --addon-name metrics-server
kubectl set resources deployment/web \
  --requests=cpu=100m,memory=128Mi \
  --limits=memory=256Mi
kubectl autoscale deployment web \
  --cpu=60% --min=2 --max=20

# Enable Auto Mode network policy enforcement
kubectl apply -f - <<'EOF'
apiVersion: v1
kind: ConfigMap
metadata:
  name: amazon-vpc-cni
  namespace: kube-system
data:
  enable-network-policy-controller: "true"
EOF
# A PDB for the node-drain exercise
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata: { name: web }
spec:
  minAvailable: 1
  selector:
    matchLabels: { app: web }

Exercise order

  • Add a readiness probe to web before testing a broken rollout.
  • Apply the PDB YAML before draining a node; check disruptionsAllowed.
  • After enabling policy enforcement, apply default-deny and allow DNS explicitly.
  • For Pod Identity, create the role, service account and association on the later identity slide; restart the workload using that service account.
Auto Mode network policy lab

Default-deny, then restore DNS

Apply after enabling the Auto Mode Network Policy Controller. The example targets the lab's default namespace.

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata: { name: default-deny, namespace: default }
spec:
  podSelector: {}
  policyTypes: [Ingress, Egress]
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata: { name: allow-dns, namespace: default }
spec:
  podSelector: {}
  policyTypes: [Egress]
  egress:
  - to:
    - namespaceSelector:
        matchLabels:
          kubernetes.io/metadata.name: kube-system
    ports:
    - { protocol: UDP, port: 53 }
    - { protocol: TCP, port: 53 }

Verify the behavior

  • Apply only the first policy. DNS from a pod in default should fail after enforcement converges.
  • Apply the DNS rule, then retry nslookup kubernetes.default.svc.cluster.local from a test pod.
  • Ingress to web remains blocked until you add an explicit ingress allow rule.
  • NetworkPolicies are additive: an existing allow policy can permit traffic despite a separate default-deny policy.
Section 04

Compute options

Data plane

Choosing compute

OptionWhat it isUse whenWatch out
EKS Auto ModeAWS manages nodes (Karpenter-based), CNI, CoreDNS, kube-proxy, EBS CSI, LB controller; Bottlerocket-based, immutable nodesNew clusters, small platform teams, want least opsPer-instance fee; less node-level customization (no SSH/SSM)
Karpenter (self-run)Open-source node autoscaler; provisions right-sized EC2 directlyDiverse workloads, Spot, cost focus, full controlYou run and upgrade Karpenter itself (on MNG or Fargate)
Managed node groupsEKS-managed ASGs with EKS-optimized AMIs, managed rolling updatesStable, homogeneous capacity; system/add-on poolsScaling via Cluster Autoscaler is slower and less bin-packing
Self-managed nodesYour own ASG + launch templateCustom OS / AMI requirementsEverything is on you
FargateServerless pod — one microVM per podIsolation, bursty or low-ops jobsNo DaemonSets, no GPUs, no EBS volumes, no privileged pods; slower start
Hybrid NodesOn-prem/edge machines joined to an EKS control planeData residency, existing hardwareNeeds private connectivity (VPN / Direct Connect)
Nodes

AMIs & node best practices

AMI families

  • Amazon Linux 2023 — default EKS-optimized AMI (AL2 AMIs ended with K8s 1.32)
  • Bottlerocket — minimal, container-only, immutable, atomic updates
  • Accelerated variants for NVIDIA / Neuron GPUs
  • Windows Server AMIs for Windows pods

Best practices

  • Prefer Bottlerocket or AL2023; don't SSH — use SSM if needed
  • Enforce IMDSv2, hop limit 1 unless pods truly need IMDS
  • Use a small dedicated MNG (or Fargate) for critical add-ons like Karpenter
  • Diversify instance types; prefer Graviton (arm64) where images support it
  • Treat nodes as immutable — replace, don't patch in place
  • Set kube-reserved/system-reserved (EKS AMIs set sane defaults)
EKS Auto Mode

Auto Mode in practice

What you get

  • Built-in NodePools: general-purpose and system
  • Custom NodePool + NodeClass (eks.amazonaws.com) for GPU, Spot, or isolated pools
  • Compute, block storage, load balancing, pod networking and DNS managed by AWS
  • Nodes expire after 14 days by default (max 21) — patched by replacement
  • Enable on new or existing clusters; migrate from MNG/Karpenter gradually

Best practices

  • Use it as the default for new clusters unless you need custom AMIs or node access
  • Define PDBs — node replacement respects them
  • Use Auto Mode's own StorageClass (ebs.csi.eks.amazonaws.com) and IngressClass
  • Don't also install self-managed Karpenter / LB controller for the same resources
  • Compare the per-instance fee against the ops time you save
Section 05

Networking

VPC CNI, IP planning, load balancing, DNS.

Amazon VPC CNI

Every pod gets a real VPC IP

This slide describes standard EKS EC2 nodes using the Amazon VPC CNI; Auto Mode uses its managed networking capability.

How it works

  • aws-node DaemonSet attaches ENIs and secondary IPs to each node
  • Pods are directly routable in the VPC — no overlay
  • Max pods per node limited by ENIs × IPs per ENI of the instance type
  • Keeps a warm pool of IPs (WARM_ENI_TARGET, WARM_IP_TARGET)

Scaling IP capacity

  • Prefix delegation — assigns /28 prefixes per slot → many more pods per node
  • Custom networking — pods use a secondary CIDR (e.g. 100.64.0.0/10) via ENIConfig
  • IPv6 clusters — greatly reduce IPv4 pressure; pod density still has node limits
  • Security groups for pods — per-pod SGs via branch ENIs
kubectl set env daemonset aws-node -n kube-system ENABLE_PREFIX_DELEGATION=true WARM_PREFIX_TARGET=1
# Standard EC2 nodes only. Prefer managed add-on configuration values in production.
Under the hood

IP math and how traffic flows

Secondary-IP calculations below apply to standard EC2 nodes; Auto Mode manages prefixes and maxPods differently.

Max pods per node

  • Secondary-IP mode:
    ENIs × (IPs per ENI − 1) + 2
  • m5.large: 3 × (10 − 1) + 2 = 29 pods
  • Prefix delegation: each slot holds a /28 (16 IPs); EKS recommends capping at 110 (< 30 vCPU) or 250
  • Each node holds warm IPs too — size subnets for nodes × pods + warm pool

Pod-to-pod

  • Same node: veth pair + host routes
  • Across nodes: plain VPC routing — no overlay, no encapsulation
  • To the internet: SNAT to the node's primary IP by default (or external SNAT via NAT GW)
  • Node security groups apply unless you use Security Groups for Pods

Services

  • ClusterIP: virtual IP; kube-proxy (iptables/IPVS/nftables) DNATs to pod IPs
  • NodePort: port on every node
  • LoadBalancer: NLB via LB Controller; IP mode targets pods directly
  • externalTrafficPolicy: Local keeps the client IP and avoids an extra hop
VPC design

Network best practices

Plan IPs generously

  • Use large private subnets (/19 or bigger) for nodes and pods
  • Add a secondary CIDR (100.64.0.0/10) when RFC1918 space is scarce
  • Give the cluster dedicated subnets for control plane ENIs (small, /28+)
  • Tag subnets: kubernetes.io/role/elb and kubernetes.io/role/internal-elb

Connectivity & cost

  • Spread nodes across ≥ 3 AZs
  • Use VPC endpoints (ECR api/dkr, S3 gateway, STS, EC2, ELB, EKS, EKS Auth, CloudWatch Logs) to cut NAT cost and enable private clusters
  • Enable VPC CNI native network policies (or Calico/Cilium)
  • Watch cross-AZ traffic — use topology-aware routing where it helps
Load balancing

Exposing services: AWS Load Balancer Controller

Example for standard EKS with the self-managed controller. Auto Mode uses its built-in load balancing and its own IngressClass controller.

Mapping

  • Ingress → ALB (L7, path/host routing, WAF, Cognito/OIDC)
  • Service type: LoadBalancer → NLB (L4, static IPs, PrivateLink)
  • IngressGroup shares one ALB across many Ingresses
  • Gateway API supported for ALB/NLB, and VPC Lattice via its own controller

Best practices

  • Use target-type: ip — traffic goes straight to pods
  • Add a preStop sleep + tune deregistration delay to avoid 502s on rollout
  • Enable pod readiness gates for zero-downtime deploys
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: web
  annotations:
    alb.ingress.kubernetes.io/scheme: internet-facing
    alb.ingress.kubernetes.io/target-type: ip
    alb.ingress.kubernetes.io/group.name: public
    alb.ingress.kubernetes.io/listen-ports: '[{"HTTPS":443}]'
    alb.ingress.kubernetes.io/certificate-arn: arn:aws:acm:...
spec:
  ingressClassName: alb
  rules:
  - host: app.example.com
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service: { name: web, port: { number: 80 } }
DNS & endpoints

DNS and API endpoint access

DNS

  • CoreDNS — in-cluster resolution; run ≥ 2 replicas across AZs, consider autoscaling
  • NodeLocal DNSCache — reduces latency and VPC resolver throttling (1024 pps per ENI)
  • ExternalDNS — syncs Ingress/Service hostnames to Route 53
  • Lower ndots or use FQDNs with trailing dot for heavy external lookups

Cluster endpoint

  • Public — reachable from internet (IAM-authenticated)
  • Public + private — nodes use private path; restrict public with CIDR allow-list
  • Private only — reach via VPN / Direct Connect / bastion / SSM
  • ✓ Best practice: private (or public with tight CIDRs) for production
Section 06

Identity & access

Humans → cluster, and pods → AWS.

Cluster access

Access entries replace aws-auth

How auth works

  • AuthN: IAM principal → token via aws eks get-token
  • AuthZ: Kubernetes RBAC and/or EKS access policies
  • Auth modes: CONFIG_MAP → API_AND_CONFIG_MAP → API (one-way migration)
  • Access policies: AmazonEKSClusterAdminPolicy (cluster scope), AmazonEKSAdminPolicy, AmazonEKSEditPolicy, AmazonEKSViewPolicy

Best practices

  • Use API mode; manage entries in IaC
  • Map IAM Identity Center roles, not IAM users
  • Disable creator admin; grant break-glass role explicitly
  • Least-privilege, namespace-scoped access for teams
aws eks create-access-entry \
  --cluster-name prod \
  --principal-arn arn:aws:iam::111122223333:role/TeamA

POLICY=arn:aws:eks::aws:cluster-access-policy
aws eks associate-access-policy \
  --cluster-name prod \
  --principal-arn arn:aws:iam::111122223333:role/TeamA \
  --policy-arn $POLICY/AmazonEKSEditPolicy \
  --access-scope \
    type=namespace,namespaces=team-a

# Or bind to RBAC groups instead of access policies:
aws eks create-access-entry ... \
  --kubernetes-groups team-a-devs
Pod IAM

IRSA vs EKS Pod Identity

IRSA (IAM Roles for Service Accounts)EKS Pod Identity
MechanismOIDC provider + projected SA token → AssumeRoleWithWebIdentityeks-pod-identity-agent DaemonSet + association via EKS API
Trust policyPer cluster OIDC issuer, per SA — must be edited for each clusterOne trust to pods.eks.amazonaws.com — reusable across clusters
SetupCreate OIDC provider, annotate SA with role ARNaws eks create-pod-identity-association
Session tags / ABACNoYes (cluster, namespace, SA tags)
Works onEC2, Fargate, any cluster incl. self-managed K8sLinux EC2 nodes incl. Auto Mode (not Fargate, not Windows)

Best practice

Default to Pod Identity for new EKS workloads; one role per app; never use node instance role permissions for apps.

Block IMDS

Set IMDSv2 hop limit to 1 so pods can't steal the node role credentials.

Kubernetes RBAC

Roles, bindings, and least privilege

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata: { name: deployer, namespace: team-a }
rules:
- apiGroups: ["apps"]
  resources: ["deployments"]
  verbs: ["get", "list", "watch", "create", "update", "patch"]
- apiGroups: [""]
  resources: ["pods", "pods/log", "services", "configmaps"]
  verbs: ["get", "list", "watch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata: { name: team-a-deployers, namespace: team-a }
subjects:
- kind: Group
  name: team-a-devs   # from access entry --kubernetes-groups
  apiGroup: rbac.authorization.k8s.io
roleRef:
  kind: Role
  name: deployer
  apiGroup: rbac.authorization.k8s.io

Concepts

  • Role = namespaced; ClusterRole = cluster-wide or reusable
  • Subjects: User, Group, ServiceAccount
  • RBAC is additive only — there is no "deny"

Best practices

  • No day-to-day cluster-admin for humans
  • Avoid * verbs/resources
  • Treat secrets get/list, pods/exec, escalate, bind, impersonate as privileged
  • Audit: kubectl auth can-i --list --as=...
Pod Identity step by step

Give a pod access to AWS — no keys

# 1. IAM role trusted by EKS Pod Identity
cat > trust.json <<'EOF'
{ "Version": "2012-10-17",
  "Statement": [{ "Effect": "Allow",
    "Principal": { "Service": "pods.eks.amazonaws.com" },
    "Action": ["sts:AssumeRole", "sts:TagSession"] }] }
EOF
aws iam create-role --role-name orders-app \
  --assume-role-policy-document file://trust.json
aws iam put-role-policy --role-name orders-app \
  --policy-name s3-read --policy-document file://s3-read.json

# 2. Standard EC2 nodes only; skip on Auto Mode
aws eks create-addon --cluster-name prod \
  --addon-name eks-pod-identity-agent

# 3. Create namespace and service account, then bind
kubectl create namespace orders
kubectl create serviceaccount orders -n orders
aws eks create-pod-identity-association \
  --cluster-name prod --namespace orders \
  --service-account orders \
  --role-arn arn:aws:iam::111122223333:role/orders-app

# 4. Verify from inside the pod
kubectl exec -n orders deploy/orders -- \
  aws sts get-caller-identity

Notes

  • Set serviceAccountName: orders on the workload. Restart existing pods after the association.
  • Provide a least-privilege s3-read.json policy for your own bucket before running this example
  • Requires a recent AWS SDK that supports the container credential provider
  • Session tags enable ABAC: eks-cluster-name, kubernetes-namespace, kubernetes-service-account
  • One role per app; never reuse the node role
  • Same pattern for add-ons: EBS CSI, LB Controller, CloudWatch agent, ESO
Section 07

Security hardening

Threat model

Security layers (the 4C model)

Cloud

  • Account & IAM
  • VPC, SGs, endpoints
  • KMS, CloudTrail
  • GuardDuty, Security Hub

Cluster

  • API endpoint access
  • Access entries + RBAC
  • Admission & policy
  • Audit logs

Container

  • Image provenance
  • Pod Security Standards
  • securityContext
  • Runtime detection

Code

  • Dependencies (SCA)
  • Secrets handling
  • SAST / reviews
  • TLS between services

Common attack paths on EKS

  • Public API endpoint + leaked kubeconfig/IAM keys → full cluster access
  • Pod reaches IMDS → steals the node role credentials
  • Over-privileged ServiceAccount token → reads secrets or creates privileged pods
  • Privileged / hostPath pod → escapes to the node, then to other pods
  • Vulnerable or malicious image → crypto mining, reverse shell, lateral movement
Defense in depth

Security best practices

Control plane & data

  • Private API endpoint / CIDR allow-list
  • Envelope encryption of K8s API data is on by default (K8s 1.28+); use your own KMS key if required
  • Enable audit & authenticator logs
  • External secrets: Secrets Manager via External Secrets Operator or Secrets Store CSI

Workloads

  • Pod Security Admission: restricted by default, baseline where needed
  • runAsNonRoot, readOnlyRootFilesystem, drop ALL caps, no privileged
  • Policy-as-code: Kyverno or OPA Gatekeeper
  • Default-deny NetworkPolicy per namespace

Supply chain & runtime

  • ECR with scan on push (Inspector enhanced scanning)
  • Immutable tags, pin digests, sign images (Notation / cosign)
  • GuardDuty EKS Protection + Runtime Monitoring
  • Security Hub / CIS EKS benchmark checks
kubectl label ns payments pod-security.kubernetes.io/enforce=restricted pod-security.kubernetes.io/warn=restricted
Example

A hardened pod spec

spec:
  serviceAccountName: orders
  automountServiceAccountToken: false
  securityContext:
    runAsNonRoot: true
    runAsUser: 10001
    fsGroup: 10001
    seccompProfile: { type: RuntimeDefault }
  containers:
  - name: app
    image: <ecr-repo>/orders@sha256:<digest>
    securityContext:
      allowPrivilegeEscalation: false
      readOnlyRootFilesystem: true
      capabilities: { drop: ["ALL"] }
    resources:
      requests: { cpu: 250m, memory: 256Mi }
      limits:   { memory: 256Mi }
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata: { name: default-deny, namespace: orders }
spec:
  podSelector: {}
  policyTypes: [Ingress, Egress]
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata: { name: allow-dns, namespace: orders }
spec:
  podSelector: {}
  policyTypes: [Egress]
  egress:
  - to:
    - namespaceSelector:
        matchLabels:
          kubernetes.io/metadata.name: kube-system
    ports:
    - { protocol: UDP, port: 53 }
    - { protocol: TCP, port: 53 }

Set automountServiceAccountToken: true only for pods that call the Kubernetes API (IRSA / Pod Identity inject their own token volumes). With default-deny egress, also allow the Pod Identity agent at 169.254.170.23.

Pod Security Standards

Pod Security Admission (PSA)

LevelWhat it allows / blocksUse for
privilegedNo restrictionsSystem namespaces only (CNI, CSI, monitoring agents)
baselineBlocks privileged containers, hostNetwork/hostPID/hostIPC, hostPath, hostPorts, dangerous capabilitiesLegacy apps during migration
restrictedBaseline + must run as non-root, drop ALL capabilities, no privilege escalation, seccomp RuntimeDefault, limited volume typesDefault for all application namespaces
apiVersion: v1
kind: Namespace
metadata:
  name: orders
  labels:
    pod-security.kubernetes.io/enforce: restricted
    pod-security.kubernetes.io/enforce-version: latest
    pod-security.kubernetes.io/warn: restricted
    pod-security.kubernetes.io/audit: restricted

Modes & rollout

  • enforce rejects pods · warn shows client warnings · audit annotates audit logs
  • Roll out: warn + audit first, fix workloads, then enforce
  • PSA checks pods; Deployments only get warnings — check ReplicaSet events
  • PodSecurityPolicy was removed in 1.25 — PSA replaces it
Policy as code

Guardrails with Kyverno, Gatekeeper, or CEL

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata: { name: disallow-latest-tag }
spec:
  background: true
  rules:
  - name: require-pinned-image
    match:
      any:
      - resources: { kinds: [Pod] }
    validate:
      failureAction: Enforce
      message: "Pin an image tag or digest, not :latest"
      pattern:
        spec:
          containers:
          - image: "!*:latest"

What to enforce

  • Images only from your ECR registries
  • No :latest; signed images (verifyImages)
  • Resource requests and required labels (team, cost-center)
  • No hostPath, no LoadBalancer without approval

Tips

  • Kyverno = YAML policies; Gatekeeper = Rego
  • ValidatingAdmissionPolicy (CEL, GA in 1.30) for simple rules, no extra controller
  • Start in Audit, read policy reports, then Enforce
  • Run the engine HA; exclude system namespaces deliberately
Secrets management

Where secrets should live

ApproachHowTrade-off
Kubernetes SecretBase64 in etcd, envelope-encrypted with KMSSimple; anyone with get secrets in the namespace can read it
External Secrets OperatorSyncs Secrets Manager / Parameter Store into K8s SecretsSource of truth in AWS, rotation via refresh; still a K8s Secret
Secrets Store CSI + AWS providerMounts secrets as files at pod startNo K8s Secret unless you enable sync
SDK call with Pod IdentityApp reads Secrets Manager directlyNothing in the cluster; app code depends on AWS
apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata: { name: db, namespace: orders }
spec:
  refreshInterval: 1h
  secretStoreRef: { kind: ClusterSecretStore, name: aws-sm }
  target: { name: db-credentials }
  data:
  - secretKey: password
    remoteRef: { key: prod/orders/db, property: password }

Best practices

  • Never commit plaintext secrets (use SOPS / Sealed Secrets if they must live in Git)
  • Restrict RBAC on secrets; mount as files rather than env vars
  • Enable rotation; separate secrets per environment
Network security

Layers of network control

The VPC CNI commands below apply to standard EC2 nodes. Auto Mode network policy is enabled through its controller ConfigMap.

Controls

  • Cluster security group — created by EKS for control plane ↔ node traffic
  • Security Groups for Pods — per-pod SG, e.g. only orders pods reach RDS (needs ENABLE_POD_ENI=true on the VPC CNI)
  • NetworkPolicy — L3/L4 rules between pods (VPC CNI eBPF agent)
  • WAF / Shield on ALBs; NACLs as a coarse backstop
  • Egress control — NAT + Network Firewall or proxy allow-list

Best practices

  • Default-deny + explicit allows (DNS, Pod Identity agent)
  • Nodes in private subnets only; TLS everywhere, mTLS via mesh if needed
# Standard EKS EC2 nodes: VPC CNI managed add-on
aws eks update-addon --cluster-name prod \
  --addon-name vpc-cni \
  --configuration-values '{"enableNetworkPolicy":"true"}'
apiVersion: vpcresources.k8s.aws/v1beta1
kind: SecurityGroupPolicy
metadata: { name: orders-db, namespace: orders }
spec:
  podSelector: { matchLabels: { app: orders } }
  securityGroups:
    groupIds: [sg-0123456789abcdef0]
Supply chain

Secure the image pipeline

Build

  • Minimal / distroless base
  • Multi-stage, non-root USER
  • Pin base by digest
→

Scan

  • Trivy / Inspector in CI
  • Fail on critical CVEs
  • SBOM (syft)
→

Sign

  • cosign or Notation + AWS Signer
  • Provenance attestations
→

Store

  • ECR, immutable tags
  • Scan on push
  • Lifecycle policies
→

Admit

  • Verify signatures
  • Allowed registries
  • Deploy by digest

Keep it fresh

Inspector re-scans images continuously when new CVEs appear; Renovate/Dependabot bump dependencies; rebuild images regularly, not only on code changes.

CI identity

CI authenticates to AWS with OIDC (no static keys), can push only to its own repos, and cannot touch the cluster directly when GitOps does the deploy.

Node security

Hardening the worker nodes

Host

  • Bottlerocket: read-only root fs, SELinux enforcing, no shell by default
  • IMDSv2 required, hop limit 1
  • No SSH keys — SSM Session Manager with session logging
  • Encrypted EBS root volumes
  • Patch by replacement (expireAfter, AMI updates)
  • Check against CIS benchmark (kube-bench)

Node IAM role — keep it minimal

  • AmazonEKSWorkerNodePolicy
  • AmazonEC2ContainerRegistryPullOnly
  • Move AmazonEKS_CNI_Policy to the aws-node service account (Pod Identity / IRSA)
  • AmazonSSMManagedInstanceCore only if you use SSM
  • Nothing app-specific — apps get their own roles
Detection & audit

Know what is happening in the cluster

Services

  • GuardDuty EKS Protection — analyzes audit logs: anonymous access, exposed dashboards, privileged pods, suspicious API calls
  • GuardDuty Runtime Monitoring — agent add-on: crypto mining, reverse shells, malicious files
  • CloudTrail — EKS API calls (create access entry, update cluster)
  • Security Hub — aggregated findings & controls
  • Open-source alternative: Falco
# Logs Insights on /aws/eks/<cluster>/cluster
# Who deleted what?
fields @timestamp, user.username, objectRef.resource,
       objectRef.namespace, objectRef.name
| filter @logStream like /kube-apiserver-audit/
| filter verb = "delete"
| sort @timestamp desc | limit 50

# Who reads secrets?
fields @timestamp, user.username, objectRef.name
| filter @logStream like /kube-apiserver-audit/
| filter objectRef.resource = "secrets"
    and verb in ["get", "list"]
| stats count(*) by user.username

# RBAC denials
fields user.username, verb, objectRef.resource
| filter @logStream like /kube-apiserver-audit/
| filter responseStatus.code = 403
| stats count(*) as denials by user.username, verb
| sort denials desc
Incident response

A compromised pod — first 30 minutes

Steps

  • Identify: GuardDuty finding, audit logs — which pod, node, SA, role
  • Contain: remove the pod from traffic, verify network isolation, and cordon the node
  • Revoke: remove the Pod Identity association and revoke active IAM role sessions if immediate cutoff is needed
  • Preserve: don't delete first — snapshot EBS, save logs, describe output. Deleting an association alone does not invalidate issued credentials.
  • Eradicate: redeploy from a clean image, rotate secrets
  • Learn: postmortem, add a policy so it can't recur
# Detach from Service & ReplicaSet (a clean replica starts)
kubectl label pod web-7d9f -n orders app=quarantine --overwrite

# Add a quarantine policy, then verify no other policy allows traffic
kubectl apply -f - <<'EOF'
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata: { name: quarantine, namespace: orders }
spec:
  podSelector: { matchLabels: { app: quarantine } }
  policyTypes: [Ingress, Egress]
EOF

# NetworkPolicy allows are additive; this policy alone is not a guaranteed deny.
# Stop new pods landing on the node
kubectl cordon ip-10-0-12-34.ap-northeast-1.compute.internal

# Preserve evidence
aws ec2 create-snapshot --volume-id vol-0abc123 \
  --description "IR case 123 - node root volume"
Section 08

Storage

CSI drivers

Persistent storage options

DriverAccessBest forNotes
EBS CSIRWO, single AZDatabases, StatefulSetsUse gp3; volume is AZ-bound → use WaitForFirstConsumer
EFS CSIRWX, multi-AZShared files, CMS, ML datasetsDynamic provisioning via access points; Fargate supports static provisioning only
FSx (Lustre/ONTAP/OpenZFS)RWXHPC/ML throughput, enterprise NASHigher performance, higher cost
Mountpoint for S3 CSIRWX (read-heavy)Large datasets, ML trainingNot POSIX-complete; sequential writes only
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata: { name: gp3 }
provisioner: ebs.csi.aws.com
volumeBindingMode: WaitForFirstConsumer
allowVolumeExpansion: true
parameters: { type: gp3, encrypted: "true" }

Best practices

  • Install CSI drivers as EKS add-ons with Pod Identity
  • Encrypt by default; back up with AWS Backup or Velero
  • Prefer managed databases (RDS/Aurora) over DBs on EKS unless you need it
Section 09

Autoscaling

Pods scale first, nodes follow.

Layers

The autoscaling stack

HPA

Scales replicas on CPU/memory or custom/external metrics. Needs metrics-server.

+

KEDA

Event-driven scaling (SQS, Kafka, Prometheus, cron), incl. scale-to-zero.

+

VPA

Recommends/sets requests. Use in recommendation mode alongside HPA on CPU.

→

Node scaling

Karpenter (or Auto Mode) provisions nodes for pending pods; Cluster Autoscaler scales ASGs.

Cluster AutoscalerKarpenter
UnitNode groups (ASGs) with fixed instance typesIndividual instances, chosen per pending pod batch
SpeedMinutes (ASG round-trip)Faster — calls EC2 Fleet directly
Cost featuresLimitedConsolidation, Spot/On-Demand mix, Graviton, drift replacement
Karpenter

NodePool & EC2NodeClass

apiVersion: karpenter.sh/v1
kind: NodePool
metadata: { name: default }
spec:
  template:
    spec:
      nodeClassRef: { group: karpenter.k8s.aws,
        kind: EC2NodeClass, name: default }
      requirements:
      - key: karpenter.sh/capacity-type
        operator: In
        values: ["spot", "on-demand"]
      - key: kubernetes.io/arch
        operator: In
        values: ["arm64", "amd64"]
      - key: karpenter.k8s.aws/instance-category
        operator: In
        values: ["c", "m", "r"]
      expireAfter: 720h
  limits: { cpu: "1000" }
  disruption:
    consolidationPolicy: WhenEmptyOrUnderutilized
    consolidateAfter: 1m
    budgets: [{ nodes: "10%" }]

Best practices

  • Run Karpenter on a small MNG or Fargate — never on nodes it manages
  • Allow broad instance families for better Spot availability
  • Set limits to cap runaway spend
  • Use disruption budgets + PDBs to control churn
  • expireAfter forces regular node refresh (patching)
  • Separate NodePools for GPU, system, or tainted workloads
  • Handle Spot interruptions via the SQS interruption queue
Section 10

Add-ons

The platform components every cluster needs.

Add-ons

The platform layer

EKS managed add-ons

  • Core: vpc-cni, coredns, kube-proxy
  • eks-pod-identity-agent, aws-ebs-csi-driver, aws-efs-csi-driver, aws-mountpoint-s3-csi-driver
  • amazon-cloudwatch-observability, adot, aws-guardduty-agent
  • Community add-ons (e.g. metrics-server) & Marketplace add-ons
  • Check compatibility: aws eks describe-addon-versions

Common Helm-installed tools

  • AWS Load Balancer Controller, Karpenter
  • ExternalDNS, cert-manager
  • External Secrets Operator
  • Kyverno / Gatekeeper, KEDA
  • Argo CD / Flux

Best practices

Prefer managed add-ons where available · pin versions in IaC · upgrade add-ons with every cluster upgrade · avoid manual kubectl edit on managed add-ons (use configuration values) · give each add-on its own Pod Identity role.

Section 11

Observability &
monitoring

Control plane, nodes, workloads, logs, traces, alerts, SLOs.

Observability overview

Logs, metrics, traces

Control plane

  • Log types: api, audit, authenticator, controllerManager, scheduler → CloudWatch Logs
  • Control plane metrics exposed via Prometheus endpoints
  • Cluster insights: upgrade readiness & config checks

Metrics

  • CloudWatch Container Insights (enhanced observability) via the add-on
  • Or Prometheus → Amazon Managed Prometheus + Amazon Managed Grafana
  • kube-state-metrics, node-exporter, DCGM for GPUs

Logs & traces

  • Fluent Bit DaemonSet → CloudWatch / OpenSearch / S3
  • ADOT / OpenTelemetry Collector → X-Ray or other backends
  • Set log retention policies — never "Never expire"

Alert on what matters

Pending pods & unschedulable reasons · node NotReady · pod restarts/OOMKilled · API server 5xx & latency · CoreDNS errors · free IPs per subnet · PVC usage · certificate expiry · SLO-based app alerts.

What to monitor

Signals per layer

LayerKey signalsSource
Control planeAPI latency & 5xx, request rate, throttling (APF), etcd size, webhook latency, pending podsCloudWatch AWS/EKS, API /metrics
NodesCPU, memory, disk, network, NotReady, pressure conditions, pods per nodenode-exporter, Container Insights
K8s objectsAvailable vs desired replicas, restarts, OOMKills, failed Jobs, HPA at max, PVC usagekube-state-metrics, kubelet
ApplicationsRED: rate, errors, duration; saturation; business KPIsApp metrics, OpenTelemetry, ALB metrics
Add-onsCoreDNS errors & latency, CNI IP allocation, Karpenter launch errors, LB controller errorsAdd-on /metrics endpoints
Network & costFree IPs per subnet, NAT bytes, cross-AZ traffic, requests vs usageCloudWatch, VPC Flow Logs, Kubecost

Methods: USE (utilization, saturation, errors) for resources · RED (rate, errors, duration) for services · Golden signals (latency, traffic, errors, saturation) for user-facing SLOs.

Control plane monitoring

Watching the API server, scheduler, and etcd

Built-in sources

  • CloudWatch AWS/EKS metrics (new clusters on K8s 1.28+, no extra cost): API request totals & 5xx, storage size, scheduler attempts, pending pods
  • EKS console Observability tab with control plane dashboards
  • Prometheus endpoints scrapeable from inside the cluster
  • Control plane logs → /aws/eks/<cluster>/cluster

Watch for

  • apiserver_request_duration_seconds p99 rising
  • apiserver_flowcontrol_rejected_requests_total > 0 (throttling)
  • etcd_mvcc_db_total_size_in_use_in_bytes near the 8 GiB etcd limit
  • Slow admission webhooks blocking every request
# API server metrics
kubectl get --raw /metrics | grep apiserver_request_total

# Scheduler metrics (EKS-proxied endpoint)
kubectl get --raw \
  "/apis/metrics.eks.amazonaws.com/v1/ksh/container/metrics"

# Controller manager metrics
kubectl get --raw \
  "/apis/metrics.eks.amazonaws.com/v1/kcm/container/metrics"

# Enable control plane logs
aws eks update-cluster-config --name prod \
  --logging '{"clusterLogging":[{"types":
    ["api","audit","authenticator",
     "controllerManager","scheduler"],
    "enabled":true}]}'

# Audit logs can be large: set retention
aws logs put-retention-policy \
  --log-group-name /aws/eks/prod/cluster \
  --retention-in-days 90
AWS-native path

CloudWatch Container Insights & Application Signals

# IAM for the CloudWatch agent via Pod Identity
# (role with CloudWatchAgentServerPolicy)
aws eks create-pod-identity-association \
  --cluster-name prod \
  --namespace amazon-cloudwatch \
  --service-account cloudwatch-agent \
  --role-arn arn:aws:iam::111122223333:role/cw-agent

# Install the add-on: CloudWatch agent + Fluent Bit
aws eks create-addon --cluster-name prod \
  --addon-name amazon-cloudwatch-observability

# Log groups created
/aws/containerinsights/prod/application
/aws/containerinsights/prod/host
/aws/containerinsights/prod/dataplane
/aws/containerinsights/prod/performance

What you get

  • Cluster → namespace → workload → pod → container metrics and curated dashboards
  • Enhanced observability: kube-state and control plane detail, GPU / Neuron metrics
  • Container logs shipped by Fluent Bit
  • Application Signals: auto-instrumented APM (Java, Python, .NET, Node.js), service map, SLOs
  • Billing per observation — exclude noisy namespaces and set log retention
Open-source path

Prometheus & Grafana

Options

  • Amazon Managed Prometheus + AWS managed collector (agentless scraper) — no Prometheus to run
  • AMP + ADOT / Prometheus agent with remote_write
  • Self-managed kube-prometheus-stack: Prometheus Operator, Alertmanager, Grafana, node-exporter, kube-state-metrics
  • Amazon Managed Grafana with AMP, CloudWatch, X-Ray data sources and SSO

Best practices

  • Control label cardinality — no user IDs or URLs with IDs
  • Recording rules for expensive queries
  • Dashboards and rules as code, in Git
  • Start from community mixins (kubernetes-mixin)
# Tell Prometheus Operator to scrape your app
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: orders
  namespace: orders
  # must match Prometheus serviceMonitorSelector
  labels: { release: kube-prometheus-stack }
spec:
  selector:
    matchLabels: { app: orders }
  endpoints:
  - port: metrics      # named Service port
    path: /metrics
    interval: 30s
Alerting

Starter alert rules (PromQL)

AlertExpression
Pod crash-loopingincrease(kube_pod_container_status_restarts_total[15m]) > 3
Pods pending (with for: 10m)sum by (namespace) (kube_pod_status_phase{phase="Pending"}) > 0
Node not readykube_node_status_condition{condition="Ready",status="true"} == 0
Deployment degradedkube_deployment_status_replicas_available < kube_deployment_spec_replicas
OOMKilledkube_pod_container_status_last_terminated_reason{reason="OOMKilled"} == 1
CPU throttlingrate(container_cpu_cfs_throttled_periods_total[5m]) / rate(container_cpu_cfs_periods_total[5m]) > 0.25
PVC almost fullkubelet_volume_stats_used_bytes / kubelet_volume_stats_capacity_bytes > 0.85
API server 5xxsum(rate(apiserver_request_total{code=~"5.."}[5m])) / sum(rate(apiserver_request_total[5m])) > 0.01
HPA maxed outkube_horizontalpodautoscaler_status_current_replicas == kube_horizontalpodautoscaler_spec_max_replicas
Alerting

Writing alerts people trust

apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata: { name: workloads, namespace: monitoring }
spec:
  groups:
  - name: workloads
    rules:
    - alert: PodCrashLooping
      expr: |
        increase(kube_pod_container_status_restarts_total[15m]) > 3
      for: 5m
      labels:
        severity: warning
        team: "{{ $labels.namespace }}"
      annotations:
        summary: "{{ $labels.pod }} is restarting"
        runbook_url: https://wiki.example.com/runbooks/crashloop

Principles

  • Page on symptoms users feel (SLO burn, 5xx, latency); ticket on causes (disk 80%)
  • Every alert has an owner, a severity, and a runbook link
  • Use for: to avoid flapping
  • Route by team label in Alertmanager / SNS → Slack, PagerDuty
  • Review noisy alerts monthly — delete or fix them
  • Alert on the monitoring itself (scrape failures, agent down)
Logging

Log pipeline design

App

Writes structured JSON to stdout/stderr

→

Node

containerd writes to /var/log/containers

→

Fluent Bit

DaemonSet tails files, adds pod / namespace / labels metadata

→

Destination

CloudWatch Logs, OpenSearch, or S3 via Firehose

Best practices

  • JSON logs with level, timestamp, request and trace IDs
  • Never log secrets, tokens, or personal data
  • Retention per log group; Infrequent Access class for rarely queried logs
  • Archive to S3 for long-term / compliance
  • Fargate: built-in Fluent Bit via the aws-observability namespace ConfigMap
# Logs Insights: top error sources in the last hour
fields @timestamp, kubernetes.namespace_name,
       kubernetes.pod_name, log
| filter log like /(?i)error|exception/
| stats count(*) as errors
    by kubernetes.namespace_name,
       kubernetes.container_name
| sort errors desc
| limit 20
Tracing

Distributed tracing with OpenTelemetry

Building blocks

  • OpenTelemetry SDKs or auto-instrumentation
  • OpenTelemetry Operator injects agents via annotations
  • Collector (ADOT) as DaemonSet or gateway Deployment
  • Backends: X-Ray / CloudWatch, Tempo, Jaeger, vendors

Best practices

  • Propagate W3C traceparent across all services
  • Put the trace ID in every log line to jump log ↔ trace
  • Sample by tail (keep errors and slow requests)
  • Name spans and services consistently
apiVersion: opentelemetry.io/v1alpha1
kind: Instrumentation
metadata: { name: default, namespace: orders }
spec:
  exporter:
    endpoint: http://otel-collector.observability:4318  # OTLP/HTTP
  propagators: [tracecontext, baggage]
  sampler:
    type: parentbased_traceidratio
    argument: "0.1"
---
# Opt a workload in (pod template annotation)
metadata:
  annotations:
    instrumentation.opentelemetry.io/inject-java: "true"
SLOs

SLIs, SLOs, and error budgets

SLI

  • Availability = good requests / all requests (from ALB or app metrics)
  • Latency = share of requests faster than 300 ms
  • Measure as close to the user as possible

SLO & budget

  • 99.9% over 30 days → error budget = 0.1% ≈ 43.2 minutes
  • Budget left → ship faster
  • Budget burnt → focus on reliability

Burn-rate alerts

  • Fast burn: 14.4× over 1h (and 5m) → page
  • Slow burn: 6× over 6h → page; 1× over 3d → ticket
  • Fewer, better pages than threshold alerts

Dashboard hierarchy

Fleet overview → cluster health → namespace / team → workload detail → pod. Each level links to the next, and every alert links to the dashboard that explains it.

Section 12

Workload reliability

Resilient workloads

Reliability best practices

Scheduling & resources

  • Always set requests; set memory limit = request
  • Avoid CPU limits for latency-sensitive apps (throttling)
  • topologySpreadConstraints across zones and nodes; use hard constraints when AZ separation is required
  • Use PodDisruptionBudget for critical multi-replica services; it limits voluntary evictions, not node/AZ failures
  • PriorityClass for critical workloads
  • ≥ 2 replicas (3 across AZs for critical)

Lifecycle

  • Separate readiness, liveness, and startup probes
  • Liveness must not depend on downstream services
  • Handle SIGTERM; preStop sleep for LB deregistration
  • Tune terminationGracePeriodSeconds
  • Rolling update: maxUnavailable: 0, maxSurge: 25%
  • Design for node loss — Spot and consolidation will happen
topologySpreadConstraints:
- { maxSkew: 1, topologyKey: topology.kubernetes.io/zone, whenUnsatisfiable: DoNotSchedule,
    labelSelector: { matchLabels: { app: web } } }
Section 13

Upgrades & lifecycle

Upgrade runbook

Upgrading an EKS cluster safely

1 · Prepare

  • Read K8s & EKS release notes
  • Check upgrade insights
  • Scan deprecated APIs (pluto, kubent)
  • Upgrade in non-prod first
→

2 · Control plane

  • One minor version at a time
  • Rolling API server update; clients must reconnect
  • N−1 rollback may be available within 7 days; check readiness and eligibility
→

3 · Add-ons

  • vpc-cni, coredns, kube-proxy
  • CSI drivers, LB controller, Karpenter
  • Match compatibility matrix
→

4 · Data plane

  • MNG: update version (rolling)
  • Karpenter / Auto Mode: drift replaces nodes
  • PDBs limit voluntary evictions; replicas and topology handle failures

Rollback reverts the control plane (and Auto Mode nodes); add-ons and managed node groups need separate rollback and compatibility checks.

Version skew

kubelet may be up to 3 minor versions older than the API server (K8s 1.28+), but never newer.

Blue/green clusters

For big jumps or risky changes: build a new cluster, shift traffic via DNS/ALB weights, then retire the old one. Requires stateless clusters + GitOps.

Section 14

Cost optimization

FinOps on EKS

Where the money goes — and how to cut it

Compute

  • Right-size requests (VPA recommendations, Goldilocks)
  • Karpenter consolidation for bin-packing
  • Spot for stateless/fault-tolerant workloads
  • Graviton: typically better price-performance
  • Compute Savings Plans for the baseline
  • Scale non-prod to zero off-hours

Network & storage

  • VPC endpoints to reduce NAT data processing
  • Minimize cross-AZ chatter
  • Share ALBs with IngressGroups
  • gp3 over gp2; delete orphaned PVs/EBS
  • Set CloudWatch log retention

Visibility & control

  • Split cost allocation data for EKS in CUR
  • Kubecost / OpenCost per namespace/team
  • Consistent labels → cost tags
  • ResourceQuotas per team
  • Stay off extended support ($0.60/h)
  • Consolidate small clusters
Section 15

GitOps, multi-tenancy,
DR & troubleshooting

Delivery & tenancy

GitOps and multi-tenancy

GitOps

  • Git is the source of truth; Argo CD or Flux reconciles
  • EKS Capabilities can run Argo CD, ACK, and kro as AWS-managed services
  • Helm / Kustomize for packaging; app-of-apps or ApplicationSets for fleets
  • CI builds & pushes images (OIDC to AWS, no static keys); CD only deploys
  • Drift detection + auto-sync with self-heal

Multi-tenancy (soft)

  • Namespace per team/app + RBAC
  • ResourceQuota & LimitRange per namespace
  • Default-deny NetworkPolicy
  • Pod Identity per workload; PSA restricted
  • Dedicated NodePools with taints for sensitive tenants
  • Hard isolation → separate clusters / accounts

Cluster strategy: separate prod and non-prod clusters (ideally separate AWS accounts); prefer fewer, larger shared clusters over many small ones unless isolation or blast radius requires it.

Beyond the basics

DR, special workloads & scale

Backup & DR

  • Cluster state = Git + IaC; rebuild rather than restore
  • Set RPO/RTO per service; back up the matching data, PVs and manifests
  • Use Velero or AWS Backup where supported; run restore and failover drills
  • ARC zonal shift / autoshift to move traffic away from an impaired AZ
  • Multi-region: active/passive clusters + data replication; measure failover against RTO/RPO

Special workloads

  • GPU/ML: accelerated AMIs, NVIDIA device plugin or GPU Operator, Neuron for Trainium/Inferentia
  • Dedicated tainted NodePools for GPUs
  • Windows nodes: separate node group, VPC CNI Windows support, node selectors
  • Service mesh (Istio) or VPC Lattice for cross-cluster traffic

Scale & images

  • Provisioned Control Plane tiers for very large clusters
  • Watch API server request rates; avoid chatty controllers
  • ECR pull-through cache for public images
  • Small images, SOCI / pre-pulled images for faster start
  • Spot on MNG: interruption handling is built in (Capacity Rebalancing); Node Termination Handler only for self-managed nodes
Troubleshooting

Common problems & first checks

SymptomLikely causeCheck
Pods PendingNo capacity, taints, affinity, PVC AZ mismatchkubectl describe pod events; Karpenter logs
Pods stuck ContainerCreatingSubnet out of IPs, CNI errors, volume attachSubnet free IPs; aws-node logs; ipamd
ImagePullBackOffWrong tag, no ECR perms, no NAT/endpointsNode role ECR policy; VPC endpoints
CrashLoopBackOff / OOMKilledApp error, low memory limit, bad probeskubectl logs --previous; describe
Node NotReadykubelet/CNI failure, disk/memory pressureNode conditions; SSM to node; log collector script
Unauthorized / ForbiddenMissing access entry or RBACaws sts get-caller-identity; kubectl auth can-i
ALB 502/504 on deployNo preStop delay, readiness, SG rulesTarget health; LB controller logs
DNS timeoutsCoreDNS overload, VPC resolver limitsCoreDNS metrics; NodeLocal DNSCache
Cheat sheet

Debugging commands

# What just happened?
kubectl get events -A --sort-by=.lastTimestamp | tail -30
kubectl describe pod <pod> -n <ns>
kubectl logs <pod> -n <ns> -c <container> --previous

# Resources
kubectl top nodes
kubectl top pods -A --sort-by=memory
kubectl describe node <node> | grep -A8 Allocated

# Get a shell next to a broken container
kubectl debug -it pod/<pod> -n <ns> \
  --image=busybox --target=<container>
kubectl debug node/<node> -it --image=ubuntu

# Networking
kubectl get endpointslices -n <ns>
kubectl run tmp --rm -it --image=busybox -- \
  nslookup web.orders.svc.cluster.local
# Access
aws sts get-caller-identity
kubectl auth can-i --list -n <ns>
aws eks list-access-entries --cluster-name prod

# Cluster health & upgrades
aws eks describe-cluster --name prod \
  --query 'cluster.{status:status,version:version}'
aws eks list-insights --cluster-name prod
aws eks describe-addon --cluster-name prod \
  --addon-name vpc-cni --query 'addon.health'

# Karpenter / Auto Mode
kubectl get nodepools,nodeclaims
kubectl describe nodeclaim <name>   # events, conditions
# self-managed Karpenter only (namespace may differ):
kubectl logs -n kube-system -l app.kubernetes.io/name=karpenter

# CNI / IPs
kubectl logs -n kube-system -l k8s-app=aws-node -c aws-node
aws ec2 describe-subnets --subnet-ids subnet-0abc \
  --query 'Subnets[].AvailableIpAddressCount'
Checklist

Production readiness checklist

Foundation

  • Cluster built from IaC, version pinned
  • ≥ 3 AZs, large subnets, VPC endpoints
  • Private / restricted API endpoint
  • API auth mode + access entries
  • Control plane logs enabled
  • Upgrade cadence within standard support

Security

  • Pod Identity per app, IMDS blocked
  • PSA restricted, policy engine
  • Default-deny NetworkPolicy
  • Secrets in Secrets Manager
  • Image scanning & signing
  • GuardDuty EKS + Runtime Monitoring
  • Audit-log queries & IR runbook ready

Operations

  • Karpenter or Auto Mode, Spot + Graviton
  • Requests, probes, PDBs, topology spread
  • HPA/KEDA on real signals
  • Control plane + node + app metrics, logs, traces
  • SLO burn-rate alerts with runbooks
  • GitOps delivery, backups tested
  • Cost visibility per team
Glossary

Terms you will meet

MNGManaged node group — EKS-managed Auto Scaling groupIRSAIAM Roles for Service Accounts (OIDC-based)
Auto ModeAWS-managed nodes and core add-onsPod IdentityEKS-native pod → IAM role mapping
KarpenterNode autoscaler that launches right-sized EC2Access entryIAM principal → cluster access mapping
NodePool / NodeClaimKarpenter's node template / a requested nodePSA / PSSPod Security Admission / Standards
VPC CNIPlugin giving pods VPC IPsPDBPodDisruptionBudget — limits voluntary evictions
ENI / prefix delegationNetwork interface / /28 blocks per slotHPA / VPA / KEDAHorizontal, vertical, event-driven pod autoscaling
CSIContainer Storage Interface driversAMP / AMGAmazon Managed Prometheus / Grafana
Add-onEKS-managed cluster componentADOTAWS Distro for OpenTelemetry
Extended supportExtra 12 months per version at $0.60/hSLOService level objective with an error budget
Further reading

References

  • EKS Best Practices Guide — docs.aws.amazon.com/eks/latest/best-practices/
  • EKS User Guide — docs.aws.amazon.com/eks/latest/userguide/
  • EKS Kubernetes version calendar — User Guide › Kubernetes versions
  • Auto Mode networking and network policy — EKS User Guide
  • Cluster version rollback — EKS User Guide
  • EKS Workshop — eksworkshop.com
  • Karpenter — karpenter.sh
  • AWS Load Balancer Controller — kubernetes-sigs.github.io/aws-load-balancer-controller
  • terraform-aws-modules/eks — registry.terraform.io
  • Kubernetes docs — kubernetes.io/docs

Prices and version support windows change — verify against the AWS pages before making decisions.

1 / 1