Architecture, compute, networking, identity, security, storage, scaling, observability, upgrades, cost — and the best practices for each.
Use ← → / Space to navigate · O for overview · T toggles theme · F fullscreen
Each section ends with concrete best practices.
Kubernetes in one slide, and what EKS adds on top.
Kubernetes is a declarative control loop: you describe desired state, controllers reconcile it.
Pod — smallest unit, 1+ containersDeployment → ReplicaSet — statelessStatefulSet — stable identity + volumesDaemonSet — one per nodeJob / CronJob — batchService — ClusterIP / NodePort / LoadBalancerIngress / Gateway API — L7 routingNetworkPolicy — pod firewallsvc.ns.svc.cluster.localConfigMap, SecretPersistentVolumeClaim, StorageClassServiceAccount, Role, RoleBindingNamespace, ResourceQuota, LimitRangeUpstream, CNCF-conformant Kubernetes with a managed control plane and deep AWS integration.
Pricing basics: $0.10 per cluster-hour in standard support, $0.60 per cluster-hour in extended support — plus the compute, storage, and data transfer you use. Auto Mode adds a per-instance management fee; Provisioned Control Plane tiers (for very large clusters) and EKS Capabilities are billed on top.
kubectl apply to a running podKnow this path by heart — almost every troubleshooting session maps to one of these steps.
aws eks get-tokenPending → Karpenter / Auto Mode launches a nodeReadyThe more managed the compute option, the more AWS takes on.
| Area | Self-managed nodes | Managed node groups | Fargate | Auto Mode |
|---|---|---|---|---|
| Control plane, etcd | AWS | AWS | AWS | AWS |
| Node AMI / OS patching | You | You trigger, AWS provides AMI | AWS | AWS (nodes recycled, max 21-day lifetime) |
| Node provisioning & scaling | You | You (ASG + autoscaler) | AWS (per pod) | AWS (Karpenter-based) |
| Core add-ons (CNI, CoreDNS, kube-proxy, EBS CSI, LB) | You | You | Partially AWS | AWS (built in) |
| Version upgrades | You | You | You (control plane) | You (control plane); nodes follow |
| IAM, RBAC, network policy, app security, data | You | You | You | You |
Clusters are cattle too — always build them from code.
Standard EKS control plane; add compute and an access entry on the next slide.
terraform-aws-modules/eks — most common in productionaws-auth ConfigMapmodule "eks" {
source = "terraform-aws-modules/eks/aws"
version = "~> 21.0"
name = "prod-apne1"
kubernetes_version = "1.37" # available Oct 2026
upgrade_policy = { support_type = "STANDARD" }
vpc_id = module.vpc.vpc_id
subnet_ids = module.vpc.private_subnets
endpoint_public_access = false
authentication_mode = "API"
enable_cluster_creator_admin_permissions = false
enabled_log_types = ["api", "audit", "authenticator"]
addons = {
vpc-cni = { before_compute = true }
coredns = {}
kube-proxy = {}
eks-pod-identity-agent = { before_compute = true }
}
}
Add these blocks inside the preceding module. Supply an existing VPC and a predeclared platform_admin_role_arn variable.
# Managed EC2 capacity
eks_managed_node_groups = {
system = {
instance_types = ["m7i.large"]
min_size = 2
max_size = 4
desired_size = 2
}
}
# Explicit break-glass administrator
access_entries = {
admin = {
principal_arn = var.platform_admin_role_arn
policy_associations = {
admin = {
policy_arn = join("", [
"arn:aws:eks::aws:cluster-access-policy/",
"AmazonEKSClusterAdminPolicy"
])
access_scope = { type = "cluster" }
}
}
}
}
The API endpoint is private, so run Terraform and kubectl from a network with VPC access. For Auto Mode, use compute_config instead of this managed node group and omit the standard CNI, CoreDNS, kube-proxy, and Pod Identity Agent add-ons.
14 months from the version's EKS release. $0.10/cluster-hour.
Additional 12 months. $0.60/cluster-hour. Enabled by default — set the upgrade policy to STANDARD to opt out.
After extended support ends, EKS auto-upgrades the control plane to the oldest supported version.
# 0. Tools: AWS CLI v2, kubectl, eksctl, helm
aws sts get-caller-identity
# 1. Create a cluster with Auto Mode (~15 min)
eksctl create cluster --name lab \
--region ap-northeast-1 --enable-auto-mode
# 2. Connect and deploy something
aws eks update-kubeconfig --name lab --region ap-northeast-1
kubectl get nodes # empty until pods need capacity
kubectl create deployment web \
--image=public.ecr.aws/nginx/nginx --replicas=2
kubectl get pods -o wide -w # watch a node appear
kubectl get nodeclaims
# 3. Expose it
kubectl expose deployment web --port=80 --type=ClusterIP
kubectl port-forward svc/web 8080:80
# 4. AFTER the exercises, clean up — clusters bill every hour
eksctl delete cluster --name lab --region ap-northeast-1
Run after creating web. Auto Mode supplies compute and Pod Identity; the exercises still need workload resources and policy setup.
# HPA needs resource requests and metrics
kubectl top nodes # check whether metrics-server exists
# Only if the metrics API is unavailable:
aws eks create-addon --cluster-name lab \
--region ap-northeast-1 --addon-name metrics-server
kubectl set resources deployment/web \
--requests=cpu=100m,memory=128Mi \
--limits=memory=256Mi
kubectl autoscale deployment web \
--cpu=60% --min=2 --max=20
# Enable Auto Mode network policy enforcement
kubectl apply -f - <<'EOF'
apiVersion: v1
kind: ConfigMap
metadata:
name: amazon-vpc-cni
namespace: kube-system
data:
enable-network-policy-controller: "true"
EOF
# A PDB for the node-drain exercise
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata: { name: web }
spec:
minAvailable: 1
selector:
matchLabels: { app: web }
web before testing a broken rollout.disruptionsAllowed.Apply after enabling the Auto Mode Network Policy Controller. The example targets the lab's default namespace.
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata: { name: default-deny, namespace: default }
spec:
podSelector: {}
policyTypes: [Ingress, Egress]
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata: { name: allow-dns, namespace: default }
spec:
podSelector: {}
policyTypes: [Egress]
egress:
- to:
- namespaceSelector:
matchLabels:
kubernetes.io/metadata.name: kube-system
ports:
- { protocol: UDP, port: 53 }
- { protocol: TCP, port: 53 }
default should fail after enforcement converges.nslookup kubernetes.default.svc.cluster.local from a test pod.web remains blocked until you add an explicit ingress allow rule.| Option | What it is | Use when | Watch out |
|---|---|---|---|
| EKS Auto Mode | AWS manages nodes (Karpenter-based), CNI, CoreDNS, kube-proxy, EBS CSI, LB controller; Bottlerocket-based, immutable nodes | New clusters, small platform teams, want least ops | Per-instance fee; less node-level customization (no SSH/SSM) |
| Karpenter (self-run) | Open-source node autoscaler; provisions right-sized EC2 directly | Diverse workloads, Spot, cost focus, full control | You run and upgrade Karpenter itself (on MNG or Fargate) |
| Managed node groups | EKS-managed ASGs with EKS-optimized AMIs, managed rolling updates | Stable, homogeneous capacity; system/add-on pools | Scaling via Cluster Autoscaler is slower and less bin-packing |
| Self-managed nodes | Your own ASG + launch template | Custom OS / AMI requirements | Everything is on you |
| Fargate | Serverless pod — one microVM per pod | Isolation, bursty or low-ops jobs | No DaemonSets, no GPUs, no EBS volumes, no privileged pods; slower start |
| Hybrid Nodes | On-prem/edge machines joined to an EKS control plane | Data residency, existing hardware | Needs private connectivity (VPN / Direct Connect) |
1 unless pods truly need IMDSkube-reserved/system-reserved (EKS AMIs set sane defaults)general-purpose and systemNodePool + NodeClass (eks.amazonaws.com) for GPU, Spot, or isolated poolsStorageClass (ebs.csi.eks.amazonaws.com) and IngressClassVPC CNI, IP planning, load balancing, DNS.
This slide describes standard EKS EC2 nodes using the Amazon VPC CNI; Auto Mode uses its managed networking capability.
aws-node DaemonSet attaches ENIs and secondary IPs to each nodeWARM_ENI_TARGET, WARM_IP_TARGET)ENIConfigkubectl set env daemonset aws-node -n kube-system ENABLE_PREFIX_DELEGATION=true WARM_PREFIX_TARGET=1
# Standard EC2 nodes only. Prefer managed add-on configuration values in production.
Secondary-IP calculations below apply to standard EC2 nodes; Auto Mode manages prefixes and maxPods differently.
ENIs × (IPs per ENI − 1) + 2ClusterIP: virtual IP; kube-proxy (iptables/IPVS/nftables) DNATs to pod IPsNodePort: port on every nodeLoadBalancer: NLB via LB Controller; IP mode targets pods directlyexternalTrafficPolicy: Local keeps the client IP and avoids an extra hopkubernetes.io/role/elb and kubernetes.io/role/internal-elbExample for standard EKS with the self-managed controller. Auto Mode uses its built-in load balancing and its own IngressClass controller.
Ingress → ALB (L7, path/host routing, WAF, Cognito/OIDC)Service type: LoadBalancer → NLB (L4, static IPs, PrivateLink)IngressGroup shares one ALB across many Ingressestarget-type: ip — traffic goes straight to podspreStop sleep + tune deregistration delay to avoid 502s on rolloutapiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: web
annotations:
alb.ingress.kubernetes.io/scheme: internet-facing
alb.ingress.kubernetes.io/target-type: ip
alb.ingress.kubernetes.io/group.name: public
alb.ingress.kubernetes.io/listen-ports: '[{"HTTPS":443}]'
alb.ingress.kubernetes.io/certificate-arn: arn:aws:acm:...
spec:
ingressClassName: alb
rules:
- host: app.example.com
http:
paths:
- path: /
pathType: Prefix
backend:
service: { name: web, port: { number: 80 } }
ndots or use FQDNs with trailing dot for heavy external lookupsHumans → cluster, and pods → AWS.
aws-authaws eks get-tokenCONFIG_MAP → API_AND_CONFIG_MAP → API (one-way migration)AmazonEKSClusterAdminPolicy (cluster scope), AmazonEKSAdminPolicy, AmazonEKSEditPolicy, AmazonEKSViewPolicyAPI mode; manage entries in IaCaws eks create-access-entry \
--cluster-name prod \
--principal-arn arn:aws:iam::111122223333:role/TeamA
POLICY=arn:aws:eks::aws:cluster-access-policy
aws eks associate-access-policy \
--cluster-name prod \
--principal-arn arn:aws:iam::111122223333:role/TeamA \
--policy-arn $POLICY/AmazonEKSEditPolicy \
--access-scope \
type=namespace,namespaces=team-a
# Or bind to RBAC groups instead of access policies:
aws eks create-access-entry ... \
--kubernetes-groups team-a-devs
| IRSA (IAM Roles for Service Accounts) | EKS Pod Identity | |
|---|---|---|
| Mechanism | OIDC provider + projected SA token → AssumeRoleWithWebIdentity | eks-pod-identity-agent DaemonSet + association via EKS API |
| Trust policy | Per cluster OIDC issuer, per SA — must be edited for each cluster | One trust to pods.eks.amazonaws.com — reusable across clusters |
| Setup | Create OIDC provider, annotate SA with role ARN | aws eks create-pod-identity-association |
| Session tags / ABAC | No | Yes (cluster, namespace, SA tags) |
| Works on | EC2, Fargate, any cluster incl. self-managed K8s | Linux EC2 nodes incl. Auto Mode (not Fargate, not Windows) |
Default to Pod Identity for new EKS workloads; one role per app; never use node instance role permissions for apps.
Set IMDSv2 hop limit to 1 so pods can't steal the node role credentials.
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata: { name: deployer, namespace: team-a }
rules:
- apiGroups: ["apps"]
resources: ["deployments"]
verbs: ["get", "list", "watch", "create", "update", "patch"]
- apiGroups: [""]
resources: ["pods", "pods/log", "services", "configmaps"]
verbs: ["get", "list", "watch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata: { name: team-a-deployers, namespace: team-a }
subjects:
- kind: Group
name: team-a-devs # from access entry --kubernetes-groups
apiGroup: rbac.authorization.k8s.io
roleRef:
kind: Role
name: deployer
apiGroup: rbac.authorization.k8s.io
Role = namespaced; ClusterRole = cluster-wide or reusableUser, Group, ServiceAccountcluster-admin for humans* verbs/resourcessecrets get/list, pods/exec, escalate, bind, impersonate as privilegedkubectl auth can-i --list --as=...# 1. IAM role trusted by EKS Pod Identity
cat > trust.json <<'EOF'
{ "Version": "2012-10-17",
"Statement": [{ "Effect": "Allow",
"Principal": { "Service": "pods.eks.amazonaws.com" },
"Action": ["sts:AssumeRole", "sts:TagSession"] }] }
EOF
aws iam create-role --role-name orders-app \
--assume-role-policy-document file://trust.json
aws iam put-role-policy --role-name orders-app \
--policy-name s3-read --policy-document file://s3-read.json
# 2. Standard EC2 nodes only; skip on Auto Mode
aws eks create-addon --cluster-name prod \
--addon-name eks-pod-identity-agent
# 3. Create namespace and service account, then bind
kubectl create namespace orders
kubectl create serviceaccount orders -n orders
aws eks create-pod-identity-association \
--cluster-name prod --namespace orders \
--service-account orders \
--role-arn arn:aws:iam::111122223333:role/orders-app
# 4. Verify from inside the pod
kubectl exec -n orders deploy/orders -- \
aws sts get-caller-identity
serviceAccountName: orders on the workload. Restart existing pods after the association.s3-read.json policy for your own bucket before running this exampleeks-cluster-name, kubernetes-namespace, kubernetes-service-accounthostPath pod → escapes to the node, then to other podsrestricted by default, baseline where neededkubectl label ns payments pod-security.kubernetes.io/enforce=restricted pod-security.kubernetes.io/warn=restricted
spec:
serviceAccountName: orders
automountServiceAccountToken: false
securityContext:
runAsNonRoot: true
runAsUser: 10001
fsGroup: 10001
seccompProfile: { type: RuntimeDefault }
containers:
- name: app
image: <ecr-repo>/orders@sha256:<digest>
securityContext:
allowPrivilegeEscalation: false
readOnlyRootFilesystem: true
capabilities: { drop: ["ALL"] }
resources:
requests: { cpu: 250m, memory: 256Mi }
limits: { memory: 256Mi }
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata: { name: default-deny, namespace: orders }
spec:
podSelector: {}
policyTypes: [Ingress, Egress]
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata: { name: allow-dns, namespace: orders }
spec:
podSelector: {}
policyTypes: [Egress]
egress:
- to:
- namespaceSelector:
matchLabels:
kubernetes.io/metadata.name: kube-system
ports:
- { protocol: UDP, port: 53 }
- { protocol: TCP, port: 53 }
Set automountServiceAccountToken: true only for pods that call the Kubernetes API (IRSA / Pod Identity inject their own token volumes). With default-deny egress, also allow the Pod Identity agent at 169.254.170.23.
| Level | What it allows / blocks | Use for |
|---|---|---|
| privileged | No restrictions | System namespaces only (CNI, CSI, monitoring agents) |
| baseline | Blocks privileged containers, hostNetwork/hostPID/hostIPC, hostPath, hostPorts, dangerous capabilities | Legacy apps during migration |
| restricted | Baseline + must run as non-root, drop ALL capabilities, no privilege escalation, seccomp RuntimeDefault, limited volume types | Default for all application namespaces |
apiVersion: v1
kind: Namespace
metadata:
name: orders
labels:
pod-security.kubernetes.io/enforce: restricted
pod-security.kubernetes.io/enforce-version: latest
pod-security.kubernetes.io/warn: restricted
pod-security.kubernetes.io/audit: restricted
enforce rejects pods · warn shows client warnings · audit annotates audit logswarn + audit first, fix workloads, then enforceapiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata: { name: disallow-latest-tag }
spec:
background: true
rules:
- name: require-pinned-image
match:
any:
- resources: { kinds: [Pod] }
validate:
failureAction: Enforce
message: "Pin an image tag or digest, not :latest"
pattern:
spec:
containers:
- image: "!*:latest"
:latest; signed images (verifyImages)hostPath, no LoadBalancer without approvalAudit, read policy reports, then Enforce| Approach | How | Trade-off |
|---|---|---|
| Kubernetes Secret | Base64 in etcd, envelope-encrypted with KMS | Simple; anyone with get secrets in the namespace can read it |
| External Secrets Operator | Syncs Secrets Manager / Parameter Store into K8s Secrets | Source of truth in AWS, rotation via refresh; still a K8s Secret |
| Secrets Store CSI + AWS provider | Mounts secrets as files at pod start | No K8s Secret unless you enable sync |
| SDK call with Pod Identity | App reads Secrets Manager directly | Nothing in the cluster; app code depends on AWS |
apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata: { name: db, namespace: orders }
spec:
refreshInterval: 1h
secretStoreRef: { kind: ClusterSecretStore, name: aws-sm }
target: { name: db-credentials }
data:
- secretKey: password
remoteRef: { key: prod/orders/db, property: password }
secrets; mount as files rather than env varsThe VPC CNI commands below apply to standard EC2 nodes. Auto Mode network policy is enabled through its controller ConfigMap.
orders pods reach RDS (needs ENABLE_POD_ENI=true on the VPC CNI)# Standard EKS EC2 nodes: VPC CNI managed add-on
aws eks update-addon --cluster-name prod \
--addon-name vpc-cni \
--configuration-values '{"enableNetworkPolicy":"true"}'
apiVersion: vpcresources.k8s.aws/v1beta1
kind: SecurityGroupPolicy
metadata: { name: orders-db, namespace: orders }
spec:
podSelector: { matchLabels: { app: orders } }
securityGroups:
groupIds: [sg-0123456789abcdef0]
USERInspector re-scans images continuously when new CVEs appear; Renovate/Dependabot bump dependencies; rebuild images regularly, not only on code changes.
CI authenticates to AWS with OIDC (no static keys), can push only to its own repos, and cannot touch the cluster directly when GitOps does the deploy.
expireAfter, AMI updates)AmazonEKSWorkerNodePolicyAmazonEC2ContainerRegistryPullOnlyAmazonEKS_CNI_Policy to the aws-node service account (Pod Identity / IRSA)AmazonSSMManagedInstanceCore only if you use SSM# Logs Insights on /aws/eks/<cluster>/cluster
# Who deleted what?
fields @timestamp, user.username, objectRef.resource,
objectRef.namespace, objectRef.name
| filter @logStream like /kube-apiserver-audit/
| filter verb = "delete"
| sort @timestamp desc | limit 50
# Who reads secrets?
fields @timestamp, user.username, objectRef.name
| filter @logStream like /kube-apiserver-audit/
| filter objectRef.resource = "secrets"
and verb in ["get", "list"]
| stats count(*) by user.username
# RBAC denials
fields user.username, verb, objectRef.resource
| filter @logStream like /kube-apiserver-audit/
| filter responseStatus.code = 403
| stats count(*) as denials by user.username, verb
| sort denials desc
describe output. Deleting an association alone does not invalidate issued credentials.# Detach from Service & ReplicaSet (a clean replica starts)
kubectl label pod web-7d9f -n orders app=quarantine --overwrite
# Add a quarantine policy, then verify no other policy allows traffic
kubectl apply -f - <<'EOF'
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata: { name: quarantine, namespace: orders }
spec:
podSelector: { matchLabels: { app: quarantine } }
policyTypes: [Ingress, Egress]
EOF
# NetworkPolicy allows are additive; this policy alone is not a guaranteed deny.
# Stop new pods landing on the node
kubectl cordon ip-10-0-12-34.ap-northeast-1.compute.internal
# Preserve evidence
aws ec2 create-snapshot --volume-id vol-0abc123 \
--description "IR case 123 - node root volume"
| Driver | Access | Best for | Notes |
|---|---|---|---|
| EBS CSI | RWO, single AZ | Databases, StatefulSets | Use gp3; volume is AZ-bound → use WaitForFirstConsumer |
| EFS CSI | RWX, multi-AZ | Shared files, CMS, ML datasets | Dynamic provisioning via access points; Fargate supports static provisioning only |
| FSx (Lustre/ONTAP/OpenZFS) | RWX | HPC/ML throughput, enterprise NAS | Higher performance, higher cost |
| Mountpoint for S3 CSI | RWX (read-heavy) | Large datasets, ML training | Not POSIX-complete; sequential writes only |
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata: { name: gp3 }
provisioner: ebs.csi.aws.com
volumeBindingMode: WaitForFirstConsumer
allowVolumeExpansion: true
parameters: { type: gp3, encrypted: "true" }
Pods scale first, nodes follow.
Scales replicas on CPU/memory or custom/external metrics. Needs metrics-server.
Event-driven scaling (SQS, Kafka, Prometheus, cron), incl. scale-to-zero.
Recommends/sets requests. Use in recommendation mode alongside HPA on CPU.
Karpenter (or Auto Mode) provisions nodes for pending pods; Cluster Autoscaler scales ASGs.
| Cluster Autoscaler | Karpenter | |
|---|---|---|
| Unit | Node groups (ASGs) with fixed instance types | Individual instances, chosen per pending pod batch |
| Speed | Minutes (ASG round-trip) | Faster — calls EC2 Fleet directly |
| Cost features | Limited | Consolidation, Spot/On-Demand mix, Graviton, drift replacement |
apiVersion: karpenter.sh/v1
kind: NodePool
metadata: { name: default }
spec:
template:
spec:
nodeClassRef: { group: karpenter.k8s.aws,
kind: EC2NodeClass, name: default }
requirements:
- key: karpenter.sh/capacity-type
operator: In
values: ["spot", "on-demand"]
- key: kubernetes.io/arch
operator: In
values: ["arm64", "amd64"]
- key: karpenter.k8s.aws/instance-category
operator: In
values: ["c", "m", "r"]
expireAfter: 720h
limits: { cpu: "1000" }
disruption:
consolidationPolicy: WhenEmptyOrUnderutilized
consolidateAfter: 1m
budgets: [{ nodes: "10%" }]
limits to cap runaway spendbudgets + PDBs to control churnexpireAfter forces regular node refresh (patching)The platform components every cluster needs.
vpc-cni, coredns, kube-proxyeks-pod-identity-agent, aws-ebs-csi-driver, aws-efs-csi-driver, aws-mountpoint-s3-csi-driveramazon-cloudwatch-observability, adot, aws-guardduty-agentmetrics-server) & Marketplace add-onsaws eks describe-addon-versionsPrefer managed add-ons where available · pin versions in IaC · upgrade add-ons with every cluster upgrade · avoid manual kubectl edit on managed add-ons (use configuration values) · give each add-on its own Pod Identity role.
Control plane, nodes, workloads, logs, traces, alerts, SLOs.
api, audit, authenticator, controllerManager, scheduler → CloudWatch LogsPending pods & unschedulable reasons · node NotReady · pod restarts/OOMKilled · API server 5xx & latency · CoreDNS errors · free IPs per subnet · PVC usage · certificate expiry · SLO-based app alerts.
| Layer | Key signals | Source |
|---|---|---|
| Control plane | API latency & 5xx, request rate, throttling (APF), etcd size, webhook latency, pending pods | CloudWatch AWS/EKS, API /metrics |
| Nodes | CPU, memory, disk, network, NotReady, pressure conditions, pods per node | node-exporter, Container Insights |
| K8s objects | Available vs desired replicas, restarts, OOMKills, failed Jobs, HPA at max, PVC usage | kube-state-metrics, kubelet |
| Applications | RED: rate, errors, duration; saturation; business KPIs | App metrics, OpenTelemetry, ALB metrics |
| Add-ons | CoreDNS errors & latency, CNI IP allocation, Karpenter launch errors, LB controller errors | Add-on /metrics endpoints |
| Network & cost | Free IPs per subnet, NAT bytes, cross-AZ traffic, requests vs usage | CloudWatch, VPC Flow Logs, Kubecost |
Methods: USE (utilization, saturation, errors) for resources · RED (rate, errors, duration) for services · Golden signals (latency, traffic, errors, saturation) for user-facing SLOs.
AWS/EKS metrics (new clusters on K8s 1.28+, no extra cost): API request totals & 5xx, storage size, scheduler attempts, pending pods/aws/eks/<cluster>/clusterapiserver_request_duration_seconds p99 risingapiserver_flowcontrol_rejected_requests_total > 0 (throttling)etcd_mvcc_db_total_size_in_use_in_bytes near the 8 GiB etcd limit# API server metrics
kubectl get --raw /metrics | grep apiserver_request_total
# Scheduler metrics (EKS-proxied endpoint)
kubectl get --raw \
"/apis/metrics.eks.amazonaws.com/v1/ksh/container/metrics"
# Controller manager metrics
kubectl get --raw \
"/apis/metrics.eks.amazonaws.com/v1/kcm/container/metrics"
# Enable control plane logs
aws eks update-cluster-config --name prod \
--logging '{"clusterLogging":[{"types":
["api","audit","authenticator",
"controllerManager","scheduler"],
"enabled":true}]}'
# Audit logs can be large: set retention
aws logs put-retention-policy \
--log-group-name /aws/eks/prod/cluster \
--retention-in-days 90
# IAM for the CloudWatch agent via Pod Identity
# (role with CloudWatchAgentServerPolicy)
aws eks create-pod-identity-association \
--cluster-name prod \
--namespace amazon-cloudwatch \
--service-account cloudwatch-agent \
--role-arn arn:aws:iam::111122223333:role/cw-agent
# Install the add-on: CloudWatch agent + Fluent Bit
aws eks create-addon --cluster-name prod \
--addon-name amazon-cloudwatch-observability
# Log groups created
/aws/containerinsights/prod/application
/aws/containerinsights/prod/host
/aws/containerinsights/prod/dataplane
/aws/containerinsights/prod/performance
remote_writekube-prometheus-stack: Prometheus Operator, Alertmanager, Grafana, node-exporter, kube-state-metrics# Tell Prometheus Operator to scrape your app
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
name: orders
namespace: orders
# must match Prometheus serviceMonitorSelector
labels: { release: kube-prometheus-stack }
spec:
selector:
matchLabels: { app: orders }
endpoints:
- port: metrics # named Service port
path: /metrics
interval: 30s
| Alert | Expression |
|---|---|
| Pod crash-looping | increase(kube_pod_container_status_restarts_total[15m]) > 3 |
Pods pending (with for: 10m) | sum by (namespace) (kube_pod_status_phase{phase="Pending"}) > 0 |
| Node not ready | kube_node_status_condition{condition="Ready",status="true"} == 0 |
| Deployment degraded | kube_deployment_status_replicas_available < kube_deployment_spec_replicas |
| OOMKilled | kube_pod_container_status_last_terminated_reason{reason="OOMKilled"} == 1 |
| CPU throttling | rate(container_cpu_cfs_throttled_periods_total[5m]) / rate(container_cpu_cfs_periods_total[5m]) > 0.25 |
| PVC almost full | kubelet_volume_stats_used_bytes / kubelet_volume_stats_capacity_bytes > 0.85 |
| API server 5xx | sum(rate(apiserver_request_total{code=~"5.."}[5m])) / sum(rate(apiserver_request_total[5m])) > 0.01 |
| HPA maxed out | kube_horizontalpodautoscaler_status_current_replicas == kube_horizontalpodautoscaler_spec_max_replicas |
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata: { name: workloads, namespace: monitoring }
spec:
groups:
- name: workloads
rules:
- alert: PodCrashLooping
expr: |
increase(kube_pod_container_status_restarts_total[15m]) > 3
for: 5m
labels:
severity: warning
team: "{{ $labels.namespace }}"
annotations:
summary: "{{ $labels.pod }} is restarting"
runbook_url: https://wiki.example.com/runbooks/crashloop
for: to avoid flappingWrites structured JSON to stdout/stderr
containerd writes to /var/log/containers
DaemonSet tails files, adds pod / namespace / labels metadata
CloudWatch Logs, OpenSearch, or S3 via Firehose
aws-observability namespace ConfigMap# Logs Insights: top error sources in the last hour
fields @timestamp, kubernetes.namespace_name,
kubernetes.pod_name, log
| filter log like /(?i)error|exception/
| stats count(*) as errors
by kubernetes.namespace_name,
kubernetes.container_name
| sort errors desc
| limit 20
traceparent across all servicesapiVersion: opentelemetry.io/v1alpha1
kind: Instrumentation
metadata: { name: default, namespace: orders }
spec:
exporter:
endpoint: http://otel-collector.observability:4318 # OTLP/HTTP
propagators: [tracecontext, baggage]
sampler:
type: parentbased_traceidratio
argument: "0.1"
---
# Opt a workload in (pod template annotation)
metadata:
annotations:
instrumentation.opentelemetry.io/inject-java: "true"
Fleet overview → cluster health → namespace / team → workload detail → pod. Each level links to the next, and every alert links to the dashboard that explains it.
topologySpreadConstraints across zones and nodes; use hard constraints when AZ separation is requiredPodDisruptionBudget for critical multi-replica services; it limits voluntary evictions, not node/AZ failuresPriorityClass for critical workloadsSIGTERM; preStop sleep for LB deregistrationterminationGracePeriodSecondsmaxUnavailable: 0, maxSurge: 25%topologySpreadConstraints:
- { maxSkew: 1, topologyKey: topology.kubernetes.io/zone, whenUnsatisfiable: DoNotSchedule,
labelSelector: { matchLabels: { app: web } } }
pluto, kubent)Rollback reverts the control plane (and Auto Mode nodes); add-ons and managed node groups need separate rollback and compatibility checks.
kubelet may be up to 3 minor versions older than the API server (K8s 1.28+), but never newer.
For big jumps or risky changes: build a new cluster, shift traffic via DNS/ALB weights, then retire the old one. Requires stateless clusters + GitOps.
ResourceQuota & LimitRange per namespacerestrictedCluster strategy: separate prod and non-prod clusters (ideally separate AWS accounts); prefer fewer, larger shared clusters over many small ones unless isolation or blast radius requires it.
| Symptom | Likely cause | Check |
|---|---|---|
Pods Pending | No capacity, taints, affinity, PVC AZ mismatch | kubectl describe pod events; Karpenter logs |
Pods stuck ContainerCreating | Subnet out of IPs, CNI errors, volume attach | Subnet free IPs; aws-node logs; ipamd |
ImagePullBackOff | Wrong tag, no ECR perms, no NAT/endpoints | Node role ECR policy; VPC endpoints |
CrashLoopBackOff / OOMKilled | App error, low memory limit, bad probes | kubectl logs --previous; describe |
Node NotReady | kubelet/CNI failure, disk/memory pressure | Node conditions; SSM to node; log collector script |
Unauthorized / Forbidden | Missing access entry or RBAC | aws sts get-caller-identity; kubectl auth can-i |
| ALB 502/504 on deploy | No preStop delay, readiness, SG rules | Target health; LB controller logs |
| DNS timeouts | CoreDNS overload, VPC resolver limits | CoreDNS metrics; NodeLocal DNSCache |
# What just happened?
kubectl get events -A --sort-by=.lastTimestamp | tail -30
kubectl describe pod <pod> -n <ns>
kubectl logs <pod> -n <ns> -c <container> --previous
# Resources
kubectl top nodes
kubectl top pods -A --sort-by=memory
kubectl describe node <node> | grep -A8 Allocated
# Get a shell next to a broken container
kubectl debug -it pod/<pod> -n <ns> \
--image=busybox --target=<container>
kubectl debug node/<node> -it --image=ubuntu
# Networking
kubectl get endpointslices -n <ns>
kubectl run tmp --rm -it --image=busybox -- \
nslookup web.orders.svc.cluster.local
# Access
aws sts get-caller-identity
kubectl auth can-i --list -n <ns>
aws eks list-access-entries --cluster-name prod
# Cluster health & upgrades
aws eks describe-cluster --name prod \
--query 'cluster.{status:status,version:version}'
aws eks list-insights --cluster-name prod
aws eks describe-addon --cluster-name prod \
--addon-name vpc-cni --query 'addon.health'
# Karpenter / Auto Mode
kubectl get nodepools,nodeclaims
kubectl describe nodeclaim <name> # events, conditions
# self-managed Karpenter only (namespace may differ):
kubectl logs -n kube-system -l app.kubernetes.io/name=karpenter
# CNI / IPs
kubectl logs -n kube-system -l k8s-app=aws-node -c aws-node
aws ec2 describe-subnets --subnet-ids subnet-0abc \
--query 'Subnets[].AvailableIpAddressCount'
API auth mode + access entries| MNG | Managed node group — EKS-managed Auto Scaling group | IRSA | IAM Roles for Service Accounts (OIDC-based) |
| Auto Mode | AWS-managed nodes and core add-ons | Pod Identity | EKS-native pod → IAM role mapping |
| Karpenter | Node autoscaler that launches right-sized EC2 | Access entry | IAM principal → cluster access mapping |
| NodePool / NodeClaim | Karpenter's node template / a requested node | PSA / PSS | Pod Security Admission / Standards |
| VPC CNI | Plugin giving pods VPC IPs | PDB | PodDisruptionBudget — limits voluntary evictions |
| ENI / prefix delegation | Network interface / /28 blocks per slot | HPA / VPA / KEDA | Horizontal, vertical, event-driven pod autoscaling |
| CSI | Container Storage Interface drivers | AMP / AMG | Amazon Managed Prometheus / Grafana |
| Add-on | EKS-managed cluster component | ADOT | AWS Distro for OpenTelemetry |
| Extended support | Extra 12 months per version at $0.60/h | SLO | Service level objective with an error budget |
docs.aws.amazon.com/eks/latest/best-practices/docs.aws.amazon.com/eks/latest/userguide/eksworkshop.comkarpenter.shkubernetes-sigs.github.io/aws-load-balancer-controllerregistry.terraform.iokubernetes.io/docsPrices and version support windows change — verify against the AWS pages before making decisions.