Self-study deck · Helm 4 · Argo CD 3.5 · Argo Rollouts · EKS

GitOps:
Helm & Argo CD

From a single chart to a fleet of clusters: package with Helm, declare in Git, reconcile with Argo CD, release safely with Argo Rollouts, and run it all on EKS — at production depth, with a lab for every chapter.

Use ← → / Space to navigate · I contents · O overview · T theme · F fullscreen

GitOps: Helm & Argo CD

Contents

Click any line to jump there · press I to return to this page

Chapter 0

Orientation

What GitOps is, the toolchain this deck builds, the versions it covers, and how to study it.

Mental model

Two halves of delivery: CI builds, CD converges

GitOps changes only the second half. CI still builds, tests and publishes artifacts.

CI — produce an artifact

  • Triggered by a commit to the app repo.
  • Build, unit test, scan, sign.
  • Publish an immutable artifact: image ghcr.io/…/web:1.4.2, chart web-0.7.0.
  • Output: a new version number, not a deployment.

CD — make a cluster match a declaration

  • The desired state is YAML in a config repo.
  • An agent compares desired vs live state and closes the gap.
  • Changing production = merging a pull request.
  • Rollback = reverting a commit.

GitOps in one line

  • Git holds the desired state; an agent in or near the cluster pulls it and continuously reconciles.
  • Nobody (and no CI job) needs cluster-admin credentials to deploy.
  • Every change has an author, a review and a timestamp.
Toolchain map

The pipeline this deck builds, tool by tool

Each box is a chapter. Read left to right: package → render → store → reconcile → release safely.

App repo + CI (deck 8)Source codeapp repoCI pipelinebuild · test · signImageghcr.io / ECRHelm chartOCI artifact (ch 2–4)Promotion PRbump tag (ch 9)Config repo (Git)envs/dev|staging|prodvalues, overlays (ch 5)apps/ + clusters/Applications, ApplicationSets (ch 8)Policy and reviewPR rules, signed commits (ch 10)KubernetesArgo CDrender · diff · sync (ch 6–7)Cluster devkind / EKSCluster prodkind / EKS (ch 9, 13)Argo Rolloutscanary · analysis (ch 11)pull

Security (ch 10), operations (ch 12) and EKS specifics (ch 13) cut across every box.

Mental model

Three questions for every GitOps change

Ask them in order when something does not deploy. Each maps to one Argo CD status.

1 · Did it render?

  • Does the chart / overlay produce valid YAML?
  • Right values file for this environment?
  • Argo CD shows ComparisonError if not.
  • Tools: helm template, kustomize build.
→

2 · Did it sync?

  • Is the Application Synced to the expected commit?
  • Was a hook, wave or webhook blocking?
  • Did auto-sync run, or is it manual?
  • Tools: argocd app get, argocd app diff.
→

3 · Is it healthy?

  • Are the Pods ready, the Rollout promoted?
  • Synced but Degraded = Git is applied, the app is broken.
  • Fix forward in Git, not with kubectl edit.
  • Tools: health status, events, logs.

Sync status and health status are independent. "Synced + Degraded" is a normal state to debug, not a contradiction.

Versions covered

What this deck assumes (checked Nov 2026)

Older tutorials show different defaults. When a slide and the official docs disagree, the docs win.

Helm 4

  • GA on 12 Nov 2025.
  • Server-Side Apply by default for new installs; upgrades and rollbacks keep the release's previous apply method.
  • kstatus-based --wait; renamed flags (--atomic → --rollback-on-failure, --force → --force-replace).
  • Post-renderers are plugins; WebAssembly plugins added.
  • apiVersion: v2 charts work unchanged.

Argo CD 3.x

  • 3.0: annotation-based resource tracking, fine-grained RBAC for update/delete, .status ignored in diffs by default.
  • 3.3: PreDelete hooks; Source Hydrator no longer cleans the target path.
  • 3.5 (latest stable 3.5.6, 20 Oct 2026): mTLS between components, Source Integrity, impersonation for all API server operations.
  • 3.6 is in release candidate.

Around them

  • Helm 3: bug fixes ended with the final release on 9 Sep 2026; security fixes end 10 Feb 2027 — migrate pipelines now.
  • Amazon EKS Capability for Argo CD (Nov 2025): managed Argo CD with its own limits (ch 13).
  • Argo Rollouts for canary and blue-green (ch 11).
  • Kustomize built into kubectl -k and Argo CD.
Lab environment

Everything runs on your laptop until Chapter 13

Labs 0–12 cost nothing. Lab 13 creates an EKS cluster and must be destroyed the same day.

Tools

  • Docker (or another container runtime) + kind (or k3d).
  • kubectl, helm 4, kustomize, argocd CLI.
  • Optional: kubeconform, helm-unittest, ct, yq, jq.
  • A GitHub account (or a local Gitea) for the config repo.

Conventions used on every slide

  • Hub cluster kind-gitops, second cluster kind-prod.
  • Config repo https://github.com/<you>/gitops-lab.
  • Demo image ghcr.io/stefanprodan/podinfo, chart name web.
  • Argo CD in namespace argocd.
gitops-lab/
├── charts/web/        # our Helm chart
├── apps/              # Argo CD Applications
│   └── root.yaml      # app-of-apps entry
├── envs/
│   ├── dev/web/values.yaml
│   ├── staging/web/values.yaml
│   └── prod/web/values.yaml
└── clusters/          # per-cluster add-ons
    ├── gitops/
    └── prod/

Lab 0 in Hands-on labs creates this repo and both kind clusters.

How to use this deck

A study loop that sticks

Read a part, run its lab, then answer the self-check out loud without notes.

Per part

  • Read the concept slides; redraw the diagrams from memory.
  • Type the YAML yourself instead of copying it — field names are the exam.
  • Read "Common mistakes" twice: these are real outages.

Per chapter

  • Run the matching lab on kind.
  • Do every "Break it" step and fix it using only the CLI and logs.
  • Answer the self-check; revisit anything you hesitate on.

After the deck

  • Build the capstone (back matter) end to end.
  • Keep the cheat sheets open while working.
  • Move on to the Observability deck: Argo CD metrics feed it.

Navigation: ← → or Space to move, I for Contents, O for overview, T for theme, F for fullscreen. Every Contents line is a link. Lab 0 in Hands-on labs sets up your environment.

Chapter 1

GitOps foundations

Principles, push vs pull, drift, repository strategy and the tools around Argo CD.

Chapter 1 · Part 1.1

Principles

The four OpenGitOps principles, push vs pull delivery, and what drift really means.

OpenGitOps

The four principles (OpenGitOps v1.0.0)

A vendor-neutral definition maintained by the CNCF GitOps Working Group. Argo CD and Flux both implement it.

1 · Declarative

  • The system's desired state is expressed declaratively.
  • YAML says what should exist, not the commands to get there.
  • Helm and Kustomize are generators of declarations.

2 · Versioned and immutable

  • Desired state is stored so that it is immutable, versioned and keeps a complete history.
  • Git commits give history, authorship and review.
  • Images and charts are referenced by fixed versions, never latest.

3 · Pulled automatically

  • Software agents automatically pull the desired state from the source.
  • The cluster reaches out to Git; Git never reaches into the cluster.
  • CI needs no cluster credentials.

4 · Continuously reconciled

  • Agents continuously observe actual state and attempt to apply the desired state.
  • Not just on commit: also when someone changes the cluster by hand.
  • This loop is what makes drift visible and correctable.
Push vs pull

Two ways to get YAML into a cluster

Push: CI runs kubectl or helm against the cluster. Pull: an agent inside the cluster fetches from Git.

Push (classic CI/CD)Developergit pushCI runnerkubectl / helmClusterAPI serveradmin credsCI holds kubeconfig or cloud credentials for every cluster.Deploys happen only when a pipeline runs.Manual changes stay until the next pipeline overwrites them.Cluster API must be reachable from CI.Pull (GitOps)Developer / CIopens PRConfig repoGitArgo CDin or near clusterWorkloadssame clusterpull + watchCredentials stay in the cluster; Git is the only input.Reconciles every few minutes and on webhook.
Push vs pull

Trade-offs in detail

Pull is the default for production on Kubernetes; push still has valid uses.

AspectPush (CI applies)Pull (GitOps agent)
CredentialsCI stores cluster-admin-like credentialsAgent uses in-cluster service account; CI only writes to Git
NetworkCI must reach every API serverCluster needs outbound access to Git and registries only
DriftUndetected until the next deployDetected continuously; optionally corrected (self-heal)
AuditPipeline logs (often expire)Git history + agent events
RollbackRe-run an old pipelinegit revert and let the agent converge
Multi-clusterPipeline fan-out logicGenerators (ApplicationSet) or one agent per cluster
Good fitOne-off jobs, bootstrap, non-Kubernetes targetsLong-lived Kubernetes workloads and add-ons
Reconciliation

The control loop behind every GitOps tool

Same pattern as a Kubernetes controller, with Git as the spec.

Desired stateGit commit abc123Renderhelm / kustomizeLive stateKubernetes APIDiffdesired vs liveapply (sync)Repeats on a timer, on Git webhook, and on watch events.

What each step means in Argo CD

  • Fetch: repo-server clones or fetches the commit for the target revision.
  • Render: runs Helm, Kustomize or plain YAML into manifests.
  • Compare: application controller diffs rendered vs live objects → Synced / OutOfSync.
  • Sync: applies the difference (manually or automatically).
  • Assess health: per-resource health rolls up to the Application.
  • Default refresh: about every 3 minutes (120 s + up to 60 s jitter) for self-managed Argo CD; webhooks make it near-instant.
Drift

Drift: when the cluster no longer matches Git

Detection is always on. Correction (self-heal) is a choice per Application.

Typical sources

  • kubectl edit / kubectl scale during an incident.
  • A controller or mutating webhook adding fields.
  • HPA changing spec.replicas.
  • Someone deleting a resource by hand.
  • An operator upgrading its own CRDs.

Detect vs correct

  • Detect: Application shows OutOfSync with a diff — always.
  • Correct: with selfHeal: true, Argo CD re-applies Git state automatically.
  • Legitimate runtime fields (HPA replicas) are excluded with ignoreDifferences (ch 7).
  • Without self-heal, drift stays until the next sync.

Team rule

  • Incidents may need a manual change — then commit the same change to Git within the incident.
  • Otherwise self-heal silently reverts the fix, or the next sync does.
  • Read-only kubectl for humans in production is the end state.
Anti-patterns

Why manual changes become dangerous under GitOps

Under GitOps, the cluster is a cache of Git. Edits to a cache are lost.

What goes wrong

  • A hotfix via kubectl set image is reverted minutes later by self-heal.
  • A deleted object comes back (auto-sync recreates it).
  • An object created by hand is pruned if it carries the tracking annotation of an app, or ignored forever if not.
  • Two sources of truth: Git says 3 replicas, the cluster says 10.

What to do instead

  • Change Git and let the agent apply it — even during incidents, via a fast-track PR.
  • For true emergencies: disable auto-sync on that Application, fix, then commit and re-enable.
  • Use sync windows to freeze automated changes when needed (ch 7).
  • Grant humans read access and Argo CD UI actions instead of cluster-admin.
Chapter 1 · Part 1.2

Repository strategy

App repo vs config repo, monorepo vs polyrepo, directories vs branches, and the rendered manifests pattern.

Two repositories

App repo vs config repo

Separate what changes on every commit (code) from what changes on every deploy (environment state).

App repo

  • Source code, Dockerfile, tests.
  • Often the Helm chart too (charts/web), versioned with the code.
  • CI builds image web:1.4.2 and pushes chart web-0.7.0 to an OCI registry.
  • Developers own it.

Config repo (gitops-lab)

  • Which version runs where: image tags, chart versions, values per environment.
  • Argo CD Applications and ApplicationSets.
  • Cluster add-ons.
  • Platform team + service owners review it; prod paths need extra approvals.

Why not one repo?

  • CI writing tags back to the repo it builds from triggers itself (loops, skipped CI hacks).
  • Deploy history gets buried in code commits.
  • Permissions differ: many can merge code, few can change prod.
  • A single repo works for small teams if CI ignores the deploy paths.
Repo topology

Monorepo vs polyrepo for configuration

Most organizations land on one config repo per platform or per team, not one per service.

OptionStrengthsWeaknessesFits
One config repo for everythingOne place to search, one PR for cross-cutting changesLarge clones, noisy history, coarse permissions (use CODEOWNERS)Small and medium platforms
Config repo per teamTeam autonomy, clear ownershipShared add-ons duplicated or in a platform repoMany teams, one platform team
Config next to code (per service)Change code and config togetherCI-writes-back loops, deploy history mixed with codeFew services, very small teams
Platform repo + team reposPlatform owns clusters and add-ons; teams own appsTwo places to lookMost enterprise setups

Argo CD handles large monorepos well if you set the argocd.argoproj.io/manifest-generate-paths annotation and use webhooks (ch 12).

Environments

Directory per environment, not branch per environment

Keep one main branch; model environments as folders. Promotion is a diff between folders.

Branch per environment (avoid)

  • dev, staging, prod branches with merges between them.
  • Merges carry unrelated changes; cherry-picks drift apart.
  • Environment-specific files conflict on every merge.
  • Hard to see "what differs between staging and prod" in one view.

Directory per environment (prefer)

  • envs/dev, envs/staging, envs/prod on main.
  • Only real differences live in each folder (values, replicas, tags).
  • diff -r envs/staging envs/prod shows exactly what promotion will change.
  • Protect envs/prod/** with CODEOWNERS and required reviews.
gitops-lab/
├── charts/web/
├── envs/
│   ├── dev/web/values.yaml
│   │   # tag: 1.5.0-rc.1
│   ├── staging/web/values.yaml
│   │   # tag: 1.4.2
│   └── prod/web/values.yaml
│       # tag: 1.4.1, replicas 4
└── apps/web-appset.yaml
Rendered manifests

The rendered manifests pattern

Commit the final YAML that will be applied, not only the inputs that generate it.

Problem it solves

  • A one-line change in a shared chart or values file can change 40 objects in 3 environments.
  • Reviewers see the input diff, not the output diff.
  • Helm/Kustomize version differences can change output silently.

How it works

  • Inputs ("dry" source): charts, values, overlays.
  • A tool renders them to plain YAML ("hydrated") and commits it to an environment branch or folder.
  • The agent applies only plain YAML; reviewers approve the rendered diff.
  • Argo CD automates this with Source Hydrator (ch 8).

Costs

  • A second commit per change and more Git history.
  • Rendering must be deterministic (no lookup, no random values).
  • Needs a bot with write access to the hydrated location.
Repository strategy

What lives where: a checklist

If you cannot answer "where does this change go?" in one second, the layout is wrong.

ChangeRepo and pathWho approves
New feature codeApp repoService team
New image builtNothing yet — CI publishes the artifact—
Deploy 1.4.2 to staginggitops-lab/envs/staging/web/values.yaml (image tag)Service team
Promote to prodgitops-lab/envs/prod/web/values.yamlService team + prod CODEOWNERS
New chart templateApp repo charts/web → new chart versionService team
New environmentgitops-lab/envs/<env>/ + generator entryPlatform team
Cluster add-on upgradegitops-lab/clusters/<cluster>/Platform team
Secret valueNever in Git: AWS Secrets Manager + External Secrets (ch 10)Owner of the secret
Chapter 1 · Part 1.3

GitOps tools landscape

Argo CD vs Flux, the tools around Argo CD, and when GitOps is the wrong tool.

Argo CD vs Flux

Both are CNCF graduated; they differ in shape

This deck uses Argo CD. Knowing Flux helps you read other teams' setups.

AspectArgo CDFlux
Core objectsApplication, ApplicationSet, AppProjectGitRepository, Kustomization, HelmRelease, OCIRepository
UIBuilt-in web UI and CLICLI; UIs are separate projects
Helm handlingRenders with Helm templates, then applies like any YAML — no Helm release recordhelm-controller performs real Helm installs and upgrades (release history, Helm hooks)
Multi-tenancyAppProjects + Argo CD RBAC + SSOKubernetes RBAC + service account impersonation per object
Multi-clusterOne instance can manage many clusters (hub-and-spoke)Usually one Flux per cluster; remote clusters via kubeconfig
Typical choiceTeams that want a UI and central fleet viewTeams that want small, composable controllers

Consequence of Argo CD rendering Helm: helm list shows nothing for apps deployed by Argo CD, and lookup returns empty (ch 3).

Around Argo CD

Adjacent tools you will meet

Argo CD syncs Git to clusters. These tools fill the gaps before and after that step.

Delivery

  • Argo Rollouts — canary and blue-green with metric analysis (ch 11).
  • Kargo — promotion between stages ("freight" moves dev → staging → prod) (ch 9).
  • Argo CD Image Updater — writes new image tags back to Git (ch 9).
  • Argo CD Notifications — Slack, email, webhooks on sync events.

Infrastructure as Kubernetes objects

  • Crossplane — cloud resources as CRDs with compositions.
  • ACK (AWS Controllers for Kubernetes) — one controller per AWS service.
  • kro — compose several resources into one custom API.
  • All three let Argo CD sync infrastructure the same way as apps.

Rendering and policy

  • Helm, Kustomize — generators (ch 2–5).
  • Kyverno, Gatekeeper — admission policies on what gets synced (ch 10).
  • External Secrets Operator — secrets from AWS Secrets Manager (ch 10).
  • Terraform — still bootstraps clusters and Argo CD itself (ch 13).
Poor fits

When GitOps is the wrong tool

GitOps shines for long-lived declared state. Forcing everything through it creates friction.

Poor fits

  • One-off imperative tasks: data fixes, manual DB migrations, kubectl debug.
  • Secrets in plain Git (even base64) — use a secret manager.
  • Very high-churn values (feature flags flipping per minute).
  • Resources whose spec is owned by another controller.
  • Ad-hoc experiments in a personal namespace.

Better alternatives

  • Jobs triggered from CI or a runbook tool, recorded in a ticket.
  • External Secrets / Sealed Secrets / SOPS.
  • A feature-flag service.
  • ignoreDifferences for controller-owned fields.
  • A sandbox namespace outside Argo CD management.

Signs you are fighting the tool

  • Constant "OutOfSync" that everyone ignores.
  • Auto-sync disabled "temporarily" for months.
  • PRs that only bump a timestamp to trigger a redeploy.
  • People asking for cluster-admin "just to fix GitOps".
Common mistakes

GitOps foundations: mistakes that cause real outages

Each one has bitten production teams.

MistakeWhat happensDo this instead
Image tag latest in GitGit does not change, so nothing redeploys — or every node pulls a different imageImmutable tags or digests; bump in Git
Branch per environmentMerge conflicts and silent drift between envsFolder per environment on one branch
CI commits tags to the app repo it buildsInfinite CI loops, [skip ci] hacksSeparate config repo
Hotfix with kubectl edit and no commitSelf-heal reverts it; incident returnsFast-track PR, or pause auto-sync then commit
Secrets committed base64-encodedCredentials leaked in Git history foreverExternal Secrets with AWS Secrets Manager
CI keeps cluster-admin "for emergencies"Push and pull both active; nobody trusts GitRemove CI cluster credentials once GitOps works
Self-check

GitOps foundations: can you answer these?

Explain each answer out loud, with an example from the lab layout.

Principles

  • State the four OpenGitOps principles from memory.
  • Why does "pulled" improve security compared to push?
  • What is the difference between drift detection and self-heal?
  • Why is latest incompatible with "versioned and immutable"?

Repositories

  • Why separate the app repo from the config repo?
  • Why are branches per environment discouraged?
  • Which file changes when you promote 1.4.2 to prod?
  • What problem does the rendered manifests pattern solve, and what does it cost?

Tools

  • How does Argo CD's Helm handling differ from Flux's helm-controller?
  • Why does helm list show nothing for an Argo CD app?
  • Which tool moves a version between stages: Rollouts, Kargo or Image Updater?
  • Name two tasks that should not go through GitOps.

Practice: Lab 1 in Hands-on labs — deploy with push, then with Argo CD, and watch drift detection and self-heal.

Chapter 2

Helm fundamentals

Charts, releases and revisions, the release lifecycle, values precedence, and what changed in Helm 4.

Chapter 2 · Part 2.1

Concepts and chart anatomy

Chart, release, revision, repository — and every file inside a chart directory.

Vocabulary

Four nouns explain all of Helm

A chart is a package; a release is an installed instance of it; each change makes a revision.

Chart

  • A directory (or .tgz) of templates + default values + metadata.
  • Versioned with SemVer in Chart.yaml (version: 0.7.0).
  • Example: our charts/web, or bitnami/redis.

Release

  • One installed copy of a chart in a namespace, with a name.
  • helm install web-dev charts/web -n dev creates release web-dev.
  • The same chart can be installed many times with different values.

Revision

  • Every install, upgrade or rollback increments the revision.
  • Each revision stores the full rendered manifest and values.
  • Rollback = create a new revision from an old one.

Repository

  • Where charts are published and pulled from.
  • Classic: HTTP server with index.yaml.
  • Modern: OCI registry (ECR, GHCR) — oci://… (ch 4).
Release flow

What happens on helm upgrade --install

Helm is a client-side renderer plus a release-history store. There is no server component.

Charttemplates + values.yamlYour values-f prod.yaml, --setTemplate engineGo templates + SprigManifestsplain YAMLKubernetes APIapply (SSA for new installs)Release Secretsh.helm.release.v1.web.v3Hooks / waitoptional1 merge values → 2 render templates → 3 validate against the API → 4 apply objects → 5 store revision → 6 run hooks / wait for readinessOrder of creation is fixed by kind (Namespaces, CRDs, ServiceAccounts … then Deployments), not by file order.

Argo CD uses only steps 1–2 (helm template) and does the rest itself. That is why Helm hooks become Argo CD hooks and no release Secret exists (ch 6–7).

Chart anatomy

Every file in a chart directory

helm create web generates this layout; delete what you do not need.

charts/web/
├── Chart.yaml          # metadata, version, dependencies
├── Chart.lock          # pinned dependency versions
├── values.yaml         # defaults (the chart's API)
├── values.schema.json  # optional JSON Schema for values
├── charts/             # vendored dependency .tgz files
├── crds/               # CRDs: installed once, never upgraded
├── templates/
│   ├── _helpers.tpl    # named templates, no output
│   ├── deployment.yaml
│   ├── service.yaml
│   ├── ingress.yaml
│   ├── hpa.yaml
│   ├── NOTES.txt       # printed after install
│   └── tests/
│       └── test-connection.yaml
├── .helmignore         # excluded from the package
└── README.md

Rules that surprise people

  • Files in templates/ starting with _ render nothing; they hold helpers.
  • Every other file in templates/ must render valid YAML (or nothing).
  • crds/ files are plain YAML — no templating — installed only on first install (ch 4).
  • charts/ holds dependencies after helm dependency update.
  • values.schema.json is validated on install, upgrade, lint and template.
  • .helmignore keeps tests data, .git and CI files out of the package.
Chart.yaml

Chart.yaml field by field

apiVersion: v2 is the Helm 3 chart format and is unchanged in Helm 4.

apiVersion: v2
name: web
description: podinfo-based demo web service
type: application        # or: library
version: 0.7.0           # CHART version (SemVer, required)
appVersion: "6.7.1"      # version of the app inside (string)
kubeVersion: ">=1.29.0-0"
keywords: [web, demo]
maintainers:
  - name: platform-team
    email: platform@example.com
dependencies:
  - name: redis
    version: 20.x.x
    repository: oci://registry-1.docker.io/bitnamicharts
    condition: redis.enabled
annotations:
  example.com/owner: team-web

version vs appVersion

  • version is the package version: bump it on every chart change; registries reject duplicates.
  • appVersion is informational: usually the image tag the chart defaults to.
  • Common pattern: image.tag defaults to .Chart.AppVersion.
  • Quote appVersion so YAML does not turn 1.10 into 1.1.
  • type: library — only helpers, cannot be installed (ch 4).
  • kubeVersion — install fails on clusters outside the range.
  • dependencies — subcharts with condition / tags / alias (ch 4).
  • deprecated: true — warns users of the chart.
values.yaml

values.yaml is the public API of your chart

Everything users may change is a value with a sensible default; everything else stays in templates.

replicaCount: 2
image:
  repository: ghcr.io/stefanprodan/podinfo
  tag: ""            # empty → .Chart.AppVersion
  pullPolicy: IfNotPresent
service:
  type: ClusterIP
  port: 9898
ingress:
  enabled: false
  className: ""
  hosts: []
resources:
  requests: {cpu: 50m, memory: 64Mi}
  limits: {memory: 128Mi}
autoscaling:
  enabled: false
  minReplicas: 2
  maxReplicas: 10
podAnnotations: {}
extraEnv: []

Good values design

  • Flat enough to set with --set image.tag=1.4.2.
  • Booleans for features: ingress.enabled, autoscaling.enabled.
  • Escape hatches (extraEnv, podAnnotations) instead of forking the chart.
  • Comment every key — helm-docs turns comments into the README (ch 4).
  • Never default secrets in values.yaml.
Chapter 2 · Part 2.2

Release lifecycle

Install, upgrade, rollback and uninstall; where Helm stores state; previewing changes; values precedence.

Lifecycle commands

The commands you use every day

Prefer upgrade --install everywhere: it is idempotent, so pipelines and humans run the same command.

# install or upgrade in one idempotent command
helm upgrade --install web-dev charts/web \
  -n dev --create-namespace \
  -f envs/dev/web/values.yaml \
  --set image.tag=1.4.2 \
  --wait --timeout 5m

helm list -n dev                # releases, status, chart
helm history web-dev -n dev     # revisions
helm status web-dev -n dev      # last deploy, notes
helm rollback web-dev 3 -n dev  # → new revision from #3
helm uninstall web-dev -n dev   # delete objects + history
helm uninstall web-dev -n dev --keep-history

Inspect a release

  • helm get values web-dev — values you supplied; add --all for merged values.
  • helm get manifest web-dev — exactly what was applied.
  • helm get hooks web-dev — hook objects.
  • helm get notes web-dev — rendered NOTES.txt.
  • Add --revision N to inspect an older revision.
  • Upgrade values: by default only your flags + chart defaults; previous release values are not reused.
  • --reuse-values: reuse last values, ignore new chart defaults (risky).
  • --reset-then-reuse-values: new chart defaults + last values + new flags.
  • Best practice: always pass the full values files from Git.
Release storage

Where Helm keeps state: Secrets in the release namespace

No Tiller, no database. Each revision is one Secret of type helm.sh/release.v1.

$ kubectl get secret -n dev -l owner=helm
NAME                            TYPE                 DATA
sh.helm.release.v1.web-dev.v1   helm.sh/release.v1   1
sh.helm.release.v1.web-dev.v2   helm.sh/release.v1   1
sh.helm.release.v1.web-dev.v3   helm.sh/release.v1   1

# decode one revision (base64 → base64 → gzip → JSON)
kubectl get secret sh.helm.release.v1.web-dev.v3 -n dev \
  -o jsonpath='{.data.release}' | base64 -d | base64 -d \
  | gunzip | jq '{name, version, info: .info.status,
                 chart: .chart.metadata.version}'

Consequences

  • Anyone who can read Secrets in the namespace can read all values — including any secrets passed as values.
  • History grows: cap it with --history-max (default 10 on upgrade).
  • Deleting these Secrets makes Helm forget the release; the objects stay.
  • A release stuck in pending-install, pending-upgrade or pending-rollback blocks the next upgrade: "another operation (install/upgrade/rollback) is in progress".
  • Recover: helm rollback web-dev <last-good-rev>; last resort: delete the pending revision's Secret, then upgrade again.
Preview before apply

See the change before it reaches the cluster

Three levels: render locally, validate against the API server, and diff against the live release.

helm template

  • Renders locally; no cluster contact by default.
  • .Capabilities uses defaults unless --kube-version / --api-versions are passed.
  • lookup returns empty.
  • This is what Argo CD runs.
  • helm template web charts/web -f envs/prod/web/values.yaml

--dry-run=server

  • Renders and sends objects to the API server in dry-run mode.
  • Catches schema errors, admission webhook rejections, immutable field changes.
  • lookup works (it queries the cluster).
  • helm upgrade web-dev charts/web -n dev --dry-run=server

helm diff (plugin)

  • Shows a unified diff between the live release and the proposed upgrade.
  • helm plugin install https://github.com/databus23/helm-diff
  • helm diff upgrade web-dev charts/web -n dev -f …
  • Run it in CI on every PR for Helm-managed releases.
  • For Argo CD apps use argocd app diff instead (ch 7).
Values precedence

Which value wins: the merge order

Later sources override earlier ones. Maps merge key by key; lists are replaced whole.

1 · Chart defaults

  • values.yaml of the chart.
  • Subchart defaults under the subchart key.
→

2 · Parent chart

  • Parent values.yaml overrides subchart defaults.
  • global: shared with all subcharts.
→

3 · -f files

  • In command-line order.
  • The last file wins.
→

4 · --set family

  • --set, --set-string, --set-file, --set-json.
  • Override every file.

Merge gotchas

  • Lists are not merged: -f prod.yaml with extraEnv: [B] replaces the default [A] entirely.
  • Set a key to null to delete a default (e.g. remove a default resource limit).
  • --set a.b=1,a.c=2 — commas separate keys; escape commas in values with \,.
  • --set guesses types: --set tag=1.10 becomes the number 1.1; use --set-string.
  • --set list[0].name=x sets one index but you must know the full list.
Values precedence

Worked example: three sources, one result

Predict the result before reading the right column.

# charts/web/values.yaml
replicaCount: 2
image: {repository: ghcr.io/stefanprodan/podinfo, tag: ""}
extraEnv: [{name: LOG_LEVEL, value: info}]
resources: {limits: {memory: 128Mi}}

# envs/prod/web/values.yaml   (-f)
replicaCount: 4
extraEnv: [{name: REGION, value: ap-northeast-1}]
resources: {limits: {memory: 256Mi, cpu: null}}

# command line
--set image.tag=1.4.1
# merged result
replicaCount: 4                # file beats default
image:
  repository: ghcr.io/stefanprodan/podinfo  # kept (map merge)
  tag: 1.4.1                   # --set beats everything
extraEnv:                      # list REPLACED, LOG_LEVEL gone
  - {name: REGION, value: ap-northeast-1}
resources:
  limits: {memory: 256Mi}      # null removes a key

# check it yourself:
helm template web charts/web -f envs/prod/web/values.yaml \
  --set image.tag=1.4.1 --show-only templates/deployment.yaml
Chapter 2 · Part 2.3

What changed in Helm 4

Server-Side Apply, kstatus waiting, renamed flags, plugins — and a migration checklist for Helm 3 pipelines.

Helm 4 at a glance

Helm 4: GA on 12 Nov 2025

The first major version since Helm 3 (2019). Charts stay compatible; the CLI and apply behavior change.

Changed behavior

  • Server-Side Apply by default for new installs.
  • kstatus-based readiness for --wait.
  • Post-renderers are plugins, referenced by name.
  • helm registry login takes a host, not a URL.

Renamed flags

  • --atomic → --rollback-on-failure.
  • --force → --force-replace.
  • Old names still work and print deprecation warnings.
  • Fix them now so CI logs stay clean and scripts read like the current docs.

Unchanged

  • apiVersion: v2 charts install as before.
  • Release storage (Secrets), commands, repositories, OCI.
  • A new chart format (v3) is planned but not required.
  • Go SDK users: module path is now helm.sh/helm/v4.

Helm 3 support

  • Bug fixes ended with the final release on 9 Sep 2026.
  • Security fixes end 10 Feb 2027 (extended from Nov 2026).
  • After that, Helm 3 in CI is an unpatched dependency.
Server-Side Apply

From three-way merge to Server-Side Apply

Helm 3 computed a patch on the client from old manifest, new manifest and live object. Helm 4 lets the API server merge.

Helm 3: client-side three-way merge

  • Patch = f(previous release manifest, new manifest, live object).
  • Fields changed by others could be silently overwritten or kept unexpectedly.
  • No record of who owns which field.

Helm 4: Server-Side Apply

  • Helm sends the full desired object; the API server merges.
  • Each field records its field manager in metadata.managedFields.
  • Conflicts with other managers (kubectl, controllers, Argo CD) become visible instead of silent.
  • Same mechanism Argo CD offers with the ServerSideApply=true sync option (ch 7).

Which releases use it

  • New installs: SSA by default.
  • Upgrades and rollbacks: keep the apply method the release already used.
  • So a release first installed with Helm 3 stays on client-side apply after you switch to Helm 4.
  • Opt in explicitly with --server-side on upgrade.
Server-Side Apply

Reading field ownership

SSA makes ownership inspectable. This is how you answer "who changed replicas?".

$ kubectl get deploy web -n dev --show-managed-fields -o yaml
metadata:
  managedFields:
  - manager: helm              # set by the Helm release
    operation: Apply
    fieldsV1:
      f:spec:
        f:template: {...}
  - manager: kube-controller-manager
    operation: Update
    subresource: status
  - manager: kubectl-edit      # a human edited replicas
    operation: Update
    fieldsV1:
      f:spec:
        f:replicas: {}

What to look for

  • manager names the client; operation: Apply means SSA.
  • A field owned by kubectl-edit or kubectl-scale is manual drift.
  • HPA-managed replicas show the HPA controller as a manager.
  • Mixed Helm + manual edits used to be invisible; now they are explicit.
  • Helm and Argo CD: never manage the same objects with both helm upgrade and Argo CD.
  • Migrating a release to Argo CD: Argo CD takes over the objects; delete the old Helm release Secrets afterwards.
Waiting for readiness

--wait with kstatus, and --rollback-on-failure

kstatus is the readiness library used by kubectl and Flux. It understands many more resource types than Helm 3's checks.

# Helm 4 style
helm upgrade --install web-prod charts/web -n prod \
  -f envs/prod/web/values.yaml \
  --wait --timeout 10m \
  --rollback-on-failure        # was --atomic

# Helm 3 style (still accepted, prints a deprecation warning)
helm upgrade --install web-prod charts/web -n prod \
  -f envs/prod/web/values.yaml --wait --atomic

What changes in practice

  • --wait takes a strategy: watcher (kstatus, what a bare --wait means), hookOnly (the default when --wait is omitted) or legacy (Helm 3 checks, an escape hatch).
  • Readiness uses each resource's status conditions (kstatus).
  • Stricter: some charts that "succeeded" under Helm 3 now correctly report not ready.
  • Custom resources with standard status conditions are understood.
  • Timeouts in CI may need to grow; fix the chart's readiness probes before raising them.
  • --rollback-on-failure implies waiting; on failure it rolls back (or uninstalls a failed first install). Good for push pipelines.
  • Not used with Argo CD: it never runs helm upgrade; it uses health status and its own retry (ch 7).
Plugins and registries

Plugins, post-renderers and registry login

Three small changes that break scripts silently.

Plugins

  • Helm 4 adds WebAssembly plugins alongside classic executable plugins.
  • Plugins are the extension point for downloaders, post-renderers and CLI commands.
  • Check each plugin you depend on (helm-diff, helm-secrets, helm-unittest) for Helm 4 support before upgrading CI.

Post-renderers

  • Helm 3: --post-renderer ./kustomize-wrapper.sh (any executable).
  • Helm 4: --post-renderer <plugin-name>; the executable must be packaged as a plugin.
  • Typical use: run Kustomize patches over chart output.
  • With Argo CD prefer Kustomize helmCharts or a multi-source app instead (ch 5, 8).

Registry login

  • helm registry login 123456789012.dkr.ecr.ap-northeast-1.amazonaws.com
  • A full URL such as https://…/charts is rejected in Helm 4.
  • ECR: pipe aws ecr get-login-password into --password-stdin.
  • Chart references stay oci://<host>/<repo>/<chart>.
Migration checklist

Moving CI pipelines from Helm 3 to Helm 4

Do it per pipeline, behind a pinned Helm version, and compare rendered output before and after.

StepWhat to checkHow
1 · Pin versionsEach pipeline uses an explicit Helm versionContainer image or setup action with a fixed version
2 · Render diffSame manifests from Helm 3 and Helm 4helm template with both, diff the output
3 · Flags--atomic, --force, --post-renderer <path>, registry login URLsgrep -R your CI files and Makefiles
4 · PluginsEvery installed plugin supports Helm 4helm plugin list; read each plugin's releases
5 · WaitingReadiness passes under kstatusRun a staging deploy with --wait; fix probes
6 · Apply methodExisting releases stay client-sideDecide whether to opt in with --server-side; test on staging first
7 · SDK usersGo imports updatedhelm.sh/helm/v3 → helm.sh/helm/v4

Argo CD bundles its own Helm binary; your CLI upgrade does not change what Argo CD renders. Check the Helm version in the Argo CD release notes.

Common mistakes

Helm fundamentals: mistakes that cause real outages

Most Helm incidents come from values handling and from stuck releases.

MistakeWhat happensDo this instead
--reuse-values on every upgradeNew chart defaults never apply; old values linger for yearsPass full values files from Git each time
Unquoted versions with --set1.10 becomes 1.1; wrong image deployed--set-string or quote in values files
Expecting lists to mergeEnv vars or volumes from defaults disappearRepeat the full list in the override
Killing Helm mid-upgradeRelease stuck in pending-upgradeGenerous --timeout; recover with helm rollback
Secrets passed as valuesReadable from release Secrets by anyone with Secret read accessExternal Secrets or mounted secrets
Same objects managed by Helm CLI and Argo CDEndless fights over fields, surprise rollbacksOne owner per object
Upgrading CI to Helm 4 blindlyDeprecation warnings ignored; post-renderer scripts failFollow the migration checklist
Self-check

Helm fundamentals: can you answer these?

Explain the answer and name the exact command or field.

Concepts

  • What is the difference between version and appVersion?
  • What does a revision store, and where?
  • Why does a rollback create a new revision number?
  • Which files in a chart are not templated?

Lifecycle

  • When would you use --dry-run=server instead of helm template?
  • What does --reset-then-reuse-values do?
  • How do you recover from "another operation is in progress"?
  • In which order are -f files and --set applied, and how do lists merge?

Helm 4

  • Which releases use Server-Side Apply after upgrading to Helm 4?
  • What replaced --atomic, and why is it irrelevant for Argo CD?
  • What must change in a pipeline that uses a post-renderer script?
  • Until when does Helm 3 receive security fixes?

Practice: Lab 2 in Hands-on labs — install, upgrade and roll back a chart, decode release Secrets, and recover from a failed upgrade.

Chapter 3

Helm templating deep dive

Go templates, Sprig, named templates and values design: build the web chart template by template.

Chapter 3 · Part 3.1

Go templates and Sprig

How Helm turns a chart plus values into YAML, and the template language that does it.

How rendering works

Chart + values → template engine → YAML → Kubernetes

Helm never "understands" Kubernetes while rendering. It produces text, then parses the text as YAML.

values.yamlchart defaults-f envs/prod/values.yamlyour overrides--set image.tag=6.7.1last winstemplates/*.yaml_helpers.tplMerged .Values+ .Release .Chart .CapabilitiesGo text/template+ Sprig + Helm funcshelm templateprints YAMLhelm installSSA to the APIArgo CDhelm template → diffErrors appear at two stages: template execution (missing key, nil pointer) and YAML parsing (bad indentation).
Syntax

Actions, pipelines and the dot

Everything between {{ and }} is an action. A pipeline passes the left value as the LAST argument of the next function.

# templates/configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: {{ .Release.Name }}-web-config
data:
  # .Values.message | default "hello" | quote
  # is the same as  quote (default "hello" .Values.message)
  MESSAGE: {{ .Values.message | default "hello" | quote }}
  LEVEL: {{ upper .Values.logLevel | quote }}
  REPLICAS: {{ .Values.replicaCount | toString | quote }}
  {{- /* a comment: produces no output */}}

Rules

  • . is the current context; at the top of a file it is the root object.
  • $ always points to the root, even inside range and with.
  • Pipelines: x | f a calls f a x. That is why default "hello" .Values.message reads backwards.
  • Comments: {{/* … */}}; YAML # comments are still rendered into the output.
  • Function calls use spaces, not commas or parentheses: printf "%s-%s" a b.
  • Parentheses group: (include "web.fullname" .) | trunc 63.
Control structures

if, with, range and variables

Every block ends with {{ end }}. Empty values are falsy: false, 0, "", nil, empty list or map.

{{- if .Values.ingress.enabled }}
apiVersion: networking.k8s.io/v1
kind: Ingress
...
{{- end }}

{{- with .Values.nodeSelector }}
nodeSelector:
  {{- toYaml . | nindent 2 }}     # . is now the map
{{- end }}

env:
{{- range $key, $val := .Values.env }}
  - name: {{ $key }}
    value: {{ $val | quote }}
{{- end }}
{{- range .Values.extraPorts }}
  - containerPort: {{ .port }}    # . is the list item
{{- end }}

Scope traps

  • Inside with and range, .Values no longer exists: . changed. Use $.Values or a variable captured before the block.
  • {{ $name := include "web.fullname" . }} captures a value once, use it anywhere below in the same file.
  • range over a map iterates keys in sorted order: output is stable, which matters for GitOps diffs.
  • else if and with … else exist; if and .a .b / if or .a .b / if not .a combine conditions.
  • eq, ne, lt, gt compare; comparing an int with a float64 from YAML fails — cast with int first.
Whitespace

Whitespace control: {{-, -}}, indent and nindent

Most broken charts are indentation bugs. Trim with dashes, then put the newline back with nindent.

# wrong: blank lines and broken indentation
spec:
  template:
    metadata:
      annotations:
      {{ toYaml .Values.podAnnotations | indent 8 }}

# right
spec:
  template:
    metadata:
      {{- with .Values.podAnnotations }}
      annotations:
        {{- toYaml . | nindent 8 }}
      {{- end }}

How it works

  • {{- trims all whitespace (including newlines) to the LEFT; -}} trims to the RIGHT.
  • indent N prefixes every line with N spaces; the first line also gets them.
  • nindent N = newline + indent N: start the action right after {{- on its own line.
  • Pattern: {{- toYaml .Values.x | nindent <column> }} where column = where the keys must start.
  • Check with helm template . --show-only templates/deployment.yaml.
  • The space after {{- is required: {{-3}} is the number −3, not trimming.
Function reference

The functions you will use every day

Helm ships Go template built-ins, most of the Sprig library, and a few Helm-only functions.

FunctionExampleUse it for
default.Values.tag | default .Chart.AppVersionFallback when the value is empty
requiredrequired "image.repository is required" .Values.image.repositoryFail rendering with a clear message
fail{{ fail "autoscaling and replicaCount=0 conflict" }}Reject invalid combinations
quote / squote{{ .Values.port | quote }}Force a YAML string (env values, annotations)
toYaml / toJsontoYaml .Values.resources | nindent 12Copy a whole map or list into the manifest
includeinclude "web.labels" . | nindent 4Call a named template and pipe the result
tpltpl .Values.configTemplate .Render a string from values as a template
printf / trunc / trimSuffixprintf "%s-%s" .Release.Name "web" | trunc 63 | trimSuffix "-"Build names within the 63-char limit
sha256sum / b64encinclude (print $.Template.BasePath "/configmap.yaml") . | sha256sumChecksums, Secret data
dict / list / merge / hasKeyinclude "web.container" (dict "root" $ "c" .)Pass several values into a named template
lookuplookup "v1" "Secret" .Release.Namespace "db"Read live cluster objects (empty without a cluster)
Built-in objects

What the template can see besides .Values

These objects are filled by Helm at render time.

ObjectUseful fields
.ReleaseName, Namespace, Revision, IsInstall, IsUpgrade, Service ("Helm")
.ChartName, Version, AppVersion, Type — from Chart.yaml
.ValuesMerged values (defaults + files + --set)
.CapabilitiesKubeVersion.Version, APIVersions.Has "policy/v1"
.FilesGet "conf/app.yaml", Glob "conf/*", AsConfig, AsSecrets
.TemplateName (current file path), BasePath (the chart's templates dir)

GitOps notes

  • Under Argo CD, .Release.Name is the Application name unless spec.source.helm.releaseName is set.
  • Argo CD renders with helm template, so .Release.IsUpgrade is false and .Release.Revision is 1 on every sync — never branch logic on them.
  • .Capabilities comes from the target cluster when Argo CD renders; with plain helm template it is a default unless you pass --kube-version / --api-versions.
  • .Files can read files inside the chart only, never outside it, and not files in templates/.
Debugging

See exactly what Helm will send

Render locally first; only then talk to a cluster.

# render everything, show values-merge errors with context
helm template web charts/web -f envs/dev/values.yaml --debug

# one file only
helm template web charts/web --show-only templates/deployment.yaml

# render as a specific cluster would see it
helm template web charts/web --kube-version 1.33.0 \
  --api-versions monitoring.coreos.com/v1

# validate against the live API server without changing it
helm install web charts/web --dry-run=server

# what is actually deployed for a release
helm get manifest web -n web
helm get values web -n web --all

# chart-level sanity
helm lint charts/web --strict -f envs/prod/values.yaml

Reading errors

  • executing "web/templates/deployment.yaml" at <.Values.image.tag>: nil pointer evaluating interface {}.tag → the parent key image is missing.
  • error converting YAML to JSON: yaml: line 23 → indentation; render with --debug to see the broken output.
  • --debug prints the rendered manifests even when YAML parsing fails.
  • helm lint --strict turns warnings (such as missing icon) into failures — useful in CI.
Chapter 3 · Part 3.2

Reusable templates

Named templates in _helpers.tpl, standard labels, selectors, checksums and why lookup does not belong in GitOps.

_helpers.tpl

Named templates for the web chart

Files starting with _ are not rendered as manifests. They hold define blocks you call with include.

{{/* charts/web/templates/_helpers.tpl */}}
{{- define "web.name" -}}
{{- $n := default .Chart.Name .Values.nameOverride }}
{{- $n | trunc 63 | trimSuffix "-" }}
{{- end }}

{{- define "web.fullname" -}}
{{- if .Values.fullnameOverride }}
{{- .Values.fullnameOverride | trunc 63 | trimSuffix "-" }}
{{- else }}
{{- $name := default .Chart.Name .Values.nameOverride }}
{{- if contains $name .Release.Name }}
{{- .Release.Name | trunc 63 | trimSuffix "-" }}
{{- else }}
{{- $full := printf "%s-%s" .Release.Name $name }}
{{- $full | trunc 63 | trimSuffix "-" }}
{{- end }}
{{- end }}
{{- end }}
{{- define "web.chart" -}}
{{- $c := printf "%s-%s" .Chart.Name .Chart.Version }}
{{- $c | replace "+" "_" | trunc 63 | trimSuffix "-" }}
{{- end }}

{{- define "web.selectorLabels" -}}
app.kubernetes.io/name: {{ include "web.name" . }}
app.kubernetes.io/instance: {{ .Release.Name }}
{{- end }}

{{- define "web.labels" -}}
{{- $v := .Values.image.tag | default .Chart.AppVersion }}
helm.sh/chart: {{ include "web.chart" . }}
{{ include "web.selectorLabels" . }}
app.kubernetes.io/version: {{ $v | quote }}
app.kubernetes.io/managed-by: {{ .Release.Service }}
{{- end }}

Prefix every template name with the chart name (web.): template names are global across the chart and its subcharts, so a generic fullname can be silently overridden.

Labels and selectors

Recommended labels, and the two that must never change

Selector labels are copied into spec.selector, which is immutable on Deployments, StatefulSets and DaemonSets.

LabelValueIn selector?
app.kubernetes.io/nameChart or app nameYes
app.kubernetes.io/instanceRelease nameYes
app.kubernetes.io/versionApp version (image tag)No — changes every release
app.kubernetes.io/componentfrontend, worker, dbOnly if fixed from day one
app.kubernetes.io/part-ofHigher-level applicationNo
app.kubernetes.io/managed-byHelmNo
helm.sh/chartname-versionNo — changes every chart bump

Why it matters

  • Putting version or helm.sh/chart in the selector breaks the next upgrade with field is immutable.
  • Changing name or instance later forces deleting and recreating the workload.
  • Services select Pods with the same selector labels — keep them identical.
  • Argo CD 3.x tracks resources with an annotation by default, so app.kubernetes.io/instance is free for Helm to use as intended.
templates/deployment.yaml

A Deployment built from helpers and values

Every value comes from values.yaml; every name and label from a helper.

{{- $tag := .Values.image.tag | default .Chart.AppVersion }}
{{- $cfg := include (print $.Template.BasePath "/configmap.yaml") . }}
apiVersion: apps/v1
kind: Deployment
metadata:
  name: {{ include "web.fullname" . }}
  labels:
    {{- include "web.labels" . | nindent 4 }}
spec:
  {{- if not .Values.autoscaling.enabled }}
  replicas: {{ .Values.replicaCount }}
  {{- end }}
  selector:
    matchLabels:
      {{- include "web.selectorLabels" . | nindent 6 }}
  template:
    metadata:
      annotations:
        checksum/config: {{ $cfg | sha256sum }}
      labels:
        {{- include "web.selectorLabels" . | nindent 8 }}
    spec:
      containers:
        - name: web
          image: "{{ .Values.image.repository }}:{{ $tag }}"
          imagePullPolicy: {{ .Values.image.pullPolicy }}
          ports:
            - name: http
              containerPort: {{ .Values.service.targetPort }}
          envFrom:
            - configMapRef:
                name: {{ include "web.fullname" . }}
          {{- with .Values.extraEnv }}
          env:
            {{- toYaml . | nindent 12 }}
          {{- end }}
          readinessProbe:
            httpGet: { path: /readyz, port: http }
          resources:
            {{- toYaml .Values.resources | nindent 12 }}
      {{- with .Values.nodeSelector }}
      nodeSelector:
        {{- toYaml . | nindent 8 }}
      {{- end }}

replicas is omitted when an HPA owns the count — otherwise every sync fights the autoscaler (Chapter 7 shows the Argo CD side of this).

Service and HPA

Optional resources behind feature flags

Whole files are wrapped in if so a disabled feature renders nothing.

# templates/service.yaml
apiVersion: v1
kind: Service
metadata:
  name: {{ include "web.fullname" . }}
  labels:
    {{- include "web.labels" . | nindent 4 }}
spec:
  type: {{ .Values.service.type }}
  ports:
    - port: {{ .Values.service.port }}
      targetPort: http
      name: http
  selector:
    {{- include "web.selectorLabels" . | nindent 4 }}
# templates/hpa.yaml
{{- if .Values.autoscaling.enabled }}
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: {{ include "web.fullname" . }}
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: {{ include "web.fullname" . }}
  minReplicas: {{ .Values.autoscaling.minReplicas }}
  maxReplicas: {{ .Values.autoscaling.maxReplicas }}
  metrics:
    - type: Resource
      resource:
        name: cpu
        target:
          type: Utilization
          averageUtilization: {{ .Values.autoscaling.targetCPU }}
{{- end }}

targetPort: http refers to the named container port, so changing the container port never needs a Service change.

Ingress

Looping over hosts: the dot changes inside range

The Ingress renders one rule per entry in .Values.ingress.hosts.

# templates/ingress.yaml
{{- if .Values.ingress.enabled }}
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: {{ include "web.fullname" . }}
  {{- with .Values.ingress.annotations }}
  annotations:
    {{- toYaml . | nindent 4 }}
  {{- end }}
spec:
  ingressClassName: {{ .Values.ingress.className }}
  rules:
  {{- range .Values.ingress.hosts }}
    - host: {{ .host | quote }}
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: {{ include "web.fullname" $ }}
                port:
                  name: http
  {{- end }}
{{- end }}

Read it carefully

  • Inside range .Values.ingress.hosts the dot is one host entry (.host).
  • The helper needs the root context, so it is called with $, not ..
  • with .Values.ingress.annotations renders the annotations: key only when the map is not empty.
  • On EKS, className: alb plus AWS Load Balancer Controller annotations in env values (Chapter 13).
  • Render it: helm template web charts/web --set ingress.enabled=true --show-only templates/ingress.yaml.
Config rollouts

Roll Pods when a ConfigMap changes

Kubernetes does not restart Pods when a mounted or referenced ConfigMap changes. A checksum annotation makes the Pod template change.

# templates/configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: {{ include "web.fullname" . }}
  labels:
    {{- include "web.labels" . | nindent 4 }}
data:
  PODINFO_UI_MESSAGE: {{ .Values.message | quote }}
  PODINFO_UI_COLOR: {{ .Values.color | quote }}

# in deployment.yaml
{{- $cfg := include (print $.Template.BasePath "/configmap.yaml") . }}
...
      annotations:
        checksum/config: {{ $cfg | sha256sum }}

How and why

  • $.Template.BasePath is the chart's templates directory; the include renders the ConfigMap text.
  • Any value change → new hash → new Pod template → rolling update.
  • Same trick for Secrets you render yourself.
  • Not needed for Kustomize: configMapGenerator appends a content hash to the name (Chapter 5).
  • For Secrets that come from External Secrets (Chapter 10) the content is not in Git; use a reloader controller or rotate by name.
Passing context

include vs template, and passing more than one value

A named template receives exactly one argument. Bundle several with dict.

{{- define "web.container" -}}
{{- $tag := .c.image.tag | default .root.Chart.AppVersion }}
- name: {{ .c.name }}
  image: "{{ .c.image.repository }}:{{ $tag }}"
  resources:
    {{- toYaml .c.resources | nindent 4 }}
{{- end }}

# caller: one container per entry in .Values.sidecars
containers:
  {{- range .Values.sidecars }}
  {{- $args := dict "root" $ "c" . }}
  {{- include "web.container" $args | nindent 2 }}
  {{- end }}

Rules

  • include returns a string, so you can pipe it into nindent, sha256sum, trim.
  • template writes output directly and cannot be piped — prefer include everywhere.
  • Pass $ (as root) whenever the named template needs .Chart, .Release or .Values.
  • tpl renders a string that came from values: {{ tpl .Values.annotationsTemplate . }} lets users write {{ .Release.Name }} in values.
  • Keep logic in helpers small: if a helper needs 40 lines of conditionals, the values API is too clever.
lookup

lookup reads the live cluster — and returns nothing in GitOps

lookup queries the API server during helm install/upgrade. It cannot do that when there is no cluster connection.

# keep a generated password across upgrades
{{- $old := lookup "v1" "Secret" .Release.Namespace "web-db" }}
apiVersion: v1
kind: Secret
metadata:
  name: web-db
data:
  {{- if $old }}
  password: {{ index $old.data "password" }}
  {{- else }}
  password: {{ randAlphaNum 24 | b64enc }}
  {{- end }}

Behavior

  • helm install / upgrade: returns the object, or an empty map if it does not exist.
  • helm template and client-side dry runs: always empty.
  • Argo CD renders with helm template: always empty → the example above generates a NEW password on every render, so the app is permanently OutOfSync.
  • Same problem with randAlphaNum, now, uuidv4 anywhere in templates.

GitOps alternative: create secrets outside the chart (External Secrets Operator, Sealed Secrets, SOPS — Chapter 10) or let the app generate them once. Rendering must be a pure function of chart + values.

Chapter 3 · Part 3.3

Values design and validation

values.yaml is the public API of your chart. Design it, document it, and validate it with a JSON Schema.

values.yaml

The web chart values: the contract with its users

Every key is documented, has a working default, and appears in exactly one template.

# -- Number of Pods when autoscaling is off
replicaCount: 2

image:
  # -- Image repository
  repository: ghcr.io/stefanprodan/podinfo
  # -- Image tag; defaults to Chart.appVersion
  tag: ""
  pullPolicy: IfNotPresent

nameOverride: ""
fullnameOverride: ""

message: "hello from web"
color: "#34577c"

service:
  type: ClusterIP
  port: 80
  targetPort: 9898
ingress:
  enabled: false
  className: nginx
  annotations: {}
  hosts:
    - host: web.local

autoscaling:
  enabled: false
  minReplicas: 2
  maxReplicas: 5
  targetCPU: 70

resources:
  requests: { cpu: 50m, memory: 64Mi }
  limits:   { memory: 128Mi }

# -- Extra env vars, rendered as-is
extraEnv: []
podAnnotations: {}
nodeSelector: {}
tolerations: []
affinity: {}

Design rules

  • camelCase keys, nouns not verbs; booleans named enabled.
  • Defaults that install and run with zero overrides.
  • Group by resource (service, ingress, autoscaling), at most 2–3 levels deep.
  • Escape hatches (extraEnv, podAnnotations, nodeSelector, tolerations) instead of a new key per field.
  • Use maps for things people override per environment; lists are replaced whole on merge, never merged.
  • Never rename a key in a minor version: users' env files break silently.
Values per environment

Layering values files in the config repo

Later files win. Keep environment files small: only the differences from the chart defaults.

gitops-lab/
├── charts/web/              # the chart (defaults)
│   ├── Chart.yaml
│   ├── values.yaml
│   ├── values.schema.json
│   └── templates/
└── envs/
    ├── dev/values.yaml      # replicaCount: 1, message: dev
    ├── staging/values.yaml
    └── prod/values.yaml     # autoscaling.enabled: true, ingress

# render prod exactly as Argo CD will
helm template web charts/web -n web \
  -f envs/prod/values.yaml

Merge semantics

  • Maps merge key by key, recursively.
  • Lists are replaced entirely: an env file with tolerations: [a] drops the defaults.
  • Setting a key to null in a later file deletes the default key.
  • --set a.b=1 beats every file; --set-string keeps "true" / "01" as strings.
  • In Argo CD the same layering is spec.source.helm.valueFiles (order matters) plus values / valuesObject (Chapter 6).
values.schema.json

Validate values with JSON Schema

If the chart contains values.schema.json, Helm validates the merged values on install, upgrade, lint and template.

{
  "$schema": "http://json-schema.org/draft-07/schema#",
  "type": "object",
  "required": ["image", "service"],
  "properties": {
    "replicaCount": { "type": "integer", "minimum": 0 },
    "image": {
      "type": "object",
      "required": ["repository"],
      "properties": {
        "repository": { "type": "string", "minLength": 1 },
        "tag": { "type": "string" },
        "pullPolicy": { "enum": ["Always", "IfNotPresent"] }
      }
    },
    "service": {
      "type": "object",
      "properties": {
        "type": { "enum": ["ClusterIP", "LoadBalancer"] },
        "port": { "type": "integer" } } }
  }
}

What you gain

  • Typos and wrong types fail at render time, in CI, not at 2 a.m. in prod.
  • "replicaCount": "3" (a string) is rejected by "type": "integer".
  • Enums document allowed values better than comments.
  • Argo CD renders with helm template, so a schema error shows up as a ComparisonError on the Application before anything is applied.
  • Add "additionalProperties": false on strict objects to catch misspelled keys — but not at the top level if parents pass global values.
  • Generate a starting schema from values with a plugin, then tighten it by hand.
Typing bugs

YAML and template type surprises

Values are parsed as YAML, then handled as Go types. Most surprises come from that conversion.

SymptomCauseFix
replicas: 1e+06 in outputLarge integers become float64 and print in scientific notation{{ .Values.big | int64 }} or keep as string
Env var rejected: "cannot unmarshal bool into string"value: true unquoted in a container envvalue: {{ .Values.flag | quote }}
Version 1.10 became 1.1YAML parsed it as a floatQuote in values: tag: "1.10"
nil pointer evaluating interface {}.tagParent key missing (image: absent)Defaults in values.yaml, or with / (.Values.image | default dict).tag
wrong type for value; expected string; got intString function on a number (upper, trunc)toString first
Annotation value on became trueYAML 1.1 booleans (on, yes)Quote annotation values
Empty map rendered as {} under a key that expects a listtoYaml of a default {}Wrap in with so nothing renders when empty
Defensive templates

Fail early with required, fail and schema

A chart that renders invalid YAML silently is worse than one that refuses to render.

{{- /* templates/_validate.tpl */ -}}
{{- define "web.validate" -}}
{{- $a := .Values.autoscaling }}
{{- if and $a.enabled (lt (int $a.maxReplicas) (int $a.minReplicas)) }}
{{- fail "autoscaling.maxReplicas must be >= minReplicas" }}
{{- end }}
{{- if and .Values.ingress.enabled (not .Values.ingress.hosts) }}
{{- fail "ingress.enabled needs at least one ingress.hosts entry" }}
{{- end }}
{{- end }}

# deployment.yaml, first line
{{- include "web.validate" . }}

# fail fast on one key
{{- $repo := required "image.repository is required"
      .Values.image.repository }}

Which tool when

  • Schema: types, required keys, enums, ranges — declarative, shows up in editors.
  • **required**: one key that must be set, with a message naming it.
  • **fail**: rules across several keys that JSON Schema expresses badly.
  • helm-unittest (Chapter 4): prove the rules fire, so refactors do not remove them.
Common mistakes

Templating mistakes that reach production

Each has bitten real teams; most are caught by helm template + lint + schema in CI.

Rendering

  • Using .Values inside range/with instead of $.Values.
  • Forgetting {{- and getting blank lines or a key at the wrong indent.
  • toYaml with indent instead of nindent on the same line as the key.
  • Unquoted env values, annotation values and versions.

Design

  • Version or chart labels in spec.selector → immutable-field errors on the next upgrade.
  • Generic template names (fullname) colliding with subcharts.
  • A values key for every Kubernetes field instead of escape hatches.
  • Renaming values keys without a deprecation path.

GitOps

  • lookup, randAlphaNum, now, uuidv4 → output changes every render → permanent OutOfSync.
  • Branching on .Release.IsInstall / IsUpgrade — always "install" under Argo CD.
  • Rendering Secrets with real values from values.yaml into Git.
  • Relying on .Capabilities in plain helm template without --kube-version.
Self-check

Templating: can you answer these?

Answer out loud, then check with helm template.

Syntax

  • What does x | default "a" | quote expand to as nested calls?
  • Inside range .Values.hosts, how do you reach .Release.Name?
  • What is the difference between {{- and -}}?
  • Why nindent rather than indent after {{-?
  • Which values are falsy in an if?

Helpers and labels

  • Why prefix template names with the chart name?
  • Which labels belong in the selector and why only those?
  • How does the checksum annotation trigger a rollout?
  • include vs template: which can be piped?
  • How do you pass both the root context and a loop item to a helper?

Values and GitOps

  • How are lists merged across values files?
  • What does values.schema.json catch that helm lint does not?
  • What does lookup return under Argo CD, and why?
  • Why must rendering be deterministic in GitOps?
  • Fix replicas: 1e+06.

Practice: Lab 3 in Hands-on labs — write the web chart from scratch, then break it with an indentation bug and a schema violation.

Chapter 4

Helm in production

Dependencies, library charts, hooks, CRDs, OCI distribution, signing and a CI quality gate for charts.

Chapter 4 · Part 4.1

Dependencies and library charts

Reuse charts instead of copying them: subcharts, conditions, aliases, global values and shared template libraries.

Dependencies

Declaring subcharts in Chart.yaml

Helm 3 and 4 (chart apiVersion: v2) declare dependencies in Chart.yaml; requirements.yaml was Helm 2.

# charts/web/Chart.yaml
apiVersion: v2
name: web
description: podinfo-based demo service
type: application
version: 0.3.0          # chart version (SemVer)
appVersion: "6.7.1"     # app/image version, informational
dependencies:
  - name: redis
    version: "~20.0.0"  # SemVer range
    repository: oci://registry-1.docker.io/bitnamicharts
    condition: redis.enabled
  - name: common-lib
    version: "1.x.x"
    repository: file://../common-lib
  - name: postgresql
    alias: db
    version: "15.5.x"
    repository: oci://registry-1.docker.io/bitnamicharts
    tags: [database]

Fields

  • version: an exact version or a SemVer range (~, ^, 1.x.x, >=1.2 <2).
  • repository: oci://…, an https:// chart repo URL, a repo alias @name, or file:// for a local path.
  • condition: a values path; the subchart is skipped when it is false.
  • tags: enable or disable groups with tags.database: false; condition wins over tags.
  • alias: install the same chart under another name (and values key), or twice.
  • import-values: pull a child's exported values up into the parent.
Dependency workflow

Chart.lock, the charts/ directory, and who downloads what

Ranges are resolved once and pinned in Chart.lock. Builds should use the lock, not re-resolve.

# resolve ranges, download .tgz files, write Chart.lock
helm dependency update charts/web

# download exactly what Chart.lock says (CI, reproducible)
helm dependency build charts/web

helm dependency list charts/web
# NAME        VERSION  REPOSITORY             STATUS
# redis       ~20.0.0  oci://registry-1...    ok
# common-lib  1.x.x    file://../common-lib   ok

ls charts/web/charts/
# common-lib-1.0.0.tgz  redis-20.0.3.tgz ...

Decisions

  • **Commit Chart.lock**: always. It is the reproducibility record.
  • **Commit charts/*.tgz?** Vendoring makes renders offline and immune to upstream deletions, at the cost of binary files in Git. Pick one rule per repo.
  • Argo CD's repo-server builds missing dependencies when it renders; it needs network access and credentials for private dependency repositories.
  • Upstream chart repos sometimes remove old versions — vendoring or mirroring to your own OCI registry protects production.
  • Bump the parent chart version whenever a dependency version changes.
Values and subcharts

Configuring subcharts from the parent

A subchart sees only its own section of the parent values, plus global.

# charts/web/values.yaml (parent)
global:
  imageRegistry: 123456789012.dkr.ecr.ap-northeast-1.amazonaws.com

redis:                 # key = subchart name
  enabled: true        # used by `condition: redis.enabled`
  architecture: standalone
  auth:
    enabled: false

db:                    # key = alias
  auth:
    database: web

# inside the redis subchart templates:
#   .Values.architecture          -> "standalone"
#   .Values.global.imageRegistry  -> the ECR host

Rules

  • The parent can override any subchart value; a subchart can never read the parent's values (except global).
  • global is copied into every subchart: use it sparingly (registry, cluster domain, environment name).
  • Subchart templates render with their own .Chart but the parent's .Release.
  • All templates share one namespace of named templates — why helpers are prefixed with the chart name.
  • Reading values of a third-party subchart: helm show values oci://…/redis --version 20.0.3.
Library charts

Share templates across 30 services without copy-paste

A chart with type: library contains only define blocks. It cannot be installed; it is used as a dependency.

# charts/common-lib/Chart.yaml
apiVersion: v2
name: common-lib
type: library
version: 1.0.0

# charts/common-lib/templates/_deployment.tpl
{{- define "common.deployment" -}}
apiVersion: apps/v1
kind: Deployment
metadata:
  name: {{ include "common.fullname" . }}
  labels:
    {{- include "common.labels" . | nindent 4 }}
spec:
  ...
{{- end }}
# charts/web/templates/deployment.yaml
{{- include "common.deployment" . }}

# charts/web/templates/service.yaml
{{- include "common.service" . }}

# a service-specific extra
# charts/web/templates/pdb.yaml
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: {{ include "common.fullname" . }}
spec:
  maxUnavailable: 1
  selector:
    matchLabels:
      {{- include "common.selectorLabels" . | nindent 6 }}

Pays off when many in-house services share one shape: one fix rolls out by bumping the library version. Version it like an API (breaking template change = major). Bitnami common is a public example. Trade-off: the manifest shape lives in another chart, so it is harder to read.

Umbrella charts

One chart that installs a whole system — and why GitOps teams avoid it

An umbrella chart has no templates of its own, only dependencies.

QuestionUmbrella chartOne Application per component (Argo CD)
Unit of deploy and rollbackEverything togetherEach component on its own
Blast radius of a bad valueWhole systemOne component
Different namespaces per componentAwkward (one release namespace)Natural
OrderingHooks and weights onlySync waves across Applications (Chapter 7–8)
Release history sizeOne huge release Secret (1 MiB Secret limit)Small, per component
Good fitShipping a product to customers as one artifactRunning your own platform with GitOps

In GitOps, the "umbrella" becomes an ApplicationSet or app-of-apps (Chapter 8): many small Applications, each pointing at one chart.

Chapter 4 · Part 4.2

Hooks, tests and CRDs

Run Jobs at the right moment, test a release, and handle CRDs and immutable fields without outages.

Hooks

Helm hooks: resources that run at lifecycle points

A hook is a normal manifest with the helm.sh/hook annotation. It is not part of the release's managed resources.

HookRuns
pre-installAfter render, before any resource is created
post-installAfter all resources are created
pre-upgrade / post-upgradeAround an upgrade
pre-rollback / post-rollbackAround a rollback
pre-delete / post-deleteAround uninstall
testOnly when you run helm test

Controls

  • helm.sh/hook-weight: "-5": lower runs first; weights are strings.
  • helm.sh/hook-delete-policy: before-hook-creation (default), hook-succeeded, hook-failed.
  • Helm waits for Jobs and Pods to complete; other kinds just need to be created.
  • A failed hook fails the install/upgrade.
  • Hook resources are not deleted on helm uninstall unless a delete policy removes them.
  • Several hooks in one annotation: "pre-install,pre-upgrade".
Migration hook

Database migrations before the new version starts

The classic hook: run a schema migration Job before Pods with the new code roll out.

# templates/migrate-job.yaml
{{- if .Values.migrations.enabled }}
apiVersion: batch/v1
kind: Job
metadata:
  name: {{ include "web.fullname" . }}-migrate
  annotations:
    "helm.sh/hook": pre-install,pre-upgrade
    "helm.sh/hook-weight": "-5"
    "helm.sh/hook-delete-policy": before-hook-creation,hook-succeeded
spec:
  backoffLimit: 1
  activeDeadlineSeconds: 600
  template:
    spec:
      restartPolicy: Never
      containers:
        - name: migrate
          image: "{{ .Values.image.repository }}:{{ .Values.image.tag }}"
          args: ["migrate", "up"]
{{- end }}

Design rules

  • Migrations must be backward compatible: old Pods keep running while the Job runs and during the rollout.
  • Expand → migrate → contract: add columns first, remove them a release later.
  • Set activeDeadlineSeconds and backoffLimit so a stuck migration fails instead of blocking forever.
  • Keep logs: hook-succeeded deletes the Job; drop it while debugging.
  • A ConfigMap or Secret the hook needs must also be a hook with a lower weight — normal resources do not exist yet at pre-install.
Hooks under Argo CD

How Argo CD treats Helm hooks

Argo CD renders with helm template and applies the result itself, so Helm's lifecycle never runs.

Mapping

  • Argo CD translates Helm hook annotations into its own hooks: install/upgrade hooks run as PreSync / PostSync.
  • helm.sh/hook-weight is used like a sync wave inside the phase.
  • Every sync is "an upgrade and an install at once": pre-install and pre-upgrade hooks both run on every sync.
  • pre-delete / post-delete map to PreDelete / PostDelete.
  • pre-rollback, post-rollback and test hooks are not supported.

Consequences

  • A post-install-only seeding Job runs again on every sync → make hooks idempotent.
  • Prefer native Argo CD annotations (argocd.argoproj.io/hook, sync-wave) in charts you own — clearer behavior (Chapter 7).
  • helm list shows nothing for Argo CD-managed apps: there is no Helm release Secret.
  • helm test is not run by Argo CD; run tests as a PostSync hook if you need them in the sync.
helm test

Smoke-test a release with helm test

Test Pods are hooks with helm.sh/hook: test. They run only when you ask, after install or upgrade.

# templates/tests/test-connection.yaml
apiVersion: v1
kind: Pod
metadata:
  name: {{ include "web.fullname" . }}-test
  annotations:
    "helm.sh/hook": test
    "helm.sh/hook-delete-policy": before-hook-creation
spec:
  restartPolicy: Never
  containers:
    - name: curl
      image: curlimages/curl:8.10.1
      command: ["curl", "-fsS"]
      args:
        - http://{{ include "web.fullname" . }}:{{ .Values.service.port }}/healthz
helm upgrade --install web charts/web -n web \
  --create-namespace --wait
helm test web -n web --logs
# NAME: web
# ...
# TEST SUITE:     web-test
# Phase:          Succeeded
# POD LOGS: web-test
# {"status":"OK"}

Good tests: reach the Service, check /healthz, call one real endpoint. Not good: full integration suites — those belong in CI or in a Rollouts analysis (Chapter 11).

CRDs

CRDs: the crds/ directory vs templates

CRDs are cluster-scoped and deleting one deletes every custom resource of that kind. Helm is deliberately cautious.

crds/ directoryCRDs in templates/
InstalledOn first install onlyOn install and upgrade
Upgraded by HelmNeverYes
Deleted on uninstallNeverYes — and all its CRs with it
TemplatedNo (plain YAML)Yes
Skip--skip-crdsValues flag in the chart

Practical patterns

  • Upgrade crds/ CRDs yourself: kubectl apply --server-side -f crds/ before helm upgrade.
  • Or ship CRDs as a separate chart (foo-crds) with its own lifecycle.
  • Protect CRDs in templates with helm.sh/resource-policy: keep.
  • Argo CD renders CRDs from crds/ and manages them like other resources, so they ARE upgraded; spec.source.helm.skipCrds: true turns that off.
  • Order: CRDs must exist before CRs — Argo CD applies CRDs first; sync waves help with operators (Chapter 7).
Immutable fields

Upgrades that fail with "field is immutable"

Some fields cannot be changed in place; the API server rejects the update.

ResourceImmutable fieldTypical cause
Deployment / StatefulSet / DaemonSetspec.selectorChanged selector labels in a chart bump
StatefulSetvolumeClaimTemplates, serviceNameChanging storage size or class
Jobspec.templateNew image tag for a migration Job that was not deleted
ServiceclusterIPHard-coded IP changed
PVCmost of spec (size can grow)Changing storage class

Fixes

  • Prevent: never change selector labels; use before-hook-creation on Job hooks.
  • Rename the resource (new name = new object) and let the old one be pruned.
  • Delete and recreate during a window: kubectl delete sts web --cascade=orphan keeps Pods running.
  • Helm 4 --force-replace (was --force) replaces resources that cannot be patched: it deletes and recreates, so expect downtime.
  • Argo CD equivalents: Replace=true or Force=true,Replace=true sync options on that resource (Chapter 7).
Chapter 4 · Part 4.3

Distribution: OCI registries and signing

Publish charts like container images, version them properly, and prove where they came from.

OCI charts

Charts in an OCI registry (ECR, GHCR, Docker Hub)

Since Helm 3.8, OCI support is GA: a chart is just another artifact next to your images.

REG=123456789012.dkr.ecr.ap-northeast-1.amazonaws.com

# ECR: one repository per chart, named <namespace>/<chart>
aws ecr create-repository --repository-name charts/web \
  --image-tag-mutability IMMUTABLE

# Helm 4: host only, no oci:// and no path
aws ecr get-login-password | \
  helm registry login $REG --username AWS --password-stdin

helm package charts/web            # -> web-0.3.0.tgz
helm push web-0.3.0.tgz oci://$REG/charts
# Pushed: $REG/charts/web:0.3.0
# Digest: sha256:...

helm show values oci://$REG/charts/web --version 0.3.0
helm install web oci://$REG/charts/web --version 0.3.0

Notes

  • The chart version becomes the OCI tag; the chart name becomes the last path segment.
  • helm push targets the namespace (oci://$REG/charts); Helm appends the chart name.
  • ECR tag immutability makes a published chart version impossible to overwrite — what you want.
  • GHCR: helm registry login ghcr.io -u <you> --password-stdin with a token, push to oci://ghcr.io/<you>/charts.
  • No helm repo add and no index.yaml for OCI: you always reference the full path.
  • Argo CD needs the registry declared as a Helm repository with OCI enabled (Chapter 6).
OCI vs index.yaml

Why OCI registries replaced classic chart repositories

A classic Helm repository is a static web server with an index.yaml listing every version.

AspectClassic repo (index.yaml)OCI registry
HostingAny web server, GitHub Pages, S3, ChartMuseumECR, GHCR, Harbor, Artifact Registry, ACR
Discoveryhelm repo add + helm search repoFull reference; no search
ScaleOne index file grows forever, downloaded on every updatePer-artifact metadata
Auth and IAMBasic auth or bucket policiesRegistry auth, IAM on ECR
ImmutabilityConvention onlyTag immutability setting
SigningPGP .prov filescosign / Notation signatures stored in the registry
Replication, scanningDIYRegistry features (ECR replication, scanning)

Many public projects still publish both. For your own charts on AWS, ECR with immutable tags is the simplest choice.

Versioning

Chart version vs appVersion

Two independent versions. Mixing them up is the most common release mistake.

version (chart)

  • SemVer, required, the identity of the package.
  • Bump on ANY change to the chart: templates, values defaults, dependencies.
  • MAJOR: breaking values or template changes (renamed keys, removed resources).
  • MINOR: new optional feature; PATCH: fixes.
  • Published versions are immutable — never re-push 0.3.0 with different content.

appVersion (application)

  • Informational: the default image tag in most charts (tag | default .Chart.AppVersion).
  • Shows up in helm list and the app.kubernetes.io/version label.
  • Bumping only the app (new image) can be done without a chart release: set image.tag in env values.
  • Quote it: appVersion: "6.7.1".

GitOps workflows

  • Image-only change: CI updates image.tag in envs/<env>/values.yaml (Chapter 9).
  • Chart change: PR to charts/web, CI checks the version bump (ct lint does), publish to OCI.
  • Pin chart versions per environment: dev can float a range, prod pins exact.
  • Same chart version promoted dev → staging → prod.
Signing

Proving a chart came from your pipeline

Two mechanisms: Helm's PGP provenance files, and Sigstore cosign signatures for OCI artifacts.

# --- PGP provenance (classic) ---
helm package charts/web --sign \
  --key 'CI Signing Key' --keyring ~/.gnupg/secring.gpg
# -> web-0.3.0.tgz + web-0.3.0.tgz.prov
helm verify web-0.3.0.tgz --keyring pubring.gpg
helm install web ./web-0.3.0.tgz --verify --keyring pubring.gpg

# --- cosign on the OCI artifact (keyless in CI) ---
DIGEST=$(helm push web-0.3.0.tgz oci://$REG/charts 2>&1 \
  | awk '/Digest:/ {print $2}')
cosign sign --yes $REG/charts/web@$DIGEST
cosign verify $REG/charts/web@$DIGEST \
  --certificate-identity-regexp 'https://github.com/<you>/.*' \
  --certificate-oidc-issuer https://token.actions.githubusercontent.com

Choosing

  • PGP .prov works with classic repos; key distribution is the hard part.
  • cosign keyless signs with the CI identity (GitHub OIDC); the signature and certificate go to the registry and the transparency log.
  • Always sign the digest, not the tag.
  • Verification must happen somewhere that can block: in the pipeline before promotion, or at admission for images (Chapter 10).
  • The same cosign flow signs your container images — one trust policy for both.
Chapter 4 · Part 4.4

Chart quality pipeline

Lint, render, validate, unit test, policy check and install-test every chart change before it reaches Git main.

Pipeline overview

Every chart PR goes through six gates

Fast checks first; the kind install test runs last and only for changed charts.

helm lint--stricthelm templateper env valueskubeconformschema per k8s versionhelm-unittesttemplate assertspolicyCheckov / Trivy / Kyvernoct installreal kind clustermerge → package → push to OCI → cosign signonly on main, version bumpedsecondsminutesEach gate catches a different class of bug: syntax → YAML → API schema → chart logic → policy → runtime behavior.
Validate rendered output

helm lint and kubeconform

Lint checks the chart; kubeconform checks the rendered manifests against the Kubernetes OpenAPI schemas.

# chart structure, template errors, values schema
for env in dev staging prod; do
  helm lint charts/web --strict -f envs/$env/values.yaml
done

# render each environment and validate against k8s 1.33 schemas
CAT=https://raw.githubusercontent.com/datreeio/CRDs-catalog/main
CRD="$CAT/{{.Group}}/{{.ResourceKind}}_{{.ResourceAPIVersion}}.json"
for env in dev staging prod; do
  helm template web charts/web -n web -f envs/$env/values.yaml \
  | kubeconform -strict -summary -kubernetes-version 1.33.0 \
      -schema-location default -schema-location "$CRD"
done
# Summary: 6 resources found parsing stdin - Valid: 6, Invalid: 0

What each catches

  • helm lint: missing Chart.yaml fields, template execution errors, schema violations.
  • kubeconform: wrong field names (contianers), wrong types, removed APIs (extensions/v1beta1 Ingress) for the target version.
  • -strict rejects unknown fields — the typo class lint cannot see.
  • CRDs (ServiceMonitor, Rollout) need extra schema locations, like the community CRDs catalog above.
  • Run against every environment's values: prod enables Ingress and HPA that dev never renders.
helm-unittest

Unit tests for templates

helm-unittest is a Helm plugin: YAML test suites that render templates and assert on the output.

# charts/web/tests/deployment_test.yaml
suite: deployment
templates:
  - templates/deployment.yaml
  - templates/configmap.yaml
tests:
  - it: omits replicas when autoscaling is on
    template: templates/deployment.yaml
    set:
      autoscaling.enabled: true
    asserts:
      - notExists:
          path: spec.replicas
  - it: uses appVersion when tag is empty
    template: templates/deployment.yaml
    asserts:
      - matchRegex:
          path: spec.template.spec.containers[0].image
          pattern: ":6\\.7\\.1$"
  - it: fails on bad autoscaling bounds
    template: templates/deployment.yaml
    set:
      autoscaling: { enabled: true, minReplicas: 5, maxReplicas: 2 }
    asserts:
      - failedTemplate:
          errorMessage: autoscaling.maxReplicas must be >= minReplicas

Usage

  • Install on Helm 4: helm plugin install https://github.com/helm-unittest/helm-unittest --verify=false (Helm 4 verifies plugin provenance; use helm-unittest v1.x).
  • Run: helm unittest charts/web.
  • Common asserts: equal, notExists, matchRegex, contains, isKind, hasDocuments, failedTemplate.
  • Test the rules from Chapter 3 (fail, required, conditional resources) so refactors keep them.
  • Snapshot tests (matchSnapshot) catch unintended output changes; review snapshot diffs in PRs.
chart-testing

ct: lint and install only the charts that changed

chart-testing compares your branch with the target branch, finds changed charts, and tests them.

# ct.yaml
target-branch: main
chart-dirs:
  - charts
validate-maintainers: false
check-version-increment: true
helm-extra-args: --timeout 300s

# in CI, inside the repo
ct lint --config ct.yaml        # lint + version bump check
kind create cluster --name ct
ct install --config ct.yaml     # install, run helm test, uninstall
kind delete cluster --name ct

# values variants: charts/web/ci/*-values.yaml
#   each file is a separate install test

What it adds

  • check-version-increment: a changed chart without a version bump fails the PR.
  • ct install installs into a throwaway namespace, waits, runs helm test, then deletes.
  • Files in charts/<chart>/ci/ are extra values sets, each installed separately — test prod-like and minimal configs.
  • Only changed charts are tested, so a monorepo with 50 charts stays fast.
  • Official GitHub Actions exist for ct and kind; locally it is the same commands.
Docs and policy

helm-docs and policy checks on rendered YAML

Generate the README from values comments; check rendered manifests against security policy.

# values.yaml comments that helm-docs reads
# -- Number of Pods when autoscaling is off
replicaCount: 2

helm-docs --chart-search-root charts
# -> charts/web/README.md with a values table
git diff --exit-code charts/web/README.md   # CI: docs up to date

# policy on rendered output
helm template web charts/web -f envs/prod/values.yaml > /tmp/prod.yaml
checkov -f /tmp/prod.yaml --framework kubernetes
trivy config charts/web
kyverno apply policies/ --resource /tmp/prod.yaml

Typical policy findings

  • No resources.requests / limits.
  • Containers running as root; missing runAsNonRoot, readOnlyRootFilesystem.
  • Images with :latest or no digest.
  • hostPath, privileged, added capabilities.
  • Running the SAME Kyverno policies in CI that the cluster enforces at admission (Chapter 10) means no surprises at sync time.
A complete CI script

scripts/ci-chart.sh: the gates in one place

Keep CI logic in a script so it runs the same on a laptop and in any CI system.

#!/usr/bin/env bash
set -euo pipefail
CHART=charts/web
helm dependency build $CHART
helm unittest $CHART
for env in dev staging prod; do
  V=envs/$env/values.yaml
  helm lint $CHART --strict -f $V
  helm template web $CHART -n web -f $V > /tmp/$env.yaml
  kubeconform -strict -summary -kubernetes-version 1.33.0 /tmp/$env.yaml
  kyverno apply policies/ --resource /tmp/$env.yaml
done
helm-docs --chart-search-root charts
git diff --exit-code -- '*.md'
ct lint --config ct.yaml
# install test only in CI (needs Docker)
if [[ "${CI:-}" == "true" ]]; then
  kind create cluster --name ct --wait 120s
  ct install --config ct.yaml
fi

Publish job (main branch only)

  • helm package → helm push to ECR → cosign sign the digest.
  • Skip if the version already exists in the registry (immutable tags would fail the push anyway).
  • Then open a PR in the config repo bumping the chart version for dev (Chapter 9).
  • Never let CI helm install into shared clusters — Argo CD deploys; CI only builds, tests and publishes.
Helmfile

Helmfile: many releases, declared in one file

Helmfile describes a set of Helm releases with per-environment values and applies them with diffs.

# helmfile.yaml
environments:
  dev:
    values: [envs/dev/env.yaml]
  prod:
    values: [envs/prod/env.yaml]
---
repositories:
  - name: ingress-nginx
    url: https://kubernetes.github.io/ingress-nginx
releases:
  - name: ingress-nginx
    namespace: ingress-nginx
    chart: ingress-nginx/ingress-nginx
    version: 4.11.3
  - name: web
    namespace: web
    chart: ./charts/web
    values:
      - envs/{{ .Environment.Name }}/values.yaml
    needs: [ingress-nginx/ingress-nginx]

Where it fits

  • helmfile -e dev diff, helmfile -e dev apply: push-style, run by a person or CI.
  • Great for local dev clusters and bootstrapping a cluster before Argo CD exists.
  • Overlaps with Argo CD ApplicationSets: once Argo CD runs, the cluster state should come from Argo CD, not Helmfile.
  • Argo CD has no built-in Helmfile support; it would need a config management plugin.
  • Uses the helm-diff plugin for diffs.
Common mistakes

Production Helm mistakes

Most of these show up only on the second install or the first upgrade.

Packaging

  • Changing the chart without bumping version.
  • Re-pushing an existing version with new content (mutable tags).
  • Floating dependency ranges in prod without Chart.lock committed.
  • Depending on an upstream repo that later deletes old versions.

Lifecycle

  • Hooks that are not idempotent — they run on every Argo CD sync.
  • Expecting crds/ CRDs to upgrade with helm upgrade.
  • Selector changes → field is immutable.
  • Using --force-replace casually: it deletes and recreates.

Pipeline

  • Testing only the default values, never the prod values file.
  • kubeconform without -strict or without CRD schemas.
  • CI that both tests AND deploys with helm upgrade to shared clusters, bypassing GitOps.
  • Signing tags instead of digests.
Self-check

Helm in production: can you answer these?

Answer, then prove it in Lab 4.

Dependencies

  • helm dependency update vs build: which belongs in CI?
  • How does a parent set a value in an aliased subchart?
  • What can a subchart read from the parent?
  • When is a library chart worth it?
  • Why do GitOps teams prefer many Applications over an umbrella chart?

Hooks and CRDs

  • Why does a pre-install hook run on every Argo CD sync?
  • Which delete policy keeps a failed Job for debugging?
  • What happens to CRDs in crds/ on helm upgrade? Under Argo CD?
  • Name three immutable fields and how to change them safely.
  • What does helm.sh/resource-policy: keep do?

Distribution and CI

  • Where does helm registry login point in Helm 4?
  • version vs appVersion: which does an image-only release change?
  • Why sign the digest and not the tag?
  • What does ct lint check that helm lint does not?
  • What does kubeconform -strict catch?

Practice: Lab 4 in Hands-on labs — library chart, dependency, migration hook, helm test, OCI push and the CI script.

Chapter 5

Kustomize and choosing a renderer

Template-free customization with bases, overlays and patches — and when to use Helm, Kustomize, or both.

Chapter 5 · Part 5.1

Kustomize

Plain YAML in, plain YAML out: bases, overlays, patches, generators, components and Helm chart inflation.

Mental model

Kustomize: transform YAML instead of templating it

A base is valid Kubernetes YAML. An overlay points at the base and declares changes. kustomize build prints the result.

apps/web/basedeployment.yamlreplicas: 1, image: podinfoservice.yamlkustomization.yamlresources: [...]envs/dev/webkustomization.yamlresources: [base] · namespace: web-devenvs/prod/webkustomization.yamlreplicas 3 · image 6.7.1 · patchkustomize builddev manifestskustomize buildprod manifestsNo variables, no logic: every overlay is a list of declared transformations applied in a fixed order.
Base and overlay

A base and a prod overlay for web

Bases never know about environments. Overlays reference the base by relative path (or a remote Git URL).

# apps/web/base/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - deployment.yaml
  - service.yaml
labels:
  - pairs:
      app.kubernetes.io/name: web
    includeSelectors: true

# apps/web/base/deployment.yaml (excerpt)
spec:
  replicas: 1
  template:
    spec:
      containers:
        - name: web
          image: ghcr.io/stefanprodan/podinfo
# envs/prod/web/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
namespace: web-prod
resources:
  - ../../../apps/web/base
  - hpa.yaml                       # prod-only resource
images:
  - name: ghcr.io/stefanprodan/podinfo
    newTag: "6.7.1"
replicas:
  - name: web
    count: 3
patches:
  - path: resources-patch.yaml
labels:
  - pairs:
      env: prod                    # not in selectors

# render
kustomize build envs/prod/web

kubectl apply -k uses a Kustomize version embedded in kubectl, which can lag the standalone kustomize binary. Argo CD ships its own kustomize; pin versions in CI to match.

Patches

Strategic merge patches and JSON 6902 patches

Both live under patches:. Older patchesStrategicMerge and patchesJson6902 fields are deprecated.

# envs/prod/web/resources-patch.yaml  (strategic merge)
apiVersion: apps/v1
kind: Deployment
metadata:
  name: web
spec:
  template:
    spec:
      containers:
        - name: web          # list items matched by name
          resources:
            requests: { cpu: 200m, memory: 256Mi }
            limits:   { memory: 512Mi }
# JSON 6902 patch, inline, selected by target
patches:
  - target:
      kind: Deployment
      name: web
    patch: |-
      - op: add
        path: /spec/template/spec/containers/0/env/-
        value: { name: PODINFO_UI_MESSAGE, value: prod }
      - op: replace
        path: /spec/revisionHistoryLimit
        value: 5

# one patch for many objects: target by label selector
  - target:
      kind: Deployment
      labelSelector: tier=frontend
    path: add-topology-spread.yaml

Strategic merge looks like the object and merges containers, ports and env by key. JSON 6902 does precise add / remove / replace by path — use it for CRDs, where strategic merge replaces whole lists. target can match group, version, kind, name, namespace, labelSelector, annotationSelector.

Generators

ConfigMap and Secret generators with content hashes

Generated names get a hash suffix of their content, and every reference is rewritten — so a change rolls the Pods automatically.

# envs/prod/web/kustomization.yaml
configMapGenerator:
  - name: web-config
    literals:
      - PODINFO_UI_COLOR=#34577c
    files:
      - config/app.yaml          # key = file name

# overlay adding to a ConfigMap generated in the base
  - name: web-config
    behavior: merge              # create | replace | merge
    literals:
      - PODINFO_UI_MESSAGE=prod

# output
# kind: ConfigMap
# metadata:
#   name: web-config-7c5fk2h9g6
# ... Deployment envFrom.configMapRef.name: web-config-7c5fk2h9g6

Why the hash matters

  • New content → new name → the Deployment's Pod template changes → rolling update. Same effect as Helm's checksum annotation.
  • Old ConfigMaps stay until pruned; with Argo CD prune: true they are removed after the sync.
  • Only references Kustomize knows about are rewritten (Pod specs, built-in kinds). Names referenced from CRDs may need a configurations entry.
  • disableNameSuffixHash: true gives stable names but loses the automatic rollout.

secretGenerator has the same syntax and produces Secret data — but never commit its input files with real secrets. Use SOPS (KSOPS) or External Secrets instead (Chapter 10).

Transformers

Images, replicas, names, namespaces and labels

Built-in transformers cover most per-environment differences without a patch file.

FieldExampleEffect
images- name: ghcr.io/stefanprodan/podinfo + newTag: "6.7.1" (or digest:)Rewrites matching image references in all workloads
replicas- name: web + count: 3Sets spec.replicas on the named workload
namespacenamespace: web-prodSets namespace on every namespaced resource
namePrefix / nameSuffixnamePrefix: prod-Renames resources and updates references
labelspairs: {env: prod}, includeSelectors: falseAdds labels; selectors only if you opt in
commonAnnotationsteam: paymentsAdds annotations everywhere
commonLabels (deprecated)—Also changes selectors → immutable-field errors; use labels

CI image bump without editing YAML by hand: cd envs/dev/web && kustomize edit set image ghcr.io/stefanprodan/podinfo:6.8.0 — a one-line diff, ideal for promotion PRs (Chapter 9).

Components

Components: optional features you can mix into any overlay

A Component is a reusable bundle of resources and patches. Overlays opt in; unlike a base, it is applied on top of what already exists.

# apps/web/components/ingress/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1alpha1
kind: Component
resources:
  - ingress.yaml
patches:
  - target: { kind: Deployment, name: web }
    patch: |-
      - op: add
        path: /spec/template/metadata/annotations/ingress-enabled
        value: "true"

# envs/prod/web/kustomization.yaml
resources:
  - ../../../apps/web/base
components:
  - ../../../apps/web/components/ingress
  - ../../../apps/web/components/hpa

When to use

  • Features that some environments need and others do not: Ingress, HPA, PDB, ServiceMonitor, debug sidecar.
  • Avoids the "diamond" problem of overlays inheriting from several intermediate overlays.
  • Equivalent to ingress.enabled: true in a Helm chart — but without templates.
  • Order: base resources first, then components in the listed order, then the overlay's own patches.
Helm inside Kustomize

Inflate a Helm chart with helmCharts, then patch it

Kustomize can render a chart and treat the output as resources — useful for third-party charts that lack a value you need.

# envs/prod/ingress-nginx/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
namespace: ingress-nginx
helmCharts:
  - name: ingress-nginx
    repo: https://kubernetes.github.io/ingress-nginx
    version: 4.11.3
    releaseName: ingress-nginx
    valuesInline:
      controller:
        replicaCount: 2
patches:
  - target: { kind: Deployment, name: ingress-nginx-controller }
    path: add-priority-class.yaml

# render: helm must be installed
kustomize build --enable-helm envs/prod/ingress-nginx

Trade-offs and Argo CD

  • Requires --enable-helm; in Argo CD set kustomize.buildOptions: --enable-helm in argocd-cm.
  • Renders with helm template: no hooks lifecycle, no release Secret — the same as Argo CD's own Helm support.
  • Patching chart output is the escape hatch when upstream values do not expose a field.
  • Helm 4 post-renderers must be plugins; in GitOps the Kustomize helmCharts route is usually simpler.
  • Prefer plain Helm with values if no patch is needed — fewer moving parts.
Chapter 5 · Part 5.2

Helm vs Kustomize vs both

Pick the renderer per use case, and know exactly how Argo CD detects and renders each one.

Decision table

Which renderer for which job

Most real platforms use both: Helm for packaging, Kustomize for last-mile environment changes.

Use caseBest fitWhy
Third-party software (ingress-nginx, cert-manager, Prometheus)Upstream Helm chart + values per envVendors ship and test charts; you only own values
Many in-house services with the same shapeHelm library chart (Chapter 4)One template set, versioned, per-service values
A few in-house services, small teamKustomize base + overlaysPlain YAML, nothing to learn beyond Kubernetes
Upstream chart missing a fieldKustomize helmCharts + patchPatch output instead of forking the chart
Distributing software to other teams or customersHelm chart in OCIVersioned package with a documented values API
Per-environment tweaks on any of the aboveValues files (Helm) or overlays (Kustomize)Keep env files small and diffable
Side by side

The same change in Helm and in Kustomize

Prod needs 3 replicas, image 6.7.1 and bigger requests.

# Helm: envs/prod/values.yaml
replicaCount: 3
image:
  tag: "6.7.1"
resources:
  requests: { cpu: 200m, memory: 256Mi }
  limits:   { memory: 512Mi }

# render
helm template web charts/web -n web-prod \
  -f envs/prod/values.yaml
# envs/prod/web/kustomization.yaml
resources: [../../../apps/web/base]
namespace: web-prod
images:
  - name: ghcr.io/stefanprodan/podinfo
    newTag: "6.7.1"
replicas:
  - { name: web, count: 3 }
patches:
  - path: resources-patch.yaml

# render
kustomize build envs/prod/web

Compare

  • Helm: the chart author decides what is changeable; users only touch values.
  • Kustomize: anything is changeable, but every change is a patch you maintain.
  • Helm values are an API contract; Kustomize patches break silently when the base changes shape.
  • Kustomize output is easier to read in review: no template logic in the way.
How Argo CD renders

Source type detection in Argo CD

The repo-server looks at the path in the Application source and picks a tool, unless the spec says which.

Path containsToolRuns
Chart.yamlHelmhelm template with spec.source.helm settings
kustomization.yaml, kustomization.yml or KustomizationKustomizekustomize build with spec.source.kustomize settings
Only .yaml / .yml / .json (and Jsonnet)DirectoryReads files as-is; directory.recurse for subfolders
Matches a config management pluginCMP sidecarYour own render command

Consequences

  • Helm apps are NOT Helm releases: no helm list, no release Secrets, no helm rollback — Argo CD owns history.
  • Values come from spec.source.helm.valueFiles / values / valuesObject, or from another repo with multi-source (Chapter 8).
  • Kustomize options in the Application (images, namePrefix, patches, components) are handy for quick overrides, but keeping them in Git files keeps the repo the single source of truth.
  • Tool versions are Argo CD's, not yours: match CI to the Helm and Kustomize versions bundled in your Argo CD release.
Repository layout

One config repo, both renderers

The lab repo uses Helm for the web chart and third-party apps, Kustomize for plain manifests and patches.

gitops-lab/
├── charts/
│   └── web/                     # our chart (Chapters 3–4)
├── apps/
│   ├── web/base/                # Kustomize base (alternative path)
│   └── web/components/{ingress,hpa}/
├── envs/
│   ├── dev/
│   │   ├── values.yaml          # Helm values for web
│   │   └── web/kustomization.yaml
│   ├── staging/ ...
│   └── prod/
│       ├── values.yaml
│       ├── web/kustomization.yaml
│       └── ingress-nginx/kustomization.yaml  # helmCharts + patch
└── clusters/
    ├── gitops/                  # hub cluster: Argo CD apps (Ch 6–8)
    └── prod/

Rules that keep it sane

  • One directory = one Argo CD Application source.
  • Environment differences live only under envs/<env>/.
  • Never point two Applications at the same rendered resources.
  • CI renders every env path on each PR and posts the diff (next slide).
Diff rendered output

Review what changes, not how the template changed

Template diffs hide impact. Render before and after, per environment, and diff the YAML.

#!/usr/bin/env bash
# scripts/render-diff.sh — run in CI on every PR
set -euo pipefail
render() {   # $1 = git ref, $2 = env
  git worktree add -q /tmp/r-$1 $1 2>/dev/null || true
  (cd /tmp/r-$1 &&
   helm template web charts/web -n web -f envs/$2/values.yaml &&
   kustomize build envs/$2/web)
}
for env in dev staging prod; do
  render origin/main $env > /tmp/$env-before.yaml
  render HEAD        $env > /tmp/$env-after.yaml
  diff -u /tmp/$env-before.yaml /tmp/$env-after.yaml \
    > /tmp/$env.diff || echo "changes in $env"
done

Why

  • A one-line change in a library chart can touch 40 services — the rendered diff shows it.
  • Reviewers see exactly what Argo CD will apply in each environment.
  • Comparing envs/staging with envs/prod output shows unintended drift between environments.
  • Tools like dyff give semantic YAML diffs; Argo CD also shows a live diff before you sync (argocd app diff, Chapter 7).
  • The Source Hydrator (Chapter 8) takes this further: commit the rendered YAML itself.
Common mistakes

Kustomize and renderer mistakes

Most are invisible until the second environment or the first base change.

Kustomize

  • commonLabels changing selectors of existing Deployments → immutable-field errors.
  • Strategic merge patches on CRDs replacing whole lists.
  • Deep overlay chains (overlay of overlay of overlay) — use components instead.
  • Committing secretGenerator input files with real secrets.

Helm + Kustomize

  • Forgetting --enable-helm in Argo CD kustomize.buildOptions.
  • Patching chart output for something a value already supports.
  • CI using different Helm/Kustomize versions than Argo CD.
  • Two Applications rendering the same resource (one via Helm, one via Kustomize).

Process

  • Reviewing template diffs instead of rendered diffs.
  • Environment-specific logic inside bases or chart templates.
  • Quick overrides in the Application spec that never make it back to Git files.
  • No rule for which renderer a new service uses.
Self-check

Kustomize and renderers: can you answer these?

Answer, then verify with kustomize build.

Kustomize

  • What is the difference between a base, an overlay and a component?
  • Strategic merge vs JSON 6902: when must you use 6902?
  • Why does configMapGenerator trigger a rollout?
  • Why is commonLabels dangerous on existing workloads?
  • How does kustomize edit set image help promotion?

Helm and Kustomize together

  • What does helmCharts need to work in Argo CD?
  • When would you patch chart output instead of setting a value?
  • Which renderer for cert-manager, and why?
  • Which for 25 similar in-house services?
  • Why can you not helm rollback an Argo CD Helm app?

Argo CD rendering

  • How does Argo CD choose Helm vs Kustomize vs directory?
  • Where do Helm values come from in an Application?
  • Why match CI tool versions to Argo CD's?
  • What does a rendered diff show that a template diff hides?
  • Where should environment differences live in the repo?

Practice: Lab 5 in Hands-on labs — Kustomize overlays for dev and prod on top of the chart output, and a rendered diff between environments.

Chapter 6

Argo CD architecture and install

What each component does, how the reconcile loop works, how to install and access Argo CD, and the CRDs you write every day.

Chapter 6 · Part 6.1

Components

Six controllers and servers, one cache, no database: every piece of state is a Kubernetes object.

Architecture

Argo CD is a set of controllers that read Git and write to Kubernetes

Desired state comes from Git (or a Helm / OCI registry). Live state comes from the Kubernetes API. Argo CD compares and converges.

namespace argocd (hub cluster)argocd-serverAPI + UI + gRPCargocd-dex-serverSSO broker (optional)notificationscontroller (optional)application-controllerStatefulSet · compare + syncrepo-servergit clone + renderapplicationset-controllergenerates Applicationsrediscache onlycommit-serverhydrator (optional)CRDs + SecretsApplication, AppProjectGit / Helm repo / OCIdesired stateKubernetes APIthis clusterRemote clusterscluster Secrets in argocdclonewatch + applyUsers: UI / argocd CLI / kubectl on CRDs → argocd-server. Webhooks from Git → argocd-server → refresh.
Components

Who does what

When something breaks, the component tells you which log to read.

ComponentKindResponsibilityRead its logs when…
argocd-serverDeploymentUI, REST/gRPC API, auth, RBAC enforcement, webhook receiverlogin fails, RBAC denies, webhook does nothing
argocd-repo-serverDeploymentClones repos, runs Helm / Kustomize / plugins, returns plain manifestsComparisonError, render errors, repo auth errors
argocd-application-controllerStatefulSetWatches clusters, diffs live vs desired, computes health, runs syncssync stuck, wrong health, slow reconcile
argocd-applicationset-controllerDeploymentTurns generators + template into Application objectsapps not created or deleted unexpectedly
argocd-redisDeploymentCache for rendered manifests and cluster state; safe to losehigh latency, cache misses
argocd-dex-serverDeploymentOptional OIDC/SAML/GitHub broker for SSOSSO redirects fail
argocd-notifications-controllerDeploymentSends Slack / email / webhook messages on triggersnotifications missing
argocd-commit-serverDeploymentWrites hydrated manifests back to Git (Source Hydrator)hydration commits missing
Components

Where state lives: there is no Argo CD database

Back up Git and a handful of Kubernetes objects and you can rebuild Argo CD anywhere.

Kubernetes objects in argocd

  • Application, ApplicationSet, AppProject (CRDs).
  • Secrets labelled argocd.argoproj.io/secret-type: repository, repo-creds or cluster.
  • argocd-cm: general settings, health/diff customizations.
  • argocd-rbac-cm: RBAC policy. argocd-cmd-params-cm: component flags.
  • argocd-secret: admin password hash, signing key, webhook secrets.

Not state

  • Redis: a cache. Deleting it costs a re-render, nothing more.
  • The repo-server working copies: re-cloned on demand.
  • The UI: a view over the API; nothing is stored in the browser.

Consequences

  • Manage Argo CD itself declaratively (it can even manage itself).
  • DR = reinstall + re-apply these objects (see Chapter 12).
  • Anyone with write access to the argocd namespace can change what Argo CD deploys: protect it like cluster-admin.
Reconcile loop

Refresh → compare → sync: three different things

People say "sync" for all three. Argo CD separates them, and so should you.

1 · Refresh

  • Ask repo-server for manifests at the target revision.
  • Triggered by the reconcile timer, a Git webhook, or a user.
  • Normal refresh uses the manifest cache for the same commit.
  • Hard refresh discards the cache and re-renders.
→

2 · Compare

  • Diff desired manifests vs live objects (after normalization and ignore rules).
  • Set sync status: Synced / OutOfSync.
  • Compute health from live objects.
  • Nothing is changed in the cluster.
→

3 · Sync

  • Apply the desired state: phases, waves, hooks, prune.
  • Manual (button / CLI) or automated (syncPolicy.automated).
  • Writes an operation and history entry to the Application.
  • Self-heal re-syncs when live drifts from Git.

Refresh interval: timeout.reconciliation in argocd-cm, default 120 s + up to 60 s jitter (timeout.reconciliation.jitter), so about 2–3 minutes on self-managed Argo CD. The EKS Capability for Argo CD polls Git every 6 minutes. Webhooks make both near-instant.

Reconcile loop

Make it fast: webhooks instead of polling

Polling every app every 3 minutes costs Git API calls and adds delay. A webhook tells Argo CD exactly which repo changed.

Set up a Git webhook

  • Point the Git provider at https://<argocd-host>/api/webhook.
  • Content type JSON, event: push (and pull request for previews).
  • Store the shared secret in argocd-secret (webhook.github.secret, webhook.gitlab.secret, …).
  • Argo CD refreshes only Applications whose repoURL matches the pushed repo.

When polling still matters

  • Private Argo CD with no inbound route from the Git provider.
  • Helm / OCI registries (no push events): rely on the timer.
  • Raise timeout.reconciliation for very large installs; keep webhooks for speed.
  • Monorepos: add the argocd.argoproj.io/manifest-generate-paths annotation so a push only refreshes apps whose paths changed (Chapter 12).
# argocd-cm (self-managed)
data:
  timeout.reconciliation: 180s

# argocd-secret
stringData:
  webhook.github.secret: <random>

# force a refresh from the CLI
argocd app get web-dev --refresh
argocd app get web-dev --hard-refresh
Rendering

How repo-server turns a source into manifests

Argo CD never runs helm install. It renders, then applies plain manifests like any other app.

Source type detection

Source containsRendered with
Chart.yaml (or chart: from a Helm repo)helm template with values / parameters
kustomization.yamlkustomize build (bundled binary)
Plain YAML / JSON (or .jsonnet)Directory source (recurse, include/exclude)
Anything elseConfig Management Plugin sidecar

Helm consequences

  • No Helm release Secret: helm list shows nothing.
  • lookup returns empty: there is no cluster during render.
  • Helm hooks are mapped to Argo CD hooks (Chapter 7).
  • .Release.Name = Application name unless helm.releaseName is set.
  • .Capabilities come from the target cluster version and API list Argo CD knows.

Render once, apply everywhere: the repo-server output is what argocd app manifests web-dev prints. Debug renders there before blaming the cluster.

Chapter 6 · Part 6.2

Install and access

Helm chart or plain manifests, HA or not, and three ways to reach the API.

Install

Two supported install paths

Pick one and keep Argo CD itself in Git. Most platform teams use the Helm chart.

Official manifests

  • manifests/install.yaml: cluster-wide, non-HA.
  • manifests/ha/install.yaml: HA (Redis HA, several replicas, needs at least 3 nodes).
  • namespace-install.yaml: namespace-scoped (no cluster-wide permissions).
  • *-install-with-hydrator.yaml: adds the commit server.
  • Apply with server-side apply: the CRDs are too large for client-side annotations.

Helm chart argo/argo-cd

  • Community-maintained chart in argoproj/argo-helm.
  • All config as values: configs.cm, configs.params, configs.rbac, configs.repositories.
  • HA: redis-ha.enabled, more replicas for server, repoServer, applicationSet; controller replicas enable sharding.
  • Upgrade = bump the chart version in Git.
# manifests
kubectl create namespace argocd
U=https://raw.githubusercontent.com
U=$U/argoproj/argo-cd/stable
kubectl apply -n argocd \
  --server-side --force-conflicts \
  -f $U/manifests/install.yaml

# or Helm (lab default)
helm repo add argo \
  https://argoproj.github.io/argo-helm
helm install argocd argo/argo-cd \
  -n argocd --create-namespace \
  -f argocd-values.yaml
kubectl -n argocd get pods
Install

A lab-sized argocd-values.yaml

Everything you would otherwise click in the UI goes here, so the install is reproducible.

configs:
  params:
    server.insecure: true   # TLS ends in front
  cm:
    timeout.reconciliation: 180s
    admin.enabled: true     # off after SSO
    kustomize.buildOptions: --enable-helm
  rbac:
    policy.default: role:readonly
  repositories:
    gitops-lab:
      url: https://github.com/<you>/gitops-lab
      type: git
server:
  replicas: 1
repoServer:
  replicas: 1
controller:
  replicas: 1
dex:
  enabled: false          # Chapter 10
notifications:
  enabled: true

Notes

  • configs.params becomes argocd-cmd-params-cm; configs.cm becomes argocd-cm; configs.rbac becomes argocd-rbac-cm.
  • server.insecure only disables TLS on the server pod; something in front must still terminate TLS.
  • Private repos: put credentials in a Secret managed outside Git (External Secrets, Chapter 10), not in values.
  • Pin the chart version (--version) and read the chart changelog on upgrade.
  • For production: HA values, resource requests, PodDisruptionBudgets, metrics ServiceMonitors.
Access

Log in: UI, CLI and core mode

The initial admin password is random and stored in a Secret. Change it, then disable admin once SSO works.

# reach the API locally
kubectl -n argocd port-forward svc/argocd-server 8080:443

# initial password (delete the Secret after changing it)
argocd admin initial-password -n argocd

argocd login localhost:8080 --username admin \
  --password '<pw>' --insecure
argocd account update-password

# core mode: no argocd-server, uses your kubeconfig + RBAC
argocd login --core
kubectl config set-context --current --namespace=argocd
argocd app list

Three access paths

  • UI / API through argocd-server: Argo CD RBAC applies; needs a route (port-forward, Ingress, LoadBalancer).
  • **argocd --core**: talks to the Kubernetes API directly; Kubernetes RBAC applies; handy for headless installs.
  • **kubectl on CRDs**: create Application objects directly; GitOps-friendly and what app-of-apps does.
  • CLI behind an L7 proxy without HTTP/2 end-to-end: add --grpc-web.
Access

Exposing argocd-server on EKS

The server speaks HTTPS and gRPC on one port. Load balancers must handle both, or the CLI uses gRPC-Web.

ALB (AWS Load Balancer Controller)

  • Terminate TLS with ACM; set server.insecure: true.
  • Two target groups: HTTP for the UI, gRPC for the CLI (conditioned on the Content-Type: application/grpc header), or use --grpc-web.
  • Internal ALB plus VPN / Verified Access for admin access.
  • Lock the security group to office / VPN ranges.

Alternatives

  • NLB with TLS passthrough to argocd-server (it keeps its own certificate).
  • Ingress-NGINX or Gateway API with GRPCRoute.
  • No exposure at all: argocd --core or port-forward for a small team.

Must do

  • Never expose the UI publicly with the admin account enabled.
  • Set url in argocd-cm to the external URL (SSO redirects and notifications links use it).
  • Webhook endpoint /api/webhook must be reachable from the Git provider (or use polling).
Declarative setup

Repositories and credentials as Secrets

Label tells Argo CD what the Secret is. Credential templates match every repo URL with a prefix.

apiVersion: v1
kind: Secret
metadata:
  name: gitops-lab
  namespace: argocd
  labels:
    argocd.argoproj.io/secret-type: repository
stringData:
  type: git
  url: https://github.com/<you>/gitops-lab
  username: git
  password: <fine-grained PAT, read-only>
---
apiVersion: v1
kind: Secret
metadata:
  name: github-org-creds
  namespace: argocd
  labels:
    argocd.argoproj.io/secret-type: repo-creds
stringData:
  type: git
  url: https://github.com/<you>   # prefix
  githubAppID: "123456"
  githubAppInstallationID: "7890123"
  githubAppPrivateKey: |
    -----BEGIN RSA PRIVATE KEY-----

Options

  • repository: one repo. repo-creds: a template for every URL under the prefix.
  • Auth: HTTPS user + token, SSH private key (sshPrivateKey), GitHub App, Google Cloud source, Azure workload identity.
  • Helm repo: type: helm + url. OCI Helm (ECR, GHCR): type: helm with enableOCI: "true".
  • Read-only credentials are enough; only the hydrator needs write access (separate write credentials).
  • Argo CD 3.0 removed repositories defined inside argocd-cm: use Secrets.
  • CLI equivalent: argocd repo add <url> --username … --password ….
Chapter 6 · Part 6.3

Core CRDs

Application and AppProject: the two objects every GitOps change goes through.

Application

Anatomy of an Application

Source (where), destination (to which cluster and namespace), project (allowed by whom), sync policy (how).

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: web-dev
  namespace: argocd
  finalizers:
    - resources-finalizer.argocd.argoproj.io
spec:
  project: team-web
  source:
    repoURL: https://github.com/<you>/gitops-lab
    targetRevision: main   # branch/tag/SHA
    path: charts/web
    helm:
      releaseName: web
      valueFiles:
        - ../../envs/dev/web/values.yaml
  destination:
    name: in-cluster       # or server: <url>
    namespace: web-dev
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
    syncOptions:
      - CreateNamespace=true

Field notes

  • metadata.namespace: argocd unless "apps in any namespace" is enabled (Chapter 10).
  • targetRevision: branch = moving target; tag or SHA = pinned. For Helm repos it is the chart version (semver ranges allowed).
  • destination: use name or server, not both.
  • source.helm: valueFiles, values, valuesObject, parameters, releaseName, ignoreMissingValueFiles.
  • source.kustomize: images, namePrefix, patches, namespace.
  • source.directory: recurse, include, exclude, jsonnet.
Application

Finalizers and cascade delete

Deleting an Application deletes its resources only if the finalizer says so.

Finalizer on the Application

FinalizerEffect on delete
noneApplication removed, workloads left running (orphaned)
resources-finalizer.argocd.argoproj.ioCascade delete, foreground
resources-finalizer.argocd.argoproj.io/backgroundCascade delete, background

Use it deliberately

  • Production apps: keep the finalizer, but protect critical objects with argocd.argoproj.io/sync-options: Delete=false (PVCs, namespaces with data).
  • Moving an app between Argo CD instances: remove the finalizer first so nothing is deleted.
  • CLI: argocd app delete web-dev --cascade=false orphans resources.
  • A stuck finalizer (cluster gone, permission lost) blocks deletion: fix access or remove the finalizer by hand (Chapter 12).
AppProject

AppProject: the guardrail around a group of Applications

Every Application belongs to exactly one project. The project decides where it may come from and where it may go.

apiVersion: argoproj.io/v1alpha1
kind: AppProject
metadata:
  name: team-web
  namespace: argocd
spec:
  description: Web team services
  sourceRepos:
    - https://github.com/<you>/gitops-lab
  destinations:
    - name: in-cluster
      namespace: 'web-*'
    - name: prod
      namespace: 'web-*'
  clusterResourceWhitelist: []   # none
  namespaceResourceBlacklist:
    - {group: '', kind: ResourceQuota}
  roles:
    - name: ci
      policies:
        - p, proj:team-web:ci, applications,
          sync, team-web/*, allow

What each field controls

  • sourceRepos: Git / Helm / OCI URLs allowed (globs allowed, ! negates).
  • destinations: cluster + namespace pairs (globs).
  • clusterResourceWhitelist / Blacklist: cluster-scoped kinds (Namespace, ClusterRole, CRD…).
  • namespaceResourceWhitelist / Blacklist: namespaced kinds.
  • roles: project-scoped RBAC and JWT tokens for CI.
  • syncWindows: when syncs are allowed (Chapter 7).
  • sourceNamespaces: namespaces whose Applications may use this project (apps in any namespace).

The built-in default project allows everything from anywhere to anywhere. Lock it down or stop using it once you have real teams.

Argo CD 3.0

Default changes to know when reading older tutorials

Many blog posts and charts were written for 2.x. These defaults changed in 3.0.

AreaArgo CD 2.xArgo CD 3.xWhat to do
Resource trackingLabel app.kubernetes.io/instanceAnnotation argocd.argoproj.io/tracking-idNothing for new installs; on upgrade, expect a one-time re-annotation
RBAC update / deleteApplied to the app and its resourcesApplies to the Application onlyGrant update/* or delete/* explicitly when users need resource actions
Diffing .statusStatus ignored for some kindsStatus ignored for all resourcesRemove old custom ignore rules for status
Pod logsReadable by anyone who can see the appNeed the logs, get RBAC permissionAdd a logs policy for developers
Repositories in argocd-cmSupported (legacy)RemovedMove to labelled Secrets

Read the upgrade notes for every minor version you cross (operator-manual/upgrading/3.x-3.y) — Chapter 12 covers the upgrade routine.

Common mistakes

Argo CD install and CRD mistakes

Each one has bitten a real platform team.

Install

  • Applying the manifests client-side: CRD annotation too large → use --server-side.
  • Exposing the UI publicly with admin enabled and the initial password unchanged.
  • Forgetting url in argocd-cm: SSO callback and notification links break.
  • Editing ConfigMaps by hand in the cluster, then losing them on the next chart upgrade.

Applications

  • destination with both name and server.
  • Helm chart relies on lookup or helm install side effects → empty or wrong output.
  • Deleting an Application without the finalizer and expecting the workload to go away (or the opposite).
  • Using the default project for everything.

Debug first

  • argocd app get web-dev → conditions show ComparisonError / SyncError.
  • argocd app manifests web-dev → what repo-server rendered.
  • kubectl -n argocd logs deploy/argocd-repo-server for render errors.
  • kubectl -n argocd logs sts/argocd-application-controller for sync errors.
Self-check

Argo CD architecture: can you answer these?

Explain the answer out loud, naming the component or field.

Components

  • Which component renders Helm, and which applies to the cluster?
  • What do you lose if Redis is deleted?
  • Where is the admin password hash stored?
  • What is the difference between a refresh, a hard refresh and a sync?
  • Why does helm list show nothing for an Argo CD-managed chart?

Install and access

  • Why must the manifests be applied server-side?
  • What does server.insecure: true actually disable?
  • When would you use argocd login --core?
  • How does a repo-creds Secret differ from a repository Secret?
  • Why do the CLI and UI need different handling on an ALB?

CRDs

  • What happens when you delete an Application without a finalizer?
  • Which AppProject field blocks a team from creating a ClusterRole?
  • Why is the default project dangerous?
  • Name three defaults that changed in Argo CD 3.0.
  • How often does Argo CD poll Git self-managed vs on the EKS capability?

Practice: Lab 6 in Hands-on labs — install Argo CD with its Helm chart, declarative repo credentials, an AppProject and a first Application.

Chapter 7

Sync deep dive

Status and health, sync policy and options, ordering with phases, waves and hooks, and taming diffs and drift.

Chapter 7 · Part 7.1

Status model

Two independent signals: is the cluster equal to Git, and is the app actually working?

Status model

Sync status and health status are different questions

An app can be Synced and Degraded (Git is wrong) or OutOfSync and Healthy (someone scaled it by hand).

Sync status — "does live equal Git?"

ValueMeaning
SyncedEvery managed resource matches the rendered manifests
OutOfSyncAt least one resource differs, is missing, or should be pruned
UnknownComparison failed (render error, repo or cluster unreachable)

Health status — "is it working?"

ValueMeaning
HealthyResource is ready (Deployment rolled out, PVC bound…)
ProgressingNot ready yet but expected to become ready
DegradedFailed (rollout deadline exceeded, crash loops…)
SuspendedPaused on purpose (CronJob suspended, Rollout paused)
MissingShould exist but does not
UnknownHealth check could not decide

Application health = the worst health among its resources. Alerts should watch both signals (Chapter 12).

Status model

Reading the four combinations

Each combination points to a different owner and a different fix.

SyncHealthTypical causeWho fixes it
SyncedHealthyEverything as intended—
SyncedDegradedGit declares something broken: bad image tag, wrong probe, missing SecretApp team: fix Git, or roll back with git revert
OutOfSyncHealthyNew commit not synced yet, manual change, or a controller rewriting fieldsSync, enable self-heal, or add an ignore rule
OutOfSyncDegraded / MissingSync failed half-way, hook failed, or a resource was deleted by handRead the operation state, then re-sync
UnknownanyRender error (ComparisonError), repo credentials, cluster unreachablePlatform team: repo-server / cluster access
Health checks

Built-in health checks and when they fall short

Argo CD knows core kinds. For CRDs it knows many popular ones, and you add the rest in Lua.

Built in (examples)

  • Deployment, StatefulSet, DaemonSet, ReplicaSet: rollout complete and replicas available.
  • Service type: LoadBalancer and Ingress: an address is assigned.
  • PersistentVolumeClaim: Bound.
  • Pod: running and ready, or completed.
  • Job: complete vs failed.
  • Many CRDs ship with checks (Argo Rollouts, cert-manager, and others).

Gaps

  • A CRD without a check is reported Healthy as soon as it exists.
  • Child Application objects (app-of-apps) have no health check by default.
  • Your operator's custom resources: add a check that reads its status.conditions.
  • Health is computed from live state only; it does not run probes or smoke tests (use PostSync hooks).
Health checks

A custom Lua health check

Defined in argocd-cm per group and kind. It receives the live object as obj and returns status and message.

# argocd-cm
data:
  resource.customizations.health.example.com_Database: |
    hs = {}
    if obj.status ~= nil and obj.status.conditions ~= nil then
      for _, c in ipairs(obj.status.conditions) do
        if c.type == "Ready" and c.status == "True" then
          hs.status = "Healthy"
          hs.message = "database ready"
          return hs
        end
        if c.type == "Failed" and c.status == "True" then
          hs.status = "Degraded"
          hs.message = c.message
          return hs
        end
      end
    end
    hs.status = "Progressing"
    hs.message = "waiting for Ready"
    return hs

Rules

  • Key format: resource.customizations.health.<group>_<kind>; core group kinds use just <kind>.
  • Always handle missing status (first seconds after create).
  • Return one of the health values on the previous slide.
  • Test locally before rolling out: argocd admin settings resource-overrides health ./obj.yaml --argocd-cm-path ./argocd-cm.yaml.
  • Keep messages short: they appear in the UI tree and in notifications.
Resource tracking

How Argo CD knows which objects it owns

Tracking decides what gets pruned. Get it wrong and Argo CD deletes, or ignores, the wrong objects.

application.resourceTrackingMethod in argocd-cm

MethodMarkerNotes
annotation (default in 3.x)argocd.argoproj.io/tracking-id: <app>:<group>/<kind>:<ns>/<name>Unambiguous; not copied to children
label (2.x default)app.kubernetes.io/instance: <app>Clashes with Helm charts that set the same label; copied to Pods
annotation+labelbothMigration aid; label is informational

Why it matters

  • Two Applications claiming the same object fight forever: one syncs it, the other wants to prune it.
  • Operators copying labels to generated objects can make Argo CD think it owns them.
  • FailOnSharedResource=true makes a sync fail instead of stealing an object owned by another app.
  • Objects created by controllers (ReplicaSets, Pods) are shown in the tree but are not managed or pruned.
Chapter 7 · Part 7.2

Sync policy

Manual or automated, with or without prune and self-heal, and the options that change how objects are applied.

Sync policy

Manual vs automated sync

Automated sync applies every new commit. Prune and self-heal are separate switches on top.

spec:
  syncPolicy:
    automated:
      prune: true        # delete objects removed from Git
      selfHeal: true     # revert manual changes in the cluster
      allowEmpty: false  # refuse to prune everything
    retry:
      limit: 5
      backoff:
        duration: 10s
        factor: 2
        maxDuration: 3m
    syncOptions:
      - CreateNamespace=true
      - ServerSideApply=true
SwitchOff (default)On
automatedOutOfSync until someone clicks SyncNew Git revision is synced automatically
pruneRemoved objects stay (marked for pruning)Removed objects are deleted
selfHealManual edits stay until the next commitLive drift is reverted after a short delay
allowEmptyRefuses a sync that would prune every resourceAllows an app to become empty
retryA failed sync waits for the next commitFailed sync retried with backoff

Automated sync syncs a given commit once. If that sync fails, it is not retried for the same commit unless retry is set — the usual reason an app sits at "Sync failed" until someone pushes again.

Sync policy

Choosing a policy per environment

Start strict in dev, careful in prod. Auto-sync in prod is fine once you trust your review process.

EnvironmentautomatedpruneselfHealWhy
dev / previewonononFast feedback; Git is always the truth
stagingonononMirror prod behavior to catch prune and drift surprises
prod (common)onononMerge = deploy; protection comes from PR review, sync windows and Rollouts
prod (cautious)off——Humans press Sync; easier rollback with argocd app rollback
shared infra (CRDs, operators)onoffonNever auto-delete CRDs: deleting one deletes every custom resource of that kind

Rollback with argocd app rollback only works when automated sync is off. With auto-sync on, roll back with git revert (Chapter 9).

Sync options

Sync options you will actually use

Set per Application in syncPolicy.syncOptions, or per resource with the argocd.argoproj.io/sync-options annotation.

OptionEffectUse when
CreateNamespace=trueCreate the destination namespace if missingEvery app that owns its namespace (labels via managedNamespaceMetadata)
ServerSideApply=trueApply with Kubernetes server-side applyLarge CRDs (annotation size limit), shared field ownership with other controllers
PruneLast=truePrune after all other waves succeededAvoid deleting the old object before the new one is healthy
ApplyOutOfSyncOnly=trueApply only resources that differApps with thousands of objects
Replace=truekubectl replace / create instead of applyObjects that cannot be patched; dangerous, deletes unspecified fields
PrunePropagationPolicy=foregroundDeletion propagation for pruned objectsWait for dependents to go first (background, orphan also exist)
RespectIgnoreDifferences=trueDo not overwrite ignored fields when syncingHPA-owned replicas, webhook-injected fields
Validate=falseSkip schema validationRare: objects with intentionally unknown fields
FailOnSharedResource=trueFail if another app already manages an objectMulti-team clusters
Sync options

Per-resource annotations

Fine-grained control without changing the whole Application.

# never prune or delete this PVC, even if removed from Git
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: data
  annotations:
    argocd.argoproj.io/sync-options: Prune=false,Delete=false
---
# CRD too large for client-side apply
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  name: widgets.example.com
  annotations:
    argocd.argoproj.io/sync-options: ServerSideApply=true
---
# custom resource whose CRD is created in the same sync
apiVersion: example.com/v1
kind: Widget
metadata:
  name: demo
  annotations:
    argocd.argoproj.io/sync-options: SkipDryRunOnMissingResource=true

What they do

  • Prune=false: Argo CD reports it as "requires pruning" but never deletes it.
  • Delete=false: kept when the Application itself is deleted.
  • ServerSideApply=true / Replace=true: apply strategy for this object only.
  • SkipDryRunOnMissingResource=true: skip the dry run when the CRD does not exist yet (it is created earlier in the same sync).
  • argocd.argoproj.io/compare-options: IgnoreExtraneous: do not count this object toward OutOfSync (for generated objects).
Sync policy

Self-heal and the "fight" with other controllers

Self-heal reverts any difference. If another controller owns a field, the two loop forever.

Classic loops

  • HPA changes spec.replicas; Git says 2; self-heal sets 2; HPA sets 6 again.
  • Admission webhook injects a sidecar or defaults; diff shows it every time.
  • Operator writes annotations or fields into objects you also declare.
  • cert-manager / cloud controllers patch caBundle or status-like fields in spec.

Fixes, in order of preference

  • Do not declare the field: remove replicas from the Deployment when an HPA owns it.
  • Server-side apply so field ownership is respected.
  • ignoreDifferences for that field + RespectIgnoreDifferences=true (section 22).
  • Last resort: compare-options: IgnoreExtraneous or exclude the kind globally.

Check it is fixed

  • argocd app diff web-dev shows nothing after a sync.
  • The app history does not grow a new sync every few minutes.
  • Controller logs no longer show repeated "Initiated automated sync".
Chapter 7 · Part 7.3

Ordering: phases, waves and hooks

Run migrations before the app, smoke tests after it, and clean-up when a sync fails.

Ordering

A sync runs phases, and each phase runs waves

Lower wave numbers first. Argo CD waits for each wave to be healthy before starting the next.

One sync operationPreSynchooks: migrationsSyncnormal resources + Sync hooksPostSynchooks: smoke testsSyncFailonly if sync failedwave -1Namespace, CRDwave 0ConfigMap, Svcwave 1DeploymentWithin a wave: kind order (Namespace, CRD, ServiceAccount, … Deployment), then name.Next wave starts only when the previous wave is healthy.PreDelete (3.3+) runswhen the Applicationis deleted, beforeresources are removed.Prune happens in the Sync phase(or last with PruneLast=true).

Order of precedence: phase → wave → kind → name. Default wave is 0; waves can be negative.

Hooks

Hook annotations

A hook is any resource (usually a Job) annotated with a phase. It is created during the sync, not kept in sync.

apiVersion: batch/v1
kind: Job
metadata:
  generateName: db-migrate-
  annotations:
    argocd.argoproj.io/hook: PreSync
    argocd.argoproj.io/hook-delete-policy: >-
      BeforeHookCreation,HookSucceeded
    argocd.argoproj.io/sync-wave: "-1"
spec:
  backoffLimit: 1
  activeDeadlineSeconds: 600
  template:
    spec:
      restartPolicy: Never
      containers:
        - name: migrate
          image: ghcr.io/<you>/web-migrate:1.4.0
          args: ["migrate", "up"]
Annotation valueMeaning
hook: PreSyncBefore any normal resource is applied
hook: SyncAlongside normal resources (after PreSync)
hook: PostSyncAfter all resources are healthy
hook: SyncFailOnly when the sync failed
hook: PreDeleteBefore resources are deleted with the app (3.3+)
hook: PostDeleteAfter the app's resources are deleted
hook: SkipNever apply this object
hook-delete-policyHookSucceeded, HookFailed, BeforeHookCreation

No delete policy means BeforeHookCreation: the old hook object is deleted when the next sync creates a new one. Use generateName or a delete policy, or the second sync collides with the first Job.

Hooks

Database migrations the GitOps way

The migration must finish before the new code starts, and must be safe to run twice.

PreSync Job

  • Runs migrate up with the new image tag.
  • Backward-compatible schema only (expand phase).
  • Idempotent: re-running is a no-op.
  • activeDeadlineSeconds so it cannot hang forever.
→

Sync (waves)

  • wave 0: ConfigMaps, Secrets (External Secrets), Services.
  • wave 1: Deployment rolls out the new code.
  • Health gate: next wave waits for rollout.
→

PostSync Job

  • Smoke test: curl the Service, check /healthz and a key endpoint.
  • Failure marks the sync failed → SyncFail hook can alert.
  • Contract phase (drop old columns) ships in a later release.

If the PreSync Job fails, the sync stops and the old version keeps running — exactly the safety you want. Fix forward with a new commit.

Hooks

Helm hooks inside Argo CD

Argo CD renders Helm charts and maps Helm hook annotations to its own phases.

Mapping

Helm annotationArgo CD equivalent
helm.sh/hook: pre-install, pre-upgradePreSync
helm.sh/hook: post-install, post-upgradePostSync
helm.sh/hook-weightsync wave
helm.sh/hook-delete-policyhook delete policy
pre-delete / post-deletePreDelete / PostDelete
pre-rollback, post-rollback, testNot supported (no Argo CD equivalent)

Gotchas

  • Argo CD has no install vs upgrade: pre-install and pre-upgrade both run on every sync.
  • Hooks designed to run once (create a secret, init a DB) must be idempotent.
  • helm test hooks are not mapped at all: run tests as a PostSync hook instead.
  • If a chart's hooks fight GitOps, override with argocd.argoproj.io/hook annotations via Kustomize post-rendering or a values flag.
Hooks

When a hook blocks the sync

A sync operation waits for hooks. A hook that never finishes keeps the app in "Syncing".

Symptoms

  • Operation state Running for many minutes.
  • Hook Job Pod in ImagePullBackOff, Pending (no node), or waiting for a dependency.
  • Hook failed → sync Failed; auto-sync will not retry the same commit without retry.
argocd app get web-dev  # op + hooks
kubectl -n web-dev get jobs,pods
kubectl -n web-dev logs \
  job/db-migrate-abcde

# stop the running op
argocd app terminate-op web-dev

# fix in Git, then
argocd app sync web-dev

Prevent

  • activeDeadlineSeconds and backoffLimit on every hook Job.
  • Pull images from a registry the cluster can reach (ECR via VPC endpoints).
  • Keep hooks fast; long data jobs belong in a separate app or pipeline.
  • Alert on syncs running longer than N minutes (Chapter 12).
Chapter 7 · Part 7.4

Diffing and drift

Teach Argo CD which differences matter, and freeze changes when the business needs quiet.

Diffing

Why an app stays OutOfSync right after a successful sync

Argo CD compares normalized desired manifests with live objects. Anything that changes live objects after apply shows up as a diff.

Usual suspects

  • Fields defaulted by the API server that the manifest writes differently (e.g. "1" vs 1, empty lists).
  • Mutating webhooks (sidecars, defaults, injected env).
  • Controllers: HPA (replicas), cert-manager (caBundle), cloud controllers.
  • Aggregated ClusterRoles whose rules are filled in by the API server.
  • Helm randAlphaNum / timestamps: a new value on every render.
# see exactly what differs
argocd app diff web-dev

# compare against a local render
helm template web charts/web \
  -f envs/dev/web/values.yaml > /tmp/want.yaml
argocd app diff web-dev --local charts/web

# one-off: server-side diff for this app
metadata:
  annotations:
    argocd.argoproj.io/compare-options: ServerSideDiff=true
Diffing

ignoreDifferences: tell Argo CD what not to compare

Per Application in spec.ignoreDifferences, or system-wide in argocd-cm.

spec:
  ignoreDifferences:
    - group: apps
      kind: Deployment
      jsonPointers:
        - /spec/replicas            # HPA owns it
    - group: admissionregistration.k8s.io
      kind: MutatingWebhookConfiguration
      jqPathExpressions:
        - .webhooks[]?.clientConfig.caBundle
    - group: '*'
      kind: '*'
      managedFieldsManagers:
        - kube-controller-manager
  syncPolicy:
    syncOptions:
      - RespectIgnoreDifferences=true

Three selectors

  • jsonPointers: exact paths (RFC 6901), simple and fast.
  • jqPathExpressions: patterns over arrays and optional fields.
  • managedFieldsManagers: ignore every field owned by a named field manager (server-side apply metadata).
  • Narrow with name and namespace when only one object needs it.
  • System-wide: resource.customizations.ignoreDifferences.apps_Deployment in argocd-cm.
  • Without RespectIgnoreDifferences=true a sync still writes the ignored field back from Git.
Drift

Drift: detect, decide, correct

Drift is any live change that Git does not know about. GitOps turns it from a mystery into an alert.

Detect

  • App turns OutOfSync while Git did not change.
  • UI diff shows who changed what (field manager in managed fields).
  • Alert on OutOfSync longer than N minutes.
→

Decide

  • Hot fix during an incident? Commit it to Git now.
  • Controller-owned field? Add an ignore rule.
  • Unauthorized change? Revert and tighten RBAC.
→

Correct

  • Self-heal reverts automatically.
  • Or a manual sync from the UI / CLI.
  • Then remove the human kubectl write access that allowed it.

Break-glass: during an incident, disable auto-sync on the app (or set a deny sync window), fix live, then commit the fix and re-enable. Never leave an app with auto-sync off and an undocumented live patch.

Sync windows

Sync windows: freeze or allow changes by schedule

Defined on the AppProject. A deny window blocks syncs; an allow window permits them only inside the window.

apiVersion: argoproj.io/v1alpha1
kind: AppProject
metadata:
  name: team-web
  namespace: argocd
spec:
  syncWindows:
    - kind: deny            # Friday evening freeze
      schedule: '0 17 * * 5'
      duration: 64h
      timeZone: Asia/Ho_Chi_Minh
      applications: ['*-prod']
      manualSync: true      # humans may still sync
    - kind: allow           # prod only in office hours
      schedule: '0 9 * * 1-5'
      duration: 8h
      timeZone: Asia/Ho_Chi_Minh
      clusters: ['prod']

Rules

  • schedule is cron; duration is how long the window lasts.
  • Match by applications, namespaces, clusters (globs).
  • If any allow window exists for an app, syncs outside allow windows are blocked.
  • Deny beats allow when both match.
  • manualSync: true lets a human override during the window (incidents).
  • Windows block syncs, not refreshes: drift is still detected.
  • CLI: argocd proj windows list team-web.
Common mistakes

Sync mistakes

Most "Argo CD is flaky" reports are one of these.

Policy

  • Auto-sync without retry → one transient failure leaves the app failed until the next commit.
  • prune: true on an app that owns CRDs → all custom resources deleted when the CRD leaves Git.
  • Self-heal on fields owned by HPA or webhooks → endless sync loops.
  • Expecting argocd app rollback to work with auto-sync on.

Ordering

  • Hooks without delete policy and with a fixed name → second sync fails on "already exists".
  • Non-idempotent pre-install Helm hooks that now run on every sync.
  • Waves on resources without health checks → next wave starts immediately.
  • Long data jobs as PreSync hooks → syncs blocked for hours.

Diffing

  • Random values in templates (randAlphaNum, now) → permanent OutOfSync.
  • Broad ignoreDifferences (kind: '*' + /spec) that hides real changes.
  • Forgetting RespectIgnoreDifferences=true, so syncs still overwrite the ignored field.
  • Two Applications managing the same object.
Self-check

Sync deep dive: can you answer these?

Explain the answer and name the exact field or annotation.

Status and health

  • What does Synced + Degraded tell you, and who fixes it?
  • How is Application health computed from its resources?
  • What is the health of a CRD object with no health check?
  • Why did Argo CD 3.x move to annotation-based tracking?
  • What does FailOnSharedResource=true protect against?

Policy and options

  • What are prune, selfHeal and allowEmpty each for?
  • Why might an auto-synced app stay failed after a transient error?
  • When do you need ServerSideApply=true?
  • How do you keep a PVC when its manifest leaves Git?
  • Why is prune dangerous for an app that owns CRDs?

Ordering and diffing

  • In which order do phase, wave, kind and name apply?
  • What happens to the sync if a PreSync Job fails?
  • Which Helm hooks run on every Argo CD sync, and why?
  • How do you stop an HPA vs self-heal loop (two ways)?
  • Does a deny sync window stop drift detection?

Practice: Lab 7 in Hands-on labs — waves and hooks ordering, ignoreDifferences for HPA replicas, a failing PreSync hook and an OutOfSync loop.

Chapter 8

Scaling apps: app-of-apps and ApplicationSet

From one Application to hundreds: bootstrap with app-of-apps, generate with ApplicationSets, combine sources, and commit rendered manifests with the Source Hydrator.

Chapter 8 · Part 8.1

App-of-apps

One root Application that deploys other Applications: the simplest way to bootstrap a cluster from Git.

App-of-apps

The pattern: Applications are just manifests

An Application can deploy any Kubernetes object — including other Application objects.

root (Application)path: apps/Git: apps/ (one Application manifest per file)argocd.yamlArgo CD manages itselfplatform-addons.yamlingress, ESO, metricsweb-dev.yamlpath: charts/webweb-prod.yamldestination: prodCluster objects created by the childrenargocd Deploymentscontrollers, CRDsweb in web-devweb in prod cluster

You apply one object by hand (the root). Everything else — including Argo CD's own upgrades — arrives through Git.

App-of-apps

Root Application and a child

The root points at a directory of Application manifests. Children point at real workloads.

# bootstrap/root.yaml  (applied once with kubectl)
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: root
  namespace: argocd
  finalizers:
    - resources-finalizer.argocd.argoproj.io
spec:
  project: platform
  source:
    repoURL: https://github.com/<you>/gitops-lab
    targetRevision: main
    path: apps
  destination:
    name: in-cluster
    namespace: argocd       # children live here
  syncPolicy:
    automated: {prune: true, selfHeal: true}
# apps/web-dev.yaml
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: web-dev
  namespace: argocd
  annotations:
    argocd.argoproj.io/sync-wave: "10"
  finalizers:
    - resources-finalizer.argocd.argoproj.io
spec:
  project: team-web
  source:
    repoURL: https://github.com/<you>/gitops-lab
    targetRevision: main
    path: charts/web
    helm:
      valueFiles: [../../envs/dev/web/values.yaml]
  destination: {name: in-cluster, namespace: web-dev}
  syncPolicy:
    automated: {prune: true, selfHeal: true}
    syncOptions: [CreateNamespace=true]
App-of-apps

Bootstrap order with waves — and the missing health check

Waves on child Applications only order the bootstrap if Argo CD can tell when a child is healthy.

Typical bootstrap waves

  • wave -10: Argo CD itself, AppProjects.
  • wave -5: CRDs and operators (cert-manager, External Secrets, Rollouts).
  • wave 0: platform config (ClusterIssuers, SecretStores, ingress).
  • wave 10: business applications.
  • The catch: by default an Application resource has no health check, so the root treats every child as healthy at once and all waves run together.
  • Add a Lua health check for argoproj.io/Application in argocd-cm that returns the child's own status.health.status.
  • Then wave 0 waits until the wave -5 operators are really healthy.
# argocd-cm
resource.customizations.health.argoproj.io_Application: |
  hs = {}
  hs.status = "Progressing"
  hs.message = ""
  if obj.status ~= nil and obj.status.health ~= nil then
    hs.status = obj.status.health.status
    if obj.status.health.message ~= nil then
      hs.message = obj.status.health.message
    end
  end
  return hs
App-of-apps

Strengths and limits

Great for a small, hand-curated set of apps. Repetitive at scale — which is where ApplicationSets come in.

Use it for

  • Cluster bootstrap: one root per cluster.
  • A curated list of platform add-ons with different settings each.
  • Teams that want every Application visible as a file in review.
  • Argo CD managing its own install (argocd.yaml child).

Limits

  • Copy-paste: 50 services × 3 environments = 150 nearly identical files.
  • Adding a cluster means writing N new files.
  • Anyone who can write to apps/ can create an Application in any project the root allows: restrict the root's project.
  • Deleting the root (with finalizer) deletes every child and its workloads.

Safety switches

  • Root without auto-prune while you are learning.
  • Delete=false sync option on critical child Applications.
  • Keep the root in a platform project that only allows Application objects in argocd.
  • Children can themselves be roots (layers), but keep it to two levels.
Chapter 8 · Part 8.2

ApplicationSet generators

A template plus generators: parameters in, Applications out.

ApplicationSet

One template, many Applications

The ApplicationSet controller runs generators, renders the template once per parameter set, and creates, updates or deletes Applications.

Generatorslist · clusters · git · scm · PRParameters{env: dev, cluster: in-cluster} …ApplicationSet controllerrender template per parameter setApplication web-devApplication web-stagingApplication web-prodtemplateApplications are owned by the ApplicationSet: edit the ApplicationSet, not the generated Applications (manual edits are overwritten).
ApplicationSet

List generator with Go templates

The simplest generator: a fixed list of parameter sets. Use Go templates (goTemplate: true) for every new ApplicationSet.

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata: {name: web, namespace: argocd}
spec:
  goTemplate: true
  goTemplateOptions: ["missingkey=error"]
  generators:
    - list:
        elements:
          - {env: dev, cluster: in-cluster}
          - {env: staging, cluster: in-cluster}
          - {env: prod, cluster: prod}
  template:
    metadata:
      name: 'web-{{.env}}'
      labels: {env: '{{.env}}'}
    spec:
      project: team-web
      source:
        repoURL: https://github.com/<you>/gitops-lab
        targetRevision: main
        path: charts/web
        helm:
          valueFiles:
            - '../../envs/{{.env}}/web/values.yaml'
      destination: {name: '{{.cluster}}', namespace: 'web-{{.env}}'}

Notes

  • missingkey=error turns a typo in a parameter name into an error instead of an empty string.
  • Go templates support functions (Sprig): {{.env | upper}}, {{default "main" .branch}}.
  • Older fasttemplate syntax ({{env}}) is still supported but limited; do not mix the two.
  • Template fields must render to the Application schema: quote templated strings in YAML.
  • Generated Application names must be unique across the whole ApplicationSet.
Generators

Cluster and Git generators

Let the data drive the apps: add a cluster Secret or a directory and a new Application appears.

# clusters: one app per registered cluster
generators:
  - clusters:
      selector:
        matchLabels:
          env: prod
# params: .name .server .metadata.labels.<k>

# git directories: one app per folder
generators:
  - git:
      repoURL: https://github.com/<you>/gitops-lab
      revision: main
      directories:
        - path: envs/*/*
# params: .path.path .path.basename

# git files: one app per config file
generators:
  - git:
      repoURL: https://github.com/<you>/gitops-lab
      revision: main
      files:
        - path: clusters/*/config.json
# params: every key in the file
GeneratorOne Application per…Typical use
listelementSmall, static sets
clusterscluster Secret (filtered by labels)Add-ons on every cluster
git directoriesmatching directoryMonorepo: one folder = one app
git filesJSON / YAML filePer-cluster or per-env config files
matrixcombination of two generatorsEvery service × every cluster
mergebase set, overridden by keysDefaults + per-cluster exceptions
scmProviderrepository in an orgEvery repo with a deploy folder
pullRequestopen pull requestPreview environments

Also: clusterDecisionResource (external placement such as OCM) and plugin (your own HTTP service returns parameters).

Generators

Matrix and merge

Combine generators: matrix multiplies, merge overrides.

# every service folder × every prod cluster
generators:
  - matrix:
      generators:
        - git:
            repoURL: https://github.com/<you>/gitops-lab
            revision: main
            directories: [{path: envs/prod/*}]
        - clusters:
            selector:
              matchLabels: {env: prod}
template:
  metadata:
    name: '{{.path.basename}}-{{.name}}'
  spec:
    destination:
      server: '{{.server}}'
      namespace: '{{.path.basename}}'
# defaults for all clusters, override per cluster
generators:
  - merge:
      mergeKeys: [name]
      generators:
        - clusters:
            values:
              ingressReplicas: "2"
        - list:
            elements:
              - name: prod
                values.ingressReplicas: "4"
# result: .values.ingressReplicas
#   = 4 on prod, 2 elsewhere

Matrix takes exactly two child generators (nest for more) — put both dimensions in metadata.name to avoid collisions. Merge needs mergeKeys in every generator. Preview output with argocd appset generate appset.yaml before applying.

Generators

Pull Request generator: preview environments

One short-lived environment per open pull request, deleted when the PR closes.

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata: {name: web-previews, namespace: argocd}
spec:
  goTemplate: true
  generators:
    - pullRequest:
        github:
          owner: <you>
          repo: web
          tokenRef: {secretName: github-token, key: token}
  template:
    metadata: {name: 'web-pr-{{.number}}'}
    spec:
      project: previews
      source:
        repoURL: https://github.com/<you>/gitops-lab
        targetRevision: main
        path: charts/web
        helm:
          parameters:
            - name: image.tag
              value: 'pr-{{.number}}-{{.head_short_sha}}'
      destination: {name: in-cluster, namespace: 'web-pr-{{.number}}'}
      syncPolicy:
        automated: {prune: true}
        syncOptions: [CreateNamespace=true]

How it works

  • Parameters include .number, .branch, .head_sha, .head_short_sha and labels.
  • Providers: GitHub, GitLab, Bitbucket, Gitea, Azure DevOps.
  • Polls the provider every requeueAfterSeconds (generator field), or near-instant with the ApplicationSet webhook.
  • CI must build and push the pr-<n>-<sha> image before the app can become healthy.
  • Closed PR → Application deleted → namespace and workloads removed (keep the finalizer).
  • Put previews in their own AppProject with a namespace pattern and a quota.
Generators

templatePatch: per-app differences without a second ApplicationSet

The template must be valid YAML; templatePatch is a Go-templated string patched on top, so it can contain conditionals.

spec:
  goTemplate: true
  generators:
    - list:
        elements:
          - {env: dev, autoSync: true}
          - {env: prod, autoSync: false}
  template:
    metadata:
      name: 'web-{{.env}}'
    spec:
      project: team-web
      source: {repoURL: https://github.com/<you>/gitops-lab,
               targetRevision: main, path: charts/web}
      destination: {name: in-cluster, namespace: 'web-{{.env}}'}
  templatePatch: |
    {{- if .autoSync }}
    spec:
      syncPolicy:
        automated:
          prune: true
          selfHeal: true
    {{- end }}

When to use what

  • Values that are plain strings → template parameters.
  • Whole blocks present for some apps only (auto-sync, extra sources, ignore rules) → templatePatch.
  • Very different apps → separate ApplicationSets; do not build a template language inside YAML.
  • templatePatch requires goTemplate: true.
  • Keep the patch small: it is harder to review than plain YAML.
Chapter 8 · Part 8.3

ApplicationSet safety

A generator bug can delete many applications at once. Decide what the controller may do.

Safety

The blast radius of a generator

One wrong commit to a Git generator path, or one cluster label change, can delete every generated Application — and with the finalizer, their workloads.

Things that delete Applications

  • A directory renamed or moved out of the generator path.
  • A cluster Secret label edited or removed.
  • An SCM / PR provider token expiring (provider returns nothing).
  • The ApplicationSet itself deleted.
  • A template change that renames Applications (old names are deleted, new ones created).

Controls

  • spec.syncPolicy.preserveResourcesOnDeletion: true → when an Application is deleted, its workloads stay.
  • spec.syncPolicy.applicationsSync: what the controller may do to Applications.
  • Controller-wide policy flag to cap every ApplicationSet (set by the platform team).
  • Review generator changes like code; preview with argocd appset generate.
Safety

applicationsSync policies

Limit the controller to the operations you trust.

PolicyCreateUpdateDelete
sync (default)yesyesyes
create-updateyesyesno
create-onlyyesnono
create-deleteyesnoyes
apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: web
  namespace: argocd
spec:
  syncPolicy:
    applicationsSync: create-update
    preserveResourcesOnDeletion: true
  goTemplate: true
  generators: [ ... ]
  template: { ... }

Choosing

  • Production fleets: create-update + preserveResourcesOnDeletion: true; delete Applications deliberately.
  • Preview environments: sync (they must disappear when PRs close).
  • create-only makes later template changes silent no-ops — usually surprising.
  • A controller-level policy overrides per-ApplicationSet settings when configured to do so.
Progressive syncs

Progressive syncs with RollingSync (beta)

Roll a change across generated Applications in steps: dev first, then staging, then prod clusters a few at a time.

spec:
  goTemplate: true
  generators:
    - list:
        elements:
          - {env: dev}
          - {env: staging}
          - {env: prod}
  strategy:
    type: RollingSync
    rollingSync:
      steps:
        - matchExpressions:
            - {key: env, operator: In, values: [dev]}
        - matchExpressions:
            - {key: env, operator: In, values: [staging]}
        - matchExpressions:
            - {key: env, operator: In, values: [prod]}
          maxUpdate: 25%      # prod apps a quarter at a time
  template:
    metadata:
      name: 'web-{{.env}}'
      labels: {env: '{{.env}}'}
    spec: { ... }

How it behaves

  • Steps select Applications by labels on the generated Applications.
  • Each step waits until its Applications are Healthy before the next starts.
  • Auto-sync is disabled on generated Applications; the controller triggers syncs itself.
  • RollingSync respects sync windows, like a user pressing Sync.
  • Optional reverse deletion removes Applications in reverse step order.
  • Status: alpha since v2.6, beta since v3.3; must be enabled on the ApplicationSet controller (flag, env var, or argocd-cmd-params-cm). Edge cases may remain.
Safety

Who may create ApplicationSets

An ApplicationSet can create Applications in any project its template names. Treat it as a privileged object.

Risks

  • A template that takes project from a Git file lets anyone who can edit that file pick a more powerful project.
  • SCM / PR generators run with a token that can read many repos.
  • Generated Applications inherit whatever destination the parameters produce.

Controls

  • Only platform admins create ApplicationSets in argocd; teams change parameters (files, labels) through PR review.
  • Hard-code project in the template; never template it from user-controlled data.
  • Restrict the target projects' destinations and sourceRepos.
  • ApplicationSets in other namespaces need explicit enablement and allowed SCM providers.

Visibility

  • Argo CD 3.5 adds ApplicationSet management in the UI.
  • kubectl -n argocd get applicationsets and .status.conditions show generator errors.
  • argocd appset get web lists generated Applications and their status.
Chapter 8 · Part 8.4

Multi-source apps and Source Hydrator

Charts from one place and values from another; rendered manifests committed back to Git.

Multi-source

Multi-source Applications: chart from a registry, values from Git

Keep third-party charts in their registry and only your values in your repo.

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: podinfo-prod
  namespace: argocd
spec:
  project: team-web
  sources:
    - repoURL: https://stefanprodan.github.io/podinfo
      chart: podinfo
      targetRevision: 6.*          # semver range
      helm:
        releaseName: podinfo
        valueFiles:
          - $values/envs/prod/podinfo/values.yaml
    - repoURL: https://github.com/<you>/gitops-lab
      targetRevision: main
      ref: values                  # exposes this repo as $values
  destination: {name: prod, namespace: podinfo}
  syncPolicy:
    syncOptions: [CreateNamespace=true]

Rules

  • sources (plural) replaces source; do not set both.
  • A source with ref: provides files to other sources through $<ref>; it can also have a path and render its own manifests.
  • Resources from all sources are merged into one Application; duplicates are an error you must avoid.
  • A change in either repo triggers a refresh.
  • Some features have limits with multiple sources (for example history rollback); check the docs for your version.
  • For OCI charts: repoURL: <registry>/<path> with a type: helm, enableOCI repository credential.
Rendered manifests

Why commit rendered manifests?

With Helm or Kustomize in the loop, the YAML that reaches the cluster is not in Git. Reviews see values, not results.

Problems with render-at-sync

  • A chart upgrade or a helper change silently changes many apps.
  • Reviewers cannot see the effective diff of a values change.
  • Render happens on every refresh: CPU on repo-server.
  • Tools that need plain YAML (policy scans) must re-render.

Rendered manifests pattern

  • A pipeline renders Helm / Kustomize per environment.
  • The output is committed to an environment branch or folder.
  • Argo CD syncs plain YAML from there.
  • PR diffs show exactly what will change in the cluster.

Cost

  • Two places to look (dry source vs hydrated output).
  • You must build and run the rendering pipeline.
  • Branch clutter if done by hand.
  • → Source Hydrator automates it inside Argo CD.
Source Hydrator

Source Hydrator: Argo CD renders and commits for you

The application controller asks repo-server to render the dry source, then the commit server pushes the result to a hydrated branch, which is what gets synced.

Dry sourcemain: charts + valuesrepo-serverrendercommit-servergit pushHydrated branchenvironments/devapplication-controllersync plain YAMLCluster
spec:
  sourceHydrator:
    drySource:
      repoURL: https://github.com/<you>/gitops-lab
      targetRevision: main
      path: apps/web/overlays/dev
    syncSource:
      targetBranch: environments/dev
      path: web
    # optional: stage for review / promotion
    # hydrateTo:
    #   targetBranch: environments/dev-next
Source Hydrator

Enabling and operating the hydrator

Off by default. Beta in the 3.5 docs. Read the docs for your exact version before relying on it.

Enable

  • Install the commit server (the *-install-with-hydrator.yaml manifests, or the Helm chart option).
  • Set hydrator.enabled: "true" in argocd-cmd-params-cm.
  • Restart the application controller and API server.
  • Add write credentials for the hydrated repo, separate from the read credentials.
  • Credential templates (repo-creds) are not supported for the hydrator's push credentials.

Behavior to know

  • Hydration triggers on a new commit in the dry source.
  • Since 3.3 the hydrator writes its files (such as manifest.yaml) into the configured path and does not delete other files already there.
  • Audit hydrated paths after upgrading from 3.2 or earlier, which cleaned the path first.
  • 3.2 added custom commit messages and automatic .gitattributes in hydrated repos.
  • Each hydrated commit links back to its dry commit, so reviews trace both.

Promotion with hydrateTo

  • Hydrate to a staging branch (environments/prod-next).
  • A PR (human or Kargo, Chapter 9) merges it into environments/prod.
  • Argo CD syncs environments/prod.
  • The PR diff is the exact YAML change in prod.
Choosing

Which pattern for which job

They combine: an ApplicationSet can generate multi-source or hydrated Applications.

NeedPatternWhy
Bootstrap a cluster with a curated list of add-onsApp-of-appsSimple, each child explicit in review
Same service in many environments or clustersApplicationSet (list / clusters / matrix)One template, parameters per target
Monorepo, one folder per appApplicationSet git directoriesNew folder = new app, no extra manifest
Preview environment per PRApplicationSet pullRequestCreated and deleted with the PR
Third-party chart + your valuesMulti-source ApplicationChart stays in its registry
Reviewers must see the exact rendered YAMLSource Hydrator (or a CI render step)Rendered manifests in Git
Roll a change across many clusters in stepsApplicationSet + RollingSync (beta)Health-gated waves of Applications
Common mistakes

App-of-apps and ApplicationSet mistakes

These scale with your fleet: one mistake, a hundred broken apps.

App-of-apps

  • No health check for Application → bootstrap waves do nothing.
  • Root in the default project → anyone with Git write can create any Application.
  • Deleting the root with its finalizer to "restart" → everything is deleted.
  • Editing child Applications in the UI → reverted by the root's self-heal.

ApplicationSet

  • Matrix without both dimensions in metadata.name → name collisions.
  • Renaming a directory → Applications deleted and recreated (workloads too, unless preserved).
  • Expired SCM token → generator returns nothing → previews deleted.
  • Templating project from user data.
  • Editing generated Applications by hand.

Sources

  • Setting both source and sources.
  • Forgetting ref: on the values source, so $values/... does not resolve.
  • Assuming the hydrator still cleans the output path (it does not since 3.3).
  • Hydrator push credentials given as a credential template.
Self-check

Scaling apps: can you answer these?

Explain the answer and name the field, generator or setting.

App-of-apps

  • Why do sync waves on child Applications not work out of the box?
  • What happens when you delete the root Application with its finalizer?
  • Which project should the root Application use, and why?
  • When is app-of-apps a better fit than an ApplicationSet?

ApplicationSet

  • Which generator gives one app per cluster with label env=prod?
  • How do you avoid name collisions in a matrix generator?
  • What does missingkey=error protect against?
  • Which settings stop a generator bug from deleting production workloads?
  • What is the maturity of RollingSync, and how does it decide a step is done?

Sources and hydrator

  • How does a values file in Git reach a chart from a Helm repo?
  • What problem does the rendered manifests pattern solve?
  • Which component pushes hydrated manifests, and with which credentials?
  • What changed in the hydrator in 3.3, and what must you audit?
  • How does hydrateTo support promotion?

Practice: Lab 8 in Hands-on labs — app-of-apps bootstrap, then an ApplicationSet (Git directory × list matrix) and PR preview environments.

Chapter 9

Multi-cluster and environment promotion

One Git repo, many clusters and environments: topologies, layouts, and moving a version from dev to prod safely.

Chapter 9 · Part 9.1

Cluster topologies

Where Argo CD runs, which clusters it manages, and how it authenticates to them.

Topologies

Three ways to place Argo CD

The choice decides blast radius, network paths and how many UIs your team has to watch.

Hub-and-spokeArgo CDhub clusterdevstagingprodone control plane, pushes to spokesOne Argo CD per clusterArgo CDdevArgo CDstagingArgo CDproddevstagingprodeach cluster pulls its own stateHybridArgo CDnon-prod hubArgo CDprod (local)devstgprodshared hub for non-prod, prod isolated

All three read the same Git repository. Only the reconciler placement changes.

Topologies

Hub-and-spoke vs one Argo CD per cluster

Neither is "right". Pick on blast radius, network reachability and team size.

AspectHub-and-spokeOne per cluster
Single pane of glassYes: one UI, one RBAC, one SSONo: N UIs (or a SaaS / aggregator on top)
Blast radiusHub outage or compromise affects every clusterLimited to one cluster
CredentialsHub holds credentials for every spokeEach cluster uses its in-cluster service account
NetworkHub must reach every spoke API serverOnly outbound to Git and the registry
ScaleController load grows with clusters; needs sharding (ch 12)Load spread naturally
OperationsUpgrade onceUpgrade N installs (manage Argo CD with Argo CD)
Typical fitPlatform team, tens of clusters, private networking in placeStrict isolation, air-gapped, edge, regulated prod

Common compromise: a hub per environment tier (non-prod hub, prod hub) so a non-prod mistake cannot touch prod.

Registering clusters

A cluster is just a labelled Secret

argocd cluster add is a shortcut; in GitOps you declare the Secret (through External Secrets, ch 10).

apiVersion: v1
kind: Secret
metadata:
  name: cluster-prod
  namespace: argocd
  labels:
    argocd.argoproj.io/secret-type: cluster
    env: prod                  # used by ApplicationSet cluster generator
    region: ap-northeast-1
type: Opaque
stringData:
  name: prod
  server: https://ABC123.gr7.ap-northeast-1.eks.amazonaws.com
  config: |
    {
      "awsAuthConfig": {
        "clusterName": "prod",
        "roleARN": "arn:aws:iam::222222222222:role/argocd-deployer"
      },
      "tlsClientConfig": { "caData": "<base64 CA>" }
    }

What each field does

  • secret-type: cluster makes Argo CD load it as a destination.
  • name is what Applications use in destination.name; server is the API URL used in destination.server.
  • config holds the auth: bearer token, client certs, exec provider or awsAuthConfig.
  • Labels on the Secret become selectable by the ApplicationSet cluster generator (ch 8).
  • The hub itself is the built-in in-cluster destination: https://kubernetes.default.svc.
  • argocd cluster add <context> creates a argocd-manager ServiceAccount with a cluster-admin role in the target: convenient, but over-privileged.
Registering clusters on EKS

Use IAM roles, not long-lived tokens

The hub authenticates with its pod identity, assumes a role per spoke, and the spoke maps that role with an access entry.

Hub account 111111111111application-controllerSA in argocd nsargocd-serverSA in argocd nsIAM role argocd-hubPod Identity / IRSASpoke account 222222222222IAM roleargocd-deployerEKS access entryrole → Kubernetes RBACEKS prodAPI serversts:AssumeRole

Hub side

  • Associate the controller and server service accounts with role argocd-hub (EKS Pod Identity or IRSA).
  • argocd-hub may only sts:AssumeRole on each spoke deployer role.

Spoke side

  • Role argocd-deployer trusts argocd-hub.
  • An EKS access entry maps it to a Kubernetes group or access policy; scope it to namespaces where possible.
  • Private API endpoints need network reachability from the hub (TGW, peering).
Registering clusters

Cluster-level knobs worth knowing

A few settings on the cluster Secret and in Argo CD change how a cluster behaves as a destination.

Scope a cluster

  • namespaces: comma-separated list; Argo CD then manages only those namespaces (no cluster-wide watch).
  • clusterResources: "true" re-allows cluster-scoped resources when namespaces is set.
  • project: a project-scoped cluster that only that AppProject can use.

Destinations by name

  • destination.name: prod instead of a URL keeps manifests stable when an endpoint changes.
  • Do not set both name and server in one Application.
  • Rename a cluster by editing the Secret name, not by re-adding.

Local practice (kind)

  • kind writes 127.0.0.1:<port> into kubeconfig; a hub inside Docker cannot reach that.
  • Point the cluster Secret at the container name on the kind Docker network, e.g. https://prod-control-plane:6443.
  • Lab 9 uses this to register the prod kind cluster with the gitops hub.
Chapter 9 · Part 9.2

Repository layout for many environments

A tree that makes every environment explicit and every difference reviewable.

Layout

The gitops-lab repository

Shared definitions once; per-environment differences small and visible.

gitops-lab/
├── charts/
│   └── web/                    # in-house chart (ch 2-4)
├── apps/
│   └── web/
│       ├── base/values.yaml    # shared by every env
│       └── appset.yaml         # one ApplicationSet per app
├── envs/
│   ├── dev/web/values.yaml     # image.tag, replicas, host
│   ├── staging/web/values.yaml
│   └── prod/web/values.yaml
├── clusters/
│   ├── gitops/                 # hub: Argo CD itself, projects
│   │   ├── argocd/values.yaml
│   │   └── projects/
│   └── prod/
│       └── addons/             # ingress, external-secrets ...
└── .github/workflows/
    └── render-diff.yaml        # diff rendered output on PRs

Rules that keep it sane

  • One directory per environment, never one branch per environment.
  • base/values.yaml holds everything that must be identical everywhere.
  • An env file contains only what truly differs: image tag, replicas, resources, hostnames, secret references.
  • clusters/ holds cluster-level concerns (add-ons, Argo CD config) separate from apps.
  • CODEOWNERS: envs/prod/ requires platform or service-owner approval.
  • Pin chart versions and prefer image digests in prod.
Layout

Wiring values files to environments

Each environment's Application points at the same chart with a different, ordered list of values files.

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: web-prod
  namespace: argocd
spec:
  project: web
  source:
    repoURL: https://github.com/<you>/gitops-lab
    targetRevision: main
    path: charts/web
    helm:
      valueFiles:              # later files win
        - ../../apps/web/base/values.yaml
        - ../../envs/prod/web/values.yaml
  destination:
    name: prod
    namespace: web
  syncPolicy:
    syncOptions: [CreateNamespace=true]

Notes

  • valueFiles paths are relative to source.path; files must stay inside the repo.
  • Chart in a registry, values in Git? Use multi-source with a $values ref (ch 8).
  • An ApplicationSet generates web-dev, web-staging, web-prod from this template (ch 8).
  • Avoid helm.parameters overrides set from the UI or CLI: they live only in the Application, not in Git.
  • Kustomize alternative: envs/<env>/web/kustomization.yaml with resources: [../../../apps/web/base] plus patches.
Layout

What may differ between environments

If prod differs from staging in something not on the left column, staging is not testing prod.

Allowed to differ

Kind of valueExample
Version under testimage.tag, chart version
Capacityreplicas, resources, HPA min/max
Endpointshostnames, certificate ARNs, URLs of dependencies
Secret referencesSecrets Manager path per environment
Feature flagsonly when deliberately staged

Should be identical

  • Chart templates and library chart version.
  • Probes, security context, PodDisruptionBudget, NetworkPolicy.
  • Sidecars, logging and metrics configuration.
  • Rollout strategy and analysis templates.
Layout

Render and diff every environment in CI

Reviewers approve the rendered change, not just a values edit they have to imagine.

# .github/workflows/render-diff.yaml (core step)
git worktree add /tmp/main origin/main     # base branch copy
for env in dev staging prod; do
  helm template web charts/web \
    -f apps/web/base/values.yaml \
    -f envs/$env/web/values.yaml > /tmp/new-$env.yaml
  helm template web /tmp/main/charts/web \
    -f /tmp/main/apps/web/base/values.yaml \
    -f /tmp/main/envs/$env/web/values.yaml > /tmp/old-$env.yaml
  diff -u /tmp/old-$env.yaml /tmp/new-$env.yaml \
    > /tmp/diff-$env.txt || true
done
# post /tmp/diff-*.txt as a PR comment

# compare environments with each other
diff <(helm template web charts/web -f apps/web/base/values.yaml \
         -f envs/staging/web/values.yaml) \
     <(helm template web charts/web -f apps/web/base/values.yaml \
         -f envs/prod/web/values.yaml)

Why it pays off

  • A one-line change in base/ that touches all three environments becomes obvious.
  • Template refactors show "no rendered change" — the safest kind of PR.
  • staging vs prod diff should show only the allowed differences from the previous slide.
  • Add kubeconform and policy checks (ch 4) on the same rendered files.
  • Alternative: argocd app diff <app> --local <path> against the live cluster (needs cluster access from CI).
Layout

Monorepo or one config repo per team?

Start with one config repo; split when ownership or scale forces it.

QuestionOne config repoRepo per team or app
Cross-cutting change (add a label everywhere)One PRN PRs
Access controlCODEOWNERS per pathRepository permissions
Argo CD performanceLarge repo: use webhooks and manifest-generate-paths (ch 12)Small repos, more repo credentials
Blast radius of a bad mergeWhole platformOne team
AppProject sourceReposSame repo for everyone: restrict by path via ApplicationSet templatesNatural: one repo per project
Chapter 9 · Part 9.3

Promotion

Moving one tested version from dev to staging to prod — by pull request, by bot, or by a promotion engine.

Promotion

Promotion is a Git change, not a deploy command

CI produces an artifact once; promotion copies its version from one environment file to the next.

CI (app repo)

  • Build, test, scan.
  • Push podinfo:6.7.1 by digest.
  • Sign the image (ch 10).
→

Dev

  • Bot PR: envs/dev/web/values.yaml → 6.7.1.
  • Auto-merge on green checks.
  • Argo CD syncs dev.
→

Staging

  • PR copies the same tag.
  • Smoke tests as PostSync hook.
  • Rollouts analysis (ch 11).
→

Prod

  • PR with CODEOWNERS approval.
  • Sync window open (ch 7).
  • Canary + analysis.

Never rebuild per environment. The digest tested in staging must be the digest running in prod.

PR-based promotion

A CI job that opens the promotion PR

The simplest promotion engine: yq + git + gh. Everything stays reviewable.

# runs in the app repo after the image is pushed
TAG=6.7.1
git clone https://github.com/<you>/gitops-lab && cd gitops-lab
git switch -c promote-web-dev-$TAG
TAG=$TAG yq -i '.image.tag = strenv(TAG)' envs/dev/web/values.yaml
git commit -am "web: promote $TAG to dev"
git push -u origin HEAD
gh pr create --title "web: $TAG -> dev" \
  --body "Image digest: $DIGEST" --label promotion

# later: promote what dev runs to staging
TAG=$(yq '.image.tag' envs/dev/web/values.yaml)
TAG=$TAG yq -i '.image.tag = strenv(TAG)' envs/staging/web/values.yaml

Design points

  • Use a GitHub App or fine-grained token limited to the config repo.
  • Dev: auto-merge when CI is green; prod: human approval.
  • Copy the value from the previous environment instead of trusting a CI variable.
  • One promotion per PR keeps git revert precise.
  • Branch protection on main blocks direct pushes, including from bots.
Promotion tools

Argo CD Image Updater and Kargo

Two different answers: "follow the newest tag" vs "move a verified bundle through stages".

Argo CD Image Updater

  • Watches container registries for new tags matching a constraint (SemVer range, latest build, digest, regex).
  • Updates the Application's image: either as an Argo CD parameter override or by committing to Git (Git write-back).
  • Prefer Git write-back: a parameter override is state that lives outside Git.
  • Good for dev environments that should always run the newest build.
  • Weak at ordered multi-stage promotion with gates.

Kargo

  • A promotion engine from the Argo CD creators (Akuity), built around Argo CD.
  • Warehouse: subscribes to image, chart or Git sources and produces Freight.
  • Freight: an immutable set of artifact versions that move together.
  • Stage: a target (dev, staging, prod) that receives Freight through Promotions.
  • Promotion steps update Git (and can trigger Argo CD sync); verification gates the next stage.

Choosing

  • Few services, PR culture: CI-driven PRs.
  • Dev should auto-track builds: Image Updater with Git write-back.
  • Many services and stages, need visibility of "what version is where": Kargo.
  • Check feature support before using these with the EKS Capability for Argo CD.
Rollback

Rolling back in a GitOps world

The fastest safe rollback is a revert commit; Argo CD's own rollback is a short-term override.

Preferred: git revert

  • git revert <promotion-commit> restores the previous tag in Git.
  • Argo CD syncs it like any other change; history stays honest.
  • Works the same for Application and ApplicationSet-managed apps.
  • Pair with Argo Rollouts abort for instant traffic shift (ch 11).

argocd app rollback

  • Redeploys a previous sync from argocd app history <app>.
  • Only works when automated sync is off; turn it off first.
  • Git still says the new version: the app is OutOfSync until Git is fixed.
  • ApplicationSet-managed apps: changing syncPolicy.automated on the Application has no effect (the controller regenerates it), so roll back in Git.
# preferred
git revert 3f2a91c
git push

# emergency override (self-managed app)
argocd app set web-prod --sync-policy none
argocd app history web-prod
argocd app rollback web-prod 12
# then fix Git, and re-enable:
argocd app set web-prod \
  --sync-policy automated
Promotion gates

What must be true before the next environment

Automate the evidence so approval is a quick check, not an investigation.

GateWhere it runsFails when
Rendered diff reviewedPR checks (28)Unexpected change outside the promoted values
Sync healthyArgo CD health, argocd app wait web-staging --healthDegraded or Progressing timeout
Smoke testsPostSync hook Job (ch 7)Hook Job fails → sync marked failed
Metrics analysisArgo Rollouts AnalysisRun (ch 11)Error rate or latency above threshold
Change windowAppProject sync windows (ch 7)Outside allowed time for prod
Human approvalCODEOWNERS on envs/prod/No approving review
Common mistakes

Multi-cluster and promotion mistakes

Each one has caused a real outage somewhere.

Topology and access

  • One hub managing prod and dev with the same credentials and AppProject.
  • argocd cluster add in prod: leaves a cluster-admin token in the hub.
  • Hub with a public EKS endpoint list "0.0.0.0/0" just so Argo CD can connect.
  • Forgetting that hub downtime stops all reconciliation (apps keep running, nothing changes).

Layout

  • Branch per environment: merges drift, cherry-picks pile up.
  • Copy-pasted full values files per environment; differences hide in 300 lines.
  • Unpinned chart versions: dev and prod render different templates.
  • Values overridden from the UI (helm.parameters) and never committed.

Promotion and rollback

  • Rebuilding the image per environment.
  • Promoting latest or a mutable tag.
  • argocd app rollback with auto-sync on (refused) or on an ApplicationSet app (overwritten).
  • Rollback via cluster edit instead of Git: self-heal reverts it.
Self-check

Multi-cluster and promotion: can you answer these?

Explain the answer and name the exact field or command.

Topologies

  • Give two reasons to run one Argo CD per cluster.
  • What makes a Secret a cluster destination?
  • How does a hub authenticate to a cross-account EKS cluster without static tokens?
  • What does namespaces on a cluster Secret change?

Layout

  • Why are branches per environment discouraged?
  • Which values should never differ between staging and prod?
  • In which order are valueFiles applied?
  • What does a rendered diff in CI catch that a values diff does not?

Promotion

  • Why promote by digest rather than tag?
  • Warehouse, Freight, Stage: what is each in Kargo?
  • Why is Git write-back preferred for Image Updater?
  • When does argocd app rollback refuse to run, and why is git revert still needed afterwards?

Practice: Lab 9 in Hands-on labs — register a second kind cluster, promote a tag by PR, roll back with git revert.

Chapter 10

Security, tenancy and secrets

Who may do what in Argo CD, how teams share it safely, where secrets live, and how to trust what gets deployed.

Chapter 10 · Part 10.1

Access control

SSO for humans, scoped tokens for machines, and RBAC policies that say exactly what each can do.

Threat model

Argo CD is a deployment robot with cluster-admin

Anyone who can steer it — through the UI, the API or Git — can change every cluster it manages.

HumansUI, CLICI pipelinesAPI tokensGit committersconfig repoArgo CDAPI server · controllerdev clusterstaging clusterprod clustercluster-admin by defaultSSO + RBAC (30)project tokensSource Integrity (33)AppProjects and impersonation shrink what each path can reach (31)

Protect all three entry points. Hardening RBAC is pointless if anyone can push to main of the config repo.

SSO

Single sign-on: direct OIDC or through Dex

Use your identity provider for humans; disable the local admin once SSO works.

# argocd-cm (Helm values: configs.cm)
url: https://argocd.example.com
admin.enabled: "false"          # after SSO is verified
oidc.config: |
  name: Okta
  issuer: https://example.okta.com
  clientID: 0oa1example
  clientSecret: $oidc.clientSecret   # key in argocd-secret
  requestedScopes: ["openid", "profile", "email", "groups"]

# alternative: Dex as a broker (GitHub, LDAP, SAML)
dex.config: |
  connectors:
    - type: github
      id: github
      name: GitHub
      config:
        clientID: $dex.github.clientID
        clientSecret: $dex.github.clientSecret
        orgs:
          - name: my-org

Choosing

  • Direct OIDC: the IdP speaks OIDC and returns group claims (Okta, Entra ID, Keycloak, Google with groups). One component fewer.
  • Dex: the IdP is GitHub, LDAP or SAML, or does not return groups. Dex is bundled with Argo CD.
  • url must be the external URL; the IdP redirect URI is <url>/auth/callback.
  • $key values are read from argocd-secret; $<secret-name>:<key> reads another Secret labelled app.kubernetes.io/part-of: argocd.
  • Map IdP groups to Argo CD roles in RBAC (next slide); scopes: '[groups]' tells RBAC which claim to read.
RBAC

argocd-rbac-cm: policies, roles and groups

Casbin-style CSV lines: p grants a permission to a subject, g puts a user or group into a role.

# argocd-rbac-cm (Helm values: configs.rbac)
policy.default: role:readonly
scopes: '[groups, email]'
policy.csv: |
  # p, <subject>, <resource>, <action>, <project>/<object>, allow|deny
  p, role:web-dev, applications, get,    web/*, allow
  p, role:web-dev, applications, sync,   web/*, allow
  p, role:web-dev, applications, action/*, web/*, allow
  p, role:web-dev, logs,         get,    web/*, allow
  p, role:web-dev, applications, sync,   web/web-prod, deny

  p, role:platform, *, *, */*, allow

  g, my-org:web-team, role:web-dev
  g, my-org:platform, role:platform

Vocabulary

PartValues
Built-in rolesrole:readonly, role:admin
Resourcesapplications, applicationsets, projects, clusters, repositories, accounts, certificates, logs, exec, extensions
Actionsget, create, update, delete, sync, override, action/<group>/<kind>/<name>
Object<project>/<application>; wildcards allowed
PrecedenceAn explicit deny wins over allow

Test before rollout: argocd admin settings rbac can role:web-dev sync applications web/web-prod --policy-file policy.csv.

RBAC

Fine-grained RBAC since Argo CD 3.0

3.0 changed defaults that older tutorials rely on. Re-check any policy written for 2.x.

Update and delete

  • Before 3.0: update/delete on an Application also allowed editing or deleting its managed resources.
  • Since 3.0: they apply to the Application object only.
  • Grant resource-level rights explicitly with sub-resource actions, e.g. update/* or delete/*/Pod/*/*.
  • Typical need: let developers delete a stuck Pod without letting them delete the Deployment.

Logs and exec

  • logs, get is enforced: users without it cannot read pod logs in the UI.
  • exec, create controls the web terminal (also requires exec.enabled: "true" in argocd-cm).
  • Keep exec off in prod; use kubectl with audited access instead.

Review checklist

  • No * actions for non-platform roles.
  • override (sync with local manifests or parameter overrides) only for break-glass roles.
  • projects, update lets a user widen their own sandbox: platform only.
  • clusters and repositories writes: platform only.
Machine access

Project roles and tokens for CI

CI should get a token scoped to one AppProject, not an admin account.

# AppProject (excerpt)
spec:
  roles:
    - name: ci
      description: CI may sync web apps
      policies:
        - p, proj:web:ci, applications, sync, web/*, allow
        - p, proj:web:ci, applications, get,  web/*, allow
      groups: []

# issue a token (expires in 30 days)
argocd proj role create-token web ci --expires-in 720h

# in CI
argocd app sync web-dev --auth-token "$ARGOCD_TOKEN" \
  --server argocd.example.com --grpc-web
argocd app wait web-dev --health --timeout 300 \
  --auth-token "$ARGOCD_TOKEN" --server argocd.example.com

Good practice

  • In pure GitOps, CI rarely needs Argo CD at all: it commits to Git and lets auto-sync work.
  • Use a token only for "sync now and wait" or for reading status in a pipeline.
  • Tokens are JWTs; they are listed on the project role and can be revoked with argocd proj role delete-token.
  • Local accounts (accounts.ci: apiKey in argocd-cm) are the alternative; prefer project roles because they cannot escape the project.
  • Store the token in the CI secret store, never in the repo.
Audit

Who changed what

Git history covers desired state; Argo CD and Kubernetes cover the actions taken on it.

QuestionEvidence
Who changed the desired state?Git commit, PR, approvers (require signed commits, 33)
Who clicked Sync, Rollback or ran an action?Kubernetes Events on the Application (kubectl get events -n argocd) and API server logs
What was deployed at time T?argocd app history <app>: revision and time of each sync
Who changed RBAC, projects or clusters?Git (if declarative) or Kubernetes audit logs for argocd namespace objects
Did anyone bypass Argo CD with kubectl?Drift shows as OutOfSync; EKS control plane audit logs show the user

Ship Argo CD logs and Kubernetes Events to your log platform: Events expire from the cluster after about an hour by default.

Chapter 10 · Part 10.2

Multi-tenancy

AppProjects as fences, Applications in team namespaces, and syncing as the tenant instead of as cluster-admin.

AppProject

The AppProject is the tenancy boundary

It answers: from which repos, to which clusters and namespaces, which kinds of objects.

apiVersion: argoproj.io/v1alpha1
kind: AppProject
metadata:
  name: web
  namespace: argocd
spec:
  description: Web team
  sourceRepos:
    - https://github.com/<you>/gitops-lab
  sourceNamespaces: [team-web]         # apps in any namespace
  destinations:
    - name: '*'
      namespace: 'web-*'
  clusterResourceWhitelist: []          # no cluster-scoped objects
  namespaceResourceBlacklist:
    - group: ''
      kind: ResourceQuota
    - group: ''
      kind: LimitRange
    - group: networking.k8s.io
      kind: NetworkPolicy
  orphanedResources:
    warn: true

Field by field

  • sourceRepos: Git or Helm repos this team may deploy from.
  • destinations: cluster (name or server) and namespace patterns.
  • clusterResourceWhitelist: []: no CRDs, ClusterRoles, Namespaces.
  • namespaceResourceBlacklist: keep quota and network policy under platform control.
  • orphanedResources: warn about objects in the namespace not tracked in Git.
  • The default project allows everything: restrict it or never use it for tenants.
Apps in any namespace

Let teams own Application objects in their namespace

By default Applications must live in argocd, so only admins can create them. This opens a controlled door.

Enable

  • argocd-cmd-params-cm: application.namespaces: team-* (server and controller read it).
  • AppProject spec.sourceNamespaces: [team-web] allows Applications from that namespace to use the project.
  • Apps are then addressed as team-web/web-dev in CLI and RBAC.
  • Same pattern exists for ApplicationSets (applicationsetcontroller.namespaces) with extra SCM restrictions.

Why

  • Teams create and delete their own Applications with plain Kubernetes RBAC on their namespace.
  • An Application in team-web can only use projects that list team-web in sourceNamespaces.
  • Fits app-of-apps per team: the team root app writes children into its own namespace.

Watch out

  • Never add argocd or a shared namespace to a tenant's sourceNamespaces.
  • Kubernetes RBAC on Application objects becomes part of your security model.
  • Notifications and some UI views need the namespace list too.
Impersonation

Sync as the tenant, not as cluster-admin

Without impersonation the controller applies every manifest with its own, very broad, service account.

# argocd-cm
application.sync.impersonation.enabled: "true"

# AppProject web
spec:
  destinationServiceAccounts:
    - server: https://kubernetes.default.svc
      namespace: web-dev
      defaultServiceAccount: web-deployer
    - server: '*'
      namespace: web-prod
      defaultServiceAccount: web-deployer

# in each destination namespace: a ServiceAccount
# web-deployer + Role/RoleBinding with only what the
# chart needs (Deployments, Services, ConfigMaps, ...)

How it works

  • The controller impersonates the configured ServiceAccount when it applies resources for that project.
  • Kubernetes RBAC then enforces the tenant's limits, even if the project or chart is wrong.
  • Since Argo CD 3.5, impersonation also applies to all API server operations (for example resource actions and deletes from the UI).
  • The controller still needs permission to impersonate those ServiceAccounts.
  • Start with one project, verify syncs, then roll out: missing Role rules show up as sync errors.
Tenancy layers

Defense in depth for a shared Argo CD

Each layer catches what the previous one missed.

LayerControlStops
GitCODEOWNERS, branch protection, signed commitsUnreviewed or forged changes
Argo CD RBACargocd-rbac-cm roles per teamSyncing or deleting another team's apps
AppProjectsourceRepos, destinations, resource allow/deny listsDeploying to other namespaces or cluster-scoped objects
ImpersonationPer-namespace deployer ServiceAccountAnything the chart tries beyond its Role
AdmissionKyverno / Gatekeeper, Pod Security AdmissionPrivileged pods, unsigned images, missing labels
RuntimeNetworkPolicy, ResourceQuota, LimitRangeNoisy or lateral-moving workloads
Tenancy

What a tenant must never control

If any of these is writable by a tenant, the tenant is effectively a cluster admin.

Never tenant-writable

  • Cluster-scoped resources: CRDs, ClusterRoles, ClusterRoleBindings, Namespaces, webhooks.
  • The argocd namespace and its ConfigMaps and Secrets.
  • AppProject objects (including their own).
  • Cluster Secrets and repository credentials.
  • Other teams' namespaces, ResourceQuota and NetworkPolicy in their own.

Platform-owned instead

  • A platform repo path clusters/<name>/ creates namespaces, quotas, policies and AppProjects.
  • Teams request a namespace by PR to that path.
  • CRDs for team operators are installed by the platform.

Verify

  • argocd admin settings rbac can for each role.
  • Try the "break it" in Lab 10: a tenant app targeting another namespace must fail with a project error.
  • kubectl auth can-i --as=system:serviceaccount:web-dev:web-deployer create clusterrole → no.
Chapter 10 · Part 10.3

Secrets in GitOps

Git holds references or ciphertext, never plaintext. Pick where decryption happens.

Options

Four ways to get secrets into a GitOps cluster

The key difference: where the plaintext exists and who can decrypt.

ApproachIn GitDecrypted byFit
External Secrets Operator (ESO)An ExternalSecret referenceESO controller reading AWS Secrets Manager / Parameter StoreDefault on AWS; rotation in one place
Secrets Store CSI DriverA SecretProviderClassCSI driver at pod mount timeSecrets as files; no Kubernetes Secret needed
Sealed SecretsA SealedSecret (ciphertext)In-cluster controller with its private keyNo cloud secret store; small teams
SOPS (with KMS)Encrypted YAML valuesA rendering plugin (KSOPS, helm-secrets) in the repo-serverEncrypted files reviewed in PRs

SOPS in Argo CD needs a config management plugin, which the EKS Capability for Argo CD does not support. ESO works with both.

External Secrets Operator

AWS Secrets Manager → Kubernetes Secret

The secret value never enters Git or Argo CD; ESO creates and refreshes the Secret in the cluster.

apiVersion: external-secrets.io/v1
kind: ClusterSecretStore
metadata:
  name: aws-sm
spec:
  provider:
    aws:
      service: SecretsManager
      region: ap-northeast-1
      auth:
        jwt:
          serviceAccountRef:
            name: external-secrets
            namespace: external-secrets
apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
  name: web-db
  namespace: web-prod
spec:
  refreshInterval: 1h
  secretStoreRef:
    kind: ClusterSecretStore
    name: aws-sm
  target:
    name: web-db            # Kubernetes Secret to create
  data:
    - secretKey: password
      remoteRef:
        key: prod/web/db    # Secrets Manager name
        property: password  # JSON key inside it

ESO's ServiceAccount gets an IAM role (IRSA shown; Pod Identity also works) scoped to prod/web/*. Rotate in Secrets Manager; ESO refreshes the Secret at refreshInterval (env vars need a pod restart). The generated Secret is owned by the ExternalSecret, so Argo CD shows it as a child resource, not as Git-tracked state.

Sealed Secrets

Encrypt with a public key, commit the ciphertext

Only the controller in the cluster holds the private key.

# create a normal Secret locally, never apply it
kubectl create secret generic web-db -n web-dev \
  --from-literal=password='s3cr3t' \
  --dry-run=client -o yaml > /tmp/secret.yaml

# encrypt with the controller's public certificate
kubeseal --format yaml < /tmp/secret.yaml \
  > envs/dev/web/sealed-web-db.yaml
rm /tmp/secret.yaml
git add envs/dev/web/sealed-web-db.yaml

# back up the sealing keys (disaster recovery!)
kubectl get secret -n kube-system \
  -l sealedsecrets.bitnami.com/sealed-secrets-key \
  -o yaml > sealed-secrets-keys.backup.yaml

Know the trade-offs

  • Default scope is strict: the ciphertext is bound to the Secret name and namespace; renaming requires re-sealing.
  • Each cluster has its own key pair: seal once per cluster.
  • Lose the private key (cluster rebuilt without backup) and every SealedSecret is unreadable.
  • Rotation of the value means re-sealing and committing; old ciphertext stays in Git history.
  • No integration with a central secret store or audit of reads.
SOPS

Encrypted values files with KMS

SOPS encrypts only the values, so diffs still show which keys changed.

# .sops.yaml at the repo root
creation_rules:
  - path_regex: envs/prod/.*\.enc\.yaml$
    kms: arn:aws:kms:ap-northeast-1:222222222222:key/abcd-...
    encrypted_regex: '^(data|stringData|password|token)$'

# edit in place (decrypts in $EDITOR, re-encrypts on save)
sops envs/prod/web/secrets.enc.yaml

# what gets committed (excerpt)
password: ENC[AES256_GCM,data:Tr3...,iv:...,tag:...,type:str]
sops:
  kms:
    - arn: arn:aws:kms:ap-northeast-1:222222222222:key/abcd-...

Argo CD integration

  • Argo CD cannot decrypt SOPS by itself.
  • Options: KSOPS (Kustomize generator) or helm-secrets, both run in the repo-server as a plugin.
  • The repo-server then needs KMS decrypt rights: it becomes a high-value target.
  • Decrypted output passes through the repo-server cache and Redis.
  • Good fit: teams that want encrypted secrets reviewed in PRs and accept the plugin.
Anti-pattern

Why "render secrets into manifests" is discouraged

Plugins that fetch a secret and inject it during manifest generation put plaintext where it does not belong.

What happens

  • The repo-server fetches the secret value while rendering.
  • The generated manifest, with the plaintext value, is cached in Redis.
  • Diffs, the API and the UI can expose it to anyone with get on the app.
  • Rotating the secret does not trigger a sync: Git did not change.

Prefer

  • An operator that creates Secrets in-cluster (ESO, Sealed Secrets).
  • Or mount secrets at runtime (CSI driver).
  • Argo CD then only sees references or ciphertext.

If you must use a plugin

  • Restrict who can view app manifests and diffs.
  • Run the plugin with the least KMS / secret-store rights.
  • Mark Secrets with argocd.argoproj.io/compare-options: IgnoreExtraneous only where needed, not as a habit.
Chapter 10 · Part 10.4

Supply chain

Trust the commit, trust the artifact, and lock down the Argo CD installation itself.

Trust the commit

Source Integrity: only sync signed commits

Argo CD 3.5 replaced the older GnuPG signature verification with Source Integrity.

What it gives you

  • Argo CD verifies the signature on the commit it is about to sync.
  • Unsigned commits, or commits signed by keys you did not trust, are refused.
  • Configured as a policy on which sources must be signed by which keys; can also be set from the CLI in 3.5.
  • Protects against a compromised Git account or a forged push.

Prerequisites

  • Developers and bots sign commits (GPG or SSH signing).
  • Branch protection requires signed commits on main.
  • Merge strategy must keep a trusted signature (platform-signed merge or squash commits).
  • Know your key rotation process before enforcing.

Before 3.5

  • Older docs describe signatureKeys in the AppProject and GnuPG keys imported into Argo CD.
  • Read the 3.4 → 3.5 upgrade notes before moving an existing setup.
  • Check the 3.5 docs for the exact configuration fields.
Trust the artifact

Verify signed images at admission

Argo CD does not check images. A Kubernetes admission policy does, for every deploy path.

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: verify-web-images
spec:
  validationFailureAction: Enforce
  webhookTimeoutSeconds: 30
  rules:
    - name: require-cosign-signature
      match:
        any:
          - resources:
              kinds: [Pod]
              namespaces: ["web-*"]
      verifyImages:
        - imageReferences:
            - "ghcr.io/<you>/*"
          attestors:
            - entries:
                - keyless:
                    subject: "https://github.com/<you>/*"
                    issuer: "https://token.actions.githubusercontent.com"

Notes

  • CI signs images keyless with cosign (OIDC identity of the workflow), covered in the CI/CD deck.
  • The policy accepts only images signed by your repository's workflows.
  • Alternatives: Sigstore policy-controller, or Gatekeeper with an external data provider.
  • Charts in OCI registries can be signed with cosign the same way.
  • Roll out in audit mode first; unsigned third-party images will fail.
Harden the installation

Lock down Argo CD itself

Treat the argocd namespace like the control plane it is.

Inside the cluster

  • mTLS between Argo CD components (Argo CD 3.5) so a pod in the cluster cannot talk to the repo-server or Redis directly.
  • Keep the NetworkPolicies that ship with the upstream manifests: only the controller and API server reach the repo-server.
  • Restrict who can exec into or read Secrets in argocd.
  • Redis holds rendered manifests: protect it (auth, network policy).

Controller privileges

  • The default install gives the controller cluster-wide rights on the local cluster.
  • Spokes: map the deployer role to scoped permissions, not cluster-admin, where workable.
  • Namespaced install mode (namespace-install.yaml) when Argo CD should manage only some namespaces.
  • Impersonation (31) moves enforcement to Kubernetes RBAC.

Edge

  • UI and API behind SSO, on a private ALB or VPN where possible.
  • Disable the admin account; rotate argocd-secret keys after setup.
  • Webhook endpoint: configure a shared secret per Git provider.
  • Keep Argo CD within supported minor versions for security fixes.
Common mistakes

Security and tenancy mistakes

Each one turns "can edit an app" into "owns the cluster".

Access

  • Local admin left enabled with the initial password.
  • policy.default: role:admin (or readonly for anonymous users) on an internet-facing UI.
  • RBAC written for 2.x relying on update to edit resources; silently narrower in 3.x.
  • CI using an admin token instead of a project role token.

Tenancy

  • Tenants deploying into the default project.
  • Empty clusterResourceWhitelist forgotten → teams create ClusterRoleBindings.
  • sourceNamespaces including argocd.
  • Impersonation enabled but the deployer Role copied from cluster-admin.

Secrets and supply chain

  • Base64 Secrets committed "temporarily" (they stay in Git history).
  • Sealed Secrets keys never backed up.
  • Secret-injecting plugin with diffs visible to everyone.
  • Image signing in CI but no admission policy that checks it.
Self-check

Security, tenancy and secrets: can you answer these?

Explain the answer and name the exact field or setting.

Access control

  • Direct OIDC vs Dex: when do you need Dex?
  • What changed for update and delete permissions in Argo CD 3.0?
  • How do you grant CI the right to sync only web/* apps?
  • Which command tests an RBAC policy without logging in as the user?

Tenancy

  • Which AppProject fields stop a team from creating a ClusterRole?
  • What does "apps in any namespace" require on the server and on the project?
  • With impersonation on, whose permissions does a sync use?
  • List four things a tenant must never be able to change.

Secrets and supply chain

  • Where does the plaintext exist with ESO, Sealed Secrets and SOPS?
  • Why can SOPS not be used with the EKS Capability for Argo CD?
  • What did Source Integrity replace in 3.5?
  • Why must image signatures be checked at admission rather than in Argo CD?

Practice: Lab 10 in Hands-on labs — SSO with Dex, two teams with separate AppProjects, a sealed secret, and a tenant trying to escape its namespace.

Chapter 11

Progressive delivery with Argo Rollouts

Ship a new version to a slice of traffic, measure it, and promote or abort automatically — while Git stays the source of truth.

Chapter 11 · Part 11.1

Rollouts

The Rollout resource, canary and blue-green strategies, and how traffic is actually shifted.

Why

A Deployment rollout is all-or-nothing

RollingUpdate checks readiness, not correctness. A bad version that passes probes reaches 100% of users.

Deployment RollingUpdate

  • Replaces pods in batches (maxSurge, maxUnavailable).
  • Gate = readiness probe only.
  • No pause for metrics, no traffic percentages.
  • Rollback = another rollout of the old version.

Argo Rollouts adds

  • Canary with explicit traffic weights and pauses.
  • Blue-green with a preview Service and controlled switch.
  • Automated analysis from Prometheus, CloudWatch, Datadog, web hooks, Jobs.
  • Instant abort: traffic goes back to the stable ReplicaSet.

Moving parts

  • argo-rollouts controller (one per cluster).
  • CRDs: Rollout, AnalysisTemplate, ClusterAnalysisTemplate, AnalysisRun, Experiment.
  • kubectl argo rollouts plugin and an optional dashboard.
  • Installed like any add-on: by Argo CD from the platform repo.
Rollout resource

A Rollout is a Deployment with a different strategy block

Same selector and pod template; the controller manages ReplicaSets itself.

apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata:
  name: web
spec:
  replicas: 5
  revisionHistoryLimit: 3
  selector:
    matchLabels: {app: web}
  template:                     # identical to a Deployment
    metadata:
      labels: {app: web}
    spec:
      containers:
        - name: web
          image: ghcr.io/stefanprodan/podinfo:6.7.1
          ports: [{containerPort: 9898}]
  strategy:
    canary:
      steps:
        - setWeight: 20
        - pause: {}              # wait for manual promote
        - setWeight: 50
        - pause: {duration: 5m}
        - setWeight: 100

Migrating a Deployment

  • Change kind: Deployment to kind: Rollout and apiVersion to argoproj.io/v1alpha1, add a strategy.
  • Or keep the Deployment and point a Rollout at it with spec.workloadRef (scale the Deployment to 0 when ready).
  • In the web chart: a rollout.enabled value that switches the template.
  • HPA can target a Rollout (scaleTargetRef.kind: Rollout).
  • Changing anything in spec.template starts a new rollout; changing replicas does not.
Canary

Canary without a traffic router: weight by replicas

With no trafficRouting, setWeight: 20 means about 20% of pods run the new version.

Service web → selects app=web (both ReplicaSets)stable ReplicaSet6.7.0 · 4 podscanary ReplicaSet6.7.1 · 1 podService webkube-proxy spreads by pod count~80% of connections~20% of connectionsWeights are approximate and change in pod-sized steps; 1 of 5 pods = 20% at best.

Fine for internal services and low traffic. For exact percentages, header-based routing or a 1% canary, add a traffic router (next slides).

Canary with ALB

Exact weights with the AWS Load Balancer Controller

Rollouts rewrites the Ingress action annotation so the ALB splits traffic between two target groups.

spec:
  strategy:
    canary:
      stableService: web-stable    # Rollouts adds a pod-hash selector
      canaryService: web-canary
      trafficRouting:
        alb:
          ingress: web             # the existing Ingress
          servicePort: 80
      steps:
        - setWeight: 5
        - pause: {duration: 10m}
        - setWeight: 25
        - pause: {duration: 10m}
        - setWeight: 50
        - pause: {duration: 10m}

# Ingress web: backend service name must be web-stable;
# Rollouts manages the weighted forward action.

How it works

  • Two Services select the same app labels; Rollouts pins each to a ReplicaSet hash.
  • Each Service becomes a target group (IP target type).
  • At setWeight: 5 the ALB sends 5% of requests to the canary target group, independent of pod counts.
  • The canary can run 1 pod for a 5% step: cheaper than scaling to 1 of 20.
  • Argo CD: ignore the annotation Rollouts manages on the Ingress, or Argo CD will show OutOfSync (ch 7 ignoreDifferences).
Traffic routers

Other ways to shift traffic

Pick what your edge already runs.

RouterConfigured asNotes
AWS ALBtrafficRouting.albWeighted target groups; header routing via ALB conditions
Gateway APItrafficRouting.plugins (Gateway API plugin)Updates HTTPRoute backend weights; plugin registered in the Rollouts ConfigMap
IstiotrafficRouting.istioVirtualService weights, header and mirror routes
NGINX IngresstrafficRouting.nginxCreates a canary Ingress with canary annotations
SMI / othersBuilt-in or pluginsCheck the Rollouts docs for current support
Noneomit trafficRoutingReplica-count weights (previous slide)
Blue-green

Blue-green: run both, switch at once

The new version runs at full size behind a preview Service; promotion flips the active Service.

spec:
  strategy:
    blueGreen:
      activeService: web            # production traffic
      previewService: web-preview   # testers, smoke tests
      autoPromotionEnabled: false   # wait for promote
      prePromotionAnalysis:
        templates:
          - templateName: smoke-tests
      postPromotionAnalysis:
        templates:
          - templateName: success-rate
      scaleDownDelaySeconds: 30     # keep old pods briefly

Canary vs blue-green

  • Blue-green: needs 2× capacity during the release; instant switch and instant switch-back.
  • Canary: gradual exposure, less capacity, slower.
  • Blue-green suits stateful sessions that must not be split, or apps that cannot run mixed versions behind one route.
  • Canary suits high-traffic stateless services where metrics need real traffic.
  • Both: database schema changes must be backward compatible with the old version.
Operating

kubectl argo rollouts: the commands you will use

In GitOps, use these to observe and to move a paused rollout — not to change the spec.

# install the plugin (macOS)
brew install argoproj/tap/kubectl-argo-rollouts

kubectl argo rollouts get rollout web -n web-dev --watch
kubectl argo rollouts promote web -n web-dev        # next step
kubectl argo rollouts promote web --full -n web-dev # skip all steps
kubectl argo rollouts abort web -n web-dev          # back to stable
kubectl argo rollouts retry rollout web -n web-dev  # after abort
kubectl argo rollouts dashboard                     # local UI :3100

GitOps rules

  • Change the image in Git, never with kubectl argo rollouts set image (Argo CD would see drift).
  • promote and abort change status, not the spec: safe with Argo CD.
  • undo rewrites the pod template → OutOfSync; use git revert instead.
  • After an abort, Git still declares the bad version: the Rollout stays Degraded until Git is reverted or a fixed version is pushed.
Chapter 11 · Part 11.2

Analysis and automation

Let metrics decide: AnalysisTemplates, automatic abort, and how Rollouts and Argo CD cooperate.

AnalysisTemplate

Success rate from Prometheus

An AnalysisRun queries the metric on an interval and counts successes and failures.

apiVersion: argoproj.io/v1alpha1
kind: AnalysisTemplate
metadata:
  name: success-rate
spec:
  args:
    - name: service
  metrics:
    - name: success-rate
      interval: 1m
      count: 5
      failureLimit: 1
      successCondition: result[0] >= 0.99
      provider:
        prometheus:
          address: http://prometheus.monitoring:9090
          query: |
            sum(rate(http_requests_total{
              service="{{args.service}}",code!~"5.."}[2m]))
            / sum(rate(http_requests_total{
              service="{{args.service}}"}[2m]))

Fields

  • interval + count: measure 5 times, one minute apart.
  • successCondition: expression on result (here a vector from PromQL).
  • failureLimit: 1: two failed measurements fail the run (failures must exceed the limit).
  • args make one template reusable across services.
  • Other providers include CloudWatch, Datadog, New Relic, web (HTTP) and Job (run a test Pod).
  • A query that returns no data is not a success: handle empty results (e.g. low traffic at 5%).
Wiring analysis

Inline steps vs background analysis

Inline analysis blocks a step; background analysis runs during the whole canary.

strategy:
  canary:
    analysis:                  # background: starts at step 2
      templates:
        - templateName: success-rate
      startingStep: 2
      args:
        - name: service
          value: web-canary
    steps:
      - setWeight: 5
      - pause: {duration: 5m}
      - setWeight: 25
      - analysis:              # inline: blocks until done
          templates:
            - templateName: latency-p95
          args:
            - name: service
              value: web-canary
      - setWeight: 50
      - pause: {duration: 10m}

Outcomes

  • Successful: inline step continues; background run keeps measuring.
  • Failed: the Rollout aborts — traffic returns to stable, canary scaled down, status Degraded.
  • Inconclusive: the Rollout pauses for a human (use inconclusiveLimit / consecutiveErrorLimit thoughtfully).
  • Every run is an AnalysisRun object: kubectl get analysisrun -n web-prod shows the measurements.
  • Measure the canary (web-canary labels), not the whole service, or the 95% stable traffic hides problems.
Automatic abort

A bad release, minute by minute

What the user sees vs what the system does when the canary fails analysis.

T+0T+5mT+6mT+7mT+8mGit merge6.7.1 in prodsetWeight 5canary pod readyanalysis #10.97 < 0.99analysis #20.96 → failedabortALB 100% stableusers affected~5% of requests for ~2 minutesArgo CD: app Degradednotify → git revert → Synced/Healthy

Compare: without Rollouts, the same release reaches 100% of users and the error rate is noticed by people, not by a query.

With Argo CD

How Rollouts and Argo CD cooperate

Argo CD applies the Rollout; the Rollouts controller executes it. Each owns a different part.

Health in Argo CD

  • Argo CD has a built-in health check for Rollout.
  • Paused at a step → health Suspended (the app is not Healthy yet).
  • Aborted or analysis failed → Degraded.
  • Fully promoted → Healthy; argocd app wait --health waits for this.

Actions from the UI

  • Argo CD ships resource actions for Rollouts (for example resume, promote-full, abort, retry, restart).
  • Grant them with RBAC action/argoproj.io/Rollout/* (ch 10).
  • They change Rollout status, so they do not create drift.

Sync settings

  • Argo CD sync does not wait for a canary to finish; health reports progress.
  • Sync waves: put analysis templates in an earlier wave than the Rollout.
  • Use ignoreDifferences for router-managed fields (Ingress annotations, VirtualService weights, HTTPRoute weights).
  • Keep auto-sync on: the next Git change after an abort deploys normally.
Notifications

Tell people when a rollout pauses, aborts or completes

Two notification engines exist; know which one you are configuring.

Argo CD notifications

  • Controller argocd-notifications-controller with argocd-notifications-cm.
  • Triggers on Application state: sync succeeded or failed, health degraded.
  • Subscribe per app with annotations, e.g. notifications.argoproj.io/subscribe.on-health-degraded.slack: deploys.
  • Self-managed Argo CD only: the EKS Capability for Argo CD has no Notifications controller.

Argo Rollouts notifications

  • Built into the Rollouts controller; configured in argo-rollouts-notification-configmap.
  • Triggers on Rollout events, e.g. rollout aborted, paused, completed, analysis run failed.
  • Same annotation style on the Rollout object.
  • Works regardless of how Argo CD is hosted.

What to alert on

  • Rollout aborted → team channel, high priority.
  • Rollout paused for manual promote → reviewer.
  • App Degraded > 10 min → on-call.
  • Do not page on every successful sync.
Common mistakes

Progressive delivery mistakes

Most failed canaries fail because of the analysis, not the code.

Analysis

  • Query measures the whole service, so 5% canary errors disappear in the average.
  • No traffic at 5% weight → empty results → inconclusive or false success.
  • Thresholds copied from another service.
  • count too small to see anything before full promotion.

GitOps interplay

  • Setting the image with the Rollouts plugin instead of Git.
  • No ignoreDifferences for router-managed fields → endless OutOfSync.
  • Treating abort as rollback: Git still declares the bad version.
  • Auto-sync off "during the canary" and forgotten.

Release design

  • Schema changes not backward compatible: the stable version breaks during the canary.
  • Blue-green without 2× capacity (Pending pods, stuck rollout).
  • Changing replicas and the image in the same commit, confusing the steps.
  • Rollout and HPA both setting replicas without understanding which wins.
Self-check

Progressive delivery: can you answer these?

Explain the answer and name the exact field or command.

Rollouts

  • What changes when you convert a Deployment into a Rollout?
  • Why is a 5% canary not possible with 5 replicas and no traffic router?
  • Which two Services does the ALB integration need, and why?
  • Canary or blue-green for a service with long-lived WebSocket sessions?

Analysis

  • Write a successCondition for "p95 latency below 300 ms".
  • What is the difference between failureLimit and inconclusiveLimit?
  • Inline vs background analysis: when does each start and stop?
  • Why should the query target the canary pods only?

With Argo CD

  • Which Argo CD health status does a paused Rollout show?
  • Why is kubectl argo rollouts undo a bad idea in GitOps?
  • After an automatic abort, what still needs to happen in Git?
  • Which notification engine works with the EKS Capability for Argo CD?

Practice: Lab 11 in Hands-on labs — a canary with a Prometheus AnalysisTemplate, then inject errors and watch the automatic abort.

Chapter 12

Operations and troubleshooting

Day-2 Argo CD: metrics and alerts, scaling the controller and repo-server, backup and upgrades, and a playbook for the errors you will meet.

Chapter 12 · Part 12.1

Observability for Argo CD

Which components expose metrics, the few metrics that matter, dashboards and alerts.

Where metrics live

Every Argo CD component exposes Prometheus metrics

Scrape all of them: the controller tells you about apps, the repo-server about Git and rendering, the API server about users.

Endpoints (install manifests)

ComponentService : portAnswers
Application controllerargocd-metrics:8082 (Helm chart: argocd-application-controller-metrics)App sync/health, reconcile time, cluster cache
API serverargocd-server-metrics:8083Requests from UI, CLI, CI
Repo-serverargocd-repo-server:8084Git requests, manifest generation
ApplicationSet controllerargocd-applicationset-controller:8080Generators, reconcile

Scraping

  • With kube-prometheus-stack: enable the chart values *.metrics.enabled and *.metrics.serviceMonitor.enabled for each component.
  • Without the operator: add scrape configs for the four services.
  • App labels are not exported by default; opt in with the controller flag --metrics-application-labels.
  • Application conditions are opt-in with --metrics-application-conditions.
The metrics that matter

Five metrics answer most questions

Learn these names; everything else is detail.

MetricTypeUse it for
argocd_app_infogaugeOne series per Application with sync_status, health_status, project, dest_server labels: count OutOfSync / Degraded apps
argocd_app_sync_totalcounterSync history by phase (Succeeded, Failed, Error, Running, Terminating): failure rate
argocd_app_reconcilehistogramHow long the controller takes to compare an app: the first sign of an overloaded controller
argocd_git_request_totalcounterGit ls-remote and fetch calls per repo from the repo-server: polling load, auth failures
argocd_git_request_duration_secondshistogramSlow Git hosts or huge repos

Check the metrics page of the docs for your exact minor version before writing alerts; labels are added over time.

PromQL

Queries for a GitOps dashboard

Use these as panels and as the base of alerts.

# apps not in sync, per project
count by (project) (argocd_app_info{sync_status="OutOfSync"})

# apps not healthy (Degraded, Missing, Unknown)
count by (health_status) (argocd_app_info{
  health_status!~"Healthy|Progressing|Suspended"})

# sync failure ratio, last hour
sum(increase(argocd_app_sync_total{phase=~"Error|Failed"}[1h]))
  / sum(increase(argocd_app_sync_total[1h]))

# p95 reconcile time per controller shard
histogram_quantile(0.95,
  sum by (le, namespace) (rate(argocd_app_reconcile_bucket[5m])))

# Git requests per repo per minute
sum by (repo) (rate(argocd_git_request_total[5m])) * 60

Reading the panels

  • A steady handful of OutOfSync apps is normal during deploys; a growing line is not.
  • Reconcile p95 above tens of seconds: shard the controller or raise processors (section 37).
  • Git requests climbing with app count: add webhooks and manifest-generate-paths.
  • Grafana: start from the dashboard JSON in the Argo CD repo (examples/dashboard.json).
Alerts

Alert on user impact, not on every OutOfSync

Auto-sync apps are briefly OutOfSync on every commit. Alert on duration.

groups:
- name: argocd
  rules:
  - alert: ArgoAppDegraded
    expr: argocd_app_info{health_status="Degraded"} == 1
    for: 10m
    labels: {severity: page}
    annotations:
      summary: "{{ $labels.name }} Degraded for 10m"
  - alert: ArgoAppOutOfSyncTooLong
    expr: |
      argocd_app_info{sync_status="OutOfSync",
        autosync_enabled="true"} == 1
    for: 30m
    labels: {severity: ticket}
  - alert: ArgoSyncFailures
    expr: |
      increase(argocd_app_sync_total{
        phase=~"Error|Failed"}[15m]) > 0
    labels: {severity: ticket}

Also watch

  • Controller and repo-server restarts (repo-server OOMKilled is common).
  • Redis down: UI slow, controller recomputes everything.
  • Expiring repo and cluster credentials; API server 5xx if CI calls argocd app sync.
  • Route alerts by the project label to the team owning the AppProject.
  • Per-app sync messages to Slack: Argo CD Notifications (self-managed only; the EKS capability has none).
Logs and events

Logs: where each kind of error shows up

Know which pod to read before you start scrolling.

SymptomLook inCommand
Render error, Helm/Kustomize failure, Git authrepo-serverkubectl -n argocd logs deploy/argocd-repo-server
Sync stuck, diff wrong, cluster cache errorsapplication controllerkubectl -n argocd logs statefulset/argocd-application-controller
Login, RBAC denied, webhook received?API serverkubectl -n argocd logs deploy/argocd-server
Apps not generated or deleted unexpectedlyApplicationSet controllerkubectl -n argocd logs deploy/argocd-applicationset-controller
Who synced what, whenKubernetes events on the Applicationkubectl -n argocd get events --field-selector involvedObject.name=web-prod

Set --loglevel debug only temporarily; switch log format to JSON (--logformat json) so logs are queryable in Loki or CloudWatch Logs Insights.

Chapter 12 · Part 12.2

Performance and scale

Where Argo CD spends its time, and the knobs for hundreds of apps and dozens of clusters.

Where time goes

The reconcile pipeline and its bottlenecks

Every refresh: fetch Git → render manifests → compare with the live cache → maybe sync.

Git hostGitHub, GitLabrepo-serverclone + helm/kustomizeapp controllerdiff + syncClusterswatch → live cacheRedismanifest cachefetchrender reqapplyBottleneck 1: Git polling and cloning (many apps × refresh every 3 min).Bottleneck 2: rendering (big Helm charts, CPU-bound, repo-server memory).Bottleneck 3: controller processors and the cluster cache (many resources, many clusters).Cache holds every watched resourceof every managed cluster in memory.
Controller

Scaling the application controller

It is a StatefulSet; each replica owns a shard of the clusters.

Processors (argocd-cmd-params-cm)

  • controller.status.processors: parallel refresh/compare workers (default 20).
  • controller.operation.processors: parallel sync workers (default 10).
  • Raise for ~1,000+ apps, together with CPU and memory requests.
  • timeout.reconciliation in argocd-cm: refresh interval, default 120s plus up to 60s jitter (timeout.reconciliation.jitter); longer means less Git load but slower drift detection.

Sharding

  • Set the StatefulSet replicas and the env ARGOCD_CONTROLLER_REPLICAS to the same value.
  • Shards split by cluster, not by app: one huge cluster still lands on one replica.
  • Algorithm: controller.sharding.algorithm = legacy (default), round-robin, or consistent-hashing.
  • Pin a cluster to a shard with shard: "2" in its cluster Secret.
apiVersion: v1
kind: ConfigMap
metadata:
  name: argocd-cmd-params-cm
  namespace: argocd
data:
  controller.status.processors: "50"
  controller.operation.processors: "25"
  controller.sharding.algorithm: round-robin
---
# Helm values (argo/argo-cd)
controller:
  replicas: 3
  env:
  - name: ARGOCD_CONTROLLER_REPLICAS
    value: "3"
  resources:
    requests: {cpu: "1", memory: 2Gi}
Repo-server

Scaling the repo-server and Git traffic

Rendering is CPU-heavy and stateless: scale it horizontally.

Repo-server

  • Run 2+ replicas (HA) and add an HPA on CPU.
  • reposerver.parallelism.limit caps concurrent renders per pod so a burst does not OOM it.
  • Give it memory headroom: large Helm charts and monorepo clones live in its temp dirs.
  • Manifests are cached in Redis keyed by commit SHA: same commit → no re-render.

Less Git traffic

  • Webhooks: Git host → https://argocd.example.com/api/webhook; refresh on push instead of waiting for polling.
  • Protect the webhook with the shared secret in argocd-secret (webhook.github.secret, etc.).
  • Monorepo: annotate apps with argocd.argoproj.io/manifest-generate-paths so a commit only refreshes apps whose paths changed.
  • Prefer one repo-server fetch per commit: avoid many repos with tiny apps if you can group them.
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
 name: web-prod
 annotations:
  # refresh only when these
  # paths change
  argocd.argoproj.io/manifest-generate-paths:
   .;/charts/web
spec:
 source:
  repoURL: https://github.com/<you>/gitops-lab
  path: envs/prod/web
Cluster cache

Large clusters: keep the controller cache small

The controller watches every resource type it may manage. Exclude what it never needs.

Shrink the watch

  • resource.exclusions in argocd-cm: drop noisy kinds (Events, Leases, metrics objects, high-churn CRs such as Cilium identities).
  • resource.inclusions: allow-list mode for tightly scoped installs.
  • Namespace-scoped install (namespace-install.yaml, or chart value createClusterRoles: false) watches only allowed namespaces.

Reduce churn

  • resource.customizations.ignoreResourceUpdates.<group_kind> keys stop refreshes caused by fields you do not care about (e.g. status-only updates on CRs).
  • Since 3.0 .status changes are ignored in diffs by default for all resources.
  • Avoid per-app tiny refresh intervals; use webhooks.
# argocd-cm
data:
  resource.exclusions: |
    - apiGroups: [""]
      kinds: ["Event"]
      clusters: ["*"]
    - apiGroups: ["coordination.k8s.io"]
      kinds: ["Lease"]
      clusters: ["*"]
    - apiGroups: ["cilium.io"]
      kinds: ["CiliumIdentity"]
      clusters: ["*"]
Sizing guide

Symptoms → which knob to turn

Change one knob at a time and watch the reconcile histogram.

SymptomLikely bottleneckKnob
UI shows "Refreshing" for minutes, reconcile p95 highController workerscontroller.status.processors, more CPU, shard
Syncs queue behind each otherSync workerscontroller.operation.processors
ComparisonError: context deadline exceededRepo-server render timeMore repo-server replicas, reposerver.parallelism.limit, smaller charts
repo-server OOMKilledParallel renders of big chartsMemory limit up, parallelism down
Git host rate-limits Argo CDPollingWebhooks, longer timeout.reconciliation, manifest-generate-paths
Controller memory grows with clustersCluster cacheShard by cluster, resource.exclusions
Chapter 12 · Part 12.3

Backup, DR and upgrades

If everything is in Git, recovery is a re-bootstrap. Keep the few things that are not.

What to back up

Argo CD state: what is in Git, and what is not

Design so that a new cluster plus one bootstrap command restores everything.

Already in Git (if you did it right)

  • Applications, ApplicationSets, AppProjects (app-of-apps root).
  • Argo CD settings: argocd-cm, argocd-rbac-cm, argocd-cmd-params-cm as Helm values.
  • Workload manifests, charts, values.

Not in Git (back it up)

  • Repository and cluster credentials (Secrets) — keep them in AWS Secrets Manager via External Secrets instead.
  • argocd-secret (admin password hash, server signing key, webhook secrets).
  • Anything someone created by clicking in the UI (find it and move it to Git).
# full export of Argo CD objects and settings
argocd admin export -n argocd \
  > argocd-backup.yaml

# restore into a fresh install
argocd admin import -n argocd - \
  < argocd-backup.yaml

# store encrypted (contains secrets)
aws s3 cp argocd-backup.yaml \
  s3://my-backups/argocd/ \
  --sse aws:kms
DR

Disaster recovery: rebuild, do not restore

The Argo CD cluster is cattle. The target clusters keep running if Argo CD dies.

1 · Argo CD is down

  • Workloads keep running: Argo CD is not in the data path.
  • Only deploys and drift correction stop.
  • Emergency changes: kubectl with a ticket, then commit to Git.
→

2 · Rebuild

  • Terraform creates the cluster and installs the Argo CD Helm release.
  • External Secrets restores repo and cluster credentials.
  • Apply the root Application (ch 8).
→

3 · Converge

  • Argo CD adopts existing resources by tracking annotation: no redeploy.
  • Check for OutOfSync apps: they show drift made during the outage.
  • Re-enable alerts.

Test it: once a quarter, build a second Argo CD from Git in a spare cluster with all apps set to manual sync, and confirm every app shows Synced.

Upgrades

Upgrading Argo CD safely

Upgrade notes per minor version list breaking changes. Read every note between your version and the target.

Process

  • Upgrade one minor version at a time (3.3 → 3.4 → 3.5).
  • Read operator-manual/upgrading/<from>-<to> for each step.
  • Bump the argo/argo-cd chart version in Git; Argo CD can manage itself, but keep a manual path (helm upgrade) for when it breaks.
  • Upgrade staging Argo CD first; compare app status counts before and after.
  • CRDs are part of the chart: confirm they upgraded (kubectl get crd applications.argoproj.io -o yaml).

Changes that bit people in 3.x

  • 3.0: resource tracking switched to annotation; fine-grained RBAC for update/delete; .status ignored in diffs.
  • 3.3: Source Hydrator no longer clears the app path before writing.
  • 3.5: GnuPG verification replaced by Source Integrity; mTLS between components; impersonation applies to all API server operations.
  • Reading an old tutorial? Check which version it targets.
# check versions
argocd version
helm -n argocd list
# preview chart changes
helm diff upgrade argocd argo/argo-cd \
  -n argocd --version <new> -f values.yaml
Self-managing Argo CD

Should Argo CD manage its own install?

Common and convenient, with one rule: never let a bad commit lock you out.

Pattern

  • Terraform or helm install does the first install.
  • A child Application argocd (in the root app) then owns the Helm release values in Git.
  • Changes to Argo CD go through PRs like everything else.

Guard rails

  • Disable prune on the argocd Application.
  • Keep selfHeal but review chart bumps in staging first.
  • Keep the admin break-glass path documented: helm upgrade from a laptop with the last good values.
  • Never manage the Argo CD namespace deletion through Argo CD.
Chapter 12 · Part 12.4

Troubleshooting playbook

Symptom → cause → fix, plus the CLI commands that show you the truth.

Tools

The five commands you will use most

Compare what Git renders with what is live, then read the controller's reason.

# what Argo CD thinks: status, conditions, last operation
argocd app get web-prod --show-operation

# force re-read of Git and re-render (ignores the cache)
argocd app get web-prod --hard-refresh

# what would change
argocd app diff web-prod

# rendered manifests from Git vs live objects
argocd app manifests web-prod --source git
argocd app manifests web-prod --source live

# stop a stuck sync operation
argocd app terminate-op web-prod

# server-side resources of the app with health
argocd app resources web-prod

Workflow

  • 1. app get: read the condition message first, it usually names the cause.
  • 2. --hard-refresh if Git changed but Argo CD did not notice.
  • 3. app diff: is the drift real or a defaulting/mutation artifact?
  • 4. Render locally with the same tool versions: helm template / kustomize build.
  • 5. Logs of repo-server (render) or controller (sync).
Playbook 1/3

Rendering and repository errors

Shown as ComparisonError on the Application; nothing is applied.

SymptomCauseFix
rpc error ... authentication requiredRepo credential missing or expired (token, SSH key, GitHub App)Fix the repository Secret (label argocd.argoproj.io/secret-type: repository or repo-creds); test with argocd repo list
helm template ... error converting YAMLBad values or template bugRun helm template locally with the same values files; check valueFiles paths
Unable to resolve 'main' to a commit SHAWrong targetRevision or branch renamedFix targetRevision; use tags for prod
context deadline exceeded during manifest generationRender too slow or repo-server overloadedScale repo-server, lower parallelism, split the chart
app path does not existWrong path, or commit not pushed yetCheck the path at that SHA in Git
Helm dependency not foundcharts/ not vendored, private dependency repo not registeredRegister the dependency repo in Argo CD or commit Chart.lock + vendored charts
Playbook 2/3

Sync and health problems

Argo CD renders fine but the sync or the health check does not finish.

SymptomCauseFix
OutOfSync right after every sync (loop)Field mutated by a controller or webhook (HPA replicas, injected sidecars, defaulted fields)ignoreDifferences + RespectIgnoreDifferences=true; remove replicas from Git when an HPA owns it
Sync waits forever on a hookPreSync Job never completes or image pull failskubectl logs job/...; argocd app terminate-op; add activeDeadlineSeconds and a hook delete policy
the server could not find the requested resource for a CRCRD not installed yet in the same syncPut CRDs in an earlier sync wave, or SkipDryRunOnMissingResource=true on the CR
Health stuck ProgressingDeployment cannot roll out, or Ingress without a load balancer addressCheck the workload events; custom Lua health check for CRs Argo CD does not know
field is immutable on syncChanged selector, Job spec, or StatefulSet volumeClaimTemplatesDelete and recreate that object (Replace=true sparingly), fix labels in the chart
Sync blocked: "sync window"A deny window in the AppProject is activeWait, or use a manual override if the window allows it
Playbook 3/3

Deletion, ownership and access problems

The problems that appear when apps are removed, moved or shared.

SymptomCauseFix
Application stuck in deletionFinalizer resources-finalizer.argocd.argoproj.io waiting on a resource that cannot be deletedFind the blocking resource (argocd app resources); as a last resort remove the finalizer with kubectl patch
Two apps fight over a resourceBoth render the same object; tracking annotation flipsOne owner per object; Argo CD reports SharedResourceWarning
Resources orphaned after removing them from Gitprune disabledEnable prune, or use the orphaned resources warning in the AppProject
permission denied in UI or CLIRBAC policy or project role missing; 3.0 fine-grained update/deleteargocd account can-i sync applications 'proj/web-prod'; fix argocd-rbac-cm
Cluster Unknown / connection refusedExpired cluster token, IAM role trust, or private endpoint not reachableargocd cluster list; on EKS check the IAM role and access entry

Removing a finalizer by hand skips cleanup: list the app's resources first so you know what will be left behind.

Common mistakes

Operational anti-patterns

Each one looks harmless until the day it is not.

Hiding problems

  • Alerting on every OutOfSync: the channel gets muted within a week.
  • Fixing prod with kubectl edit and forgetting Git: self-heal reverts it, or worse, the next deploy does.
  • Clicking "Sync" with "Replace" or "Force" to make an error go away.

Scaling problems

  • Polling 500 apps every 3 minutes against GitHub without webhooks.
  • One repo-server replica with no memory headroom.
  • Adding controller replicas without setting ARGOCD_CONTROLLER_REPLICAS.

Recovery problems

  • Credentials only in the cluster: a rebuild cannot reconnect to Git.
  • Skipping minor versions on upgrade and missing a breaking change.
  • Argo CD pruning its own namespace through the self-managed app.
Self-check

Operations: can you answer these?

Answer out loud, then check the slide that covers it.

Metrics

  • Which metric and labels give you the number of Degraded apps per project?
  • Why alert on OutOfSync only after a duration?
  • Which component's metrics show Git polling load?
  • What does a rising argocd_app_reconcile p95 tell you?

Scale and DR

  • How does controller sharding split work, and what is its limit?
  • Name three ways to reduce Git load.
  • What is not in Git in a typical install, and where should it live?
  • Why do workloads keep running when Argo CD is down?

Troubleshooting

  • An app loops OutOfSync after each sync because of an HPA: two fixes?
  • A CR fails because its CRD is missing: two fixes?
  • An app is stuck deleting: what do you check before removing the finalizer?
  • Which upgrade notes do you read to go from 3.3 to 3.5?

Practice: Lab 12 in Hands-on labs — an ops drill with five pre-broken faults and Argo CD metrics in Prometheus.

Chapter 13

GitOps on EKS

Self-managed Argo CD on EKS, the AWS-managed EKS Capability for Argo CD, Terraform bootstrap with the GitOps Bridge, and cluster add-ons by label.

Chapter 13 · Part 13.1

Self-managed Argo CD on EKS

Reference architecture, IAM-based cluster access, and a Terraform bootstrap from zero to a root Application.

Reference architecture

One management cluster, many workload clusters

Argo CD lives in a small management cluster and deploys to dev, staging and prod clusters, often in other accounts.

Account: platformEKS mgmtArgo CDcontroller · repo-server · serverALBOIDC SSOExternal Secretsrepo credsPod Identity roleargocd-application-controllerAccount: devEKS devaccess entry → hub roleAccount: prodEKS prodaccess entry → hub roleGitHub: gitops-labcharts/ apps/ envs/ clusters/Amazon ECRimages + OCI chartsSecrets Managerrepo + cluster credsassume role

Workload clusters need no inbound access from the internet: the hub reaches their API endpoints (public with CIDR allow-list, or private over TGW/peering).

Cluster access with IAM

How self-managed Argo CD authenticates to EKS clusters

No static tokens: the controller uses an IAM role, and each target cluster maps that role with an access entry.

apiVersion: v1
kind: Secret
metadata:
  name: cluster-prod
  namespace: argocd
  labels:
    argocd.argoproj.io/secret-type: cluster
    env: prod
type: Opaque
stringData:
  name: prod
  server: https://ABCD.gr7.ap-northeast-1.eks.amazonaws.com
  config: |
    {
      "awsAuthConfig": {
        "clusterName": "prod",
        "roleARN":
          "arn:aws:iam::222233334444:role/argocd-deployer"
      },
      "tlsClientConfig": {"caData": "<base64 CA>"}
    }

The chain

  • Controller pod → EKS Pod Identity (or IRSA) role in the platform account.
  • That role assumes argocd-deployer in the target account (roleARN).
  • Argo CD calls aws eks get-token style auth for clusterName.
  • Target cluster: access entry for argocd-deployer + an access policy (cluster admin, or namespace-scoped).
  • The argocd-server pod needs the same role to show live state in the UI.
  • Keep the Secret in Git as an ExternalSecret: only the CA and ARNs, no token.
  • Labels (env: prod) feed ApplicationSet cluster generators (section 42).
Install

Argo CD on EKS: production Helm values

Install with the argo/argo-cd chart; these are the values that change on EKS.

# values-eks.yaml (argo/argo-cd)
global:
  domain: argocd.example.com
configs:
  params:
    server.insecure: true        # TLS ends at the ALB
  cm:
    timeout.reconciliation: 180s
  rbac:
    policy.default: role:readonly
server:
  ingress:
    enabled: true
    controller: aws
    ingressClassName: alb
    annotations:
      alb.ingress.kubernetes.io/scheme: internal
      alb.ingress.kubernetes.io/certificate-arn: <acm-arn>
      alb.ingress.kubernetes.io/target-type: ip
redis-ha:
  enabled: true
controller:
  replicas: 1
repoServer:
  autoscaling: {enabled: true, minReplicas: 2}

Why these values

  • server.insecure: the ALB terminates TLS with an ACM certificate; pods speak HTTP inside the VPC.
  • Internal ALB + SSO: the UI is not on the internet.
  • redis-ha and 2+ repo-servers for HA; the controller scales by sharding (ch 12).
  • gRPC for the CLI through ALB needs extra setup; many teams use argocd login --grpc-web.
  • Pin the chart version in Git; upgrades are PRs.
Terraform bootstrap

From nothing to a running GitOps cluster in one apply

Terraform owns the cluster and the first Argo CD install. Argo CD owns everything after that.

module "eks" {
  source             = "terraform-aws-modules/eks/aws" # pin version
  name               = "gitops"
  kubernetes_version = "1.33"
  vpc_id             = module.vpc.vpc_id
  subnet_ids         = module.vpc.private_subnets
  enable_cluster_creator_admin_permissions = true
  eks_managed_node_groups = { default = {
    instance_types = ["m6i.large"], desired_size = 2 } }
}

resource "helm_release" "argocd" {
  name             = "argocd"
  namespace        = "argocd"
  create_namespace = true
  repository       = "https://argoproj.github.io/argo-helm"
  chart            = "argo-cd"
  version          = var.argocd_chart_version
  values           = [file("values-eks.yaml")]
}
# argocd-apps chart: no CRD needed at plan time
resource "helm_release" "root" {
  name       = "root"
  namespace  = "argocd"
  repository = "https://argoproj.github.io/argo-helm"
  chart      = "argocd-apps"
  depends_on = [helm_release.argocd]
  values = [yamlencode({
    applications = {
      root = {
        project = "default"
        source = {
          repoURL        = "https://github.com/<you>/gitops-lab"
          path           = "clusters/gitops"
          targetRevision = "main"
        }
        destination = { namespace = "argocd",
          server = "https://kubernetes.default.svc" }
        syncPolicy = { automated = { prune = true, selfHeal = true } }
      }
    }
  })]
}

Module input names change between major versions of terraform-aws-modules/eks/aws; check the README of the version you pin.

Boundaries

Who owns what: Terraform vs Argo CD

Draw the line once and write it down; overlap causes two tools to fight.

ResourceOwnerWhy
VPC, EKS cluster, node groups, IAM roles, access entriesTerraformAWS APIs, needed before Kubernetes exists
Argo CD first install + root ApplicationTerraform (once)Bootstrap; after that Argo CD can manage its own chart
Cluster add-ons (LB controller, ExternalDNS, Karpenter config, metrics)Argo CDKubernetes resources, versioned per cluster
Workloads and their configArgo CDTeam-owned, promoted through Git
IAM roles for add-ons (Pod Identity associations)TerraformAWS side; pass ARNs to Argo CD as metadata (next slide)
Karpenter itselfEither, pick oneOften Terraform installs it so nodes exist before Argo CD runs
GitOps Bridge

GitOps Bridge: hand Terraform outputs to Argo CD

Terraform knows the AWS facts (role ARNs, Region, VPC ID); Argo CD templates need them. Put them on the cluster Secret.

# cluster Secret written by Terraform
# (gitops-bridge module or a kubernetes_secret)
metadata:
  name: gitops
  labels:
    argocd.argoproj.io/secret-type: cluster
    environment: dev
    enable_aws_load_balancer_controller: "true"
    enable_external_dns: "true"
  annotations:
    aws_region: ap-northeast-1
    aws_account_id: "111122223333"
    aws_cluster_name: gitops
    aws_vpc_id: vpc-0abc
    aws_load_balancer_controller_iam_role_arn: arn:aws:iam::...
    addons_repo_url: https://github.com/<you>/gitops-lab
stringData:
  name: gitops
  server: https://kubernetes.default.svc

How it works

  • Labels = feature flags: which add-ons this cluster gets.
  • Annotations = values: ARNs and IDs the add-on charts need.
  • ApplicationSets select clusters by label and read annotations in templates.
  • Open-source reference: the GitOps Bridge project (gitops-bridge-dev) with a Terraform module and add-on ApplicationSets.
  • Same idea works for any number of clusters: each Terraform stack writes its own cluster Secret to the hub.
Chapter 13 · Part 13.2

EKS Capability for Argo CD

AWS-managed Argo CD: how it differs from upstream, its limits, and when to choose it.

What it is

Argo CD as an EKS capability

Launched Nov 2025 with EKS Capabilities (alongside ACK and kro). AWS runs Argo CD; you use its CRDs.

How it works

  • Argo CD controllers run in AWS-owned infrastructure, not on your nodes.
  • You create the capability on an EKS cluster with an IAM capability role.
  • Applications, ApplicationSets and AppProjects are normal CRs in that cluster.
  • Target clusters are identified by EKS cluster ARN, not API server URL.
  • UI and API access through AWS IAM Identity Center; API calls use AWS authentication.

What AWS handles

  • Install, scaling, HA and upgrades of Argo CD.
  • Connectivity to private remote clusters (no VPC peering needed).
  • Integration with IAM, Identity Center and EKS access entries.
  • You cannot choose the Argo CD version.

What you still own

  • Git repos, Applications, AppProjects, RBAC mapping.
  • Access entries and access policies on every target cluster.
  • Secrets for workloads (External Secrets etc.).
  • Monitoring and alerting of app health.
Register clusters

Targets: the local cluster and remote clusters by ARN

The local cluster is not registered automatically; register it explicitly (the docs suggest the name in-cluster).

apiVersion: v1
kind: Secret
metadata:
  name: prod
  namespace: argocd
  labels:
    argocd.argoproj.io/secret-type: cluster
    env: prod
stringData:
  name: prod
  # EKS cluster ARN, not the API server URL
  server: arn:aws:eks:ap-northeast-1:222233334444:cluster/prod
---
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: web-prod
  namespace: argocd
spec:
  project: default
  destination:
    name: prod            # or server: <cluster ARN>
    namespace: web

Access entries

  • Local cluster: an access entry for the capability role is created for you, with no Kubernetes permissions by default.
  • Remote cluster: create an access entry for the capability role yourself (aws eks create-access-entry), then associate an access policy.
  • Policy choice: AmazonEKSClusterAdminPolicy for platform add-ons; namespace-scoped policies for tenant-only clusters.
  • Cross-account: create the access entry in the target account with the capability role ARN from the source account.
Limits

Documented limits and fixed settings

Size your design against these before migrating.

Per capability instance

LimitValue
Application resources3,000
Remote clusters100
Kubernetes resources per Application150
Git polling interval6 minutes (self-managed default: 120 s + up to 60 s jitter)
Sync timeoutFixed at 120 s

Design consequences

  • 150 resources per app: split big umbrella apps (for example one app per add-on, not one for all).
  • 6-minute polling: configure Git webhooks or accept slower rollout start.
  • 120 s sync timeout: long PreSync migrations must be designed to finish quickly or run outside the sync.
  • More than 3,000 apps or 100 clusters: multiple capability instances, or self-managed.
Unsupported

What the managed capability does not support

Check every feature you use today against this list (and the current docs).

Not supported

  • Config management plugins (CMPs): only built-in Helm, Kustomize and directory sources.
  • Notifications controller: no Slack/email notifications from Argo CD itself.
  • SSO providers other than IAM Identity Center (no Dex, no direct Okta OIDC).
  • UI extensions.
  • Most argocd-cm configuration keys: only a documented subset.
  • Choosing the Argo CD version.

Workarounds

  • CMP (e.g. secrets plugin) → External Secrets Operator + Secrets Manager.
  • Notifications → Prometheus/CloudWatch alerts on app health, or CI posting results.
  • Okta/Entra users → federate them into IAM Identity Center.
  • Custom render steps → pre-render in CI (rendered manifests pattern, ch 1).
Decision

Managed capability vs self-managed vs vendor SaaS

Pick by the features you need and how much you want to operate.

QuestionEKS capabilitySelf-managedVendor SaaS (e.g. Akuity)
Who runs, scales, upgrades Argo CDAWSYouVendor
Version choiceNoYesUsually yes
TargetsEKS clusters (by ARN)Any KubernetesAny Kubernetes
SSOIAM Identity Center onlyAny OIDC/SAML via Dex or directVendor-provided
CMPs, notifications, UI extensionsNoYesVaries
Scale tuning (shards, processors)Not exposed; fixed limitsFull controlVendor-managed
Best fitAWS-only shops, small platform team, standard Helm/KustomizeHeavy customization, non-EKS targetsMulti-cloud, promotion features
Chapter 13 · Part 13.3

Platform add-ons through GitOps

One ApplicationSet per add-on, enabled per cluster by label, configured by annotations.

Add-on pattern

Add-ons as ApplicationSets selected by cluster labels

Enable an add-on on a cluster by adding one label to its cluster Secret.

Cluster Secrets (hub)labels + annotationsdevlbc=true, eds=trueprodlbc=true, karp=trueApplicationSet: aws-lb-controllerselector lbc=trueApplicationSet: external-dnsselector eds=trueApplicationSet: karpenter-configselector karp=truelbc-dev, lbc-prodApplicationsexternal-dns-devApplicationkarpenter-prodApplicationGenerator: clusters + selector.matchLabelsTemplate reads annotations for ARNs and IDs
Add-on ApplicationSet

AWS Load Balancer Controller on every cluster that asks for it

Chart from the public EKS repo; IAM role, Region and VPC ID come from the cluster Secret annotations.

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: addon-aws-load-balancer-controller
  namespace: argocd
spec:
  goTemplate: true
  goTemplateOptions: ["missingkey=error"]
  generators:
  - clusters:
      selector:
        matchLabels:
          enable_aws_load_balancer_controller: "true"
  template:
    metadata:
      name: 'lbc-{{.name}}'
    spec:
      project: platform
      source:
        repoURL: https://aws.github.io/eks-charts
        chart: aws-load-balancer-controller
        targetRevision: '{{index .metadata.annotations
                            "lbc_chart_version"}}'
        helm:
          valuesObject:
            clusterName: '{{index .metadata.annotations
                             "aws_cluster_name"}}'
            region: '{{index .metadata.annotations "aws_region"}}'
            vpcId: '{{index .metadata.annotations "aws_vpc_id"}}'
      destination:
        name: '{{.name}}'
        namespace: kube-system
      syncPolicy:
        automated: {prune: true, selfHeal: true}
Add-on ApplicationSet

Why the add-on template looks like this

Each line on the previous slide prevents a real failure mode.

Template choices

  • goTemplate: true + missingkey=error: a cluster missing an annotation fails loudly instead of rendering empty values.
  • index .metadata.annotations "key": annotation keys may contain characters a dotted path cannot.
  • Chart version per cluster (lbc_chart_version): prod upgrades after dev by changing one annotation.
  • destination.name: the cluster Secret name; works for both self-managed and the EKS capability.

Around it

  • IAM: create the Pod Identity association for the controller's service account in Terraform.
  • Same shape for ExternalDNS, External Secrets, metrics-server, Karpenter NodePools.
  • AppProject platform may create cluster-scoped resources; tenant projects may not.
  • Upgrade an add-on everywhere with ApplicationSet progressive sync (RollingSync) by env label.
Ordering

Add-on order and dependencies

Some add-ons need others first: CRDs, webhooks, IAM.

Typical order (sync waves on the add-on Applications)

  • Wave -2: CRD-only charts (e.g. Prometheus Operator CRDs, Gateway API CRDs).
  • Wave -1: AWS Load Balancer Controller, External Secrets Operator, cert-manager.
  • Wave 0: ExternalDNS, metrics-server, Karpenter NodePools.
  • Wave 1: workloads (often a separate root app).

Gotchas

  • Waves order Applications only inside one parent app (app-of-apps), not across ApplicationSets.
  • Webhook-based add-ons (LB controller, cert-manager) must be Healthy before CRs that hit their webhooks are applied.
  • The EKS capability counts each add-on Application toward the 3,000 limit and each must stay under 150 resources.

Alternatives

  • EKS managed add-ons for core components (VPC CNI, CoreDNS, kube-proxy, Pod Identity Agent): let AWS manage those; Argo CD manages the rest.
  • ApplicationSet progressive sync (RollingSync) to roll an add-on upgrade dev → staging → prod.
Common mistakes

EKS GitOps mistakes

Most are about IAM and boundaries, not about Argo CD itself.

Access

  • Static ServiceAccount tokens in cluster Secrets instead of IAM roles.
  • Forgetting the access entry on a new cluster: Argo CD shows the cluster as unreachable or forbidden.
  • Capability: assuming the local access entry grants permissions (it does not by default).

Boundaries

  • Terraform and Argo CD both managing the same Helm release.
  • Hard-coding role ARNs and VPC IDs in Git per cluster instead of passing them as cluster metadata.
  • Putting all add-ons in one Application (fails the 150-resource limit on the capability).

Operations

  • Public Argo CD UI without SSO.
  • Choosing the managed capability, then discovering a CMP or notifications dependency.
  • No webhook with 6-minute polling, then wondering why deploys are slow.
Self-check

GitOps on EKS: can you answer these?

Answer out loud, then check the slide that covers it.

Self-managed

  • Trace how the controller authenticates to a prod cluster in another account.
  • Why install the root app with the argocd-apps chart instead of kubernetes_manifest?
  • Which resources stay in Terraform, and which move to Argo CD?
  • What does the GitOps Bridge put in labels vs annotations?

EKS capability

  • How are target clusters identified?
  • Which access entry is created automatically, and what can it do?
  • Name four limits and two unsupported features.
  • Your team uses a secrets CMP and Slack notifications: what changes if you move?

Add-ons

  • Write the generator that selects clusters with enable_external_dns: "true".
  • Why use missingkey=error?
  • How do you upgrade an add-on in dev before prod?
  • Which core add-ons are better left to EKS managed add-ons?

Practice: Lab 13 in Hands-on labs — Terraform a small EKS cluster, enable Argo CD (capability or self-managed), sync an app with External Secrets and an ALB Ingress, then destroy.

Capstone

One service from commit to production

Everything in the deck in one pipeline. Build it after Lab 13, on kind or on EKS.

Developergit push app repoCI (GitHub Actions)test · build · cosign signAmazon ECRimage + OCI chartPromotion PRbump tag in envs/devConfig repo gitops-labcharts/ apps/ envs/ clusters/Argo CD (hub)ApplicationSet per envdevclusterstagingclusterprodclusterAWS Secrets Managersource of secretsArgo Rollouts canaryPrometheus AnalysisTemplatepull + reconcileExternal SecretsCI never touches a cluster: its last step is a pull request.

Promotion is a Git change: dev → staging → prod are three small PRs to envs/<env>/values.yaml, each merged only after the previous environment is Healthy.

Capstone

Build order and acceptance checklist

Done means every box is ticked and proven with a command, not a screenshot.

Build order

  • 1 · App repo: podinfo fork, Dockerfile, unit tests.
  • 2 · CI: build, push to ECR, cosign sign, helm package + helm push oci://….
  • 3 · CI last step: open a PR that sets image.tag in envs/dev/values.yaml.
  • 4 · Hub Argo CD: one ApplicationSet (Git files generator over envs/*/config.yaml).
  • 5 · Rollout + AnalysisTemplate in the chart (canary 20% → 60% → 100%).
  • 6 · ExternalSecret + ClusterSecretStore for the DB password.
  • 7 · Admission policy that only allows signed images from your ECR.

Acceptance

  • A merged PR reaches dev with no manual step (argocd app wait web-dev --health).
  • Promotion to prod is a reviewed PR; prod has a sync window outside office hours.
  • A bad release (--random-error) aborts the canary automatically.
  • kubectl edit in prod is reverted by self-heal within one reconcile.
  • No secret value exists in Git; rotating it in Secrets Manager updates the pod.
  • An unsigned image is rejected at admission.
  • Rollback = git revert, measured end to end in minutes.

Stretch goals

  • Preview environment per PR with the Pull Request generator.
  • Kargo stages instead of hand-written promotion PRs.
  • Source Hydrator: commit rendered manifests to an env/* branch.
  • Grafana dashboard + alert on argocd_app_info{health_status="Degraded"}.
  • DR drill: delete the hub cluster, re-bootstrap from clusters/ only.
Cheat sheet

Helm 4 commands

Release name first, chart second. -n everywhere: releases are namespaced.

Lifecycle

CommandWhat it does
helm upgrade --install web ./charts/web -n demo --create-namespaceIdempotent install or upgrade
helm upgrade … --rollback-on-failure --timeout 5mWait, roll back if not ready (Helm 3: --atomic)
helm upgrade … --server-sideForce Server-Side Apply on an old release
helm rollback web 3 -n demoRe-apply revision 3 as a new revision
helm history web -n demoRevisions, status, chart and app version
helm uninstall web -n demo --keep-historyRemove resources, keep release records
helm test web -n demoRun pods annotated helm.sh/hook: test

Inspect, render, distribute

CommandWhat it does
helm template web ./charts/web -f dev.yamlRender locally, no cluster
helm install … --dry-run=serverRender + validate against the API server
helm get values|manifest|hooks webWhat the release actually used
helm show values repo/chartDefault values of a chart
helm dependency update ./charts/webFetch deps, write Chart.lock
helm package ./charts/web / helm push x.tgz oci://reg/chartsPublish to an OCI registry
helm lint --strict / helm registry login reg.example.comLint; log in (host only, no URL)
Cheat sheet

Template functions you will use every day

Pipelines pass the left value as the last argument: .Values.x | default "a" = default "a" .Values.x.

FunctionExampleNote
include + nindent{{- include "web.labels" . | nindent 4 }}Use instead of template (pipeable)
toYaml{{- toYaml .Values.resources | nindent 12 }}Dump a values subtree
default{{ .Values.image.tag | default .Chart.AppVersion }}Empty string, 0, false also count as empty
required{{ required "host is required" .Values.host }}Fail render with a message
quote / squotevalue: {{ .Values.flag | quote }}Keep strings strings ("true", "0123")
tpl{{ tpl .Values.extraConfig . }}Render a template stored in values
lookup(lookup "v1" "Secret" .Release.Namespace "db")Empty under helm template and Argo CD
sha256sumchecksum/config: {{ include (print $.Template.BasePath "/cm.yaml") . | sha256sum }}Roll pods on config change
with / range / ${{- range $k, $v := .Values.env }} … {{ $.Release.Name }}. changes inside; $ is root
fail{{- if and .Values.a .Values.b }}{{ fail "a and b are exclusive" }}{{ end }}Guard invalid combinations
Cheat sheet

argocd CLI

The CLI talks to the API server: log in first (argocd login), or use --core to talk to Kubernetes directly.

Applications

CommandWhat it does
argocd app list / app get web-devStatus, health, resources
argocd app get web-dev --hard-refreshRe-render, bypass the manifest cache
argocd app diff web-devLive vs desired (exit 1 if different)
argocd app sync web-dev [--prune] [--resource :Deployment:web]Sync all or one resource
argocd app wait web-dev --health --timeout 300Block until Healthy (CI gates)
argocd app manifests web-dev [--source live]Rendered or live manifests
argocd app history web-dev / app rollback web-dev 3Rollback needs auto-sync off
argocd app terminate-op web-devStop a stuck sync operation

Platform

CommandWhat it does
argocd login localhost:8080 --insecureLog in through a port-forward
argocd account update-password [--account alice]Change a local password
argocd repo add|listRepositories and credentials
argocd cluster add <ctx> --name prod / cluster listRegister a cluster
argocd proj create|get|roleAppProjects and project roles
argocd appset list|get|createApplicationSets
argocd admin settings rbac can alice sync applications 'team-a/*' --policy-file policy.csvTest an RBAC rule offline (without --policy-file it reads argocd-rbac-cm from the cluster)
argocd admin export > backup.yamlBack up apps, projects, settings
Cheat sheet

Application and AppProject skeletons

Copy, then delete what you do not need. Both live in the argocd namespace.

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: web-prod
  namespace: argocd
  finalizers:                    # cascade delete
    - resources-finalizer.argocd.argoproj.io
spec:
  project: team-a
  source:
    repoURL: https://github.com/<you>/gitops-lab
    targetRevision: main         # branch, tag or SHA
    path: charts/web
    helm:
      valueFiles: [values.yaml]
      valuesObject: {replicaCount: 3}
  destination:
    server: https://kubernetes.default.svc
    namespace: prod
  syncPolicy:
    automated: {prune: true, selfHeal: true}
    syncOptions: [CreateNamespace=true, ServerSideApply=true]
    retry: {limit: 5, backoff: {duration: 10s, factor: 2}}
  ignoreDifferences:
    - {group: apps, kind: Deployment, jsonPointers: [/spec/replicas]}
apiVersion: argoproj.io/v1alpha1
kind: AppProject
metadata:
  name: team-a
  namespace: argocd
spec:
  description: Team A services
  sourceRepos:
    - https://github.com/<you>/gitops-lab
  destinations:
    - server: https://kubernetes.default.svc
      namespace: team-a
  clusterResourceWhitelist: []   # no cluster-scoped objects
  namespaceResourceBlacklist:
    - {group: "", kind: ResourceQuota}
  roles:
    - name: ci
      policies:
        - p, proj:team-a:ci, applications, sync, team-a/*, allow
  syncWindows:
    - kind: deny
      schedule: "0 18 * * 5"     # Friday 18:00
      duration: 64h
      applications: ["*"]
Cheat sheet

ApplicationSet skeleton: one app per environment directory

Go templates (goTemplate: true) with missingkey=error turn typos into errors instead of empty strings.

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: web
  namespace: argocd
spec:
  goTemplate: true
  goTemplateOptions: ["missingkey=error"]
  generators:
    - git:
        repoURL: https://github.com/<you>/gitops-lab
        revision: main
        files:
          - path: "envs/*/config.yaml"   # keys become params
  syncPolicy:
    preserveResourcesOnDeletion: true
  template:
    metadata:
      name: 'web-{{.path.basename}}'
    spec:
      project: default
      sources:
        - repoURL: https://github.com/<you>/gitops-lab
          targetRevision: main
          path: charts/web
          helm:
            valueFiles:
              - '$values/envs/{{.path.basename}}/values.yaml'
        - repoURL: https://github.com/<you>/gitops-lab
          targetRevision: main
          ref: values                    # makes $values
      destination:
        server: '{{.server}}'            # from config.yaml
        namespace: '{{.path.basename}}'
      syncPolicy:
        automated: {prune: true, selfHeal: true}
        syncOptions: [CreateNamespace=true]

Other generators: list, clusters (label selector on cluster Secrets), matrix, merge, scmProvider, pullRequest.

Cheat sheet

Sync options and annotations

Sync options go in spec.syncPolicy.syncOptions (whole app) or the argocd.argoproj.io/sync-options annotation (one resource).

Sync options

OptionEffect
CreateNamespace=trueCreate the destination namespace
ServerSideApply=trueApply with SSA (large CRDs, field ownership)
PruneLast=truePrune after all other resources are applied
ApplyOutOfSyncOnly=trueApply only changed resources (big apps)
RespectIgnoreDifferences=trueDo not overwrite ignored fields on sync
Replace=truekubectl replace/create instead of apply
Prune=false (annotation)Never prune this resource

Resource annotations (argocd.argoproj.io/…)

AnnotationEffect
sync-wave: "-1"Order within a phase (lower first)
hook: PreSync|Sync|PostSync|SyncFail|PreDeleteRun as a hook in that phase
hook-delete-policy: HookSucceeded|HookFailed|BeforeHookCreationWhen to delete the hook
compare-options: IgnoreExtraneousIgnore for sync status
sync-options: Delete=falseKeep the object when the app is deleted
manifest-generate-paths: .Only refresh when these paths change (monorepos)
Lab setup

Before you start

Labs 0–12 run free on a laptop with kind. Lab 13 is the only one that costs money.

Laptop and accounts

  • 8 GB RAM free for Docker (16 GB machine), 4 CPUs; two kind clusters in Lab 9.
  • Docker Desktop, OrbStack or colima running.
  • A GitHub account and gh auth login done (repo + PR labs).
  • Lab 13 only: an AWS sandbox account, AWS CLI v2, eksctl, a budget alarm.

Conventions used in every lab

  • Hub cluster kind-gitops; second cluster kind-prod (Lab 9).
  • Config repo https://github.com/<you>/gitops-lab: charts/web, apps/, envs/{dev,staging,prod}/, clusters/.
  • Demo app ghcr.io/stefanprodan/podinfo (port 9898); chart web.
  • Argo CD in namespace argocd, reached with a port-forward on localhost:8080.
  • export REPO=https://github.com/<you>/gitops-lab in every shell.
brew install kind kubectl helm kustomize \
  argocd kubeconform chart-testing \
  yq jq gh kubeseal
brew install \
  argoproj/tap/kubectl-argo-rollouts

# Lab 13 only
brew install awscli eksctl

helm version            # v4.x
argocd version --client # v3.5.x
kind version
kubectl version --client
kustomize version
kubeconform -v

Each lab = Build, Verify, Break it, Clean up. Commit everything you write to gitops-lab: later labs build on earlier ones (Lab 3 chart → Labs 4–12).

Lab 0 · Build

Setup: hub cluster, config repo, versions

Goal: a two-node kind cluster and an empty config repo that every later lab commits to.

# kind-gitops.yaml
cat > kind-gitops.yaml <<'EOF'
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
name: gitops
nodes:
  - role: control-plane
  - role: worker
EOF
kind create cluster --config kind-gitops.yaml
kubectl config current-context    # kind-gitops
kubectl get nodes -o wide         # 2 nodes Ready

# helper used in every lab
cat >> ~/.zshrc <<'EOF'
alias k=kubectl
export REPO=https://github.com/<you>/gitops-lab
EOF
source ~/.zshrc
gh repo create gitops-lab --public --clone
cd gitops-lab
mkdir -p charts apps clusters envs/{dev,staging,prod}
for d in apps clusters envs/dev envs/staging envs/prod; do
  touch $d/.gitkeep; done
cat > README.md <<'EOF'
# gitops-lab
charts/   Helm charts (web, common)
apps/     Argo CD Application manifests and raw YAML
envs/     per-environment values (dev, staging, prod)
clusters/ bootstrap manifests per cluster
EOF
git add . && git commit -m "repo skeleton"
git push -u origin main

# record versions in the repo
{ helm version; argocd version --client --short
  kubectl version --client; kind version; } > versions.txt
git add versions.txt && git commit -m versions && git push
Lab 0 · Verify, break it, clean up

Know which cluster you are talking to

The most common lab mistake is a command sent to the wrong context.

Verify

  • kubectl get nodes shows gitops-control-plane and gitops-worker Ready.
  • gh repo view <you>/gitops-lab --web opens the repo with the skeleton.
  • helm version prints v4; argocd version --client prints v3.5.
  • docker ps shows two kindest/node containers.

Break it

  • Wrong context: kubectl config use-context docker-desktop (or any other) → kubectl get nodes shows other nodes. Fix: kubectl config use-context kind-gitops; add the context to your prompt.
  • Runtime down: quit Docker → kubectl times out, kind get clusters errors. Start Docker; kind containers restart with it.
  • Why do later labs pass --context explicitly?
# keep the cluster: Labs 1-12 use it
# full reset at the very end:
kind delete clusters --all
gh repo delete <you>/gitops-lab \
  --yes
Lab 1 · Build

Push vs pull: the same app two ways

Goal: feel the difference between a script that pushes YAML and an agent that keeps pulling it.

# apps/podinfo-raw/podinfo.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: podinfo
spec:
  replicas: 2
  selector:
    matchLabels: {app: podinfo}
  template:
    metadata:
      labels: {app: podinfo}
    spec:
      containers:
        - name: podinfo
          image: ghcr.io/stefanprodan/podinfo:6.7.0
          ports: [{containerPort: 9898}]
---
apiVersion: v1
kind: Service
metadata:
  name: podinfo
spec:
  selector: {app: podinfo}
  ports: [{port: 9898}]
# PUSH: what a CI job with cluster-admin would do
kubectl create namespace push-demo
kubectl -n push-demo apply -f apps/podinfo-raw/
git add apps/podinfo-raw && git commit -m podinfo && git push

# PULL: a quick Argo CD (reinstalled properly in Lab 6)
helm repo add argo https://argoproj.github.io/argo-helm
helm install argocd argo/argo-cd -n argocd \
  --create-namespace --wait
kubectl -n argocd port-forward svc/argocd-server \
  8080:443 >/dev/null 2>&1 &
PW=$(kubectl -n argocd get secret \
  argocd-initial-admin-secret \
  -o jsonpath='{.data.password}' | base64 -d)
argocd login localhost:8080 --username admin \
  --password "$PW" --insecure
argocd app create podinfo --repo $REPO \
  --path apps/podinfo-raw \
  --dest-server https://kubernetes.default.svc \
  --dest-namespace pull-demo \
  --sync-policy automated --self-heal --auto-prune \
  --sync-option CreateNamespace=true
Lab 1 · Verify, break it, clean up

Drift: who notices, who fixes it?

Make the same manual change in both namespaces and watch what happens.

Verify

  • kubectl get deploy -A | grep podinfo: 2/2 in push-demo and pull-demo.
  • argocd app get podinfo: Sync Status Synced, Health Healthy.
  • Open https://localhost:8080 and find the Deployment → ReplicaSet → Pods tree.
  • Which credentials did the push need, and where would a CI system keep them?

Break it

  • Drift: kubectl -n push-demo scale deploy podinfo --replicas=5 → stays 5 forever; nobody knows.
  • Same in pull-demo → back to 2 within seconds (selfHeal); argocd app history podinfo shows nothing new because Git did not change.
  • Delete: kubectl -n pull-demo delete deploy podinfo → recreated. Then set replicas: 3 in Git, push: only pull-demo follows (within ~3 min, or click Refresh).
argocd app delete podinfo --yes
kubectl delete ns push-demo
helm uninstall argocd -n argocd
kubectl delete ns argocd
pkill -f "port-forward svc/argocd"
# keep apps/podinfo-raw in Git
Lab 2 · Build

Helm release lifecycle with a public chart

Goal: install, upgrade, roll back, and see exactly where Helm keeps release state.

helm repo add podinfo https://stefanprodan.github.io/podinfo
helm repo update
helm show values podinfo/podinfo | head -40

helm install web podinfo/podinfo -n demo \
  --create-namespace --set replicaCount=2
helm upgrade web podinfo/podinfo -n demo \
  --reuse-values --set ui.message="hello v2"
helm upgrade web podinfo/podinfo -n demo \
  --reuse-values --set ui.color="#34577c"
helm history web -n demo           # revisions 1-3

helm rollback web 2 -n demo        # → revision 4 = copy of 2
helm history web -n demo

helm get values web -n demo        # values you supplied
helm get values web -n demo --all  # merged with defaults
helm get manifest web -n demo | head -30
# release state = one Secret per revision
kubectl -n demo get secret -l owner=helm
# sh.helm.release.v1.web.v1 ... v4   helm.sh/release.v1

# decode: base64 (Kubernetes) -> base64 (Helm) -> gzip -> JSON
kubectl -n demo get secret sh.helm.release.v1.web.v4 \
  -o jsonpath='{.data.release}' \
  | base64 -d | base64 -d | gunzip \
  | jq '{name, version, status: .info.status,
         chart: .chart.metadata.version}'

# render without touching the cluster
helm template web podinfo/podinfo --set replicaCount=3 \
  | grep -B2 -A6 "kind: Deployment"

# validate against the API server, change nothing
helm upgrade web podinfo/podinfo -n demo --reuse-values \
  --set replicaCount=3 --dry-run=server | head -20

kubectl -n demo port-forward svc/web-podinfo 9898:9898 &
curl -s localhost:9898 | jq .message
Lab 2 · Verify, break it, clean up

A failed upgrade, recovered automatically

Helm 4 renamed --atomic to --rollback-on-failure; the old flag still works with a warning.

Verify

  • helm history web -n demo shows 4 revisions; revision 4 description "Rollback to 2".
  • curl localhost:9898 shows "hello v2" (from revision 2), not the color from 3.
  • Four sh.helm.release.v1.web.v* Secrets exist; --history-max (default 10) caps them.
  • Why is a rollback a new revision instead of moving back to an old one?

Break it

  • Bad image: helm upgrade web podinfo/podinfo -n demo --reuse-values --set image.tag=0.0.0-missing --rollback-on-failure --timeout 90s → pods ImagePullBackOff; after 90 s Helm rolls back; history shows failed then a new deployed.
  • Without the flag: same command minus --rollback-on-failure → release status failed, broken pods stay. Recover: helm rollback web -n demo (previous revision).
  • Pending: Ctrl-C an upgrade mid-way → pending-upgrade blocks the next upgrade; fix with helm rollback.
pkill -f "port-forward svc/web-podinfo"
helm uninstall web -n demo
kubectl delete ns demo
helm repo remove podinfo
Lab 3 · Build (1/2)

Write the web chart from scratch (1/2): metadata, values, schema

Goal: a small production-shaped chart you fully understand. Every later lab deploys it.

mkdir -p charts/web/templates
# charts/web/Chart.yaml
apiVersion: v2
name: web
description: podinfo-based web service
type: application
version: 0.1.0          # chart version: bump on every change
appVersion: "6.7.0"     # default image tag

# charts/web/values.yaml
replicaCount: 2
image:
  repository: ghcr.io/stefanprodan/podinfo
  tag: ""               # empty -> .Chart.AppVersion
service:
  port: 9898
message: "hello from web"
resources:
  requests: {cpu: 50m, memory: 64Mi}
  limits: {memory: 128Mi}
# charts/web/values.schema.json
{
  "$schema": "https://json-schema.org/draft-07/schema",
  "type": "object",
  "required": ["replicaCount", "image", "service"],
  "properties": {
    "replicaCount": {"type": "integer", "minimum": 1},
    "image": {
      "type": "object",
      "required": ["repository"],
      "properties": {
        "repository": {"type": "string"},
        "tag": {"type": "string"}
      }
    },
    "service": {
      "type": "object",
      "properties": {"port": {"type": "integer"}}
    },
    "message": {"type": "string"},
    "resources": {"type": "object"}
  }
}
Lab 3 · Build (2/2)

Write the web chart from scratch (2/2): templates

Helpers keep names and labels identical everywhere; the checksum rolls pods when the ConfigMap changes.

{{/* charts/web/templates/_helpers.tpl */}}
{{- define "web.fullname" -}}
{{- if contains .Chart.Name .Release.Name -}}
{{- .Release.Name | trunc 63 | trimSuffix "-" -}}
{{- else -}}
{{- printf "%s-%s" .Release.Name .Chart.Name
    | trunc 63 | trimSuffix "-" -}}
{{- end -}}
{{- end -}}
{{- define "web.selectorLabels" -}}
app.kubernetes.io/name: {{ .Chart.Name }}
app.kubernetes.io/instance: {{ .Release.Name }}
{{- end -}}
{{- define "web.labels" -}}
{{ include "web.selectorLabels" . }}
app.kubernetes.io/version: {{ .Chart.AppVersion | quote }}
helm.sh/chart: {{ printf "%s-%s" .Chart.Name .Chart.Version }}
{{- end -}}

{{/* templates/configmap.yaml */}}
apiVersion: v1
kind: ConfigMap
metadata:
  name: {{ include "web.fullname" . }}
  labels: {{- include "web.labels" . | nindent 4 }}
data:
  PODINFO_UI_MESSAGE: {{ .Values.message | quote }}
{{/* templates/deployment.yaml */}}
apiVersion: apps/v1
kind: Deployment
metadata:
  name: {{ include "web.fullname" . }}
  labels: {{- include "web.labels" . | nindent 4 }}
spec:
  replicas: {{ .Values.replicaCount }}
  selector:
    matchLabels: {{- include "web.selectorLabels" . | nindent 6 }}
  template:
    metadata:
      labels: {{- include "web.selectorLabels" . | nindent 8 }}
      annotations:
        checksum/config: {{ include (print $.Template.BasePath
          "/configmap.yaml") . | sha256sum }}
    spec:
      containers:
        - name: web
          image: "{{ .Values.image.repository }}:{{
            .Values.image.tag | default .Chart.AppVersion }}"
          ports: [{name: http, containerPort: 9898}]
          envFrom:
            - configMapRef:
                name: {{ include "web.fullname" . }}
          resources: {{- toYaml .Values.resources | nindent 12 }}
{{/* templates/service.yaml: same metadata; spec.selector =
     web.selectorLabels (nindent 4); port .Values.service.port,
     targetPort: http */}}
Lab 3 · Verify, break it, clean up

Lint, render, install, then break it twice

Read the error messages slowly: they name the file and the line.

Verify

  • helm lint charts/web --strict → 0 failures.
  • helm template web charts/web | kubeconform -strict -summary → all valid.
  • helm install web charts/web -n demo --create-namespace; kubectl -n demo get deploy web 2/2.
  • Change message, helm upgrade web charts/web -n demo → pods roll (kubectl -n demo rollout status deploy/web) because the checksum changed.
  • Commit charts/web and push: Labs 4–12 use it.

Break it

  • Indentation: change the pod labels nindent 8 to nindent 4 → helm template fails with a YAML parse error on web/templates/deployment.yaml. helm template --debug prints the broken YAML.
  • Schema: helm upgrade web charts/web -n demo --set replicaCount=two → "values don't meet the specifications of the schema"; --set replicaCount=0 fails minimum.
  • Hidden bug: remove | quote from the message and set message: "true" → ConfigMap rejected: data values must be strings.
helm uninstall web -n demo
kubectl delete ns demo
git add charts/web
git commit -m "web chart v0.1.0"
git push
Lab 4 · Build (1/2)

Production chart: library chart, hooks, tests

Goal: share labels through a library chart, run a migration before every upgrade, and test the release.

mkdir -p charts/common/templates
cat > charts/common/Chart.yaml <<'EOF'
apiVersion: v2
name: common
type: library          # no resources of its own
version: 0.1.0
EOF
cat > charts/common/templates/_labels.tpl <<'EOF'
{{- define "common.labels" -}}
app.kubernetes.io/name: {{ .Chart.Name }}
app.kubernetes.io/instance: {{ .Release.Name }}
app.kubernetes.io/managed-by: {{ .Release.Service }}
{{- end -}}
EOF
# append to charts/web/Chart.yaml, bump version: 0.2.0
dependencies:
  - name: common
    version: 0.1.0
    repository: file://../common
helm dependency update charts/web   # charts/ + Chart.lock
# templates/tests/test-connection.yaml
apiVersion: v1
kind: Pod
metadata:
  name: {{ include "web.fullname" . }}-test
  annotations: {"helm.sh/hook": test}
spec:
  restartPolicy: Never
  containers: [{name: curl, image: "curlimages/curl:8.10.1",
    args: ["-fsS", "http://{{ include "web.fullname" . }}:9898/healthz"]}]
# templates/migrate-job.yaml
apiVersion: batch/v1
kind: Job
metadata:
  name: {{ include "web.fullname" . }}-migrate
  annotations:
    "helm.sh/hook": pre-install,pre-upgrade
    "helm.sh/hook-weight": "-5"
    "helm.sh/hook-delete-policy": before-hook-creation,hook-succeeded
spec:
  backoffLimit: 0
  template:
    spec:
      restartPolicy: Never
      containers:
        - name: migrate
          image: busybox:1.36
          command: ["sh", "-c", "echo migrating; sleep 5"]

# local OCI registry + publish
docker run -d --name registry -p 5001:5000 registry:2
helm package charts/web -d dist/        # web-0.2.0.tgz
helm push dist/web-0.2.0.tgz oci://localhost:5001/charts \
  --plain-http
helm show chart oci://localhost:5001/charts/web \
  --version 0.2.0 --plain-http
Lab 4 · Build (2/2)

Production chart: the quality pipeline

One script that CI runs on every pull request touching charts/.

# charts/web/tests/deployment_test.yaml  (helm-unittest)
suite: deployment
templates:
  - templates/deployment.yaml
  - templates/configmap.yaml
tests:
  - it: defaults the image tag to appVersion
    template: templates/deployment.yaml
    asserts:
      - equal:
          path: spec.template.spec.containers[0].image
          value: ghcr.io/stefanprodan/podinfo:6.7.0
  - it: honours replicaCount
    template: templates/deployment.yaml
    set: {replicaCount: 4}
    asserts:
      - equal: {path: spec.replicas, value: 4}
# scripts/chart-ci.sh
#!/usr/bin/env bash
set -euo pipefail
CHART=charts/web
helm dependency build $CHART
helm lint $CHART --strict
helm template web $CHART \
  | kubeconform -strict -summary -kubernetes-version 1.33.0
helm unittest $CHART
ct lint --charts $CHART --validate-maintainers=false
echo "chart OK"

# one-time setup
helm plugin install --verify=false \
  https://github.com/helm-unittest/helm-unittest
# Helm 4 verifies plugin provenance; git plugins need --verify=false
chmod +x scripts/chart-ci.sh && ./scripts/chart-ci.sh

ct lint compares against the Git default branch, so commit first. Install ct with brew install chart-testing.

Lab 4 · Verify, break it, clean up

Hooks, tests and immutable versions

Watch the order: the migration Job finishes before the Deployment changes.

Verify

  • helm upgrade --install web charts/web -n demo --create-namespace while kubectl -n demo get jobs,pods -w → web-migrate completes first, then is deleted (hook-succeeded).
  • helm test web -n demo → Phase: Succeeded.
  • helm pull oci://localhost:5001/charts/web --version 0.2.0 --plain-http downloads the package.
  • ./scripts/chart-ci.sh prints "chart OK".

Break it

  • Failing hook: change the migrate command to exit 1, upgrade → "pre-upgrade hooks failed"; the Deployment is unchanged. Fix, bump version, upgrade.
  • Same version, new content: edit a template, helm package + push 0.2.0 again → registry:2 silently overwrites it. With ECR tag immutability the push is rejected. Rule: never reuse a chart version.
  • Unit test: set appVersion: "6.7.1" → the first test fails with expected/actual values.
helm uninstall web -n demo
kubectl delete ns demo
docker rm -f registry
rm -rf dist
git add charts scripts
git commit -m "web 0.2.0"
git push
Lab 5 · Build

Kustomize overlays on top of the chart

Goal: render once with Helm, then tweak per environment with Kustomize, and diff the environments.

mkdir -p kustomize/{base,dev,prod,helm}
helm template web charts/web --skip-tests \
  > kustomize/base/web.yaml
cat > kustomize/base/kustomization.yaml <<'EOF'
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources: [web.yaml]
EOF
cat > kustomize/dev/kustomization.yaml <<'EOF'
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
namespace: dev
resources: [../base]
replicas: [{name: web, count: 1}]
EOF
cat > kustomize/prod/kustomization.yaml <<'EOF'
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
namespace: prod
resources: [../base]
replicas: [{name: web, count: 3}]
images:
  - name: ghcr.io/stefanprodan/podinfo
    newTag: 6.7.1
patches:
  - path: limits.yaml
EOF
cat > kustomize/prod/limits.yaml <<'EOF'
apiVersion: apps/v1
kind: Deployment
metadata:
  name: web
spec:
  template:
    spec:
      containers:
        - name: web
          resources:
            limits: {memory: 256Mi}
EOF
kustomize build kustomize/dev | kubeconform -strict -summary
diff <(kustomize build kustomize/dev) \
     <(kustomize build kustomize/prod)

# alternative: let Kustomize inflate the chart itself
cat > kustomize/helm/kustomization.yaml <<'EOF'
helmGlobals:
  chartHome: ../../charts
helmCharts:
  - name: web
    releaseName: web
    namespace: demo
    valuesInline: {replicaCount: 2, message: "via kustomize"}
EOF
kustomize build --enable-helm kustomize/helm | head -20
Lab 5 · Verify, break it, clean up

Environment diffs you can review

The diff between dev and prod should be short and boring. If it is long, the overlays are doing too much.

Verify

  • The diff shows only: namespace, replicas, image tag, memory limit.
  • kubectl create ns dev && kubectl apply -k kustomize/dev → kubectl -n dev get deploy web 1/1.
  • The ConfigMap name is unchanged (no hash) because it came from the chart, not a generator.
  • When would you drop the Helm step and write the base by hand?

Break it

  • Patch misses: rename metadata.name in limits.yaml to webapp → kustomize build fails: no resource matches the patch. Fix the name.
  • Forgotten namespace: remove namespace: dev from the overlay → objects have no namespace and land in your current namespace (default). Always set it in the overlay.
  • Stale base: change charts/web/values.yaml → the base does not change until you re-run helm template. Lesson: render in CI, or use helmCharts.
kubectl delete -k kustomize/dev
kubectl delete ns dev
git add kustomize
git commit -m "kustomize overlays"
git push
Lab 6 · Build

Install Argo CD properly and deploy the chart

Goal: Argo CD from its Helm chart with a values file, declarative repo, project and Application.

# clusters/gitops/argocd-values.yaml
configs:
  cm:
    timeout.reconciliation: 180s   # Git poll interval
  params:
    server.insecure: false
dex:
  enabled: false                   # SSO comes in Lab 10
notifications:
  enabled: true

helm repo add argo https://argoproj.github.io/argo-helm
helm upgrade --install argocd argo/argo-cd -n argocd \
  --create-namespace -f clusters/gitops/argocd-values.yaml \
  --wait
kubectl -n argocd get deploy,sts
kubectl -n argocd port-forward svc/argocd-server \
  8080:443 >/dev/null 2>&1 &
argocd login localhost:8080 --username admin --insecure \
  --password "$(kubectl -n argocd get secret \
  argocd-initial-admin-secret \
  -o jsonpath='{.data.password}' | base64 -d)"
argocd account update-password
kubectl apply -f clusters/gitops/bootstrap.yaml
# clusters/gitops/bootstrap.yaml
apiVersion: v1
kind: Secret
metadata:
  name: gitops-lab
  namespace: argocd
  labels: {argocd.argoproj.io/secret-type: repository}
stringData:   # private repo: add username + read-only token
  {type: git, url: "https://github.com/<you>/gitops-lab"}
---
apiVersion: argoproj.io/v1alpha1
kind: AppProject
metadata: {name: lab, namespace: argocd}
spec:
  sourceRepos: [https://github.com/<you>/gitops-lab]
  destinations: [{server: https://kubernetes.default.svc, namespace: dev}]
---
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata: {name: web-dev, namespace: argocd}
spec:
  project: lab
  source:
    repoURL: https://github.com/<you>/gitops-lab
    targetRevision: main
    path: charts/web
    helm: {valuesObject: {replicaCount: 1, message: "dev"}}
  destination: {server: https://kubernetes.default.svc, namespace: dev}
  syncPolicy: {automated: {prune: true, selfHeal: true},
               syncOptions: [CreateNamespace=true]}
Lab 6 · Verify, break it, clean up

Read the UI, then hit the project guard-rails

The AppProject is what stops an Application from deploying anywhere it likes.

Verify

  • argocd app get web-dev → Synced, Healthy; the Helm pre-install Job ran as a PreSync hook.
  • kubectl -n argocd get secret -l argocd.argoproj.io/secret-type=repository lists the repo.
  • UI: open web-dev, click the Deployment → Live manifest, Diff, Events tabs.
  • kubectl -n argocd get pods: name each pod and its job (Chapter 6).

Break it

  • Destination not allowed: set the Application destination.namespace: prod and apply → condition "InvalidSpecError": the destination does not match any allowed destination in project lab. Fix: revert, or add the destination to the project.
  • Bad repo URL: typo in repoURL → ComparisonError / repository not accessible; argocd app get web-dev shows it under Conditions.
  • Unknown path: path: charts/webb → "app path does not exist".
# keep Argo CD and web-dev for Lab 7+
git add clusters/gitops
git commit -m "argocd bootstrap"
git push
# never commit a token in
# bootstrap.yaml
Lab 7 · Build

Sync mechanics: waves and hooks

Goal: see the exact order Argo CD applies things in, and what a failing hook does to a sync.

# apps/sync-demo/00-config.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: app-config
  annotations: {argocd.argoproj.io/sync-wave: "-1"}
data: {MODE: demo}
---
# apps/sync-demo/10-migrate.yaml
apiVersion: batch/v1
kind: Job
metadata:
  name: migrate
  annotations:
    argocd.argoproj.io/hook: PreSync
    argocd.argoproj.io/hook-delete-policy: BeforeHookCreation
spec:
  backoffLimit: 0
  template:
    spec:
      restartPolicy: Never
      containers:
        - name: m
          image: busybox:1.36
          command: ["sh", "-c", "echo migrate; sleep 5"]
# 20-web.yaml: Deployment + Service "web" (podinfo, replicas: 1)
#   with annotation argocd.argoproj.io/sync-wave: "1"
# 30-smoke.yaml: Job "smoke", hook PostSync,
#   hook-delete-policy HookSucceeded, curlimages/curl
#   args ["-fsS", "http://web:9898/healthz"]
# metrics-server so the HPA in "Break it" can work on kind
helm repo add metrics-server \
  https://kubernetes-sigs.github.io/metrics-server/
helm install metrics-server metrics-server/metrics-server \
  -n kube-system --set args={--kubelet-insecure-tls}

git add apps/sync-demo && git commit -m sync-demo && git push
argocd app create sync-demo --project default --repo $REPO \
  --path apps/sync-demo \
  --dest-server https://kubernetes.default.svc \
  --dest-namespace sync-demo \
  --sync-option CreateNamespace=true
argocd app sync sync-demo

# watch the order in a second terminal
kubectl -n sync-demo get jobs,cm,deploy,pods -w
# PreSync: migrate -> wave -1: app-config
#   -> wave 1: web -> PostSync: smoke
argocd app get sync-demo --show-operation
Lab 7 · Verify, break it, clean up

A failing hook, then a fight with the HPA

Two classic production problems, reproduced in five minutes.

Verify

  • argocd app get sync-demo --show-operation lists phases and waves in order.
  • kubectl -n sync-demo get jobs: migrate stays (BeforeHookCreation); smoke is gone (HookSucceeded).
  • UI → Sync status → the last operation shows each resource with its phase and wave.

Break it

  • Failing hook: set the migrate command to exit 1, push, argocd app sync sync-demo → operation Failed, wave 1 never applied. Fix, push, sync again (argocd app terminate-op if it hangs).
  • HPA fight: argocd app set sync-demo --sync-policy automated --self-heal; add an HPA (minReplicas: 2, target web) to Git → the app flips OutOfSync on spec.replicas as self-heal and the HPA fight.
  • Fix: add ignoreDifferences for /spec/replicas on Deployment web plus RespectIgnoreDifferences=true (in Git, not with kubectl patch).
argocd app delete sync-demo --yes
# keep metrics-server
git add apps/sync-demo
git commit -m "sync-demo fixed"
git push
Lab 8 · Build

App-of-apps, then an ApplicationSet

Goal: bootstrap many Applications from Git, then replace hand-written ones with a generator.

# apps/web-dev.yaml (from Lab 6, project default) and
# apps/web-staging.yaml (namespace staging, replicaCount 2)

# clusters/gitops/root.yaml
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata: {name: root, namespace: argocd}
spec:
  project: default
  source:
    repoURL: https://github.com/<you>/gitops-lab
    targetRevision: main
    path: apps            # top-level *.yaml only (not recursive)
  destination: {server: https://kubernetes.default.svc, namespace: argocd}
  syncPolicy: {automated: {prune: true, selfHeal: true}}
git add apps clusters && git commit -m root && git push
kubectl apply -f clusters/gitops/root.yaml
argocd app list     # root, web-dev, web-staging
# per-environment values for the ApplicationSet
printf 'replicaCount: 1\nmessage: dev\n' > envs/dev/values.yaml
printf 'replicaCount: 2\nmessage: staging\n' > envs/staging/values.yaml
printf 'replicaCount: 3\nmessage: prod\n' > envs/prod/values.yaml
# apps/web-appset.yaml (replaces web-dev / web-staging)
apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata: {name: web, namespace: argocd}
spec:
  goTemplate: true
  goTemplateOptions: ["missingkey=error"]
  generators:
    - git: {repoURL: "https://github.com/<you>/gitops-lab",
            revision: main, directories: [{path: envs/*}]}
  template:
    metadata: {name: 'web-{{.path.basename}}'}
    spec:
      project: default
      sources:
        - repoURL: https://github.com/<you>/gitops-lab
          targetRevision: main
          path: charts/web
          helm:
            valueFiles:
              - '$values/envs/{{.path.basename}}/values.yaml'
        - repoURL: https://github.com/<you>/gitops-lab
          targetRevision: main
          ref: values
      destination: {server: "https://kubernetes.default.svc",
                    namespace: '{{.path.basename}}'}
      syncPolicy: {automated: {prune: true, selfHeal: true},
                   syncOptions: [CreateNamespace=true]}
  syncPolicy: {preserveResourcesOnDeletion: true}
# git rm apps/web-dev.yaml apps/web-staging.yaml; add appset; push
Lab 8 · Verify, break it, clean up

Generators create, update and remove apps

Adding an environment is now a directory, not a YAML file per app.

Verify

  • argocd appset list; argocd app list → web-dev, web-staging, web-prod.
  • Add envs/qa/values.yaml, push → web-qa appears within one reconcile.
  • Remove envs/qa, push → web-qa Application is deleted, but its Deployment stays (preserveResourcesOnDeletion).
  • Optional: a pullRequest generator (GitHub token in a Secret) creates web-pr-<number> per open PR.

Break it

  • Template typo: change {{.path.basename}} to {{.path.basname}} → no apps change; kubectl -n argocd describe applicationset web shows an error condition (missingkey=error). Fix and push.
  • Name clash: apply the ApplicationSet before removing apps/web-dev.yaml → two owners fight over web-dev. Remove the hand-written file first.
  • Think: what would kubectl delete applicationset web do without preserveResourcesOnDeletion?
kubectl delete ns qa
# keep root and the web ApplicationSet
# for Lab 9
Lab 9 · Build

Two clusters: hub-and-spoke and promotion by PR

Goal: one Argo CD (in kind-gitops) deploys prod to a second cluster; promotion is a pull request.

kind create cluster --name prod
kubectl config use-context kind-gitops   # create switched it
# an identity for Argo CD inside prod
kubectl --context kind-prod -n kube-system create sa argocd-manager
kubectl --context kind-prod create clusterrolebinding \
  argocd-manager --clusterrole cluster-admin \
  --serviceaccount kube-system:argocd-manager
kubectl --context kind-prod -n kube-system apply -f - <<'EOF'
apiVersion: v1
kind: Secret
metadata:
  name: argocd-manager-token
  annotations: {kubernetes.io/service-account.name: argocd-manager}
type: kubernetes.io/service-account-token
EOF
TOKEN=$(kubectl --context kind-prod -n kube-system get secret \
  argocd-manager-token -o jsonpath='{.data.token}' | base64 -d)
# register (kind nodes share the "kind" Docker network;
# insecure is for kind only: real clusters use caData / IAM)
kubectl -n argocd apply -f - <<EOF
apiVersion: v1
kind: Secret
metadata: {name: cluster-prod, labels:
  {argocd.argoproj.io/secret-type: cluster, env: prod}}
stringData:
  name: prod
  server: https://prod-control-plane:6443
  config: '{"bearerToken":"$TOKEN","tlsClientConfig":{"insecure":true}}'
EOF
argocd cluster list
# one config.yaml per env: its keys become generator params
echo 'server: https://kubernetes.default.svc' > envs/dev/config.yaml
echo 'server: https://kubernetes.default.svc' > envs/staging/config.yaml
echo 'server: https://prod-control-plane:6443' > envs/prod/config.yaml
# in apps/web-appset.yaml switch the generator to files:
#   - git: {repoURL: ..., revision: main,
#           files: [{path: "envs/*/config.yaml"}]}
# and set destination.server: '{{.server}}'
git add envs apps && git commit -m "prod on its own cluster"
git push

# promotion = a reviewed change to one file
git switch -c promote-6.7.1
yq -i '.image.tag = "6.7.1"' envs/staging/values.yaml
git commit -am "staging: web 6.7.1" && git push -u origin HEAD
gh pr create --fill && gh pr merge --squash --delete-branch
git switch main && git pull
argocd app wait web-staging --health --timeout 300
# then the same PR for envs/prod/values.yaml

# rollback = revert the promotion commit
git revert --no-edit HEAD && git push
Lab 9 · Verify, break it, clean up

Prod on its own cluster, and why rollback is a Git operation

The hub never needs inbound access from prod; prod never needs Git credentials.

Verify

  • argocd cluster list → prod with status Successful.
  • kubectl --context kind-prod -n prod get deploy web → 3/3 replicas.
  • argocd app get web-prod → destination https://prod-control-plane:6443.
  • After the revert, web-staging runs 6.7.0 again and git log records who did it.

Break it

  • Unreachable server: register with the host kubeconfig address https://127.0.0.1:<port> → cluster connection fails (inside the hub pod 127.0.0.1 is itself). Fix: the internal name from kind get kubeconfig --name prod --internal.
  • Rollback with auto-sync: argocd app rollback web-prod 1 → refused: rollback cannot run while automated sync is enabled. That is the point: roll back in Git.
  • Promotion skipped staging: merge a prod PR while staging is Degraded. Which check (branch protection + argocd app wait) would have stopped it?
kubectl -n argocd delete secret \
  cluster-prod
kind delete cluster --name prod
# set envs/prod/config.yaml back to
# https://kubernetes.default.svc,
# commit and push
Lab 10 · Build

Teams, RBAC, AppProjects and sealed secrets

Goal: two teams share one Argo CD without touching each other, and secrets go to Git encrypted.

# clusters/gitops/team-values.yaml (merged into Argo CD)
configs:
  cm:
    accounts.alice: login
    accounts.bob: login
  rbac:
    policy.default: ""             # no access unless granted
    policy.csv: |
      p, role:team-a, applications, *, team-a/*, allow
      p, role:team-a, projects, get, team-a, allow
      p, role:team-b, applications, *, team-b/*, allow
      p, role:team-b, projects, get, team-b, allow
      g, alice, role:team-a
      g, bob, role:team-b
helm upgrade argocd argo/argo-cd -n argocd --reuse-values \
  -f clusters/gitops/team-values.yaml
kubectl create ns team-a && kubectl create ns team-b

# AppProject team-a (team-b: same with b)
apiVersion: argoproj.io/v1alpha1
kind: AppProject
metadata: {name: team-a, namespace: argocd}
spec:
  sourceRepos: [https://github.com/<you>/gitops-lab]
  destinations:
    - {server: https://kubernetes.default.svc, namespace: team-a}

argocd account update-password --account alice \
  --current-password "$ADMIN_PW" --new-password 'alice-lab-1'
helm repo add sealed-secrets \
  https://bitnami-labs.github.io/sealed-secrets
helm install sealed-secrets sealed-secrets/sealed-secrets \
  -n kube-system

mkdir -p apps/team-a
cp apps/podinfo-raw/podinfo.yaml apps/team-a/
kubectl create secret generic db -n team-a \
  --from-literal=password='s3cr3t' \
  --dry-run=client -o yaml \
  | kubeseal --controller-name sealed-secrets \
      --controller-namespace kube-system -o yaml \
  > apps/team-a/db-sealed.yaml
grep -c s3cr3t apps/team-a/db-sealed.yaml   # 0
git add apps/team-a && git commit -m team-a && git push

argocd app create team-a-web --project team-a --repo $REPO \
  --path apps/team-a \
  --dest-server https://kubernetes.default.svc \
  --dest-namespace team-a --sync-policy automated

# optional SSO: dex.config with a GitHub connector
#   (callback https://localhost:8080/api/dex/callback)
Lab 10 · Verify, break it, clean up

Prove the boundaries hold

Log in as each user and try to cross into the other team.

Verify

  • argocd login localhost:8080 --username alice --insecure → argocd app list shows only team-a-*.
  • kubectl -n team-a get secret db -o jsonpath='{.data.password}' | base64 -d → s3cr3t, decrypted in-cluster.
  • argocd admin settings rbac can bob sync applications 'team-a/*' --namespace argocd → No.
  • The SealedSecret in Git is safe to publish; the controller key is not (back it up).

Break it

  • Cross-namespace: as alice, create an app in project team-a with destination team-b → rejected: destination not permitted in project team-a.
  • Cross-team action: as bob, argocd app sync team-a-web → permission denied.
  • Moved secret: copy db-sealed.yaml into a team-b app → the controller cannot decrypt it (strict scope binds name + namespace). Re-seal for team-b.
argocd login localhost:8080 \
  --username admin --insecure
argocd app delete team-a-web --yes
kubectl delete ns team-a team-b
helm uninstall sealed-secrets \
  -n kube-system
# keep RBAC values for Lab 12
Lab 11 · Build (1/2)

Argo Rollouts canary with Prometheus analysis (1/2)

Goal: a canary that promotes itself when metrics are good and aborts itself when they are not.

helm install argo-rollouts argo/argo-rollouts \
  -n argo-rollouts --create-namespace
helm repo add prometheus-community \
  https://prometheus-community.github.io/helm-charts
helm install prom prometheus-community/prometheus \
  -n monitoring --create-namespace \
  --set alertmanager.enabled=false \
  --set prometheus-pushgateway.enabled=false
# server URL: http://prom-prometheus-server.monitoring.svc

# apps/rollout/service.yaml
apiVersion: v1
kind: Service
metadata: {name: web}
spec:
  selector: {app: web}
  ports: [{port: 9898}]

# load generator: 20 requests/s through the Service
kubectl create ns demo
kubectl -n demo run load --image=curlimages/curl:8.10.1 \
  --command -- sh -c \
  'while true; do curl -s -o /dev/null http://web:9898/; \
   sleep 0.05; done'
# apps/rollout/rollout.yaml
apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata: {name: web}
spec:
  replicas: 5
  selector: {matchLabels: {app: web}}
  template:
    metadata:
      labels: {app: web}
      annotations:
        prometheus.io/scrape: "true"
        prometheus.io/port: "9898"
    spec:
      containers:
        - name: web
          image: ghcr.io/stefanprodan/podinfo:6.7.0
          ports: [{containerPort: 9898}]
  strategy:
    canary:
      analysis:
        templates: [{templateName: success-rate}]
        startingStep: 1          # start after the first setWeight
      steps:
        - setWeight: 20
        - pause: {duration: 2m}
        - setWeight: 60
        - pause: {duration: 2m}
Lab 11 · Build (2/2)

Argo Rollouts canary with Prometheus analysis (2/2)

Without a traffic router, weights are approximated by the ratio of canary to stable pods.

# apps/rollout/analysis.yaml
apiVersion: argoproj.io/v1alpha1
kind: AnalysisTemplate
metadata: {name: success-rate}
spec:
  metrics:
    - name: success-rate
      interval: 30s
      failureLimit: 1
      successCondition: result[0] >= 0.95
      provider:
        prometheus:
          address: http://prom-prometheus-server.monitoring.svc
          query: |
            sum(rate(http_request_duration_seconds_count{
              app="web", status!~"5.."}[1m]))
            /
            sum(rate(http_request_duration_seconds_count{
              app="web"}[1m]))
git add apps/rollout && git commit -m rollout && git push
argocd app create rollout-demo --repo $REPO \
  --path apps/rollout \
  --dest-server https://kubernetes.default.svc \
  --dest-namespace demo --sync-policy automated

# a good release: change only the UI message
#   env: [{name: PODINFO_UI_MESSAGE, value: "v2"}]
git commit -am "web v2" && git push
kubectl argo rollouts get rollout web -n demo --watch
# 20% -> pause -> 60% -> pause -> 100%, AnalysisRun Successful

# dashboard (optional)
kubectl argo rollouts dashboard   # http://localhost:3100

Argo CD shows a paused Rollout as Suspended; resume from the UI (Rollout → Resume) or kubectl argo rollouts promote web -n demo.

Lab 11 · Verify, break it, clean up

A bad release aborts itself

podinfo can return random errors: that is our "bug".

Verify

  • The good release finishes; kubectl argo rollouts list rollouts -n demo → Healthy.
  • kubectl -n demo get analysisrun → Successful, one per release.
  • In Prometheus (kubectl -n monitoring port-forward svc/prom-prometheus-server 9090:80) the query returns ≈ 1.0.

Break it

  • Bad release: add command: ["./podinfo", "--port=9898", "--random-error=true"] to the container, push → at 20% the success rate drops below 0.95; the AnalysisRun fails; the Rollout aborts and scales the canary to 0. Argo CD shows the app Degraded.
  • Recover the GitOps way: git revert the bad commit → the Rollout goes back to stable and turns Healthy.
  • Think: why is the abort fast while a full rollback in Git is still needed?
argocd app delete rollout-demo --yes
kubectl delete ns demo
helm uninstall argo-rollouts \
  -n argo-rollouts
# keep Prometheus for Lab 12
Lab 12 · Build

Ops drill: five faults on a branch, plus metrics

Goal: diagnose broken Applications with the CLI and logs only, then watch them in Prometheus.

git switch -c drill
# 1 bad Helm value (schema violation)
yq -i '.replicaCount = "two"' charts/web/values.yaml
# 2 failing hook
sed -i '' 's/sleep 5/exit 1/' apps/sync-demo/10-migrate.yaml
# 3 an object with a finalizer nobody removes
cat > apps/sync-demo/50-stuck.yaml <<'EOF'
apiVersion: v1
kind: ConfigMap
metadata:
  name: stuck
  finalizers: [example.com/never-removed]
EOF
cp ~/lab7-hpa.yaml apps/sync-demo/40-hpa.yaml  # 4 HPA, no ignore
git add -A && git commit -m drill && git push -u origin drill
# apps that point at the broken branch
for a in web:charts/web sync:apps/sync-demo; do
  argocd app create drill-${a%%:*} --repo $REPO \
    --revision drill --path ${a#*:} \
    --dest-server https://kubernetes.default.svc \
    --dest-namespace drill --sync-option CreateNamespace=true
done
# 5 missing credential: a private repo, no Secret for it
gh repo create gitops-private --private
argocd app create drill-private \
  --repo https://github.com/<you>/gitops-private --path . \
  --dest-server https://kubernetes.default.svc \
  --dest-namespace drill
# clusters/gitops/metrics-values.yaml
controller:
  metrics:
    enabled: true
    service:
      annotations:
        prometheus.io/scrape: "true"
        prometheus.io/port: "8082"
server:
  metrics: {enabled: true}
repoServer:
  metrics: {enabled: true}

helm upgrade argocd argo/argo-cd -n argocd --reuse-values \
  -f clusters/gitops/metrics-values.yaml
kubectl -n argocd port-forward \
  svc/argocd-application-controller-metrics 8082:8082 &
curl -s localhost:8082/metrics | grep '^argocd_app_info' | head
# PromQL (Prometheus from Lab 11)
sum by (sync_status, health_status) (argocd_app_info)
argocd_app_info{sync_status="OutOfSync"}
histogram_quantile(0.95, sum by (le)
  (rate(argocd_app_reconcile_bucket[5m])))

# your toolkit
argocd app get drill-web --hard-refresh
argocd app diff drill-sync
kubectl -n argocd logs deploy/argocd-repo-server --tail=50
kubectl -n argocd logs sts/argocd-application-controller --tail=50
Lab 12 · Verify, break it, clean up

Symptom → tool → evidence → fix

Evidence first, then the fix.

FaultSymptomWhere to lookFix
1 Bad valueComparisonErrorargocd app get conditionsFix the value (schema caught it)
2 Failing hookSync Failed at PreSyncapp get --show-operation; job logsFix command, resync
3 FinalizerRemove 50-stuck.yaml, prune → stuck Terminatingkubectl get cm stuck -o yaml (finalizers)kubectl patch the ConfigMap: finalizers: null
4 HPA fightOutOfSync flapping on spec.replicasargocd app diff drill-syncignoreDifferences + RespectIgnoreDifferences
5 No credentialRepository not accessibleargocd repo list; app conditionsAdd a repo Secret with a read-only token
for a in web sync private; do
  argocd app delete drill-$a --yes; done
kubectl delete ns drill
git switch main; git branch -D drill
git push origin --delete drill
gh repo delete <you>/gitops-private --yes
pkill -f port-forward
Lab 13 · Build (1/2)

GitOps on EKS: cluster, Argo CD, External Secrets

Goal: the same repo on EKS Auto Mode, secrets from AWS Secrets Manager. Costs money: finish and destroy the same day.

export AWS_REGION=ap-northeast-1
# a small cluster (eksctl here; Chapter 13 shows Terraform)
cat > eks.yaml <<'EOF'
apiVersion: eksctl.io/v1alpha5
kind: ClusterConfig
metadata: {name: gitops-eks, region: ap-northeast-1}
autoModeConfig: {enabled: true}
EOF
eksctl create cluster -f eks.yaml          # ~15-20 min
kubectl config current-context

# Argo CD: self-managed (as in Lab 6) ...
helm upgrade --install argocd argo/argo-cd -n argocd \
  --create-namespace -f clusters/gitops/argocd-values.yaml
# Managed EKS Capability instead? It does not register this
# cluster for you: register it by ARN (needs an access entry)
# and target that name, not kubernetes.default.svc. This lab
# uses self-managed Argo CD; see the AWS guide for the rest.
# a secret in AWS + permission for External Secrets
aws secretsmanager create-secret --name gitops-lab/db \
  --secret-string '{"password":"s3cr3t"}'
eksctl create podidentityassociation --cluster gitops-eks \
  --namespace external-secrets \
  --service-account-name external-secrets \
  --permission-policy-arns \
  arn:aws:iam::aws:policy/SecretsManagerReadWrite
helm repo add external-secrets https://charts.external-secrets.io
helm install external-secrets external-secrets/external-secrets \
  -n external-secrets --create-namespace

# apps/eks/store.yaml
apiVersion: external-secrets.io/v1
kind: ClusterSecretStore
metadata: {name: aws-sm}
spec:
  provider:
    aws: {service: SecretsManager, region: ap-northeast-1}

The lab uses a broad managed policy for speed; in real use, scope it to secretsmanager:GetSecretValue on gitops-lab/*.

Lab 13 · Build (2/2)

GitOps on EKS: secret, Ingress, Applications

Everything below is committed to apps/eks/ and synced by Argo CD, not applied by hand.

# apps/eks/db-secret.yaml
apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata: {name: db, namespace: demo}
spec:
  refreshInterval: 1h
  secretStoreRef: {kind: ClusterSecretStore, name: aws-sm}
  target: {name: db}
  data:
    - secretKey: password
      remoteRef: {key: gitops-lab/db, property: password}

git add apps/eks && git commit -m "eks extras" && git push
argocd app create web-demo --repo $REPO --path charts/web \
  --dest-server https://kubernetes.default.svc \
  --dest-namespace demo --sync-policy automated \
  --sync-option CreateNamespace=true
argocd app create eks-extras --repo $REPO --path apps/eks \
  --dest-server https://kubernetes.default.svc \
  --dest-namespace demo --sync-policy automated
# apps/eks/ingress.yaml (Auto Mode built-in ALB)
apiVersion: eks.amazonaws.com/v1
kind: IngressClassParams
metadata: {name: alb}
spec: {scheme: internet-facing}
---
apiVersion: networking.k8s.io/v1
kind: IngressClass
metadata: {name: alb}
spec:
  controller: eks.amazonaws.com/alb
  parameters: {apiGroup: eks.amazonaws.com,
               kind: IngressClassParams, name: alb}
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata: {name: web, namespace: demo}
spec:
  ingressClassName: alb
  rules:
    - http:
        paths:
          - path: /
            pathType: Prefix
            backend: {service: {name: web, port: {number: 9898}}}
Lab 13 · Verify, break it, clean up

Prove it on AWS, then tear it all down

The same Git repo drove kind and EKS; only envs/ and clusters/ changed.

Verify

  • kubectl -n demo get externalsecret db → SecretSynced, Ready True.
  • kubectl -n demo get secret db -o jsonpath='{.data.password}' | base64 -d → s3cr3t.
  • kubectl -n demo get ingress web → an ALB hostname; curl http://<host>/ returns podinfo JSON (wait a few minutes).
  • Rotate the secret in Secrets Manager → the Kubernetes Secret updates after refreshInterval (or annotate force-sync).

Break it

  • No permission: delete the pod identity association, restart External Secrets → ExternalSecret SecretSyncedError with AccessDenied. Recreate it.
  • Wrong key: remoteRef.property: passwd → error naming the missing key; the old Secret keeps its value.
  • Capability limits (optional): if you try the managed capability, find in the docs: sync timeout, Git poll interval, supported SSO, how to register the cluster.
# 1) delete the Ingress first: the ALB
#    belongs to it
kubectl -n demo delete ingress web
sleep 120
# 2) disable the Argo CD capability
#    if you enabled it (console)
# 3) cluster, VPC, NAT gateway
eksctl delete cluster -f eks.yaml \
  --wait
aws secretsmanager delete-secret \
  --secret-id gitops-lab/db \
  --force-delete-without-recovery
# 4) check for leftovers: ALBs,
#    EBS volumes, Elastic IPs
Lab safety

Cost and cleanup

Labs 0–12 are free. Lab 13 bills by the hour from the moment eksctl create cluster starts.

Billed per hour in Lab 13

  • EKS control plane (per cluster).
  • Worker nodes, plus the EKS Auto Mode management fee per node.
  • NAT gateway and its public IPv4 addresses.
  • Application Load Balancer created by the Ingress.
  • The EKS Capability for Argo CD, if enabled.
  • Secrets Manager: per secret per month, plus API calls.

Habits

  • Run Lab 13 in one sitting; destroy the same day.
  • Delete Ingresses before the cluster so the ALB is not orphaned.
  • Set a budget alarm before eksctl create cluster.
  • Never put real credentials in gitops-lab; it is public.
  • Use --context in scripts once you have more than one cluster.

Clean up

  • Local: kind delete clusters --all; docker rm -f registry.
  • GitHub: delete gitops-lab and gitops-private when done, or keep gitops-lab as your reference.
  • AWS: eksctl delete cluster -f eks.yaml --wait; delete the secret with --force-delete-without-recovery.
  • Then check the console for leftover load balancers, EBS volumes, Elastic IPs and log groups.
Glossary

GitOps and Helm terms

Short anchors; explain each in your own words.

TermShort meaning
GitOpsDesired state in Git, pulled and continuously reconciled by an agent
ReconciliationLoop that compares desired and live state and acts on the difference
DriftLive state that no longer matches Git
Push vs pullCI applies to the cluster vs an in-cluster agent fetches from Git
Config repoRepo holding deployable state per environment, separate from app code
PromotionMoving a version from one environment to the next, as a Git change
Rendered manifestsPlain YAML produced from Helm/Kustomize, committed for review
TermShort meaning
ChartHelm package: templates + default values + metadata
ReleaseOne installed instance of a chart, with numbered revisions
ValuesInputs that parameterise a chart; merged by precedence
Library chartChart with only named templates, used as a dependency
Helm hookResource run at a lifecycle point (pre-upgrade, test, …)
OCI chartChart stored as an artifact in a container registry
Server-Side ApplyAPI server merges fields and tracks owners (Helm 4 default for new installs)
Glossary

Argo CD and Rollouts terms

Short anchors; explain each in your own words.

TermShort meaning
ApplicationCRD: one source (or several) → one destination, plus sync policy
AppProjectTenancy boundary: allowed repos, destinations, resource kinds, roles
ApplicationSetTemplate + generators that create many Applications
GeneratorSource of parameters: list, clusters, Git, matrix, PR, SCM
Sync / health statusMatches Git? / Is it working? — two separate questions
Sync waveInteger order of resources within a sync phase
Sync hookResource run in PreSync, Sync, PostSync, SyncFail or PreDelete
Prune / self-healDelete what left Git / revert live changes automatically
TermShort meaning
Hard refreshRe-render manifests, bypassing the repo-server cache
Source HydratorArgo CD feature that commits rendered manifests to a branch
Hub-and-spokeOne Argo CD managing many clusters
RolloutArgo Rollouts CRD replacing a Deployment for canary / blue-green
CanaryShift a small share of traffic to the new version first
AnalysisTemplateMetric checks (e.g. PromQL) with success conditions
AnalysisRunOne execution of an AnalysisTemplate during a rollout
KargoPromotion tool: stages, freight, verification between environments
Further reading

Primary references

Versions move fast: when a slide and the docs disagree, the docs for your installed version win.

Helm, Kustomize, GitOps

  • Helm documentation — helm.sh/docs/
  • Helm 4 release announcement — helm.sh/blog/helm-4-released/
  • Helm changelog — helm.sh/docs/changelog/
  • Chart best practices — helm.sh/docs/chart_best_practices/
  • Kustomize reference — kubectl.docs.kubernetes.io/references/kustomize/
  • OpenGitOps principles — opengitops.dev
  • External Secrets Operator — external-secrets.io

Argo projects and EKS

  • Argo CD documentation — argo-cd.readthedocs.io
  • Upgrade notes 3.2 → 3.3 — argo-cd.readthedocs.io/en/stable/operator-manual/upgrading/3.2-3.3/
  • Upgrade notes 3.4 → 3.5 — argo-cd.readthedocs.io/en/stable/operator-manual/upgrading/3.4-3.5/
  • Argo CD releases — github.com/argoproj/argo-cd/releases
  • Argo Rollouts — argo-rollouts.readthedocs.io
  • Kargo — docs.kargo.io
  • EKS Capability for Argo CD vs self-managed — docs.aws.amazon.com/eks/latest/userguide/argocd-comparison.html
1 / 1