From a single chart to a fleet of clusters: package with Helm, declare in Git, reconcile with Argo CD, release safely with Argo Rollouts, and run it all on EKS — at production depth, with a lab for every chapter.
Use ← → / Space to navigate · I contents · O overview · T theme · F fullscreen
Click any line to jump there · press I to return to this page
What GitOps is, the toolchain this deck builds, the versions it covers, and how to study it.
GitOps changes only the second half. CI still builds, tests and publishes artifacts.
ghcr.io/…/web:1.4.2, chart web-0.7.0.Each box is a chapter. Read left to right: package → render → store → reconcile → release safely.
Security (ch 10), operations (ch 12) and EKS specifics (ch 13) cut across every box.
Ask them in order when something does not deploy. Each maps to one Argo CD status.
ComparisonError if not.helm template, kustomize build.Synced to the expected commit?argocd app get, argocd app diff.Synced but Degraded = Git is applied, the app is broken.kubectl edit.Sync status and health status are independent. "Synced + Degraded" is a normal state to debug, not a contradiction.
Older tutorials show different defaults. When a slide and the official docs disagree, the docs win.
--wait; renamed flags (--atomic → --rollback-on-failure, --force → --force-replace).apiVersion: v2 charts work unchanged..status ignored in diffs by default.kubectl -k and Argo CD.Labs 0–12 cost nothing. Lab 13 creates an EKS cluster and must be destroyed the same day.
kind (or k3d).kubectl, helm 4, kustomize, argocd CLI.kubeconform, helm-unittest, ct, yq, jq.kind-gitops, second cluster kind-prod.https://github.com/<you>/gitops-lab.ghcr.io/stefanprodan/podinfo, chart name web.argocd.gitops-lab/
├── charts/web/ # our Helm chart
├── apps/ # Argo CD Applications
│ └── root.yaml # app-of-apps entry
├── envs/
│ ├── dev/web/values.yaml
│ ├── staging/web/values.yaml
│ └── prod/web/values.yaml
└── clusters/ # per-cluster add-ons
├── gitops/
└── prod/Lab 0 in Hands-on labs creates this repo and both kind clusters.
Read a part, run its lab, then answer the self-check out loud without notes.
Navigation: ← → or Space to move, I for Contents, O for overview, T for theme, F for fullscreen. Every Contents line is a link. Lab 0 in Hands-on labs sets up your environment.
Principles, push vs pull, drift, repository strategy and the tools around Argo CD.
The four OpenGitOps principles, push vs pull delivery, and what drift really means.
A vendor-neutral definition maintained by the CNCF GitOps Working Group. Argo CD and Flux both implement it.
latest.Push: CI runs kubectl or helm against the cluster. Pull: an agent inside the cluster fetches from Git.
Pull is the default for production on Kubernetes; push still has valid uses.
| Aspect | Push (CI applies) | Pull (GitOps agent) |
|---|---|---|
| Credentials | CI stores cluster-admin-like credentials | Agent uses in-cluster service account; CI only writes to Git |
| Network | CI must reach every API server | Cluster needs outbound access to Git and registries only |
| Drift | Undetected until the next deploy | Detected continuously; optionally corrected (self-heal) |
| Audit | Pipeline logs (often expire) | Git history + agent events |
| Rollback | Re-run an old pipeline | git revert and let the agent converge |
| Multi-cluster | Pipeline fan-out logic | Generators (ApplicationSet) or one agent per cluster |
| Good fit | One-off jobs, bootstrap, non-Kubernetes targets | Long-lived Kubernetes workloads and add-ons |
Same pattern as a Kubernetes controller, with Git as the spec.
Synced / OutOfSync.Detection is always on. Correction (self-heal) is a choice per Application.
kubectl edit / kubectl scale during an incident.spec.replicas.OutOfSync with a diff — always.selfHeal: true, Argo CD re-applies Git state automatically.ignoreDifferences (ch 7).kubectl for humans in production is the end state.Under GitOps, the cluster is a cache of Git. Edits to a cache are lost.
kubectl set image is reverted minutes later by self-heal.App repo vs config repo, monorepo vs polyrepo, directories vs branches, and the rendered manifests pattern.
Separate what changes on every commit (code) from what changes on every deploy (environment state).
charts/web), versioned with the code.web:1.4.2 and pushes chart web-0.7.0 to an OCI registry.gitops-lab)Most organizations land on one config repo per platform or per team, not one per service.
| Option | Strengths | Weaknesses | Fits |
|---|---|---|---|
| One config repo for everything | One place to search, one PR for cross-cutting changes | Large clones, noisy history, coarse permissions (use CODEOWNERS) | Small and medium platforms |
| Config repo per team | Team autonomy, clear ownership | Shared add-ons duplicated or in a platform repo | Many teams, one platform team |
| Config next to code (per service) | Change code and config together | CI-writes-back loops, deploy history mixed with code | Few services, very small teams |
| Platform repo + team repos | Platform owns clusters and add-ons; teams own apps | Two places to look | Most enterprise setups |
Argo CD handles large monorepos well if you set the argocd.argoproj.io/manifest-generate-paths annotation and use webhooks (ch 12).
Keep one main branch; model environments as folders. Promotion is a diff between folders.
dev, staging, prod branches with merges between them.envs/dev, envs/staging, envs/prod on main.diff -r envs/staging envs/prod shows exactly what promotion will change.envs/prod/** with CODEOWNERS and required reviews.gitops-lab/
├── charts/web/
├── envs/
│ ├── dev/web/values.yaml
│ │ # tag: 1.5.0-rc.1
│ ├── staging/web/values.yaml
│ │ # tag: 1.4.2
│ └── prod/web/values.yaml
│ # tag: 1.4.1, replicas 4
└── apps/web-appset.yamlCommit the final YAML that will be applied, not only the inputs that generate it.
lookup, no random values).If you cannot answer "where does this change go?" in one second, the layout is wrong.
| Change | Repo and path | Who approves |
|---|---|---|
| New feature code | App repo | Service team |
| New image built | Nothing yet — CI publishes the artifact | — |
| Deploy 1.4.2 to staging | gitops-lab/envs/staging/web/values.yaml (image tag) | Service team |
| Promote to prod | gitops-lab/envs/prod/web/values.yaml | Service team + prod CODEOWNERS |
| New chart template | App repo charts/web → new chart version | Service team |
| New environment | gitops-lab/envs/<env>/ + generator entry | Platform team |
| Cluster add-on upgrade | gitops-lab/clusters/<cluster>/ | Platform team |
| Secret value | Never in Git: AWS Secrets Manager + External Secrets (ch 10) | Owner of the secret |
Argo CD vs Flux, the tools around Argo CD, and when GitOps is the wrong tool.
This deck uses Argo CD. Knowing Flux helps you read other teams' setups.
| Aspect | Argo CD | Flux |
|---|---|---|
| Core objects | Application, ApplicationSet, AppProject | GitRepository, Kustomization, HelmRelease, OCIRepository |
| UI | Built-in web UI and CLI | CLI; UIs are separate projects |
| Helm handling | Renders with Helm templates, then applies like any YAML — no Helm release record | helm-controller performs real Helm installs and upgrades (release history, Helm hooks) |
| Multi-tenancy | AppProjects + Argo CD RBAC + SSO | Kubernetes RBAC + service account impersonation per object |
| Multi-cluster | One instance can manage many clusters (hub-and-spoke) | Usually one Flux per cluster; remote clusters via kubeconfig |
| Typical choice | Teams that want a UI and central fleet view | Teams that want small, composable controllers |
Consequence of Argo CD rendering Helm: helm list shows nothing for apps deployed by Argo CD, and lookup returns empty (ch 3).
Argo CD syncs Git to clusters. These tools fill the gaps before and after that step.
GitOps shines for long-lived declared state. Forcing everything through it creates friction.
kubectl debug.ignoreDifferences for controller-owned fields.Each one has bitten production teams.
| Mistake | What happens | Do this instead |
|---|---|---|
Image tag latest in Git | Git does not change, so nothing redeploys — or every node pulls a different image | Immutable tags or digests; bump in Git |
| Branch per environment | Merge conflicts and silent drift between envs | Folder per environment on one branch |
| CI commits tags to the app repo it builds | Infinite CI loops, [skip ci] hacks | Separate config repo |
Hotfix with kubectl edit and no commit | Self-heal reverts it; incident returns | Fast-track PR, or pause auto-sync then commit |
| Secrets committed base64-encoded | Credentials leaked in Git history forever | External Secrets with AWS Secrets Manager |
| CI keeps cluster-admin "for emergencies" | Push and pull both active; nobody trusts Git | Remove CI cluster credentials once GitOps works |
Explain each answer out loud, with an example from the lab layout.
latest incompatible with "versioned and immutable"?helm list show nothing for an Argo CD app?Practice: Lab 1 in Hands-on labs — deploy with push, then with Argo CD, and watch drift detection and self-heal.
Charts, releases and revisions, the release lifecycle, values precedence, and what changed in Helm 4.
Chart, release, revision, repository — and every file inside a chart directory.
A chart is a package; a release is an installed instance of it; each change makes a revision.
.tgz) of templates + default values + metadata.Chart.yaml (version: 0.7.0).charts/web, or bitnami/redis.helm install web-dev charts/web -n dev creates release web-dev.index.yaml.oci://… (ch 4).Helm is a client-side renderer plus a release-history store. There is no server component.
Argo CD uses only steps 1–2 (helm template) and does the rest itself. That is why Helm hooks become Argo CD hooks and no release Secret exists (ch 6–7).
helm create web generates this layout; delete what you do not need.
charts/web/
├── Chart.yaml # metadata, version, dependencies
├── Chart.lock # pinned dependency versions
├── values.yaml # defaults (the chart's API)
├── values.schema.json # optional JSON Schema for values
├── charts/ # vendored dependency .tgz files
├── crds/ # CRDs: installed once, never upgraded
├── templates/
│ ├── _helpers.tpl # named templates, no output
│ ├── deployment.yaml
│ ├── service.yaml
│ ├── ingress.yaml
│ ├── hpa.yaml
│ ├── NOTES.txt # printed after install
│ └── tests/
│ └── test-connection.yaml
├── .helmignore # excluded from the package
└── README.mdtemplates/ starting with _ render nothing; they hold helpers.templates/ must render valid YAML (or nothing).crds/ files are plain YAML — no templating — installed only on first install (ch 4).charts/ holds dependencies after helm dependency update.values.schema.json is validated on install, upgrade, lint and template..helmignore keeps tests data, .git and CI files out of the package.apiVersion: v2 is the Helm 3 chart format and is unchanged in Helm 4.
apiVersion: v2
name: web
description: podinfo-based demo web service
type: application # or: library
version: 0.7.0 # CHART version (SemVer, required)
appVersion: "6.7.1" # version of the app inside (string)
kubeVersion: ">=1.29.0-0"
keywords: [web, demo]
maintainers:
- name: platform-team
email: platform@example.com
dependencies:
- name: redis
version: 20.x.x
repository: oci://registry-1.docker.io/bitnamicharts
condition: redis.enabled
annotations:
example.com/owner: team-webversion is the package version: bump it on every chart change; registries reject duplicates.appVersion is informational: usually the image tag the chart defaults to.image.tag defaults to .Chart.AppVersion.appVersion so YAML does not turn 1.10 into 1.1.type: library — only helpers, cannot be installed (ch 4).kubeVersion — install fails on clusters outside the range.dependencies — subcharts with condition / tags / alias (ch 4).deprecated: true — warns users of the chart.Everything users may change is a value with a sensible default; everything else stays in templates.
replicaCount: 2
image:
repository: ghcr.io/stefanprodan/podinfo
tag: "" # empty → .Chart.AppVersion
pullPolicy: IfNotPresent
service:
type: ClusterIP
port: 9898
ingress:
enabled: false
className: ""
hosts: []
resources:
requests: {cpu: 50m, memory: 64Mi}
limits: {memory: 128Mi}
autoscaling:
enabled: false
minReplicas: 2
maxReplicas: 10
podAnnotations: {}
extraEnv: []--set image.tag=1.4.2.ingress.enabled, autoscaling.enabled.extraEnv, podAnnotations) instead of forking the chart.helm-docs turns comments into the README (ch 4).values.yaml.Install, upgrade, rollback and uninstall; where Helm stores state; previewing changes; values precedence.
Prefer upgrade --install everywhere: it is idempotent, so pipelines and humans run the same command.
# install or upgrade in one idempotent command
helm upgrade --install web-dev charts/web \
-n dev --create-namespace \
-f envs/dev/web/values.yaml \
--set image.tag=1.4.2 \
--wait --timeout 5m
helm list -n dev # releases, status, chart
helm history web-dev -n dev # revisions
helm status web-dev -n dev # last deploy, notes
helm rollback web-dev 3 -n dev # → new revision from #3
helm uninstall web-dev -n dev # delete objects + history
helm uninstall web-dev -n dev --keep-historyhelm get values web-dev — values you supplied; add --all for merged values.helm get manifest web-dev — exactly what was applied.helm get hooks web-dev — hook objects.helm get notes web-dev — rendered NOTES.txt.--revision N to inspect an older revision.--reuse-values: reuse last values, ignore new chart defaults (risky).--reset-then-reuse-values: new chart defaults + last values + new flags.No Tiller, no database. Each revision is one Secret of type helm.sh/release.v1.
$ kubectl get secret -n dev -l owner=helm
NAME TYPE DATA
sh.helm.release.v1.web-dev.v1 helm.sh/release.v1 1
sh.helm.release.v1.web-dev.v2 helm.sh/release.v1 1
sh.helm.release.v1.web-dev.v3 helm.sh/release.v1 1
# decode one revision (base64 → base64 → gzip → JSON)
kubectl get secret sh.helm.release.v1.web-dev.v3 -n dev \
-o jsonpath='{.data.release}' | base64 -d | base64 -d \
| gunzip | jq '{name, version, info: .info.status,
chart: .chart.metadata.version}'--history-max (default 10 on upgrade).pending-install, pending-upgrade or pending-rollback blocks the next upgrade: "another operation (install/upgrade/rollback) is in progress".helm rollback web-dev <last-good-rev>; last resort: delete the pending revision's Secret, then upgrade again.Three levels: render locally, validate against the API server, and diff against the live release.
.Capabilities uses defaults unless --kube-version / --api-versions are passed.lookup returns empty.helm template web charts/web -f envs/prod/web/values.yamllookup works (it queries the cluster).helm upgrade web-dev charts/web -n dev --dry-run=serverhelm plugin install https://github.com/databus23/helm-diffhelm diff upgrade web-dev charts/web -n dev -f …argocd app diff instead (ch 7).Later sources override earlier ones. Maps merge key by key; lists are replaced whole.
values.yaml of the chart.values.yaml overrides subchart defaults.global: shared with all subcharts.--set, --set-string, --set-file, --set-json.-f prod.yaml with extraEnv: [B] replaces the default [A] entirely.null to delete a default (e.g. remove a default resource limit).--set a.b=1,a.c=2 — commas separate keys; escape commas in values with \,.--set guesses types: --set tag=1.10 becomes the number 1.1; use --set-string.--set list[0].name=x sets one index but you must know the full list.Predict the result before reading the right column.
# charts/web/values.yaml
replicaCount: 2
image: {repository: ghcr.io/stefanprodan/podinfo, tag: ""}
extraEnv: [{name: LOG_LEVEL, value: info}]
resources: {limits: {memory: 128Mi}}
# envs/prod/web/values.yaml (-f)
replicaCount: 4
extraEnv: [{name: REGION, value: ap-northeast-1}]
resources: {limits: {memory: 256Mi, cpu: null}}
# command line
--set image.tag=1.4.1# merged result
replicaCount: 4 # file beats default
image:
repository: ghcr.io/stefanprodan/podinfo # kept (map merge)
tag: 1.4.1 # --set beats everything
extraEnv: # list REPLACED, LOG_LEVEL gone
- {name: REGION, value: ap-northeast-1}
resources:
limits: {memory: 256Mi} # null removes a key
# check it yourself:
helm template web charts/web -f envs/prod/web/values.yaml \
--set image.tag=1.4.1 --show-only templates/deployment.yamlServer-Side Apply, kstatus waiting, renamed flags, plugins — and a migration checklist for Helm 3 pipelines.
The first major version since Helm 3 (2019). Charts stay compatible; the CLI and apply behavior change.
--wait.helm registry login takes a host, not a URL.--atomic → --rollback-on-failure.--force → --force-replace.apiVersion: v2 charts install as before.helm.sh/helm/v4.Helm 3 computed a patch on the client from old manifest, new manifest and live object. Helm 4 lets the API server merge.
metadata.managedFields.ServerSideApply=true sync option (ch 7).--server-side on upgrade.SSA makes ownership inspectable. This is how you answer "who changed replicas?".
$ kubectl get deploy web -n dev --show-managed-fields -o yaml
metadata:
managedFields:
- manager: helm # set by the Helm release
operation: Apply
fieldsV1:
f:spec:
f:template: {...}
- manager: kube-controller-manager
operation: Update
subresource: status
- manager: kubectl-edit # a human edited replicas
operation: Update
fieldsV1:
f:spec:
f:replicas: {}manager names the client; operation: Apply means SSA.kubectl-edit or kubectl-scale is manual drift.helm upgrade and Argo CD.kstatus is the readiness library used by kubectl and Flux. It understands many more resource types than Helm 3's checks.
# Helm 4 style
helm upgrade --install web-prod charts/web -n prod \
-f envs/prod/web/values.yaml \
--wait --timeout 10m \
--rollback-on-failure # was --atomic
# Helm 3 style (still accepted, prints a deprecation warning)
helm upgrade --install web-prod charts/web -n prod \
-f envs/prod/web/values.yaml --wait --atomic--wait takes a strategy: watcher (kstatus, what a bare --wait means), hookOnly (the default when --wait is omitted) or legacy (Helm 3 checks, an escape hatch).--rollback-on-failure implies waiting; on failure it rolls back (or uninstalls a failed first install). Good for push pipelines.helm upgrade; it uses health status and its own retry (ch 7).Three small changes that break scripts silently.
helm-diff, helm-secrets, helm-unittest) for Helm 4 support before upgrading CI.--post-renderer ./kustomize-wrapper.sh (any executable).--post-renderer <plugin-name>; the executable must be packaged as a plugin.helmCharts or a multi-source app instead (ch 5, 8).helm registry login 123456789012.dkr.ecr.ap-northeast-1.amazonaws.comhttps://…/charts is rejected in Helm 4.aws ecr get-login-password into --password-stdin.oci://<host>/<repo>/<chart>.Do it per pipeline, behind a pinned Helm version, and compare rendered output before and after.
| Step | What to check | How |
|---|---|---|
| 1 · Pin versions | Each pipeline uses an explicit Helm version | Container image or setup action with a fixed version |
| 2 · Render diff | Same manifests from Helm 3 and Helm 4 | helm template with both, diff the output |
| 3 · Flags | --atomic, --force, --post-renderer <path>, registry login URLs | grep -R your CI files and Makefiles |
| 4 · Plugins | Every installed plugin supports Helm 4 | helm plugin list; read each plugin's releases |
| 5 · Waiting | Readiness passes under kstatus | Run a staging deploy with --wait; fix probes |
| 6 · Apply method | Existing releases stay client-side | Decide whether to opt in with --server-side; test on staging first |
| 7 · SDK users | Go imports updated | helm.sh/helm/v3 → helm.sh/helm/v4 |
Argo CD bundles its own Helm binary; your CLI upgrade does not change what Argo CD renders. Check the Helm version in the Argo CD release notes.
Most Helm incidents come from values handling and from stuck releases.
| Mistake | What happens | Do this instead |
|---|---|---|
--reuse-values on every upgrade | New chart defaults never apply; old values linger for years | Pass full values files from Git each time |
Unquoted versions with --set | 1.10 becomes 1.1; wrong image deployed | --set-string or quote in values files |
| Expecting lists to merge | Env vars or volumes from defaults disappear | Repeat the full list in the override |
| Killing Helm mid-upgrade | Release stuck in pending-upgrade | Generous --timeout; recover with helm rollback |
| Secrets passed as values | Readable from release Secrets by anyone with Secret read access | External Secrets or mounted secrets |
| Same objects managed by Helm CLI and Argo CD | Endless fights over fields, surprise rollbacks | One owner per object |
| Upgrading CI to Helm 4 blindly | Deprecation warnings ignored; post-renderer scripts fail | Follow the migration checklist |
Explain the answer and name the exact command or field.
version and appVersion?--dry-run=server instead of helm template?--reset-then-reuse-values do?-f files and --set applied, and how do lists merge?--atomic, and why is it irrelevant for Argo CD?Practice: Lab 2 in Hands-on labs — install, upgrade and roll back a chart, decode release Secrets, and recover from a failed upgrade.
Go templates, Sprig, named templates and values design: build the web chart template by template.
How Helm turns a chart plus values into YAML, and the template language that does it.
Helm never "understands" Kubernetes while rendering. It produces text, then parses the text as YAML.
Everything between {{ and }} is an action. A pipeline passes the left value as the LAST argument of the next function.
# templates/configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
name: {{ .Release.Name }}-web-config
data:
# .Values.message | default "hello" | quote
# is the same as quote (default "hello" .Values.message)
MESSAGE: {{ .Values.message | default "hello" | quote }}
LEVEL: {{ upper .Values.logLevel | quote }}
REPLICAS: {{ .Values.replicaCount | toString | quote }}
{{- /* a comment: produces no output */}}. is the current context; at the top of a file it is the root object.$ always points to the root, even inside range and with.x | f a calls f a x. That is why default "hello" .Values.message reads backwards.{{/* … */}}; YAML # comments are still rendered into the output.printf "%s-%s" a b.(include "web.fullname" .) | trunc 63.Every block ends with {{ end }}. Empty values are falsy: false, 0, "", nil, empty list or map.
{{- if .Values.ingress.enabled }}
apiVersion: networking.k8s.io/v1
kind: Ingress
...
{{- end }}
{{- with .Values.nodeSelector }}
nodeSelector:
{{- toYaml . | nindent 2 }} # . is now the map
{{- end }}
env:
{{- range $key, $val := .Values.env }}
- name: {{ $key }}
value: {{ $val | quote }}
{{- end }}
{{- range .Values.extraPorts }}
- containerPort: {{ .port }} # . is the list item
{{- end }}with and range, .Values no longer exists: . changed. Use $.Values or a variable captured before the block.{{ $name := include "web.fullname" . }} captures a value once, use it anywhere below in the same file.range over a map iterates keys in sorted order: output is stable, which matters for GitOps diffs.else if and with … else exist; if and .a .b / if or .a .b / if not .a combine conditions.eq, ne, lt, gt compare; comparing an int with a float64 from YAML fails — cast with int first.{{-, -}}, indent and nindentMost broken charts are indentation bugs. Trim with dashes, then put the newline back with nindent.
# wrong: blank lines and broken indentation
spec:
template:
metadata:
annotations:
{{ toYaml .Values.podAnnotations | indent 8 }}
# right
spec:
template:
metadata:
{{- with .Values.podAnnotations }}
annotations:
{{- toYaml . | nindent 8 }}
{{- end }}{{- trims all whitespace (including newlines) to the LEFT; -}} trims to the RIGHT.indent N prefixes every line with N spaces; the first line also gets them.nindent N = newline + indent N: start the action right after {{- on its own line.{{- toYaml .Values.x | nindent <column> }} where column = where the keys must start.helm template . --show-only templates/deployment.yaml.{{- is required: {{-3}} is the number −3, not trimming.Helm ships Go template built-ins, most of the Sprig library, and a few Helm-only functions.
| Function | Example | Use it for |
|---|---|---|
default | .Values.tag | default .Chart.AppVersion | Fallback when the value is empty |
required | required "image.repository is required" .Values.image.repository | Fail rendering with a clear message |
fail | {{ fail "autoscaling and replicaCount=0 conflict" }} | Reject invalid combinations |
quote / squote | {{ .Values.port | quote }} | Force a YAML string (env values, annotations) |
toYaml / toJson | toYaml .Values.resources | nindent 12 | Copy a whole map or list into the manifest |
include | include "web.labels" . | nindent 4 | Call a named template and pipe the result |
tpl | tpl .Values.configTemplate . | Render a string from values as a template |
printf / trunc / trimSuffix | printf "%s-%s" .Release.Name "web" | trunc 63 | trimSuffix "-" | Build names within the 63-char limit |
sha256sum / b64enc | include (print $.Template.BasePath "/configmap.yaml") . | sha256sum | Checksums, Secret data |
dict / list / merge / hasKey | include "web.container" (dict "root" $ "c" .) | Pass several values into a named template |
lookup | lookup "v1" "Secret" .Release.Namespace "db" | Read live cluster objects (empty without a cluster) |
.ValuesThese objects are filled by Helm at render time.
| Object | Useful fields |
|---|---|
.Release | Name, Namespace, Revision, IsInstall, IsUpgrade, Service ("Helm") |
.Chart | Name, Version, AppVersion, Type — from Chart.yaml |
.Values | Merged values (defaults + files + --set) |
.Capabilities | KubeVersion.Version, APIVersions.Has "policy/v1" |
.Files | Get "conf/app.yaml", Glob "conf/*", AsConfig, AsSecrets |
.Template | Name (current file path), BasePath (the chart's templates dir) |
.Release.Name is the Application name unless spec.source.helm.releaseName is set.helm template, so .Release.IsUpgrade is false and .Release.Revision is 1 on every sync — never branch logic on them..Capabilities comes from the target cluster when Argo CD renders; with plain helm template it is a default unless you pass --kube-version / --api-versions..Files can read files inside the chart only, never outside it, and not files in templates/.Render locally first; only then talk to a cluster.
# render everything, show values-merge errors with context
helm template web charts/web -f envs/dev/values.yaml --debug
# one file only
helm template web charts/web --show-only templates/deployment.yaml
# render as a specific cluster would see it
helm template web charts/web --kube-version 1.33.0 \
--api-versions monitoring.coreos.com/v1
# validate against the live API server without changing it
helm install web charts/web --dry-run=server
# what is actually deployed for a release
helm get manifest web -n web
helm get values web -n web --all
# chart-level sanity
helm lint charts/web --strict -f envs/prod/values.yamlexecuting "web/templates/deployment.yaml" at <.Values.image.tag>: nil pointer evaluating interface {}.tag → the parent key image is missing.error converting YAML to JSON: yaml: line 23 → indentation; render with --debug to see the broken output.--debug prints the rendered manifests even when YAML parsing fails.helm lint --strict turns warnings (such as missing icon) into failures — useful in CI.Named templates in _helpers.tpl, standard labels, selectors, checksums and why lookup does not belong in GitOps.
web chartFiles starting with _ are not rendered as manifests. They hold define blocks you call with include.
{{/* charts/web/templates/_helpers.tpl */}}
{{- define "web.name" -}}
{{- $n := default .Chart.Name .Values.nameOverride }}
{{- $n | trunc 63 | trimSuffix "-" }}
{{- end }}
{{- define "web.fullname" -}}
{{- if .Values.fullnameOverride }}
{{- .Values.fullnameOverride | trunc 63 | trimSuffix "-" }}
{{- else }}
{{- $name := default .Chart.Name .Values.nameOverride }}
{{- if contains $name .Release.Name }}
{{- .Release.Name | trunc 63 | trimSuffix "-" }}
{{- else }}
{{- $full := printf "%s-%s" .Release.Name $name }}
{{- $full | trunc 63 | trimSuffix "-" }}
{{- end }}
{{- end }}
{{- end }}{{- define "web.chart" -}}
{{- $c := printf "%s-%s" .Chart.Name .Chart.Version }}
{{- $c | replace "+" "_" | trunc 63 | trimSuffix "-" }}
{{- end }}
{{- define "web.selectorLabels" -}}
app.kubernetes.io/name: {{ include "web.name" . }}
app.kubernetes.io/instance: {{ .Release.Name }}
{{- end }}
{{- define "web.labels" -}}
{{- $v := .Values.image.tag | default .Chart.AppVersion }}
helm.sh/chart: {{ include "web.chart" . }}
{{ include "web.selectorLabels" . }}
app.kubernetes.io/version: {{ $v | quote }}
app.kubernetes.io/managed-by: {{ .Release.Service }}
{{- end }}Prefix every template name with the chart name (web.): template names are global across the chart and its subcharts, so a generic fullname can be silently overridden.
Selector labels are copied into spec.selector, which is immutable on Deployments, StatefulSets and DaemonSets.
| Label | Value | In selector? |
|---|---|---|
app.kubernetes.io/name | Chart or app name | Yes |
app.kubernetes.io/instance | Release name | Yes |
app.kubernetes.io/version | App version (image tag) | No — changes every release |
app.kubernetes.io/component | frontend, worker, db | Only if fixed from day one |
app.kubernetes.io/part-of | Higher-level application | No |
app.kubernetes.io/managed-by | Helm | No |
helm.sh/chart | name-version | No — changes every chart bump |
version or helm.sh/chart in the selector breaks the next upgrade with field is immutable.name or instance later forces deleting and recreating the workload.app.kubernetes.io/instance is free for Helm to use as intended.Every value comes from values.yaml; every name and label from a helper.
{{- $tag := .Values.image.tag | default .Chart.AppVersion }}
{{- $cfg := include (print $.Template.BasePath "/configmap.yaml") . }}
apiVersion: apps/v1
kind: Deployment
metadata:
name: {{ include "web.fullname" . }}
labels:
{{- include "web.labels" . | nindent 4 }}
spec:
{{- if not .Values.autoscaling.enabled }}
replicas: {{ .Values.replicaCount }}
{{- end }}
selector:
matchLabels:
{{- include "web.selectorLabels" . | nindent 6 }}
template:
metadata:
annotations:
checksum/config: {{ $cfg | sha256sum }}
labels:
{{- include "web.selectorLabels" . | nindent 8 }} spec:
containers:
- name: web
image: "{{ .Values.image.repository }}:{{ $tag }}"
imagePullPolicy: {{ .Values.image.pullPolicy }}
ports:
- name: http
containerPort: {{ .Values.service.targetPort }}
envFrom:
- configMapRef:
name: {{ include "web.fullname" . }}
{{- with .Values.extraEnv }}
env:
{{- toYaml . | nindent 12 }}
{{- end }}
readinessProbe:
httpGet: { path: /readyz, port: http }
resources:
{{- toYaml .Values.resources | nindent 12 }}
{{- with .Values.nodeSelector }}
nodeSelector:
{{- toYaml . | nindent 8 }}
{{- end }}replicas is omitted when an HPA owns the count — otherwise every sync fights the autoscaler (Chapter 7 shows the Argo CD side of this).
Whole files are wrapped in if so a disabled feature renders nothing.
# templates/service.yaml
apiVersion: v1
kind: Service
metadata:
name: {{ include "web.fullname" . }}
labels:
{{- include "web.labels" . | nindent 4 }}
spec:
type: {{ .Values.service.type }}
ports:
- port: {{ .Values.service.port }}
targetPort: http
name: http
selector:
{{- include "web.selectorLabels" . | nindent 4 }}# templates/hpa.yaml
{{- if .Values.autoscaling.enabled }}
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
name: {{ include "web.fullname" . }}
spec:
scaleTargetRef:
apiVersion: apps/v1
kind: Deployment
name: {{ include "web.fullname" . }}
minReplicas: {{ .Values.autoscaling.minReplicas }}
maxReplicas: {{ .Values.autoscaling.maxReplicas }}
metrics:
- type: Resource
resource:
name: cpu
target:
type: Utilization
averageUtilization: {{ .Values.autoscaling.targetCPU }}
{{- end }}targetPort: http refers to the named container port, so changing the container port never needs a Service change.
rangeThe Ingress renders one rule per entry in .Values.ingress.hosts.
# templates/ingress.yaml
{{- if .Values.ingress.enabled }}
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: {{ include "web.fullname" . }}
{{- with .Values.ingress.annotations }}
annotations:
{{- toYaml . | nindent 4 }}
{{- end }}
spec:
ingressClassName: {{ .Values.ingress.className }}
rules:
{{- range .Values.ingress.hosts }}
- host: {{ .host | quote }}
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: {{ include "web.fullname" $ }}
port:
name: http
{{- end }}
{{- end }}range .Values.ingress.hosts the dot is one host entry (.host).$, not ..with .Values.ingress.annotations renders the annotations: key only when the map is not empty.className: alb plus AWS Load Balancer Controller annotations in env values (Chapter 13).helm template web charts/web --set ingress.enabled=true --show-only templates/ingress.yaml.Kubernetes does not restart Pods when a mounted or referenced ConfigMap changes. A checksum annotation makes the Pod template change.
# templates/configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
name: {{ include "web.fullname" . }}
labels:
{{- include "web.labels" . | nindent 4 }}
data:
PODINFO_UI_MESSAGE: {{ .Values.message | quote }}
PODINFO_UI_COLOR: {{ .Values.color | quote }}
# in deployment.yaml
{{- $cfg := include (print $.Template.BasePath "/configmap.yaml") . }}
...
annotations:
checksum/config: {{ $cfg | sha256sum }}$.Template.BasePath is the chart's templates directory; the include renders the ConfigMap text.configMapGenerator appends a content hash to the name (Chapter 5).include vs template, and passing more than one valueA named template receives exactly one argument. Bundle several with dict.
{{- define "web.container" -}}
{{- $tag := .c.image.tag | default .root.Chart.AppVersion }}
- name: {{ .c.name }}
image: "{{ .c.image.repository }}:{{ $tag }}"
resources:
{{- toYaml .c.resources | nindent 4 }}
{{- end }}
# caller: one container per entry in .Values.sidecars
containers:
{{- range .Values.sidecars }}
{{- $args := dict "root" $ "c" . }}
{{- include "web.container" $args | nindent 2 }}
{{- end }}include returns a string, so you can pipe it into nindent, sha256sum, trim.template writes output directly and cannot be piped — prefer include everywhere.$ (as root) whenever the named template needs .Chart, .Release or .Values.tpl renders a string that came from values: {{ tpl .Values.annotationsTemplate . }} lets users write {{ .Release.Name }} in values.lookup reads the live cluster — and returns nothing in GitOpslookup queries the API server during helm install/upgrade. It cannot do that when there is no cluster connection.
# keep a generated password across upgrades
{{- $old := lookup "v1" "Secret" .Release.Namespace "web-db" }}
apiVersion: v1
kind: Secret
metadata:
name: web-db
data:
{{- if $old }}
password: {{ index $old.data "password" }}
{{- else }}
password: {{ randAlphaNum 24 | b64enc }}
{{- end }}helm install / upgrade: returns the object, or an empty map if it does not exist.helm template and client-side dry runs: always empty.helm template: always empty → the example above generates a NEW password on every render, so the app is permanently OutOfSync.randAlphaNum, now, uuidv4 anywhere in templates.GitOps alternative: create secrets outside the chart (External Secrets Operator, Sealed Secrets, SOPS — Chapter 10) or let the app generate them once. Rendering must be a pure function of chart + values.
values.yaml is the public API of your chart. Design it, document it, and validate it with a JSON Schema.
web chart values: the contract with its usersEvery key is documented, has a working default, and appears in exactly one template.
# -- Number of Pods when autoscaling is off
replicaCount: 2
image:
# -- Image repository
repository: ghcr.io/stefanprodan/podinfo
# -- Image tag; defaults to Chart.appVersion
tag: ""
pullPolicy: IfNotPresent
nameOverride: ""
fullnameOverride: ""
message: "hello from web"
color: "#34577c"
service:
type: ClusterIP
port: 80
targetPort: 9898ingress:
enabled: false
className: nginx
annotations: {}
hosts:
- host: web.local
autoscaling:
enabled: false
minReplicas: 2
maxReplicas: 5
targetCPU: 70
resources:
requests: { cpu: 50m, memory: 64Mi }
limits: { memory: 128Mi }
# -- Extra env vars, rendered as-is
extraEnv: []
podAnnotations: {}
nodeSelector: {}
tolerations: []
affinity: {}enabled.service, ingress, autoscaling), at most 2–3 levels deep.extraEnv, podAnnotations, nodeSelector, tolerations) instead of a new key per field.Later files win. Keep environment files small: only the differences from the chart defaults.
gitops-lab/
├── charts/web/ # the chart (defaults)
│ ├── Chart.yaml
│ ├── values.yaml
│ ├── values.schema.json
│ └── templates/
└── envs/
├── dev/values.yaml # replicaCount: 1, message: dev
├── staging/values.yaml
└── prod/values.yaml # autoscaling.enabled: true, ingress
# render prod exactly as Argo CD will
helm template web charts/web -n web \
-f envs/prod/values.yamltolerations: [a] drops the defaults.null in a later file deletes the default key.--set a.b=1 beats every file; --set-string keeps "true" / "01" as strings.spec.source.helm.valueFiles (order matters) plus values / valuesObject (Chapter 6).If the chart contains values.schema.json, Helm validates the merged values on install, upgrade, lint and template.
{
"$schema": "http://json-schema.org/draft-07/schema#",
"type": "object",
"required": ["image", "service"],
"properties": {
"replicaCount": { "type": "integer", "minimum": 0 },
"image": {
"type": "object",
"required": ["repository"],
"properties": {
"repository": { "type": "string", "minLength": 1 },
"tag": { "type": "string" },
"pullPolicy": { "enum": ["Always", "IfNotPresent"] }
}
},
"service": {
"type": "object",
"properties": {
"type": { "enum": ["ClusterIP", "LoadBalancer"] },
"port": { "type": "integer" } } }
}
}"replicaCount": "3" (a string) is rejected by "type": "integer".helm template, so a schema error shows up as a ComparisonError on the Application before anything is applied."additionalProperties": false on strict objects to catch misspelled keys — but not at the top level if parents pass global values.Values are parsed as YAML, then handled as Go types. Most surprises come from that conversion.
| Symptom | Cause | Fix |
|---|---|---|
replicas: 1e+06 in output | Large integers become float64 and print in scientific notation | {{ .Values.big | int64 }} or keep as string |
| Env var rejected: "cannot unmarshal bool into string" | value: true unquoted in a container env | value: {{ .Values.flag | quote }} |
Version 1.10 became 1.1 | YAML parsed it as a float | Quote in values: tag: "1.10" |
nil pointer evaluating interface {}.tag | Parent key missing (image: absent) | Defaults in values.yaml, or with / (.Values.image | default dict).tag |
wrong type for value; expected string; got int | String function on a number (upper, trunc) | toString first |
Annotation value on became true | YAML 1.1 booleans (on, yes) | Quote annotation values |
Empty map rendered as {} under a key that expects a list | toYaml of a default {} | Wrap in with so nothing renders when empty |
required, fail and schemaA chart that renders invalid YAML silently is worse than one that refuses to render.
{{- /* templates/_validate.tpl */ -}}
{{- define "web.validate" -}}
{{- $a := .Values.autoscaling }}
{{- if and $a.enabled (lt (int $a.maxReplicas) (int $a.minReplicas)) }}
{{- fail "autoscaling.maxReplicas must be >= minReplicas" }}
{{- end }}
{{- if and .Values.ingress.enabled (not .Values.ingress.hosts) }}
{{- fail "ingress.enabled needs at least one ingress.hosts entry" }}
{{- end }}
{{- end }}
# deployment.yaml, first line
{{- include "web.validate" . }}
# fail fast on one key
{{- $repo := required "image.repository is required"
.Values.image.repository }}required**: one key that must be set, with a message naming it.fail**: rules across several keys that JSON Schema expresses badly.Each has bitten real teams; most are caught by helm template + lint + schema in CI.
.Values inside range/with instead of $.Values.{{- and getting blank lines or a key at the wrong indent.toYaml with indent instead of nindent on the same line as the key.spec.selector → immutable-field errors on the next upgrade.fullname) colliding with subcharts.lookup, randAlphaNum, now, uuidv4 → output changes every render → permanent OutOfSync..Release.IsInstall / IsUpgrade — always "install" under Argo CD.values.yaml into Git..Capabilities in plain helm template without --kube-version.Answer out loud, then check with helm template.
x | default "a" | quote expand to as nested calls?range .Values.hosts, how do you reach .Release.Name?{{- and -}}?nindent rather than indent after {{-?if?include vs template: which can be piped?values.schema.json catch that helm lint does not?lookup return under Argo CD, and why?replicas: 1e+06.Practice: Lab 3 in Hands-on labs — write the web chart from scratch, then break it with an indentation bug and a schema violation.
Dependencies, library charts, hooks, CRDs, OCI distribution, signing and a CI quality gate for charts.
Reuse charts instead of copying them: subcharts, conditions, aliases, global values and shared template libraries.
Chart.yamlHelm 3 and 4 (chart apiVersion: v2) declare dependencies in Chart.yaml; requirements.yaml was Helm 2.
# charts/web/Chart.yaml
apiVersion: v2
name: web
description: podinfo-based demo service
type: application
version: 0.3.0 # chart version (SemVer)
appVersion: "6.7.1" # app/image version, informational
dependencies:
- name: redis
version: "~20.0.0" # SemVer range
repository: oci://registry-1.docker.io/bitnamicharts
condition: redis.enabled
- name: common-lib
version: "1.x.x"
repository: file://../common-lib
- name: postgresql
alias: db
version: "15.5.x"
repository: oci://registry-1.docker.io/bitnamicharts
tags: [database]version: an exact version or a SemVer range (~, ^, 1.x.x, >=1.2 <2).repository: oci://…, an https:// chart repo URL, a repo alias @name, or file:// for a local path.condition: a values path; the subchart is skipped when it is false.tags: enable or disable groups with tags.database: false; condition wins over tags.alias: install the same chart under another name (and values key), or twice.import-values: pull a child's exported values up into the parent.Chart.lock, the charts/ directory, and who downloads whatRanges are resolved once and pinned in Chart.lock. Builds should use the lock, not re-resolve.
# resolve ranges, download .tgz files, write Chart.lock
helm dependency update charts/web
# download exactly what Chart.lock says (CI, reproducible)
helm dependency build charts/web
helm dependency list charts/web
# NAME VERSION REPOSITORY STATUS
# redis ~20.0.0 oci://registry-1... ok
# common-lib 1.x.x file://../common-lib ok
ls charts/web/charts/
# common-lib-1.0.0.tgz redis-20.0.3.tgz ...Chart.lock**: always. It is the reproducibility record.charts/*.tgz?** Vendoring makes renders offline and immune to upstream deletions, at the cost of binary files in Git. Pick one rule per repo.version whenever a dependency version changes.A subchart sees only its own section of the parent values, plus global.
# charts/web/values.yaml (parent)
global:
imageRegistry: 123456789012.dkr.ecr.ap-northeast-1.amazonaws.com
redis: # key = subchart name
enabled: true # used by `condition: redis.enabled`
architecture: standalone
auth:
enabled: false
db: # key = alias
auth:
database: web
# inside the redis subchart templates:
# .Values.architecture -> "standalone"
# .Values.global.imageRegistry -> the ECR hostglobal).global is copied into every subchart: use it sparingly (registry, cluster domain, environment name)..Chart but the parent's .Release.helm show values oci://…/redis --version 20.0.3.A chart with type: library contains only define blocks. It cannot be installed; it is used as a dependency.
# charts/common-lib/Chart.yaml
apiVersion: v2
name: common-lib
type: library
version: 1.0.0
# charts/common-lib/templates/_deployment.tpl
{{- define "common.deployment" -}}
apiVersion: apps/v1
kind: Deployment
metadata:
name: {{ include "common.fullname" . }}
labels:
{{- include "common.labels" . | nindent 4 }}
spec:
...
{{- end }}# charts/web/templates/deployment.yaml
{{- include "common.deployment" . }}
# charts/web/templates/service.yaml
{{- include "common.service" . }}
# a service-specific extra
# charts/web/templates/pdb.yaml
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
name: {{ include "common.fullname" . }}
spec:
maxUnavailable: 1
selector:
matchLabels:
{{- include "common.selectorLabels" . | nindent 6 }}Pays off when many in-house services share one shape: one fix rolls out by bumping the library version. Version it like an API (breaking template change = major). Bitnami common is a public example. Trade-off: the manifest shape lives in another chart, so it is harder to read.
An umbrella chart has no templates of its own, only dependencies.
| Question | Umbrella chart | One Application per component (Argo CD) |
|---|---|---|
| Unit of deploy and rollback | Everything together | Each component on its own |
| Blast radius of a bad value | Whole system | One component |
| Different namespaces per component | Awkward (one release namespace) | Natural |
| Ordering | Hooks and weights only | Sync waves across Applications (Chapter 7–8) |
| Release history size | One huge release Secret (1 MiB Secret limit) | Small, per component |
| Good fit | Shipping a product to customers as one artifact | Running your own platform with GitOps |
In GitOps, the "umbrella" becomes an ApplicationSet or app-of-apps (Chapter 8): many small Applications, each pointing at one chart.
Run Jobs at the right moment, test a release, and handle CRDs and immutable fields without outages.
A hook is a normal manifest with the helm.sh/hook annotation. It is not part of the release's managed resources.
| Hook | Runs |
|---|---|
pre-install | After render, before any resource is created |
post-install | After all resources are created |
pre-upgrade / post-upgrade | Around an upgrade |
pre-rollback / post-rollback | Around a rollback |
pre-delete / post-delete | Around uninstall |
test | Only when you run helm test |
helm.sh/hook-weight: "-5": lower runs first; weights are strings.helm.sh/hook-delete-policy: before-hook-creation (default), hook-succeeded, hook-failed.helm uninstall unless a delete policy removes them."pre-install,pre-upgrade".The classic hook: run a schema migration Job before Pods with the new code roll out.
# templates/migrate-job.yaml
{{- if .Values.migrations.enabled }}
apiVersion: batch/v1
kind: Job
metadata:
name: {{ include "web.fullname" . }}-migrate
annotations:
"helm.sh/hook": pre-install,pre-upgrade
"helm.sh/hook-weight": "-5"
"helm.sh/hook-delete-policy": before-hook-creation,hook-succeeded
spec:
backoffLimit: 1
activeDeadlineSeconds: 600
template:
spec:
restartPolicy: Never
containers:
- name: migrate
image: "{{ .Values.image.repository }}:{{ .Values.image.tag }}"
args: ["migrate", "up"]
{{- end }}activeDeadlineSeconds and backoffLimit so a stuck migration fails instead of blocking forever.hook-succeeded deletes the Job; drop it while debugging.pre-install.Argo CD renders with helm template and applies the result itself, so Helm's lifecycle never runs.
PreSync / PostSync.helm.sh/hook-weight is used like a sync wave inside the phase.pre-install and pre-upgrade hooks both run on every sync.pre-delete / post-delete map to PreDelete / PostDelete.pre-rollback, post-rollback and test hooks are not supported.post-install-only seeding Job runs again on every sync → make hooks idempotent.argocd.argoproj.io/hook, sync-wave) in charts you own — clearer behavior (Chapter 7).helm list shows nothing for Argo CD-managed apps: there is no Helm release Secret.helm test is not run by Argo CD; run tests as a PostSync hook if you need them in the sync.helm testTest Pods are hooks with helm.sh/hook: test. They run only when you ask, after install or upgrade.
# templates/tests/test-connection.yaml
apiVersion: v1
kind: Pod
metadata:
name: {{ include "web.fullname" . }}-test
annotations:
"helm.sh/hook": test
"helm.sh/hook-delete-policy": before-hook-creation
spec:
restartPolicy: Never
containers:
- name: curl
image: curlimages/curl:8.10.1
command: ["curl", "-fsS"]
args:
- http://{{ include "web.fullname" . }}:{{ .Values.service.port }}/healthzhelm upgrade --install web charts/web -n web \
--create-namespace --wait
helm test web -n web --logs
# NAME: web
# ...
# TEST SUITE: web-test
# Phase: Succeeded
# POD LOGS: web-test
# {"status":"OK"}Good tests: reach the Service, check /healthz, call one real endpoint. Not good: full integration suites — those belong in CI or in a Rollouts analysis (Chapter 11).
crds/ directory vs templatesCRDs are cluster-scoped and deleting one deletes every custom resource of that kind. Helm is deliberately cautious.
crds/ directory | CRDs in templates/ | |
|---|---|---|
| Installed | On first install only | On install and upgrade |
| Upgraded by Helm | Never | Yes |
| Deleted on uninstall | Never | Yes — and all its CRs with it |
| Templated | No (plain YAML) | Yes |
| Skip | --skip-crds | Values flag in the chart |
crds/ CRDs yourself: kubectl apply --server-side -f crds/ before helm upgrade.foo-crds) with its own lifecycle.helm.sh/resource-policy: keep.crds/ and manages them like other resources, so they ARE upgraded; spec.source.helm.skipCrds: true turns that off.Some fields cannot be changed in place; the API server rejects the update.
| Resource | Immutable field | Typical cause |
|---|---|---|
| Deployment / StatefulSet / DaemonSet | spec.selector | Changed selector labels in a chart bump |
| StatefulSet | volumeClaimTemplates, serviceName | Changing storage size or class |
| Job | spec.template | New image tag for a migration Job that was not deleted |
| Service | clusterIP | Hard-coded IP changed |
| PVC | most of spec (size can grow) | Changing storage class |
before-hook-creation on Job hooks.kubectl delete sts web --cascade=orphan keeps Pods running.--force-replace (was --force) replaces resources that cannot be patched: it deletes and recreates, so expect downtime.Replace=true or Force=true,Replace=true sync options on that resource (Chapter 7).Publish charts like container images, version them properly, and prove where they came from.
Since Helm 3.8, OCI support is GA: a chart is just another artifact next to your images.
REG=123456789012.dkr.ecr.ap-northeast-1.amazonaws.com
# ECR: one repository per chart, named <namespace>/<chart>
aws ecr create-repository --repository-name charts/web \
--image-tag-mutability IMMUTABLE
# Helm 4: host only, no oci:// and no path
aws ecr get-login-password | \
helm registry login $REG --username AWS --password-stdin
helm package charts/web # -> web-0.3.0.tgz
helm push web-0.3.0.tgz oci://$REG/charts
# Pushed: $REG/charts/web:0.3.0
# Digest: sha256:...
helm show values oci://$REG/charts/web --version 0.3.0
helm install web oci://$REG/charts/web --version 0.3.0helm push targets the namespace (oci://$REG/charts); Helm appends the chart name.helm registry login ghcr.io -u <you> --password-stdin with a token, push to oci://ghcr.io/<you>/charts.helm repo add and no index.yaml for OCI: you always reference the full path.A classic Helm repository is a static web server with an index.yaml listing every version.
| Aspect | Classic repo (index.yaml) | OCI registry |
|---|---|---|
| Hosting | Any web server, GitHub Pages, S3, ChartMuseum | ECR, GHCR, Harbor, Artifact Registry, ACR |
| Discovery | helm repo add + helm search repo | Full reference; no search |
| Scale | One index file grows forever, downloaded on every update | Per-artifact metadata |
| Auth and IAM | Basic auth or bucket policies | Registry auth, IAM on ECR |
| Immutability | Convention only | Tag immutability setting |
| Signing | PGP .prov files | cosign / Notation signatures stored in the registry |
| Replication, scanning | DIY | Registry features (ECR replication, scanning) |
Many public projects still publish both. For your own charts on AWS, ECR with immutable tags is the simplest choice.
version vs appVersionTwo independent versions. Mixing them up is the most common release mistake.
version (chart)0.3.0 with different content.appVersion (application)tag | default .Chart.AppVersion).helm list and the app.kubernetes.io/version label.image.tag in env values.appVersion: "6.7.1".image.tag in envs/<env>/values.yaml (Chapter 9).charts/web, CI checks the version bump (ct lint does), publish to OCI.Two mechanisms: Helm's PGP provenance files, and Sigstore cosign signatures for OCI artifacts.
# --- PGP provenance (classic) ---
helm package charts/web --sign \
--key 'CI Signing Key' --keyring ~/.gnupg/secring.gpg
# -> web-0.3.0.tgz + web-0.3.0.tgz.prov
helm verify web-0.3.0.tgz --keyring pubring.gpg
helm install web ./web-0.3.0.tgz --verify --keyring pubring.gpg
# --- cosign on the OCI artifact (keyless in CI) ---
DIGEST=$(helm push web-0.3.0.tgz oci://$REG/charts 2>&1 \
| awk '/Digest:/ {print $2}')
cosign sign --yes $REG/charts/web@$DIGEST
cosign verify $REG/charts/web@$DIGEST \
--certificate-identity-regexp 'https://github.com/<you>/.*' \
--certificate-oidc-issuer https://token.actions.githubusercontent.com.prov works with classic repos; key distribution is the hard part.Lint, render, validate, unit test, policy check and install-test every chart change before it reaches Git main.
Fast checks first; the kind install test runs last and only for changed charts.
helm lint and kubeconformLint checks the chart; kubeconform checks the rendered manifests against the Kubernetes OpenAPI schemas.
# chart structure, template errors, values schema
for env in dev staging prod; do
helm lint charts/web --strict -f envs/$env/values.yaml
done
# render each environment and validate against k8s 1.33 schemas
CAT=https://raw.githubusercontent.com/datreeio/CRDs-catalog/main
CRD="$CAT/{{.Group}}/{{.ResourceKind}}_{{.ResourceAPIVersion}}.json"
for env in dev staging prod; do
helm template web charts/web -n web -f envs/$env/values.yaml \
| kubeconform -strict -summary -kubernetes-version 1.33.0 \
-schema-location default -schema-location "$CRD"
done
# Summary: 6 resources found parsing stdin - Valid: 6, Invalid: 0helm lint: missing Chart.yaml fields, template execution errors, schema violations.contianers), wrong types, removed APIs (extensions/v1beta1 Ingress) for the target version.-strict rejects unknown fields — the typo class lint cannot see.helm-unittest is a Helm plugin: YAML test suites that render templates and assert on the output.
# charts/web/tests/deployment_test.yaml
suite: deployment
templates:
- templates/deployment.yaml
- templates/configmap.yaml
tests:
- it: omits replicas when autoscaling is on
template: templates/deployment.yaml
set:
autoscaling.enabled: true
asserts:
- notExists:
path: spec.replicas
- it: uses appVersion when tag is empty
template: templates/deployment.yaml
asserts:
- matchRegex:
path: spec.template.spec.containers[0].image
pattern: ":6\\.7\\.1$"
- it: fails on bad autoscaling bounds
template: templates/deployment.yaml
set:
autoscaling: { enabled: true, minReplicas: 5, maxReplicas: 2 }
asserts:
- failedTemplate:
errorMessage: autoscaling.maxReplicas must be >= minReplicashelm plugin install https://github.com/helm-unittest/helm-unittest --verify=false (Helm 4 verifies plugin provenance; use helm-unittest v1.x).helm unittest charts/web.equal, notExists, matchRegex, contains, isKind, hasDocuments, failedTemplate.fail, required, conditional resources) so refactors keep them.matchSnapshot) catch unintended output changes; review snapshot diffs in PRs.ct: lint and install only the charts that changedchart-testing compares your branch with the target branch, finds changed charts, and tests them.
# ct.yaml
target-branch: main
chart-dirs:
- charts
validate-maintainers: false
check-version-increment: true
helm-extra-args: --timeout 300s
# in CI, inside the repo
ct lint --config ct.yaml # lint + version bump check
kind create cluster --name ct
ct install --config ct.yaml # install, run helm test, uninstall
kind delete cluster --name ct
# values variants: charts/web/ci/*-values.yaml
# each file is a separate install testcheck-version-increment: a changed chart without a version bump fails the PR.ct install installs into a throwaway namespace, waits, runs helm test, then deletes.charts/<chart>/ci/ are extra values sets, each installed separately — test prod-like and minimal configs.helm-docs and policy checks on rendered YAMLGenerate the README from values comments; check rendered manifests against security policy.
# values.yaml comments that helm-docs reads
# -- Number of Pods when autoscaling is off
replicaCount: 2
helm-docs --chart-search-root charts
# -> charts/web/README.md with a values table
git diff --exit-code charts/web/README.md # CI: docs up to date
# policy on rendered output
helm template web charts/web -f envs/prod/values.yaml > /tmp/prod.yaml
checkov -f /tmp/prod.yaml --framework kubernetes
trivy config charts/web
kyverno apply policies/ --resource /tmp/prod.yamlresources.requests / limits.runAsNonRoot, readOnlyRootFilesystem.:latest or no digest.hostPath, privileged, added capabilities.scripts/ci-chart.sh: the gates in one placeKeep CI logic in a script so it runs the same on a laptop and in any CI system.
#!/usr/bin/env bash
set -euo pipefail
CHART=charts/web
helm dependency build $CHART
helm unittest $CHART
for env in dev staging prod; do
V=envs/$env/values.yaml
helm lint $CHART --strict -f $V
helm template web $CHART -n web -f $V > /tmp/$env.yaml
kubeconform -strict -summary -kubernetes-version 1.33.0 /tmp/$env.yaml
kyverno apply policies/ --resource /tmp/$env.yaml
done
helm-docs --chart-search-root charts
git diff --exit-code -- '*.md'
ct lint --config ct.yaml
# install test only in CI (needs Docker)
if [[ "${CI:-}" == "true" ]]; then
kind create cluster --name ct --wait 120s
ct install --config ct.yaml
fihelm package → helm push to ECR → cosign sign the digest.helm install into shared clusters — Argo CD deploys; CI only builds, tests and publishes.Helmfile describes a set of Helm releases with per-environment values and applies them with diffs.
# helmfile.yaml
environments:
dev:
values: [envs/dev/env.yaml]
prod:
values: [envs/prod/env.yaml]
---
repositories:
- name: ingress-nginx
url: https://kubernetes.github.io/ingress-nginx
releases:
- name: ingress-nginx
namespace: ingress-nginx
chart: ingress-nginx/ingress-nginx
version: 4.11.3
- name: web
namespace: web
chart: ./charts/web
values:
- envs/{{ .Environment.Name }}/values.yaml
needs: [ingress-nginx/ingress-nginx]helmfile -e dev diff, helmfile -e dev apply: push-style, run by a person or CI.helm-diff plugin for diffs.Most of these show up only on the second install or the first upgrade.
version.Chart.lock committed.crds/ CRDs to upgrade with helm upgrade.field is immutable.--force-replace casually: it deletes and recreates.-strict or without CRD schemas.helm upgrade to shared clusters, bypassing GitOps.Answer, then prove it in Lab 4.
helm dependency update vs build: which belongs in CI?pre-install hook run on every Argo CD sync?crds/ on helm upgrade? Under Argo CD?helm.sh/resource-policy: keep do?helm registry login point in Helm 4?version vs appVersion: which does an image-only release change?ct lint check that helm lint does not?-strict catch?Practice: Lab 4 in Hands-on labs — library chart, dependency, migration hook, helm test, OCI push and the CI script.
Template-free customization with bases, overlays and patches — and when to use Helm, Kustomize, or both.
Plain YAML in, plain YAML out: bases, overlays, patches, generators, components and Helm chart inflation.
A base is valid Kubernetes YAML. An overlay points at the base and declares changes. kustomize build prints the result.
webBases never know about environments. Overlays reference the base by relative path (or a remote Git URL).
# apps/web/base/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- deployment.yaml
- service.yaml
labels:
- pairs:
app.kubernetes.io/name: web
includeSelectors: true
# apps/web/base/deployment.yaml (excerpt)
spec:
replicas: 1
template:
spec:
containers:
- name: web
image: ghcr.io/stefanprodan/podinfo# envs/prod/web/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
namespace: web-prod
resources:
- ../../../apps/web/base
- hpa.yaml # prod-only resource
images:
- name: ghcr.io/stefanprodan/podinfo
newTag: "6.7.1"
replicas:
- name: web
count: 3
patches:
- path: resources-patch.yaml
labels:
- pairs:
env: prod # not in selectors
# render
kustomize build envs/prod/webkubectl apply -k uses a Kustomize version embedded in kubectl, which can lag the standalone kustomize binary. Argo CD ships its own kustomize; pin versions in CI to match.
Both live under patches:. Older patchesStrategicMerge and patchesJson6902 fields are deprecated.
# envs/prod/web/resources-patch.yaml (strategic merge)
apiVersion: apps/v1
kind: Deployment
metadata:
name: web
spec:
template:
spec:
containers:
- name: web # list items matched by name
resources:
requests: { cpu: 200m, memory: 256Mi }
limits: { memory: 512Mi }# JSON 6902 patch, inline, selected by target
patches:
- target:
kind: Deployment
name: web
patch: |-
- op: add
path: /spec/template/spec/containers/0/env/-
value: { name: PODINFO_UI_MESSAGE, value: prod }
- op: replace
path: /spec/revisionHistoryLimit
value: 5
# one patch for many objects: target by label selector
- target:
kind: Deployment
labelSelector: tier=frontend
path: add-topology-spread.yamlStrategic merge looks like the object and merges containers, ports and env by key. JSON 6902 does precise add / remove / replace by path — use it for CRDs, where strategic merge replaces whole lists. target can match group, version, kind, name, namespace, labelSelector, annotationSelector.
Generated names get a hash suffix of their content, and every reference is rewritten — so a change rolls the Pods automatically.
# envs/prod/web/kustomization.yaml
configMapGenerator:
- name: web-config
literals:
- PODINFO_UI_COLOR=#34577c
files:
- config/app.yaml # key = file name
# overlay adding to a ConfigMap generated in the base
- name: web-config
behavior: merge # create | replace | merge
literals:
- PODINFO_UI_MESSAGE=prod
# output
# kind: ConfigMap
# metadata:
# name: web-config-7c5fk2h9g6
# ... Deployment envFrom.configMapRef.name: web-config-7c5fk2h9g6prune: true they are removed after the sync.configurations entry.disableNameSuffixHash: true gives stable names but loses the automatic rollout.secretGenerator has the same syntax and produces Secret data — but never commit its input files with real secrets. Use SOPS (KSOPS) or External Secrets instead (Chapter 10).
Built-in transformers cover most per-environment differences without a patch file.
| Field | Example | Effect |
|---|---|---|
images | - name: ghcr.io/stefanprodan/podinfo + newTag: "6.7.1" (or digest:) | Rewrites matching image references in all workloads |
replicas | - name: web + count: 3 | Sets spec.replicas on the named workload |
namespace | namespace: web-prod | Sets namespace on every namespaced resource |
namePrefix / nameSuffix | namePrefix: prod- | Renames resources and updates references |
labels | pairs: {env: prod}, includeSelectors: false | Adds labels; selectors only if you opt in |
commonAnnotations | team: payments | Adds annotations everywhere |
commonLabels (deprecated) | — | Also changes selectors → immutable-field errors; use labels |
CI image bump without editing YAML by hand: cd envs/dev/web && kustomize edit set image ghcr.io/stefanprodan/podinfo:6.8.0 — a one-line diff, ideal for promotion PRs (Chapter 9).
A Component is a reusable bundle of resources and patches. Overlays opt in; unlike a base, it is applied on top of what already exists.
# apps/web/components/ingress/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1alpha1
kind: Component
resources:
- ingress.yaml
patches:
- target: { kind: Deployment, name: web }
patch: |-
- op: add
path: /spec/template/metadata/annotations/ingress-enabled
value: "true"
# envs/prod/web/kustomization.yaml
resources:
- ../../../apps/web/base
components:
- ../../../apps/web/components/ingress
- ../../../apps/web/components/hpaingress.enabled: true in a Helm chart — but without templates.helmCharts, then patch itKustomize can render a chart and treat the output as resources — useful for third-party charts that lack a value you need.
# envs/prod/ingress-nginx/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
namespace: ingress-nginx
helmCharts:
- name: ingress-nginx
repo: https://kubernetes.github.io/ingress-nginx
version: 4.11.3
releaseName: ingress-nginx
valuesInline:
controller:
replicaCount: 2
patches:
- target: { kind: Deployment, name: ingress-nginx-controller }
path: add-priority-class.yaml
# render: helm must be installed
kustomize build --enable-helm envs/prod/ingress-nginx--enable-helm; in Argo CD set kustomize.buildOptions: --enable-helm in argocd-cm.helm template: no hooks lifecycle, no release Secret — the same as Argo CD's own Helm support.helmCharts route is usually simpler.Pick the renderer per use case, and know exactly how Argo CD detects and renders each one.
Most real platforms use both: Helm for packaging, Kustomize for last-mile environment changes.
| Use case | Best fit | Why |
|---|---|---|
| Third-party software (ingress-nginx, cert-manager, Prometheus) | Upstream Helm chart + values per env | Vendors ship and test charts; you only own values |
| Many in-house services with the same shape | Helm library chart (Chapter 4) | One template set, versioned, per-service values |
| A few in-house services, small team | Kustomize base + overlays | Plain YAML, nothing to learn beyond Kubernetes |
| Upstream chart missing a field | Kustomize helmCharts + patch | Patch output instead of forking the chart |
| Distributing software to other teams or customers | Helm chart in OCI | Versioned package with a documented values API |
| Per-environment tweaks on any of the above | Values files (Helm) or overlays (Kustomize) | Keep env files small and diffable |
Prod needs 3 replicas, image 6.7.1 and bigger requests.
# Helm: envs/prod/values.yaml
replicaCount: 3
image:
tag: "6.7.1"
resources:
requests: { cpu: 200m, memory: 256Mi }
limits: { memory: 512Mi }
# render
helm template web charts/web -n web-prod \
-f envs/prod/values.yaml# envs/prod/web/kustomization.yaml
resources: [../../../apps/web/base]
namespace: web-prod
images:
- name: ghcr.io/stefanprodan/podinfo
newTag: "6.7.1"
replicas:
- { name: web, count: 3 }
patches:
- path: resources-patch.yaml
# render
kustomize build envs/prod/webThe repo-server looks at the path in the Application source and picks a tool, unless the spec says which.
| Path contains | Tool | Runs |
|---|---|---|
Chart.yaml | Helm | helm template with spec.source.helm settings |
kustomization.yaml, kustomization.yml or Kustomization | Kustomize | kustomize build with spec.source.kustomize settings |
Only .yaml / .yml / .json (and Jsonnet) | Directory | Reads files as-is; directory.recurse for subfolders |
| Matches a config management plugin | CMP sidecar | Your own render command |
helm list, no release Secrets, no helm rollback — Argo CD owns history.spec.source.helm.valueFiles / values / valuesObject, or from another repo with multi-source (Chapter 8).images, namePrefix, patches, components) are handy for quick overrides, but keeping them in Git files keeps the repo the single source of truth.The lab repo uses Helm for the web chart and third-party apps, Kustomize for plain manifests and patches.
gitops-lab/
├── charts/
│ └── web/ # our chart (Chapters 3–4)
├── apps/
│ ├── web/base/ # Kustomize base (alternative path)
│ └── web/components/{ingress,hpa}/
├── envs/
│ ├── dev/
│ │ ├── values.yaml # Helm values for web
│ │ └── web/kustomization.yaml
│ ├── staging/ ...
│ └── prod/
│ ├── values.yaml
│ ├── web/kustomization.yaml
│ └── ingress-nginx/kustomization.yaml # helmCharts + patch
└── clusters/
├── gitops/ # hub cluster: Argo CD apps (Ch 6–8)
└── prod/envs/<env>/.Template diffs hide impact. Render before and after, per environment, and diff the YAML.
#!/usr/bin/env bash
# scripts/render-diff.sh — run in CI on every PR
set -euo pipefail
render() { # $1 = git ref, $2 = env
git worktree add -q /tmp/r-$1 $1 2>/dev/null || true
(cd /tmp/r-$1 &&
helm template web charts/web -n web -f envs/$2/values.yaml &&
kustomize build envs/$2/web)
}
for env in dev staging prod; do
render origin/main $env > /tmp/$env-before.yaml
render HEAD $env > /tmp/$env-after.yaml
diff -u /tmp/$env-before.yaml /tmp/$env-after.yaml \
> /tmp/$env.diff || echo "changes in $env"
doneenvs/staging with envs/prod output shows unintended drift between environments.dyff give semantic YAML diffs; Argo CD also shows a live diff before you sync (argocd app diff, Chapter 7).Most are invisible until the second environment or the first base change.
commonLabels changing selectors of existing Deployments → immutable-field errors.secretGenerator input files with real secrets.--enable-helm in Argo CD kustomize.buildOptions.Answer, then verify with kustomize build.
configMapGenerator trigger a rollout?commonLabels dangerous on existing workloads?kustomize edit set image help promotion?helmCharts need to work in Argo CD?helm rollback an Argo CD Helm app?Practice: Lab 5 in Hands-on labs — Kustomize overlays for dev and prod on top of the chart output, and a rendered diff between environments.
What each component does, how the reconcile loop works, how to install and access Argo CD, and the CRDs you write every day.
Six controllers and servers, one cache, no database: every piece of state is a Kubernetes object.
Desired state comes from Git (or a Helm / OCI registry). Live state comes from the Kubernetes API. Argo CD compares and converges.
When something breaks, the component tells you which log to read.
| Component | Kind | Responsibility | Read its logs when… |
|---|---|---|---|
argocd-server | Deployment | UI, REST/gRPC API, auth, RBAC enforcement, webhook receiver | login fails, RBAC denies, webhook does nothing |
argocd-repo-server | Deployment | Clones repos, runs Helm / Kustomize / plugins, returns plain manifests | ComparisonError, render errors, repo auth errors |
argocd-application-controller | StatefulSet | Watches clusters, diffs live vs desired, computes health, runs syncs | sync stuck, wrong health, slow reconcile |
argocd-applicationset-controller | Deployment | Turns generators + template into Application objects | apps not created or deleted unexpectedly |
argocd-redis | Deployment | Cache for rendered manifests and cluster state; safe to lose | high latency, cache misses |
argocd-dex-server | Deployment | Optional OIDC/SAML/GitHub broker for SSO | SSO redirects fail |
argocd-notifications-controller | Deployment | Sends Slack / email / webhook messages on triggers | notifications missing |
argocd-commit-server | Deployment | Writes hydrated manifests back to Git (Source Hydrator) | hydration commits missing |
Back up Git and a handful of Kubernetes objects and you can rebuild Argo CD anywhere.
argocdApplication, ApplicationSet, AppProject (CRDs).argocd.argoproj.io/secret-type: repository, repo-creds or cluster.argocd-cm: general settings, health/diff customizations.argocd-rbac-cm: RBAC policy. argocd-cmd-params-cm: component flags.argocd-secret: admin password hash, signing key, webhook secrets.argocd namespace can change what Argo CD deploys: protect it like cluster-admin.People say "sync" for all three. Argo CD separates them, and so should you.
syncPolicy.automated).Refresh interval: timeout.reconciliation in argocd-cm, default 120 s + up to 60 s jitter (timeout.reconciliation.jitter), so about 2–3 minutes on self-managed Argo CD. The EKS Capability for Argo CD polls Git every 6 minutes. Webhooks make both near-instant.
Polling every app every 3 minutes costs Git API calls and adds delay. A webhook tells Argo CD exactly which repo changed.
https://<argocd-host>/api/webhook.argocd-secret (webhook.github.secret, webhook.gitlab.secret, …).repoURL matches the pushed repo.timeout.reconciliation for very large installs; keep webhooks for speed.argocd.argoproj.io/manifest-generate-paths annotation so a push only refreshes apps whose paths changed (Chapter 12).# argocd-cm (self-managed)
data:
timeout.reconciliation: 180s
# argocd-secret
stringData:
webhook.github.secret: <random>
# force a refresh from the CLI
argocd app get web-dev --refresh
argocd app get web-dev --hard-refreshArgo CD never runs helm install. It renders, then applies plain manifests like any other app.
| Source contains | Rendered with |
|---|---|
Chart.yaml (or chart: from a Helm repo) | helm template with values / parameters |
kustomization.yaml | kustomize build (bundled binary) |
Plain YAML / JSON (or .jsonnet) | Directory source (recurse, include/exclude) |
| Anything else | Config Management Plugin sidecar |
helm list shows nothing.lookup returns empty: there is no cluster during render..Release.Name = Application name unless helm.releaseName is set..Capabilities come from the target cluster version and API list Argo CD knows.Render once, apply everywhere: the repo-server output is what argocd app manifests web-dev prints. Debug renders there before blaming the cluster.
Helm chart or plain manifests, HA or not, and three ways to reach the API.
Pick one and keep Argo CD itself in Git. Most platform teams use the Helm chart.
manifests/install.yaml: cluster-wide, non-HA.manifests/ha/install.yaml: HA (Redis HA, several replicas, needs at least 3 nodes).namespace-install.yaml: namespace-scoped (no cluster-wide permissions).*-install-with-hydrator.yaml: adds the commit server.argo/argo-cdargoproj/argo-helm.configs.cm, configs.params, configs.rbac, configs.repositories.redis-ha.enabled, more replicas for server, repoServer, applicationSet; controller replicas enable sharding.# manifests
kubectl create namespace argocd
U=https://raw.githubusercontent.com
U=$U/argoproj/argo-cd/stable
kubectl apply -n argocd \
--server-side --force-conflicts \
-f $U/manifests/install.yaml
# or Helm (lab default)
helm repo add argo \
https://argoproj.github.io/argo-helm
helm install argocd argo/argo-cd \
-n argocd --create-namespace \
-f argocd-values.yaml
kubectl -n argocd get podsargocd-values.yamlEverything you would otherwise click in the UI goes here, so the install is reproducible.
configs:
params:
server.insecure: true # TLS ends in front
cm:
timeout.reconciliation: 180s
admin.enabled: true # off after SSO
kustomize.buildOptions: --enable-helm
rbac:
policy.default: role:readonly
repositories:
gitops-lab:
url: https://github.com/<you>/gitops-lab
type: git
server:
replicas: 1
repoServer:
replicas: 1
controller:
replicas: 1
dex:
enabled: false # Chapter 10
notifications:
enabled: trueconfigs.params becomes argocd-cmd-params-cm; configs.cm becomes argocd-cm; configs.rbac becomes argocd-rbac-cm.server.insecure only disables TLS on the server pod; something in front must still terminate TLS.--version) and read the chart changelog on upgrade.ServiceMonitors.The initial admin password is random and stored in a Secret. Change it, then disable admin once SSO works.
# reach the API locally
kubectl -n argocd port-forward svc/argocd-server 8080:443
# initial password (delete the Secret after changing it)
argocd admin initial-password -n argocd
argocd login localhost:8080 --username admin \
--password '<pw>' --insecure
argocd account update-password
# core mode: no argocd-server, uses your kubeconfig + RBAC
argocd login --core
kubectl config set-context --current --namespace=argocd
argocd app listargocd --core**: talks to the Kubernetes API directly; Kubernetes RBAC applies; handy for headless installs.kubectl on CRDs**: create Application objects directly; GitOps-friendly and what app-of-apps does.--grpc-web.The server speaks HTTPS and gRPC on one port. Load balancers must handle both, or the CLI uses gRPC-Web.
server.insecure: true.Content-Type: application/grpc header), or use --grpc-web.GRPCRoute.argocd --core or port-forward for a small team.url in argocd-cm to the external URL (SSO redirects and notifications links use it)./api/webhook must be reachable from the Git provider (or use polling).Label tells Argo CD what the Secret is. Credential templates match every repo URL with a prefix.
apiVersion: v1
kind: Secret
metadata:
name: gitops-lab
namespace: argocd
labels:
argocd.argoproj.io/secret-type: repository
stringData:
type: git
url: https://github.com/<you>/gitops-lab
username: git
password: <fine-grained PAT, read-only>
---
apiVersion: v1
kind: Secret
metadata:
name: github-org-creds
namespace: argocd
labels:
argocd.argoproj.io/secret-type: repo-creds
stringData:
type: git
url: https://github.com/<you> # prefix
githubAppID: "123456"
githubAppInstallationID: "7890123"
githubAppPrivateKey: |
-----BEGIN RSA PRIVATE KEY-----repository: one repo. repo-creds: a template for every URL under the prefix.sshPrivateKey), GitHub App, Google Cloud source, Azure workload identity.type: helm + url. OCI Helm (ECR, GHCR): type: helm with enableOCI: "true".argocd-cm: use Secrets.argocd repo add <url> --username … --password ….Application and AppProject: the two objects every GitOps change goes through.
ApplicationSource (where), destination (to which cluster and namespace), project (allowed by whom), sync policy (how).
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: web-dev
namespace: argocd
finalizers:
- resources-finalizer.argocd.argoproj.io
spec:
project: team-web
source:
repoURL: https://github.com/<you>/gitops-lab
targetRevision: main # branch/tag/SHA
path: charts/web
helm:
releaseName: web
valueFiles:
- ../../envs/dev/web/values.yaml
destination:
name: in-cluster # or server: <url>
namespace: web-dev
syncPolicy:
automated:
prune: true
selfHeal: true
syncOptions:
- CreateNamespace=truemetadata.namespace: argocd unless "apps in any namespace" is enabled (Chapter 10).targetRevision: branch = moving target; tag or SHA = pinned. For Helm repos it is the chart version (semver ranges allowed).destination: use name or server, not both.source.helm: valueFiles, values, valuesObject, parameters, releaseName, ignoreMissingValueFiles.source.kustomize: images, namePrefix, patches, namespace.source.directory: recurse, include, exclude, jsonnet.Deleting an Application deletes its resources only if the finalizer says so.
| Finalizer | Effect on delete |
|---|---|
| none | Application removed, workloads left running (orphaned) |
resources-finalizer.argocd.argoproj.io | Cascade delete, foreground |
resources-finalizer.argocd.argoproj.io/background | Cascade delete, background |
argocd.argoproj.io/sync-options: Delete=false (PVCs, namespaces with data).argocd app delete web-dev --cascade=false orphans resources.Every Application belongs to exactly one project. The project decides where it may come from and where it may go.
apiVersion: argoproj.io/v1alpha1
kind: AppProject
metadata:
name: team-web
namespace: argocd
spec:
description: Web team services
sourceRepos:
- https://github.com/<you>/gitops-lab
destinations:
- name: in-cluster
namespace: 'web-*'
- name: prod
namespace: 'web-*'
clusterResourceWhitelist: [] # none
namespaceResourceBlacklist:
- {group: '', kind: ResourceQuota}
roles:
- name: ci
policies:
- p, proj:team-web:ci, applications,
sync, team-web/*, allowsourceRepos: Git / Helm / OCI URLs allowed (globs allowed, ! negates).destinations: cluster + namespace pairs (globs).clusterResourceWhitelist / Blacklist: cluster-scoped kinds (Namespace, ClusterRole, CRD…).namespaceResourceWhitelist / Blacklist: namespaced kinds.roles: project-scoped RBAC and JWT tokens for CI.syncWindows: when syncs are allowed (Chapter 7).sourceNamespaces: namespaces whose Applications may use this project (apps in any namespace).The built-in default project allows everything from anywhere to anywhere. Lock it down or stop using it once you have real teams.
Many blog posts and charts were written for 2.x. These defaults changed in 3.0.
| Area | Argo CD 2.x | Argo CD 3.x | What to do |
|---|---|---|---|
| Resource tracking | Label app.kubernetes.io/instance | Annotation argocd.argoproj.io/tracking-id | Nothing for new installs; on upgrade, expect a one-time re-annotation |
| RBAC update / delete | Applied to the app and its resources | Applies to the Application only | Grant update/* or delete/* explicitly when users need resource actions |
Diffing .status | Status ignored for some kinds | Status ignored for all resources | Remove old custom ignore rules for status |
| Pod logs | Readable by anyone who can see the app | Need the logs, get RBAC permission | Add a logs policy for developers |
Repositories in argocd-cm | Supported (legacy) | Removed | Move to labelled Secrets |
Read the upgrade notes for every minor version you cross (operator-manual/upgrading/3.x-3.y) — Chapter 12 covers the upgrade routine.
Each one has bitten a real platform team.
--server-side.admin enabled and the initial password unchanged.url in argocd-cm: SSO callback and notification links break.destination with both name and server.lookup or helm install side effects → empty or wrong output.default project for everything.argocd app get web-dev → conditions show ComparisonError / SyncError.argocd app manifests web-dev → what repo-server rendered.kubectl -n argocd logs deploy/argocd-repo-server for render errors.kubectl -n argocd logs sts/argocd-application-controller for sync errors.Explain the answer out loud, naming the component or field.
helm list show nothing for an Argo CD-managed chart?server.insecure: true actually disable?argocd login --core?repo-creds Secret differ from a repository Secret?default project dangerous?Practice: Lab 6 in Hands-on labs — install Argo CD with its Helm chart, declarative repo credentials, an AppProject and a first Application.
Status and health, sync policy and options, ordering with phases, waves and hooks, and taming diffs and drift.
Two independent signals: is the cluster equal to Git, and is the app actually working?
An app can be Synced and Degraded (Git is wrong) or OutOfSync and Healthy (someone scaled it by hand).
| Value | Meaning |
|---|---|
| Synced | Every managed resource matches the rendered manifests |
| OutOfSync | At least one resource differs, is missing, or should be pruned |
| Unknown | Comparison failed (render error, repo or cluster unreachable) |
| Value | Meaning |
|---|---|
| Healthy | Resource is ready (Deployment rolled out, PVC bound…) |
| Progressing | Not ready yet but expected to become ready |
| Degraded | Failed (rollout deadline exceeded, crash loops…) |
| Suspended | Paused on purpose (CronJob suspended, Rollout paused) |
| Missing | Should exist but does not |
| Unknown | Health check could not decide |
Application health = the worst health among its resources. Alerts should watch both signals (Chapter 12).
Each combination points to a different owner and a different fix.
| Sync | Health | Typical cause | Who fixes it |
|---|---|---|---|
| Synced | Healthy | Everything as intended | — |
| Synced | Degraded | Git declares something broken: bad image tag, wrong probe, missing Secret | App team: fix Git, or roll back with git revert |
| OutOfSync | Healthy | New commit not synced yet, manual change, or a controller rewriting fields | Sync, enable self-heal, or add an ignore rule |
| OutOfSync | Degraded / Missing | Sync failed half-way, hook failed, or a resource was deleted by hand | Read the operation state, then re-sync |
| Unknown | any | Render error (ComparisonError), repo credentials, cluster unreachable | Platform team: repo-server / cluster access |
Argo CD knows core kinds. For CRDs it knows many popular ones, and you add the rest in Lua.
type: LoadBalancer and Ingress: an address is assigned.Bound.Application objects (app-of-apps) have no health check by default.status.conditions.Defined in argocd-cm per group and kind. It receives the live object as obj and returns status and message.
# argocd-cm
data:
resource.customizations.health.example.com_Database: |
hs = {}
if obj.status ~= nil and obj.status.conditions ~= nil then
for _, c in ipairs(obj.status.conditions) do
if c.type == "Ready" and c.status == "True" then
hs.status = "Healthy"
hs.message = "database ready"
return hs
end
if c.type == "Failed" and c.status == "True" then
hs.status = "Degraded"
hs.message = c.message
return hs
end
end
end
hs.status = "Progressing"
hs.message = "waiting for Ready"
return hsresource.customizations.health.<group>_<kind>; core group kinds use just <kind>.status (first seconds after create).argocd admin settings resource-overrides health ./obj.yaml --argocd-cm-path ./argocd-cm.yaml.Tracking decides what gets pruned. Get it wrong and Argo CD deletes, or ignores, the wrong objects.
application.resourceTrackingMethod in argocd-cm| Method | Marker | Notes |
|---|---|---|
annotation (default in 3.x) | argocd.argoproj.io/tracking-id: <app>:<group>/<kind>:<ns>/<name> | Unambiguous; not copied to children |
label (2.x default) | app.kubernetes.io/instance: <app> | Clashes with Helm charts that set the same label; copied to Pods |
annotation+label | both | Migration aid; label is informational |
FailOnSharedResource=true makes a sync fail instead of stealing an object owned by another app.Manual or automated, with or without prune and self-heal, and the options that change how objects are applied.
Automated sync applies every new commit. Prune and self-heal are separate switches on top.
spec:
syncPolicy:
automated:
prune: true # delete objects removed from Git
selfHeal: true # revert manual changes in the cluster
allowEmpty: false # refuse to prune everything
retry:
limit: 5
backoff:
duration: 10s
factor: 2
maxDuration: 3m
syncOptions:
- CreateNamespace=true
- ServerSideApply=true| Switch | Off (default) | On |
|---|---|---|
automated | OutOfSync until someone clicks Sync | New Git revision is synced automatically |
prune | Removed objects stay (marked for pruning) | Removed objects are deleted |
selfHeal | Manual edits stay until the next commit | Live drift is reverted after a short delay |
allowEmpty | Refuses a sync that would prune every resource | Allows an app to become empty |
retry | A failed sync waits for the next commit | Failed sync retried with backoff |
Automated sync syncs a given commit once. If that sync fails, it is not retried for the same commit unless retry is set — the usual reason an app sits at "Sync failed" until someone pushes again.
Start strict in dev, careful in prod. Auto-sync in prod is fine once you trust your review process.
| Environment | automated | prune | selfHeal | Why |
|---|---|---|---|---|
| dev / preview | on | on | on | Fast feedback; Git is always the truth |
| staging | on | on | on | Mirror prod behavior to catch prune and drift surprises |
| prod (common) | on | on | on | Merge = deploy; protection comes from PR review, sync windows and Rollouts |
| prod (cautious) | off | — | — | Humans press Sync; easier rollback with argocd app rollback |
| shared infra (CRDs, operators) | on | off | on | Never auto-delete CRDs: deleting one deletes every custom resource of that kind |
Rollback with argocd app rollback only works when automated sync is off. With auto-sync on, roll back with git revert (Chapter 9).
Set per Application in syncPolicy.syncOptions, or per resource with the argocd.argoproj.io/sync-options annotation.
| Option | Effect | Use when |
|---|---|---|
CreateNamespace=true | Create the destination namespace if missing | Every app that owns its namespace (labels via managedNamespaceMetadata) |
ServerSideApply=true | Apply with Kubernetes server-side apply | Large CRDs (annotation size limit), shared field ownership with other controllers |
PruneLast=true | Prune after all other waves succeeded | Avoid deleting the old object before the new one is healthy |
ApplyOutOfSyncOnly=true | Apply only resources that differ | Apps with thousands of objects |
Replace=true | kubectl replace / create instead of apply | Objects that cannot be patched; dangerous, deletes unspecified fields |
PrunePropagationPolicy=foreground | Deletion propagation for pruned objects | Wait for dependents to go first (background, orphan also exist) |
RespectIgnoreDifferences=true | Do not overwrite ignored fields when syncing | HPA-owned replicas, webhook-injected fields |
Validate=false | Skip schema validation | Rare: objects with intentionally unknown fields |
FailOnSharedResource=true | Fail if another app already manages an object | Multi-team clusters |
Fine-grained control without changing the whole Application.
# never prune or delete this PVC, even if removed from Git
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: data
annotations:
argocd.argoproj.io/sync-options: Prune=false,Delete=false
---
# CRD too large for client-side apply
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
name: widgets.example.com
annotations:
argocd.argoproj.io/sync-options: ServerSideApply=true
---
# custom resource whose CRD is created in the same sync
apiVersion: example.com/v1
kind: Widget
metadata:
name: demo
annotations:
argocd.argoproj.io/sync-options: SkipDryRunOnMissingResource=truePrune=false: Argo CD reports it as "requires pruning" but never deletes it.Delete=false: kept when the Application itself is deleted.ServerSideApply=true / Replace=true: apply strategy for this object only.SkipDryRunOnMissingResource=true: skip the dry run when the CRD does not exist yet (it is created earlier in the same sync).argocd.argoproj.io/compare-options: IgnoreExtraneous: do not count this object toward OutOfSync (for generated objects).Self-heal reverts any difference. If another controller owns a field, the two loop forever.
spec.replicas; Git says 2; self-heal sets 2; HPA sets 6 again.caBundle or status-like fields in spec.replicas from the Deployment when an HPA owns it.ignoreDifferences for that field + RespectIgnoreDifferences=true (section 22).compare-options: IgnoreExtraneous or exclude the kind globally.argocd app diff web-dev shows nothing after a sync.Run migrations before the app, smoke tests after it, and clean-up when a sync fails.
Lower wave numbers first. Argo CD waits for each wave to be healthy before starting the next.
Order of precedence: phase → wave → kind → name. Default wave is 0; waves can be negative.
A hook is any resource (usually a Job) annotated with a phase. It is created during the sync, not kept in sync.
apiVersion: batch/v1
kind: Job
metadata:
generateName: db-migrate-
annotations:
argocd.argoproj.io/hook: PreSync
argocd.argoproj.io/hook-delete-policy: >-
BeforeHookCreation,HookSucceeded
argocd.argoproj.io/sync-wave: "-1"
spec:
backoffLimit: 1
activeDeadlineSeconds: 600
template:
spec:
restartPolicy: Never
containers:
- name: migrate
image: ghcr.io/<you>/web-migrate:1.4.0
args: ["migrate", "up"]| Annotation value | Meaning |
|---|---|
hook: PreSync | Before any normal resource is applied |
hook: Sync | Alongside normal resources (after PreSync) |
hook: PostSync | After all resources are healthy |
hook: SyncFail | Only when the sync failed |
hook: PreDelete | Before resources are deleted with the app (3.3+) |
hook: PostDelete | After the app's resources are deleted |
hook: Skip | Never apply this object |
hook-delete-policy | HookSucceeded, HookFailed, BeforeHookCreation |
No delete policy means BeforeHookCreation: the old hook object is deleted when the next sync creates a new one. Use generateName or a delete policy, or the second sync collides with the first Job.
The migration must finish before the new code starts, and must be safe to run twice.
migrate up with the new image tag.activeDeadlineSeconds so it cannot hang forever.curl the Service, check /healthz and a key endpoint.If the PreSync Job fails, the sync stops and the old version keeps running — exactly the safety you want. Fix forward with a new commit.
Argo CD renders Helm charts and maps Helm hook annotations to its own phases.
| Helm annotation | Argo CD equivalent |
|---|---|
helm.sh/hook: pre-install, pre-upgrade | PreSync |
helm.sh/hook: post-install, post-upgrade | PostSync |
helm.sh/hook-weight | sync wave |
helm.sh/hook-delete-policy | hook delete policy |
pre-delete / post-delete | PreDelete / PostDelete |
pre-rollback, post-rollback, test | Not supported (no Argo CD equivalent) |
pre-install and pre-upgrade both run on every sync.helm test hooks are not mapped at all: run tests as a PostSync hook instead.argocd.argoproj.io/hook annotations via Kustomize post-rendering or a values flag.A sync operation waits for hooks. A hook that never finishes keeps the app in "Syncing".
Running for many minutes.ImagePullBackOff, Pending (no node), or waiting for a dependency.Failed; auto-sync will not retry the same commit without retry.argocd app get web-dev # op + hooks
kubectl -n web-dev get jobs,pods
kubectl -n web-dev logs \
job/db-migrate-abcde
# stop the running op
argocd app terminate-op web-dev
# fix in Git, then
argocd app sync web-devactiveDeadlineSeconds and backoffLimit on every hook Job.Teach Argo CD which differences matter, and freeze changes when the business needs quiet.
Argo CD compares normalized desired manifests with live objects. Anything that changes live objects after apply shows up as a diff.
"1" vs 1, empty lists).replicas), cert-manager (caBundle), cloud controllers.randAlphaNum / timestamps: a new value on every render.# see exactly what differs
argocd app diff web-dev
# compare against a local render
helm template web charts/web \
-f envs/dev/web/values.yaml > /tmp/want.yaml
argocd app diff web-dev --local charts/web
# one-off: server-side diff for this app
metadata:
annotations:
argocd.argoproj.io/compare-options: ServerSideDiff=trueignoreDifferences: tell Argo CD what not to comparePer Application in spec.ignoreDifferences, or system-wide in argocd-cm.
spec:
ignoreDifferences:
- group: apps
kind: Deployment
jsonPointers:
- /spec/replicas # HPA owns it
- group: admissionregistration.k8s.io
kind: MutatingWebhookConfiguration
jqPathExpressions:
- .webhooks[]?.clientConfig.caBundle
- group: '*'
kind: '*'
managedFieldsManagers:
- kube-controller-manager
syncPolicy:
syncOptions:
- RespectIgnoreDifferences=truejsonPointers: exact paths (RFC 6901), simple and fast.jqPathExpressions: patterns over arrays and optional fields.managedFieldsManagers: ignore every field owned by a named field manager (server-side apply metadata).name and namespace when only one object needs it.resource.customizations.ignoreDifferences.apps_Deployment in argocd-cm.RespectIgnoreDifferences=true a sync still writes the ignored field back from Git.Drift is any live change that Git does not know about. GitOps turns it from a mystery into an alert.
kubectl write access that allowed it.Break-glass: during an incident, disable auto-sync on the app (or set a deny sync window), fix live, then commit the fix and re-enable. Never leave an app with auto-sync off and an undocumented live patch.
Defined on the AppProject. A deny window blocks syncs; an allow window permits them only inside the window.
apiVersion: argoproj.io/v1alpha1
kind: AppProject
metadata:
name: team-web
namespace: argocd
spec:
syncWindows:
- kind: deny # Friday evening freeze
schedule: '0 17 * * 5'
duration: 64h
timeZone: Asia/Ho_Chi_Minh
applications: ['*-prod']
manualSync: true # humans may still sync
- kind: allow # prod only in office hours
schedule: '0 9 * * 1-5'
duration: 8h
timeZone: Asia/Ho_Chi_Minh
clusters: ['prod']schedule is cron; duration is how long the window lasts.applications, namespaces, clusters (globs).manualSync: true lets a human override during the window (incidents).argocd proj windows list team-web.Most "Argo CD is flaky" reports are one of these.
retry → one transient failure leaves the app failed until the next commit.prune: true on an app that owns CRDs → all custom resources deleted when the CRD leaves Git.argocd app rollback to work with auto-sync on.name → second sync fails on "already exists".pre-install Helm hooks that now run on every sync.randAlphaNum, now) → permanent OutOfSync.ignoreDifferences (kind: '*' + /spec) that hides real changes.RespectIgnoreDifferences=true, so syncs still overwrite the ignored field.Explain the answer and name the exact field or annotation.
FailOnSharedResource=true protect against?prune, selfHeal and allowEmpty each for?ServerSideApply=true?prune dangerous for an app that owns CRDs?Practice: Lab 7 in Hands-on labs — waves and hooks ordering, ignoreDifferences for HPA replicas, a failing PreSync hook and an OutOfSync loop.
From one Application to hundreds: bootstrap with app-of-apps, generate with ApplicationSets, combine sources, and commit rendered manifests with the Source Hydrator.
One root Application that deploys other Applications: the simplest way to bootstrap a cluster from Git.
An Application can deploy any Kubernetes object — including other Application objects.
You apply one object by hand (the root). Everything else — including Argo CD's own upgrades — arrives through Git.
The root points at a directory of Application manifests. Children point at real workloads.
# bootstrap/root.yaml (applied once with kubectl)
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: root
namespace: argocd
finalizers:
- resources-finalizer.argocd.argoproj.io
spec:
project: platform
source:
repoURL: https://github.com/<you>/gitops-lab
targetRevision: main
path: apps
destination:
name: in-cluster
namespace: argocd # children live here
syncPolicy:
automated: {prune: true, selfHeal: true}# apps/web-dev.yaml
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: web-dev
namespace: argocd
annotations:
argocd.argoproj.io/sync-wave: "10"
finalizers:
- resources-finalizer.argocd.argoproj.io
spec:
project: team-web
source:
repoURL: https://github.com/<you>/gitops-lab
targetRevision: main
path: charts/web
helm:
valueFiles: [../../envs/dev/web/values.yaml]
destination: {name: in-cluster, namespace: web-dev}
syncPolicy:
automated: {prune: true, selfHeal: true}
syncOptions: [CreateNamespace=true]Waves on child Applications only order the bootstrap if Argo CD can tell when a child is healthy.
Application resource has no health check, so the root treats every child as healthy at once and all waves run together.argoproj.io/Application in argocd-cm that returns the child's own status.health.status.# argocd-cm
resource.customizations.health.argoproj.io_Application: |
hs = {}
hs.status = "Progressing"
hs.message = ""
if obj.status ~= nil and obj.status.health ~= nil then
hs.status = obj.status.health.status
if obj.status.health.message ~= nil then
hs.message = obj.status.health.message
end
end
return hsGreat for a small, hand-curated set of apps. Repetitive at scale — which is where ApplicationSets come in.
argocd.yaml child).apps/ can create an Application in any project the root allows: restrict the root's project.Delete=false sync option on critical child Applications.platform project that only allows Application objects in argocd.A template plus generators: parameters in, Applications out.
The ApplicationSet controller runs generators, renders the template once per parameter set, and creates, updates or deletes Applications.
The simplest generator: a fixed list of parameter sets. Use Go templates (goTemplate: true) for every new ApplicationSet.
apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata: {name: web, namespace: argocd}
spec:
goTemplate: true
goTemplateOptions: ["missingkey=error"]
generators:
- list:
elements:
- {env: dev, cluster: in-cluster}
- {env: staging, cluster: in-cluster}
- {env: prod, cluster: prod}
template:
metadata:
name: 'web-{{.env}}'
labels: {env: '{{.env}}'}
spec:
project: team-web
source:
repoURL: https://github.com/<you>/gitops-lab
targetRevision: main
path: charts/web
helm:
valueFiles:
- '../../envs/{{.env}}/web/values.yaml'
destination: {name: '{{.cluster}}', namespace: 'web-{{.env}}'}missingkey=error turns a typo in a parameter name into an error instead of an empty string.{{.env | upper}}, {{default "main" .branch}}.{{env}}) is still supported but limited; do not mix the two.Let the data drive the apps: add a cluster Secret or a directory and a new Application appears.
# clusters: one app per registered cluster
generators:
- clusters:
selector:
matchLabels:
env: prod
# params: .name .server .metadata.labels.<k>
# git directories: one app per folder
generators:
- git:
repoURL: https://github.com/<you>/gitops-lab
revision: main
directories:
- path: envs/*/*
# params: .path.path .path.basename
# git files: one app per config file
generators:
- git:
repoURL: https://github.com/<you>/gitops-lab
revision: main
files:
- path: clusters/*/config.json
# params: every key in the file| Generator | One Application per… | Typical use |
|---|---|---|
list | element | Small, static sets |
clusters | cluster Secret (filtered by labels) | Add-ons on every cluster |
git directories | matching directory | Monorepo: one folder = one app |
git files | JSON / YAML file | Per-cluster or per-env config files |
matrix | combination of two generators | Every service × every cluster |
merge | base set, overridden by keys | Defaults + per-cluster exceptions |
scmProvider | repository in an org | Every repo with a deploy folder |
pullRequest | open pull request | Preview environments |
Also: clusterDecisionResource (external placement such as OCM) and plugin (your own HTTP service returns parameters).
Combine generators: matrix multiplies, merge overrides.
# every service folder × every prod cluster
generators:
- matrix:
generators:
- git:
repoURL: https://github.com/<you>/gitops-lab
revision: main
directories: [{path: envs/prod/*}]
- clusters:
selector:
matchLabels: {env: prod}
template:
metadata:
name: '{{.path.basename}}-{{.name}}'
spec:
destination:
server: '{{.server}}'
namespace: '{{.path.basename}}'# defaults for all clusters, override per cluster
generators:
- merge:
mergeKeys: [name]
generators:
- clusters:
values:
ingressReplicas: "2"
- list:
elements:
- name: prod
values.ingressReplicas: "4"
# result: .values.ingressReplicas
# = 4 on prod, 2 elsewhereMatrix takes exactly two child generators (nest for more) — put both dimensions in metadata.name to avoid collisions. Merge needs mergeKeys in every generator. Preview output with argocd appset generate appset.yaml before applying.
One short-lived environment per open pull request, deleted when the PR closes.
apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata: {name: web-previews, namespace: argocd}
spec:
goTemplate: true
generators:
- pullRequest:
github:
owner: <you>
repo: web
tokenRef: {secretName: github-token, key: token}
template:
metadata: {name: 'web-pr-{{.number}}'}
spec:
project: previews
source:
repoURL: https://github.com/<you>/gitops-lab
targetRevision: main
path: charts/web
helm:
parameters:
- name: image.tag
value: 'pr-{{.number}}-{{.head_short_sha}}'
destination: {name: in-cluster, namespace: 'web-pr-{{.number}}'}
syncPolicy:
automated: {prune: true}
syncOptions: [CreateNamespace=true].number, .branch, .head_sha, .head_short_sha and labels.requeueAfterSeconds (generator field), or near-instant with the ApplicationSet webhook.pr-<n>-<sha> image before the app can become healthy.templatePatch: per-app differences without a second ApplicationSetThe template must be valid YAML; templatePatch is a Go-templated string patched on top, so it can contain conditionals.
spec:
goTemplate: true
generators:
- list:
elements:
- {env: dev, autoSync: true}
- {env: prod, autoSync: false}
template:
metadata:
name: 'web-{{.env}}'
spec:
project: team-web
source: {repoURL: https://github.com/<you>/gitops-lab,
targetRevision: main, path: charts/web}
destination: {name: in-cluster, namespace: 'web-{{.env}}'}
templatePatch: |
{{- if .autoSync }}
spec:
syncPolicy:
automated:
prune: true
selfHeal: true
{{- end }}templatePatch.templatePatch requires goTemplate: true.A generator bug can delete many applications at once. Decide what the controller may do.
One wrong commit to a Git generator path, or one cluster label change, can delete every generated Application — and with the finalizer, their workloads.
spec.syncPolicy.preserveResourcesOnDeletion: true → when an Application is deleted, its workloads stay.spec.syncPolicy.applicationsSync: what the controller may do to Applications.argocd appset generate.applicationsSync policiesLimit the controller to the operations you trust.
| Policy | Create | Update | Delete |
|---|---|---|---|
sync (default) | yes | yes | yes |
create-update | yes | yes | no |
create-only | yes | no | no |
create-delete | yes | no | yes |
apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
name: web
namespace: argocd
spec:
syncPolicy:
applicationsSync: create-update
preserveResourcesOnDeletion: true
goTemplate: true
generators: [ ... ]
template: { ... }create-update + preserveResourcesOnDeletion: true; delete Applications deliberately.sync (they must disappear when PRs close).create-only makes later template changes silent no-ops — usually surprising.Roll a change across generated Applications in steps: dev first, then staging, then prod clusters a few at a time.
spec:
goTemplate: true
generators:
- list:
elements:
- {env: dev}
- {env: staging}
- {env: prod}
strategy:
type: RollingSync
rollingSync:
steps:
- matchExpressions:
- {key: env, operator: In, values: [dev]}
- matchExpressions:
- {key: env, operator: In, values: [staging]}
- matchExpressions:
- {key: env, operator: In, values: [prod]}
maxUpdate: 25% # prod apps a quarter at a time
template:
metadata:
name: 'web-{{.env}}'
labels: {env: '{{.env}}'}
spec: { ... }argocd-cmd-params-cm). Edge cases may remain.An ApplicationSet can create Applications in any project its template names. Treat it as a privileged object.
project from a Git file lets anyone who can edit that file pick a more powerful project.destination the parameters produce.argocd; teams change parameters (files, labels) through PR review.project in the template; never template it from user-controlled data.destinations and sourceRepos.kubectl -n argocd get applicationsets and .status.conditions show generator errors.argocd appset get web lists generated Applications and their status.Charts from one place and values from another; rendered manifests committed back to Git.
Keep third-party charts in their registry and only your values in your repo.
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: podinfo-prod
namespace: argocd
spec:
project: team-web
sources:
- repoURL: https://stefanprodan.github.io/podinfo
chart: podinfo
targetRevision: 6.* # semver range
helm:
releaseName: podinfo
valueFiles:
- $values/envs/prod/podinfo/values.yaml
- repoURL: https://github.com/<you>/gitops-lab
targetRevision: main
ref: values # exposes this repo as $values
destination: {name: prod, namespace: podinfo}
syncPolicy:
syncOptions: [CreateNamespace=true]sources (plural) replaces source; do not set both.ref: provides files to other sources through $<ref>; it can also have a path and render its own manifests.repoURL: <registry>/<path> with a type: helm, enableOCI repository credential.With Helm or Kustomize in the loop, the YAML that reaches the cluster is not in Git. Reviews see values, not results.
The application controller asks repo-server to render the dry source, then the commit server pushes the result to a hydrated branch, which is what gets synced.
spec:
sourceHydrator:
drySource:
repoURL: https://github.com/<you>/gitops-lab
targetRevision: main
path: apps/web/overlays/dev
syncSource:
targetBranch: environments/dev
path: web
# optional: stage for review / promotion
# hydrateTo:
# targetBranch: environments/dev-nextOff by default. Beta in the 3.5 docs. Read the docs for your exact version before relying on it.
*-install-with-hydrator.yaml manifests, or the Helm chart option).hydrator.enabled: "true" in argocd-cmd-params-cm.repo-creds) are not supported for the hydrator's push credentials.manifest.yaml) into the configured path and does not delete other files already there..gitattributes in hydrated repos.hydrateToenvironments/prod-next).environments/prod.environments/prod.They combine: an ApplicationSet can generate multi-source or hydrated Applications.
| Need | Pattern | Why |
|---|---|---|
| Bootstrap a cluster with a curated list of add-ons | App-of-apps | Simple, each child explicit in review |
| Same service in many environments or clusters | ApplicationSet (list / clusters / matrix) | One template, parameters per target |
| Monorepo, one folder per app | ApplicationSet git directories | New folder = new app, no extra manifest |
| Preview environment per PR | ApplicationSet pullRequest | Created and deleted with the PR |
| Third-party chart + your values | Multi-source Application | Chart stays in its registry |
| Reviewers must see the exact rendered YAML | Source Hydrator (or a CI render step) | Rendered manifests in Git |
| Roll a change across many clusters in steps | ApplicationSet + RollingSync (beta) | Health-gated waves of Applications |
These scale with your fleet: one mistake, a hundred broken apps.
Application → bootstrap waves do nothing.default project → anyone with Git write can create any Application.metadata.name → name collisions.project from user data.source and sources.ref: on the values source, so $values/... does not resolve.Explain the answer and name the field, generator or setting.
env=prod?missingkey=error protect against?hydrateTo support promotion?Practice: Lab 8 in Hands-on labs — app-of-apps bootstrap, then an ApplicationSet (Git directory × list matrix) and PR preview environments.
One Git repo, many clusters and environments: topologies, layouts, and moving a version from dev to prod safely.
Where Argo CD runs, which clusters it manages, and how it authenticates to them.
The choice decides blast radius, network paths and how many UIs your team has to watch.
All three read the same Git repository. Only the reconciler placement changes.
Neither is "right". Pick on blast radius, network reachability and team size.
| Aspect | Hub-and-spoke | One per cluster |
|---|---|---|
| Single pane of glass | Yes: one UI, one RBAC, one SSO | No: N UIs (or a SaaS / aggregator on top) |
| Blast radius | Hub outage or compromise affects every cluster | Limited to one cluster |
| Credentials | Hub holds credentials for every spoke | Each cluster uses its in-cluster service account |
| Network | Hub must reach every spoke API server | Only outbound to Git and the registry |
| Scale | Controller load grows with clusters; needs sharding (ch 12) | Load spread naturally |
| Operations | Upgrade once | Upgrade N installs (manage Argo CD with Argo CD) |
| Typical fit | Platform team, tens of clusters, private networking in place | Strict isolation, air-gapped, edge, regulated prod |
Common compromise: a hub per environment tier (non-prod hub, prod hub) so a non-prod mistake cannot touch prod.
argocd cluster add is a shortcut; in GitOps you declare the Secret (through External Secrets, ch 10).
apiVersion: v1
kind: Secret
metadata:
name: cluster-prod
namespace: argocd
labels:
argocd.argoproj.io/secret-type: cluster
env: prod # used by ApplicationSet cluster generator
region: ap-northeast-1
type: Opaque
stringData:
name: prod
server: https://ABC123.gr7.ap-northeast-1.eks.amazonaws.com
config: |
{
"awsAuthConfig": {
"clusterName": "prod",
"roleARN": "arn:aws:iam::222222222222:role/argocd-deployer"
},
"tlsClientConfig": { "caData": "<base64 CA>" }
}secret-type: cluster makes Argo CD load it as a destination.name is what Applications use in destination.name; server is the API URL used in destination.server.config holds the auth: bearer token, client certs, exec provider or awsAuthConfig.in-cluster destination: https://kubernetes.default.svc.argocd cluster add <context> creates a argocd-manager ServiceAccount with a cluster-admin role in the target: convenient, but over-privileged.The hub authenticates with its pod identity, assumes a role per spoke, and the spoke maps that role with an access entry.
argocd-hub (EKS Pod Identity or IRSA).argocd-hub may only sts:AssumeRole on each spoke deployer role.argocd-deployer trusts argocd-hub.A few settings on the cluster Secret and in Argo CD change how a cluster behaves as a destination.
namespaces: comma-separated list; Argo CD then manages only those namespaces (no cluster-wide watch).clusterResources: "true" re-allows cluster-scoped resources when namespaces is set.project: a project-scoped cluster that only that AppProject can use.destination.name: prod instead of a URL keeps manifests stable when an endpoint changes.name and server in one Application.name, not by re-adding.127.0.0.1:<port> into kubeconfig; a hub inside Docker cannot reach that.https://prod-control-plane:6443.prod kind cluster with the gitops hub.A tree that makes every environment explicit and every difference reviewable.
Shared definitions once; per-environment differences small and visible.
gitops-lab/
├── charts/
│ └── web/ # in-house chart (ch 2-4)
├── apps/
│ └── web/
│ ├── base/values.yaml # shared by every env
│ └── appset.yaml # one ApplicationSet per app
├── envs/
│ ├── dev/web/values.yaml # image.tag, replicas, host
│ ├── staging/web/values.yaml
│ └── prod/web/values.yaml
├── clusters/
│ ├── gitops/ # hub: Argo CD itself, projects
│ │ ├── argocd/values.yaml
│ │ └── projects/
│ └── prod/
│ └── addons/ # ingress, external-secrets ...
└── .github/workflows/
└── render-diff.yaml # diff rendered output on PRsbase/values.yaml holds everything that must be identical everywhere.clusters/ holds cluster-level concerns (add-ons, Argo CD config) separate from apps.CODEOWNERS: envs/prod/ requires platform or service-owner approval.Each environment's Application points at the same chart with a different, ordered list of values files.
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: web-prod
namespace: argocd
spec:
project: web
source:
repoURL: https://github.com/<you>/gitops-lab
targetRevision: main
path: charts/web
helm:
valueFiles: # later files win
- ../../apps/web/base/values.yaml
- ../../envs/prod/web/values.yaml
destination:
name: prod
namespace: web
syncPolicy:
syncOptions: [CreateNamespace=true]valueFiles paths are relative to source.path; files must stay inside the repo.$values ref (ch 8).web-dev, web-staging, web-prod from this template (ch 8).helm.parameters overrides set from the UI or CLI: they live only in the Application, not in Git.envs/<env>/web/kustomization.yaml with resources: [../../../apps/web/base] plus patches.If prod differs from staging in something not on the left column, staging is not testing prod.
| Kind of value | Example |
|---|---|
| Version under test | image.tag, chart version |
| Capacity | replicas, resources, HPA min/max |
| Endpoints | hostnames, certificate ARNs, URLs of dependencies |
| Secret references | Secrets Manager path per environment |
| Feature flags | only when deliberately staged |
Reviewers approve the rendered change, not just a values edit they have to imagine.
# .github/workflows/render-diff.yaml (core step)
git worktree add /tmp/main origin/main # base branch copy
for env in dev staging prod; do
helm template web charts/web \
-f apps/web/base/values.yaml \
-f envs/$env/web/values.yaml > /tmp/new-$env.yaml
helm template web /tmp/main/charts/web \
-f /tmp/main/apps/web/base/values.yaml \
-f /tmp/main/envs/$env/web/values.yaml > /tmp/old-$env.yaml
diff -u /tmp/old-$env.yaml /tmp/new-$env.yaml \
> /tmp/diff-$env.txt || true
done
# post /tmp/diff-*.txt as a PR comment
# compare environments with each other
diff <(helm template web charts/web -f apps/web/base/values.yaml \
-f envs/staging/web/values.yaml) \
<(helm template web charts/web -f apps/web/base/values.yaml \
-f envs/prod/web/values.yaml)base/ that touches all three environments becomes obvious.kubeconform and policy checks (ch 4) on the same rendered files.argocd app diff <app> --local <path> against the live cluster (needs cluster access from CI).Start with one config repo; split when ownership or scale forces it.
| Question | One config repo | Repo per team or app |
|---|---|---|
| Cross-cutting change (add a label everywhere) | One PR | N PRs |
| Access control | CODEOWNERS per path | Repository permissions |
| Argo CD performance | Large repo: use webhooks and manifest-generate-paths (ch 12) | Small repos, more repo credentials |
| Blast radius of a bad merge | Whole platform | One team |
AppProject sourceRepos | Same repo for everyone: restrict by path via ApplicationSet templates | Natural: one repo per project |
Moving one tested version from dev to staging to prod — by pull request, by bot, or by a promotion engine.
CI produces an artifact once; promotion copies its version from one environment file to the next.
podinfo:6.7.1 by digest.envs/dev/web/values.yaml → 6.7.1.Never rebuild per environment. The digest tested in staging must be the digest running in prod.
The simplest promotion engine: yq + git + gh. Everything stays reviewable.
# runs in the app repo after the image is pushed
TAG=6.7.1
git clone https://github.com/<you>/gitops-lab && cd gitops-lab
git switch -c promote-web-dev-$TAG
TAG=$TAG yq -i '.image.tag = strenv(TAG)' envs/dev/web/values.yaml
git commit -am "web: promote $TAG to dev"
git push -u origin HEAD
gh pr create --title "web: $TAG -> dev" \
--body "Image digest: $DIGEST" --label promotion
# later: promote what dev runs to staging
TAG=$(yq '.image.tag' envs/dev/web/values.yaml)
TAG=$TAG yq -i '.image.tag = strenv(TAG)' envs/staging/web/values.yamlgit revert precise.main blocks direct pushes, including from bots.Two different answers: "follow the newest tag" vs "move a verified bundle through stages".
The fastest safe rollback is a revert commit; Argo CD's own rollback is a short-term override.
git revertgit revert <promotion-commit> restores the previous tag in Git.argocd app rollbackargocd app history <app>.syncPolicy.automated on the Application has no effect (the controller regenerates it), so roll back in Git.# preferred
git revert 3f2a91c
git push
# emergency override (self-managed app)
argocd app set web-prod --sync-policy none
argocd app history web-prod
argocd app rollback web-prod 12
# then fix Git, and re-enable:
argocd app set web-prod \
--sync-policy automatedAutomate the evidence so approval is a quick check, not an investigation.
| Gate | Where it runs | Fails when |
|---|---|---|
| Rendered diff reviewed | PR checks (28) | Unexpected change outside the promoted values |
| Sync healthy | Argo CD health, argocd app wait web-staging --health | Degraded or Progressing timeout |
| Smoke tests | PostSync hook Job (ch 7) | Hook Job fails → sync marked failed |
| Metrics analysis | Argo Rollouts AnalysisRun (ch 11) | Error rate or latency above threshold |
| Change window | AppProject sync windows (ch 7) | Outside allowed time for prod |
| Human approval | CODEOWNERS on envs/prod/ | No approving review |
Each one has caused a real outage somewhere.
argocd cluster add in prod: leaves a cluster-admin token in the hub.helm.parameters) and never committed.latest or a mutable tag.argocd app rollback with auto-sync on (refused) or on an ApplicationSet app (overwritten).Explain the answer and name the exact field or command.
namespaces on a cluster Secret change?valueFiles applied?argocd app rollback refuse to run, and why is git revert still needed afterwards?Practice: Lab 9 in Hands-on labs — register a second kind cluster, promote a tag by PR, roll back with git revert.
Who may do what in Argo CD, how teams share it safely, where secrets live, and how to trust what gets deployed.
SSO for humans, scoped tokens for machines, and RBAC policies that say exactly what each can do.
Anyone who can steer it — through the UI, the API or Git — can change every cluster it manages.
Protect all three entry points. Hardening RBAC is pointless if anyone can push to main of the config repo.
Use your identity provider for humans; disable the local admin once SSO works.
# argocd-cm (Helm values: configs.cm)
url: https://argocd.example.com
admin.enabled: "false" # after SSO is verified
oidc.config: |
name: Okta
issuer: https://example.okta.com
clientID: 0oa1example
clientSecret: $oidc.clientSecret # key in argocd-secret
requestedScopes: ["openid", "profile", "email", "groups"]
# alternative: Dex as a broker (GitHub, LDAP, SAML)
dex.config: |
connectors:
- type: github
id: github
name: GitHub
config:
clientID: $dex.github.clientID
clientSecret: $dex.github.clientSecret
orgs:
- name: my-orgurl must be the external URL; the IdP redirect URI is <url>/auth/callback.$key values are read from argocd-secret; $<secret-name>:<key> reads another Secret labelled app.kubernetes.io/part-of: argocd.scopes: '[groups]' tells RBAC which claim to read.Casbin-style CSV lines: p grants a permission to a subject, g puts a user or group into a role.
# argocd-rbac-cm (Helm values: configs.rbac)
policy.default: role:readonly
scopes: '[groups, email]'
policy.csv: |
# p, <subject>, <resource>, <action>, <project>/<object>, allow|deny
p, role:web-dev, applications, get, web/*, allow
p, role:web-dev, applications, sync, web/*, allow
p, role:web-dev, applications, action/*, web/*, allow
p, role:web-dev, logs, get, web/*, allow
p, role:web-dev, applications, sync, web/web-prod, deny
p, role:platform, *, *, */*, allow
g, my-org:web-team, role:web-dev
g, my-org:platform, role:platform| Part | Values |
|---|---|
| Built-in roles | role:readonly, role:admin |
| Resources | applications, applicationsets, projects, clusters, repositories, accounts, certificates, logs, exec, extensions |
| Actions | get, create, update, delete, sync, override, action/<group>/<kind>/<name> |
| Object | <project>/<application>; wildcards allowed |
| Precedence | An explicit deny wins over allow |
Test before rollout: argocd admin settings rbac can role:web-dev sync applications web/web-prod --policy-file policy.csv.
3.0 changed defaults that older tutorials rely on. Re-check any policy written for 2.x.
update/delete on an Application also allowed editing or deleting its managed resources.update/* or delete/*/Pod/*/*.logs, get is enforced: users without it cannot read pod logs in the UI.exec, create controls the web terminal (also requires exec.enabled: "true" in argocd-cm).kubectl with audited access instead.* actions for non-platform roles.override (sync with local manifests or parameter overrides) only for break-glass roles.projects, update lets a user widen their own sandbox: platform only.clusters and repositories writes: platform only.CI should get a token scoped to one AppProject, not an admin account.
# AppProject (excerpt)
spec:
roles:
- name: ci
description: CI may sync web apps
policies:
- p, proj:web:ci, applications, sync, web/*, allow
- p, proj:web:ci, applications, get, web/*, allow
groups: []
# issue a token (expires in 30 days)
argocd proj role create-token web ci --expires-in 720h
# in CI
argocd app sync web-dev --auth-token "$ARGOCD_TOKEN" \
--server argocd.example.com --grpc-web
argocd app wait web-dev --health --timeout 300 \
--auth-token "$ARGOCD_TOKEN" --server argocd.example.comargocd proj role delete-token.accounts.ci: apiKey in argocd-cm) are the alternative; prefer project roles because they cannot escape the project.Git history covers desired state; Argo CD and Kubernetes cover the actions taken on it.
| Question | Evidence |
|---|---|
| Who changed the desired state? | Git commit, PR, approvers (require signed commits, 33) |
| Who clicked Sync, Rollback or ran an action? | Kubernetes Events on the Application (kubectl get events -n argocd) and API server logs |
| What was deployed at time T? | argocd app history <app>: revision and time of each sync |
| Who changed RBAC, projects or clusters? | Git (if declarative) or Kubernetes audit logs for argocd namespace objects |
| Did anyone bypass Argo CD with kubectl? | Drift shows as OutOfSync; EKS control plane audit logs show the user |
Ship Argo CD logs and Kubernetes Events to your log platform: Events expire from the cluster after about an hour by default.
AppProjects as fences, Applications in team namespaces, and syncing as the tenant instead of as cluster-admin.
It answers: from which repos, to which clusters and namespaces, which kinds of objects.
apiVersion: argoproj.io/v1alpha1
kind: AppProject
metadata:
name: web
namespace: argocd
spec:
description: Web team
sourceRepos:
- https://github.com/<you>/gitops-lab
sourceNamespaces: [team-web] # apps in any namespace
destinations:
- name: '*'
namespace: 'web-*'
clusterResourceWhitelist: [] # no cluster-scoped objects
namespaceResourceBlacklist:
- group: ''
kind: ResourceQuota
- group: ''
kind: LimitRange
- group: networking.k8s.io
kind: NetworkPolicy
orphanedResources:
warn: truesourceRepos: Git or Helm repos this team may deploy from.destinations: cluster (name or server) and namespace patterns.clusterResourceWhitelist: []: no CRDs, ClusterRoles, Namespaces.namespaceResourceBlacklist: keep quota and network policy under platform control.orphanedResources: warn about objects in the namespace not tracked in Git.default project allows everything: restrict it or never use it for tenants.By default Applications must live in argocd, so only admins can create them. This opens a controlled door.
argocd-cmd-params-cm: application.namespaces: team-* (server and controller read it).spec.sourceNamespaces: [team-web] allows Applications from that namespace to use the project.team-web/web-dev in CLI and RBAC.applicationsetcontroller.namespaces) with extra SCM restrictions.team-web can only use projects that list team-web in sourceNamespaces.argocd or a shared namespace to a tenant's sourceNamespaces.Without impersonation the controller applies every manifest with its own, very broad, service account.
# argocd-cm
application.sync.impersonation.enabled: "true"
# AppProject web
spec:
destinationServiceAccounts:
- server: https://kubernetes.default.svc
namespace: web-dev
defaultServiceAccount: web-deployer
- server: '*'
namespace: web-prod
defaultServiceAccount: web-deployer
# in each destination namespace: a ServiceAccount
# web-deployer + Role/RoleBinding with only what the
# chart needs (Deployments, Services, ConfigMaps, ...)Each layer catches what the previous one missed.
| Layer | Control | Stops |
|---|---|---|
| Git | CODEOWNERS, branch protection, signed commits | Unreviewed or forged changes |
| Argo CD RBAC | argocd-rbac-cm roles per team | Syncing or deleting another team's apps |
| AppProject | sourceRepos, destinations, resource allow/deny lists | Deploying to other namespaces or cluster-scoped objects |
| Impersonation | Per-namespace deployer ServiceAccount | Anything the chart tries beyond its Role |
| Admission | Kyverno / Gatekeeper, Pod Security Admission | Privileged pods, unsigned images, missing labels |
| Runtime | NetworkPolicy, ResourceQuota, LimitRange | Noisy or lateral-moving workloads |
If any of these is writable by a tenant, the tenant is effectively a cluster admin.
argocd namespace and its ConfigMaps and Secrets.clusters/<name>/ creates namespaces, quotas, policies and AppProjects.argocd admin settings rbac can for each role.kubectl auth can-i --as=system:serviceaccount:web-dev:web-deployer create clusterrole → no.Git holds references or ciphertext, never plaintext. Pick where decryption happens.
The key difference: where the plaintext exists and who can decrypt.
| Approach | In Git | Decrypted by | Fit |
|---|---|---|---|
| External Secrets Operator (ESO) | An ExternalSecret reference | ESO controller reading AWS Secrets Manager / Parameter Store | Default on AWS; rotation in one place |
| Secrets Store CSI Driver | A SecretProviderClass | CSI driver at pod mount time | Secrets as files; no Kubernetes Secret needed |
| Sealed Secrets | A SealedSecret (ciphertext) | In-cluster controller with its private key | No cloud secret store; small teams |
| SOPS (with KMS) | Encrypted YAML values | A rendering plugin (KSOPS, helm-secrets) in the repo-server | Encrypted files reviewed in PRs |
SOPS in Argo CD needs a config management plugin, which the EKS Capability for Argo CD does not support. ESO works with both.
The secret value never enters Git or Argo CD; ESO creates and refreshes the Secret in the cluster.
apiVersion: external-secrets.io/v1
kind: ClusterSecretStore
metadata:
name: aws-sm
spec:
provider:
aws:
service: SecretsManager
region: ap-northeast-1
auth:
jwt:
serviceAccountRef:
name: external-secrets
namespace: external-secretsapiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
name: web-db
namespace: web-prod
spec:
refreshInterval: 1h
secretStoreRef:
kind: ClusterSecretStore
name: aws-sm
target:
name: web-db # Kubernetes Secret to create
data:
- secretKey: password
remoteRef:
key: prod/web/db # Secrets Manager name
property: password # JSON key inside itESO's ServiceAccount gets an IAM role (IRSA shown; Pod Identity also works) scoped to prod/web/*. Rotate in Secrets Manager; ESO refreshes the Secret at refreshInterval (env vars need a pod restart). The generated Secret is owned by the ExternalSecret, so Argo CD shows it as a child resource, not as Git-tracked state.
Only the controller in the cluster holds the private key.
# create a normal Secret locally, never apply it
kubectl create secret generic web-db -n web-dev \
--from-literal=password='s3cr3t' \
--dry-run=client -o yaml > /tmp/secret.yaml
# encrypt with the controller's public certificate
kubeseal --format yaml < /tmp/secret.yaml \
> envs/dev/web/sealed-web-db.yaml
rm /tmp/secret.yaml
git add envs/dev/web/sealed-web-db.yaml
# back up the sealing keys (disaster recovery!)
kubectl get secret -n kube-system \
-l sealedsecrets.bitnami.com/sealed-secrets-key \
-o yaml > sealed-secrets-keys.backup.yamlSOPS encrypts only the values, so diffs still show which keys changed.
# .sops.yaml at the repo root
creation_rules:
- path_regex: envs/prod/.*\.enc\.yaml$
kms: arn:aws:kms:ap-northeast-1:222222222222:key/abcd-...
encrypted_regex: '^(data|stringData|password|token)$'
# edit in place (decrypts in $EDITOR, re-encrypts on save)
sops envs/prod/web/secrets.enc.yaml
# what gets committed (excerpt)
password: ENC[AES256_GCM,data:Tr3...,iv:...,tag:...,type:str]
sops:
kms:
- arn: arn:aws:kms:ap-northeast-1:222222222222:key/abcd-...Plugins that fetch a secret and inject it during manifest generation put plaintext where it does not belong.
get on the app.argocd.argoproj.io/compare-options: IgnoreExtraneous only where needed, not as a habit.Trust the commit, trust the artifact, and lock down the Argo CD installation itself.
Argo CD 3.5 replaced the older GnuPG signature verification with Source Integrity.
main.signatureKeys in the AppProject and GnuPG keys imported into Argo CD.Argo CD does not check images. A Kubernetes admission policy does, for every deploy path.
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
name: verify-web-images
spec:
validationFailureAction: Enforce
webhookTimeoutSeconds: 30
rules:
- name: require-cosign-signature
match:
any:
- resources:
kinds: [Pod]
namespaces: ["web-*"]
verifyImages:
- imageReferences:
- "ghcr.io/<you>/*"
attestors:
- entries:
- keyless:
subject: "https://github.com/<you>/*"
issuer: "https://token.actions.githubusercontent.com"Treat the argocd namespace like the control plane it is.
exec into or read Secrets in argocd.cluster-admin, where workable.namespace-install.yaml) when Argo CD should manage only some namespaces.argocd-secret keys after setup.Each one turns "can edit an app" into "owns the cluster".
policy.default: role:admin (or readonly for anonymous users) on an internet-facing UI.update to edit resources; silently narrower in 3.x.default project.clusterResourceWhitelist forgotten → teams create ClusterRoleBindings.sourceNamespaces including argocd.Explain the answer and name the exact field or setting.
update and delete permissions in Argo CD 3.0?web/* apps?Practice: Lab 10 in Hands-on labs — SSO with Dex, two teams with separate AppProjects, a sealed secret, and a tenant trying to escape its namespace.
Ship a new version to a slice of traffic, measure it, and promote or abort automatically — while Git stays the source of truth.
The Rollout resource, canary and blue-green strategies, and how traffic is actually shifted.
RollingUpdate checks readiness, not correctness. A bad version that passes probes reaches 100% of users.
maxSurge, maxUnavailable).argo-rollouts controller (one per cluster).Rollout, AnalysisTemplate, ClusterAnalysisTemplate, AnalysisRun, Experiment.kubectl argo rollouts plugin and an optional dashboard.Same selector and pod template; the controller manages ReplicaSets itself.
apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata:
name: web
spec:
replicas: 5
revisionHistoryLimit: 3
selector:
matchLabels: {app: web}
template: # identical to a Deployment
metadata:
labels: {app: web}
spec:
containers:
- name: web
image: ghcr.io/stefanprodan/podinfo:6.7.1
ports: [{containerPort: 9898}]
strategy:
canary:
steps:
- setWeight: 20
- pause: {} # wait for manual promote
- setWeight: 50
- pause: {duration: 5m}
- setWeight: 100kind: Deployment to kind: Rollout and apiVersion to argoproj.io/v1alpha1, add a strategy.spec.workloadRef (scale the Deployment to 0 when ready).web chart: a rollout.enabled value that switches the template.scaleTargetRef.kind: Rollout).spec.template starts a new rollout; changing replicas does not.With no trafficRouting, setWeight: 20 means about 20% of pods run the new version.
Fine for internal services and low traffic. For exact percentages, header-based routing or a 1% canary, add a traffic router (next slides).
Rollouts rewrites the Ingress action annotation so the ALB splits traffic between two target groups.
spec:
strategy:
canary:
stableService: web-stable # Rollouts adds a pod-hash selector
canaryService: web-canary
trafficRouting:
alb:
ingress: web # the existing Ingress
servicePort: 80
steps:
- setWeight: 5
- pause: {duration: 10m}
- setWeight: 25
- pause: {duration: 10m}
- setWeight: 50
- pause: {duration: 10m}
# Ingress web: backend service name must be web-stable;
# Rollouts manages the weighted forward action.setWeight: 5 the ALB sends 5% of requests to the canary target group, independent of pod counts.ignoreDifferences).Pick what your edge already runs.
| Router | Configured as | Notes |
|---|---|---|
| AWS ALB | trafficRouting.alb | Weighted target groups; header routing via ALB conditions |
| Gateway API | trafficRouting.plugins (Gateway API plugin) | Updates HTTPRoute backend weights; plugin registered in the Rollouts ConfigMap |
| Istio | trafficRouting.istio | VirtualService weights, header and mirror routes |
| NGINX Ingress | trafficRouting.nginx | Creates a canary Ingress with canary annotations |
| SMI / others | Built-in or plugins | Check the Rollouts docs for current support |
| None | omit trafficRouting | Replica-count weights (previous slide) |
The new version runs at full size behind a preview Service; promotion flips the active Service.
spec:
strategy:
blueGreen:
activeService: web # production traffic
previewService: web-preview # testers, smoke tests
autoPromotionEnabled: false # wait for promote
prePromotionAnalysis:
templates:
- templateName: smoke-tests
postPromotionAnalysis:
templates:
- templateName: success-rate
scaleDownDelaySeconds: 30 # keep old pods brieflyIn GitOps, use these to observe and to move a paused rollout — not to change the spec.
# install the plugin (macOS)
brew install argoproj/tap/kubectl-argo-rollouts
kubectl argo rollouts get rollout web -n web-dev --watch
kubectl argo rollouts promote web -n web-dev # next step
kubectl argo rollouts promote web --full -n web-dev # skip all steps
kubectl argo rollouts abort web -n web-dev # back to stable
kubectl argo rollouts retry rollout web -n web-dev # after abort
kubectl argo rollouts dashboard # local UI :3100kubectl argo rollouts set image (Argo CD would see drift).promote and abort change status, not the spec: safe with Argo CD.undo rewrites the pod template → OutOfSync; use git revert instead.Let metrics decide: AnalysisTemplates, automatic abort, and how Rollouts and Argo CD cooperate.
An AnalysisRun queries the metric on an interval and counts successes and failures.
apiVersion: argoproj.io/v1alpha1
kind: AnalysisTemplate
metadata:
name: success-rate
spec:
args:
- name: service
metrics:
- name: success-rate
interval: 1m
count: 5
failureLimit: 1
successCondition: result[0] >= 0.99
provider:
prometheus:
address: http://prometheus.monitoring:9090
query: |
sum(rate(http_requests_total{
service="{{args.service}}",code!~"5.."}[2m]))
/ sum(rate(http_requests_total{
service="{{args.service}}"}[2m]))interval + count: measure 5 times, one minute apart.successCondition: expression on result (here a vector from PromQL).failureLimit: 1: two failed measurements fail the run (failures must exceed the limit).args make one template reusable across services.Inline analysis blocks a step; background analysis runs during the whole canary.
strategy:
canary:
analysis: # background: starts at step 2
templates:
- templateName: success-rate
startingStep: 2
args:
- name: service
value: web-canary
steps:
- setWeight: 5
- pause: {duration: 5m}
- setWeight: 25
- analysis: # inline: blocks until done
templates:
- templateName: latency-p95
args:
- name: service
value: web-canary
- setWeight: 50
- pause: {duration: 10m}Degraded.inconclusiveLimit / consecutiveErrorLimit thoughtfully).AnalysisRun object: kubectl get analysisrun -n web-prod shows the measurements.web-canary labels), not the whole service, or the 95% stable traffic hides problems.What the user sees vs what the system does when the canary fails analysis.
Compare: without Rollouts, the same release reaches 100% of users and the error rate is noticed by people, not by a query.
Argo CD applies the Rollout; the Rollouts controller executes it. Each owns a different part.
Rollout.Suspended (the app is not Healthy yet).Degraded.Healthy; argocd app wait --health waits for this.action/argoproj.io/Rollout/* (ch 10).ignoreDifferences for router-managed fields (Ingress annotations, VirtualService weights, HTTPRoute weights).Two notification engines exist; know which one you are configuring.
argocd-notifications-controller with argocd-notifications-cm.notifications.argoproj.io/subscribe.on-health-degraded.slack: deploys.argo-rollouts-notification-configmap.Most failed canaries fail because of the analysis, not the code.
count too small to see anything before full promotion.ignoreDifferences for router-managed fields → endless OutOfSync.replicas and the image in the same commit, confusing the steps.Explain the answer and name the exact field or command.
successCondition for "p95 latency below 300 ms".failureLimit and inconclusiveLimit?kubectl argo rollouts undo a bad idea in GitOps?Practice: Lab 11 in Hands-on labs — a canary with a Prometheus AnalysisTemplate, then inject errors and watch the automatic abort.
Day-2 Argo CD: metrics and alerts, scaling the controller and repo-server, backup and upgrades, and a playbook for the errors you will meet.
Which components expose metrics, the few metrics that matter, dashboards and alerts.
Scrape all of them: the controller tells you about apps, the repo-server about Git and rendering, the API server about users.
| Component | Service : port | Answers |
|---|---|---|
| Application controller | argocd-metrics:8082 (Helm chart: argocd-application-controller-metrics) | App sync/health, reconcile time, cluster cache |
| API server | argocd-server-metrics:8083 | Requests from UI, CLI, CI |
| Repo-server | argocd-repo-server:8084 | Git requests, manifest generation |
| ApplicationSet controller | argocd-applicationset-controller:8080 | Generators, reconcile |
*.metrics.enabled and *.metrics.serviceMonitor.enabled for each component.--metrics-application-labels.--metrics-application-conditions.Learn these names; everything else is detail.
| Metric | Type | Use it for |
|---|---|---|
argocd_app_info | gauge | One series per Application with sync_status, health_status, project, dest_server labels: count OutOfSync / Degraded apps |
argocd_app_sync_total | counter | Sync history by phase (Succeeded, Failed, Error, Running, Terminating): failure rate |
argocd_app_reconcile | histogram | How long the controller takes to compare an app: the first sign of an overloaded controller |
argocd_git_request_total | counter | Git ls-remote and fetch calls per repo from the repo-server: polling load, auth failures |
argocd_git_request_duration_seconds | histogram | Slow Git hosts or huge repos |
Check the metrics page of the docs for your exact minor version before writing alerts; labels are added over time.
Use these as panels and as the base of alerts.
# apps not in sync, per project
count by (project) (argocd_app_info{sync_status="OutOfSync"})
# apps not healthy (Degraded, Missing, Unknown)
count by (health_status) (argocd_app_info{
health_status!~"Healthy|Progressing|Suspended"})
# sync failure ratio, last hour
sum(increase(argocd_app_sync_total{phase=~"Error|Failed"}[1h]))
/ sum(increase(argocd_app_sync_total[1h]))
# p95 reconcile time per controller shard
histogram_quantile(0.95,
sum by (le, namespace) (rate(argocd_app_reconcile_bucket[5m])))
# Git requests per repo per minute
sum by (repo) (rate(argocd_git_request_total[5m])) * 60manifest-generate-paths.examples/dashboard.json).Auto-sync apps are briefly OutOfSync on every commit. Alert on duration.
groups:
- name: argocd
rules:
- alert: ArgoAppDegraded
expr: argocd_app_info{health_status="Degraded"} == 1
for: 10m
labels: {severity: page}
annotations:
summary: "{{ $labels.name }} Degraded for 10m"
- alert: ArgoAppOutOfSyncTooLong
expr: |
argocd_app_info{sync_status="OutOfSync",
autosync_enabled="true"} == 1
for: 30m
labels: {severity: ticket}
- alert: ArgoSyncFailures
expr: |
increase(argocd_app_sync_total{
phase=~"Error|Failed"}[15m]) > 0
labels: {severity: ticket}argocd app sync.project label to the team owning the AppProject.Know which pod to read before you start scrolling.
| Symptom | Look in | Command |
|---|---|---|
| Render error, Helm/Kustomize failure, Git auth | repo-server | kubectl -n argocd logs deploy/argocd-repo-server |
| Sync stuck, diff wrong, cluster cache errors | application controller | kubectl -n argocd logs statefulset/argocd-application-controller |
| Login, RBAC denied, webhook received? | API server | kubectl -n argocd logs deploy/argocd-server |
| Apps not generated or deleted unexpectedly | ApplicationSet controller | kubectl -n argocd logs deploy/argocd-applicationset-controller |
| Who synced what, when | Kubernetes events on the Application | kubectl -n argocd get events --field-selector involvedObject.name=web-prod |
Set --loglevel debug only temporarily; switch log format to JSON (--logformat json) so logs are queryable in Loki or CloudWatch Logs Insights.
Where Argo CD spends its time, and the knobs for hundreds of apps and dozens of clusters.
Every refresh: fetch Git → render manifests → compare with the live cache → maybe sync.
It is a StatefulSet; each replica owns a shard of the clusters.
controller.status.processors: parallel refresh/compare workers (default 20).controller.operation.processors: parallel sync workers (default 10).timeout.reconciliation in argocd-cm: refresh interval, default 120s plus up to 60s jitter (timeout.reconciliation.jitter); longer means less Git load but slower drift detection.ARGOCD_CONTROLLER_REPLICAS to the same value.controller.sharding.algorithm = legacy (default), round-robin, or consistent-hashing.shard: "2" in its cluster Secret.apiVersion: v1
kind: ConfigMap
metadata:
name: argocd-cmd-params-cm
namespace: argocd
data:
controller.status.processors: "50"
controller.operation.processors: "25"
controller.sharding.algorithm: round-robin
---
# Helm values (argo/argo-cd)
controller:
replicas: 3
env:
- name: ARGOCD_CONTROLLER_REPLICAS
value: "3"
resources:
requests: {cpu: "1", memory: 2Gi}Rendering is CPU-heavy and stateless: scale it horizontally.
reposerver.parallelism.limit caps concurrent renders per pod so a burst does not OOM it.https://argocd.example.com/api/webhook; refresh on push instead of waiting for polling.argocd-secret (webhook.github.secret, etc.).argocd.argoproj.io/manifest-generate-paths so a commit only refreshes apps whose paths changed.apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: web-prod
annotations:
# refresh only when these
# paths change
argocd.argoproj.io/manifest-generate-paths:
.;/charts/web
spec:
source:
repoURL: https://github.com/<you>/gitops-lab
path: envs/prod/webThe controller watches every resource type it may manage. Exclude what it never needs.
resource.exclusions in argocd-cm: drop noisy kinds (Events, Leases, metrics objects, high-churn CRs such as Cilium identities).resource.inclusions: allow-list mode for tightly scoped installs.namespace-install.yaml, or chart value createClusterRoles: false) watches only allowed namespaces.resource.customizations.ignoreResourceUpdates.<group_kind> keys stop refreshes caused by fields you do not care about (e.g. status-only updates on CRs)..status changes are ignored in diffs by default for all resources.# argocd-cm
data:
resource.exclusions: |
- apiGroups: [""]
kinds: ["Event"]
clusters: ["*"]
- apiGroups: ["coordination.k8s.io"]
kinds: ["Lease"]
clusters: ["*"]
- apiGroups: ["cilium.io"]
kinds: ["CiliumIdentity"]
clusters: ["*"]Change one knob at a time and watch the reconcile histogram.
| Symptom | Likely bottleneck | Knob |
|---|---|---|
| UI shows "Refreshing" for minutes, reconcile p95 high | Controller workers | controller.status.processors, more CPU, shard |
| Syncs queue behind each other | Sync workers | controller.operation.processors |
| ComparisonError: context deadline exceeded | Repo-server render time | More repo-server replicas, reposerver.parallelism.limit, smaller charts |
| repo-server OOMKilled | Parallel renders of big charts | Memory limit up, parallelism down |
| Git host rate-limits Argo CD | Polling | Webhooks, longer timeout.reconciliation, manifest-generate-paths |
| Controller memory grows with clusters | Cluster cache | Shard by cluster, resource.exclusions |
If everything is in Git, recovery is a re-bootstrap. Keep the few things that are not.
Design so that a new cluster plus one bootstrap command restores everything.
argocd-cm, argocd-rbac-cm, argocd-cmd-params-cm as Helm values.argocd-secret (admin password hash, server signing key, webhook secrets).# full export of Argo CD objects and settings
argocd admin export -n argocd \
> argocd-backup.yaml
# restore into a fresh install
argocd admin import -n argocd - \
< argocd-backup.yaml
# store encrypted (contains secrets)
aws s3 cp argocd-backup.yaml \
s3://my-backups/argocd/ \
--sse aws:kmsThe Argo CD cluster is cattle. The target clusters keep running if Argo CD dies.
kubectl with a ticket, then commit to Git.Test it: once a quarter, build a second Argo CD from Git in a spare cluster with all apps set to manual sync, and confirm every app shows Synced.
Upgrade notes per minor version list breaking changes. Read every note between your version and the target.
operator-manual/upgrading/<from>-<to> for each step.argo/argo-cd chart version in Git; Argo CD can manage itself, but keep a manual path (helm upgrade) for when it breaks.kubectl get crd applications.argoproj.io -o yaml)..status ignored in diffs.# check versions
argocd version
helm -n argocd list
# preview chart changes
helm diff upgrade argocd argo/argo-cd \
-n argocd --version <new> -f values.yamlCommon and convenient, with one rule: never let a bad commit lock you out.
helm install does the first install.argocd (in the root app) then owns the Helm release values in Git.prune on the argocd Application.selfHeal but review chart bumps in staging first.helm upgrade from a laptop with the last good values.Symptom → cause → fix, plus the CLI commands that show you the truth.
Compare what Git renders with what is live, then read the controller's reason.
# what Argo CD thinks: status, conditions, last operation
argocd app get web-prod --show-operation
# force re-read of Git and re-render (ignores the cache)
argocd app get web-prod --hard-refresh
# what would change
argocd app diff web-prod
# rendered manifests from Git vs live objects
argocd app manifests web-prod --source git
argocd app manifests web-prod --source live
# stop a stuck sync operation
argocd app terminate-op web-prod
# server-side resources of the app with health
argocd app resources web-prodapp get: read the condition message first, it usually names the cause.--hard-refresh if Git changed but Argo CD did not notice.app diff: is the drift real or a defaulting/mutation artifact?helm template / kustomize build.Shown as ComparisonError on the Application; nothing is applied.
| Symptom | Cause | Fix |
|---|---|---|
rpc error ... authentication required | Repo credential missing or expired (token, SSH key, GitHub App) | Fix the repository Secret (label argocd.argoproj.io/secret-type: repository or repo-creds); test with argocd repo list |
helm template ... error converting YAML | Bad values or template bug | Run helm template locally with the same values files; check valueFiles paths |
Unable to resolve 'main' to a commit SHA | Wrong targetRevision or branch renamed | Fix targetRevision; use tags for prod |
context deadline exceeded during manifest generation | Render too slow or repo-server overloaded | Scale repo-server, lower parallelism, split the chart |
app path does not exist | Wrong path, or commit not pushed yet | Check the path at that SHA in Git |
| Helm dependency not found | charts/ not vendored, private dependency repo not registered | Register the dependency repo in Argo CD or commit Chart.lock + vendored charts |
Argo CD renders fine but the sync or the health check does not finish.
| Symptom | Cause | Fix |
|---|---|---|
| OutOfSync right after every sync (loop) | Field mutated by a controller or webhook (HPA replicas, injected sidecars, defaulted fields) | ignoreDifferences + RespectIgnoreDifferences=true; remove replicas from Git when an HPA owns it |
| Sync waits forever on a hook | PreSync Job never completes or image pull fails | kubectl logs job/...; argocd app terminate-op; add activeDeadlineSeconds and a hook delete policy |
the server could not find the requested resource for a CR | CRD not installed yet in the same sync | Put CRDs in an earlier sync wave, or SkipDryRunOnMissingResource=true on the CR |
Health stuck Progressing | Deployment cannot roll out, or Ingress without a load balancer address | Check the workload events; custom Lua health check for CRs Argo CD does not know |
field is immutable on sync | Changed selector, Job spec, or StatefulSet volumeClaimTemplates | Delete and recreate that object (Replace=true sparingly), fix labels in the chart |
| Sync blocked: "sync window" | A deny window in the AppProject is active | Wait, or use a manual override if the window allows it |
The problems that appear when apps are removed, moved or shared.
| Symptom | Cause | Fix |
|---|---|---|
| Application stuck in deletion | Finalizer resources-finalizer.argocd.argoproj.io waiting on a resource that cannot be deleted | Find the blocking resource (argocd app resources); as a last resort remove the finalizer with kubectl patch |
| Two apps fight over a resource | Both render the same object; tracking annotation flips | One owner per object; Argo CD reports SharedResourceWarning |
| Resources orphaned after removing them from Git | prune disabled | Enable prune, or use the orphaned resources warning in the AppProject |
permission denied in UI or CLI | RBAC policy or project role missing; 3.0 fine-grained update/delete | argocd account can-i sync applications 'proj/web-prod'; fix argocd-rbac-cm |
Cluster Unknown / connection refused | Expired cluster token, IAM role trust, or private endpoint not reachable | argocd cluster list; on EKS check the IAM role and access entry |
Removing a finalizer by hand skips cleanup: list the app's resources first so you know what will be left behind.
Each one looks harmless until the day it is not.
kubectl edit and forgetting Git: self-heal reverts it, or worse, the next deploy does.ARGOCD_CONTROLLER_REPLICAS.Answer out loud, then check the slide that covers it.
argocd_app_reconcile p95 tell you?Practice: Lab 12 in Hands-on labs — an ops drill with five pre-broken faults and Argo CD metrics in Prometheus.
Self-managed Argo CD on EKS, the AWS-managed EKS Capability for Argo CD, Terraform bootstrap with the GitOps Bridge, and cluster add-ons by label.
Reference architecture, IAM-based cluster access, and a Terraform bootstrap from zero to a root Application.
Argo CD lives in a small management cluster and deploys to dev, staging and prod clusters, often in other accounts.
Workload clusters need no inbound access from the internet: the hub reaches their API endpoints (public with CIDR allow-list, or private over TGW/peering).
No static tokens: the controller uses an IAM role, and each target cluster maps that role with an access entry.
apiVersion: v1
kind: Secret
metadata:
name: cluster-prod
namespace: argocd
labels:
argocd.argoproj.io/secret-type: cluster
env: prod
type: Opaque
stringData:
name: prod
server: https://ABCD.gr7.ap-northeast-1.eks.amazonaws.com
config: |
{
"awsAuthConfig": {
"clusterName": "prod",
"roleARN":
"arn:aws:iam::222233334444:role/argocd-deployer"
},
"tlsClientConfig": {"caData": "<base64 CA>"}
}argocd-deployer in the target account (roleARN).aws eks get-token style auth for clusterName.argocd-deployer + an access policy (cluster admin, or namespace-scoped).argocd-server pod needs the same role to show live state in the UI.env: prod) feed ApplicationSet cluster generators (section 42).Install with the argo/argo-cd chart; these are the values that change on EKS.
# values-eks.yaml (argo/argo-cd)
global:
domain: argocd.example.com
configs:
params:
server.insecure: true # TLS ends at the ALB
cm:
timeout.reconciliation: 180s
rbac:
policy.default: role:readonly
server:
ingress:
enabled: true
controller: aws
ingressClassName: alb
annotations:
alb.ingress.kubernetes.io/scheme: internal
alb.ingress.kubernetes.io/certificate-arn: <acm-arn>
alb.ingress.kubernetes.io/target-type: ip
redis-ha:
enabled: true
controller:
replicas: 1
repoServer:
autoscaling: {enabled: true, minReplicas: 2}server.insecure: the ALB terminates TLS with an ACM certificate; pods speak HTTP inside the VPC.redis-ha and 2+ repo-servers for HA; the controller scales by sharding (ch 12).argocd login --grpc-web.Terraform owns the cluster and the first Argo CD install. Argo CD owns everything after that.
module "eks" {
source = "terraform-aws-modules/eks/aws" # pin version
name = "gitops"
kubernetes_version = "1.33"
vpc_id = module.vpc.vpc_id
subnet_ids = module.vpc.private_subnets
enable_cluster_creator_admin_permissions = true
eks_managed_node_groups = { default = {
instance_types = ["m6i.large"], desired_size = 2 } }
}
resource "helm_release" "argocd" {
name = "argocd"
namespace = "argocd"
create_namespace = true
repository = "https://argoproj.github.io/argo-helm"
chart = "argo-cd"
version = var.argocd_chart_version
values = [file("values-eks.yaml")]
}# argocd-apps chart: no CRD needed at plan time
resource "helm_release" "root" {
name = "root"
namespace = "argocd"
repository = "https://argoproj.github.io/argo-helm"
chart = "argocd-apps"
depends_on = [helm_release.argocd]
values = [yamlencode({
applications = {
root = {
project = "default"
source = {
repoURL = "https://github.com/<you>/gitops-lab"
path = "clusters/gitops"
targetRevision = "main"
}
destination = { namespace = "argocd",
server = "https://kubernetes.default.svc" }
syncPolicy = { automated = { prune = true, selfHeal = true } }
}
}
})]
}Module input names change between major versions of terraform-aws-modules/eks/aws; check the README of the version you pin.
Draw the line once and write it down; overlap causes two tools to fight.
| Resource | Owner | Why |
|---|---|---|
| VPC, EKS cluster, node groups, IAM roles, access entries | Terraform | AWS APIs, needed before Kubernetes exists |
| Argo CD first install + root Application | Terraform (once) | Bootstrap; after that Argo CD can manage its own chart |
| Cluster add-ons (LB controller, ExternalDNS, Karpenter config, metrics) | Argo CD | Kubernetes resources, versioned per cluster |
| Workloads and their config | Argo CD | Team-owned, promoted through Git |
| IAM roles for add-ons (Pod Identity associations) | Terraform | AWS side; pass ARNs to Argo CD as metadata (next slide) |
| Karpenter itself | Either, pick one | Often Terraform installs it so nodes exist before Argo CD runs |
Terraform knows the AWS facts (role ARNs, Region, VPC ID); Argo CD templates need them. Put them on the cluster Secret.
# cluster Secret written by Terraform
# (gitops-bridge module or a kubernetes_secret)
metadata:
name: gitops
labels:
argocd.argoproj.io/secret-type: cluster
environment: dev
enable_aws_load_balancer_controller: "true"
enable_external_dns: "true"
annotations:
aws_region: ap-northeast-1
aws_account_id: "111122223333"
aws_cluster_name: gitops
aws_vpc_id: vpc-0abc
aws_load_balancer_controller_iam_role_arn: arn:aws:iam::...
addons_repo_url: https://github.com/<you>/gitops-lab
stringData:
name: gitops
server: https://kubernetes.default.svcgitops-bridge-dev) with a Terraform module and add-on ApplicationSets.AWS-managed Argo CD: how it differs from upstream, its limits, and when to choose it.
Launched Nov 2025 with EKS Capabilities (alongside ACK and kro). AWS runs Argo CD; you use its CRDs.
The local cluster is not registered automatically; register it explicitly (the docs suggest the name in-cluster).
apiVersion: v1
kind: Secret
metadata:
name: prod
namespace: argocd
labels:
argocd.argoproj.io/secret-type: cluster
env: prod
stringData:
name: prod
# EKS cluster ARN, not the API server URL
server: arn:aws:eks:ap-northeast-1:222233334444:cluster/prod
---
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: web-prod
namespace: argocd
spec:
project: default
destination:
name: prod # or server: <cluster ARN>
namespace: webaws eks create-access-entry), then associate an access policy.AmazonEKSClusterAdminPolicy for platform add-ons; namespace-scoped policies for tenant-only clusters.Size your design against these before migrating.
| Limit | Value |
|---|---|
| Application resources | 3,000 |
| Remote clusters | 100 |
| Kubernetes resources per Application | 150 |
| Git polling interval | 6 minutes (self-managed default: 120 s + up to 60 s jitter) |
| Sync timeout | Fixed at 120 s |
Check every feature you use today against this list (and the current docs).
argocd-cm configuration keys: only a documented subset.Pick by the features you need and how much you want to operate.
| Question | EKS capability | Self-managed | Vendor SaaS (e.g. Akuity) |
|---|---|---|---|
| Who runs, scales, upgrades Argo CD | AWS | You | Vendor |
| Version choice | No | Yes | Usually yes |
| Targets | EKS clusters (by ARN) | Any Kubernetes | Any Kubernetes |
| SSO | IAM Identity Center only | Any OIDC/SAML via Dex or direct | Vendor-provided |
| CMPs, notifications, UI extensions | No | Yes | Varies |
| Scale tuning (shards, processors) | Not exposed; fixed limits | Full control | Vendor-managed |
| Best fit | AWS-only shops, small platform team, standard Helm/Kustomize | Heavy customization, non-EKS targets | Multi-cloud, promotion features |
One ApplicationSet per add-on, enabled per cluster by label, configured by annotations.
Enable an add-on on a cluster by adding one label to its cluster Secret.
Chart from the public EKS repo; IAM role, Region and VPC ID come from the cluster Secret annotations.
apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
name: addon-aws-load-balancer-controller
namespace: argocd
spec:
goTemplate: true
goTemplateOptions: ["missingkey=error"]
generators:
- clusters:
selector:
matchLabels:
enable_aws_load_balancer_controller: "true"
template:
metadata:
name: 'lbc-{{.name}}' spec:
project: platform
source:
repoURL: https://aws.github.io/eks-charts
chart: aws-load-balancer-controller
targetRevision: '{{index .metadata.annotations
"lbc_chart_version"}}'
helm:
valuesObject:
clusterName: '{{index .metadata.annotations
"aws_cluster_name"}}'
region: '{{index .metadata.annotations "aws_region"}}'
vpcId: '{{index .metadata.annotations "aws_vpc_id"}}'
destination:
name: '{{.name}}'
namespace: kube-system
syncPolicy:
automated: {prune: true, selfHeal: true}Each line on the previous slide prevents a real failure mode.
goTemplate: true + missingkey=error: a cluster missing an annotation fails loudly instead of rendering empty values.index .metadata.annotations "key": annotation keys may contain characters a dotted path cannot.lbc_chart_version): prod upgrades after dev by changing one annotation.destination.name: the cluster Secret name; works for both self-managed and the EKS capability.platform may create cluster-scoped resources; tenant projects may not.env label.Some add-ons need others first: CRDs, webhooks, IAM.
Most are about IAM and boundaries, not about Argo CD itself.
Answer out loud, then check the slide that covers it.
argocd-apps chart instead of kubernetes_manifest?enable_external_dns: "true".missingkey=error?Practice: Lab 13 in Hands-on labs — Terraform a small EKS cluster, enable Argo CD (capability or self-managed), sync an app with External Secrets and an ALB Ingress, then destroy.
Everything in the deck in one pipeline. Build it after Lab 13, on kind or on EKS.
Promotion is a Git change: dev → staging → prod are three small PRs to envs/<env>/values.yaml, each merged only after the previous environment is Healthy.
Done means every box is ticked and proven with a command, not a screenshot.
cosign sign, helm package + helm push oci://….image.tag in envs/dev/values.yaml.envs/*/config.yaml).argocd app wait web-dev --health).--random-error) aborts the canary automatically.kubectl edit in prod is reverted by self-heal within one reconcile.git revert, measured end to end in minutes.env/* branch.argocd_app_info{health_status="Degraded"}.clusters/ only.Release name first, chart second. -n everywhere: releases are namespaced.
| Command | What it does |
|---|---|
helm upgrade --install web ./charts/web -n demo --create-namespace | Idempotent install or upgrade |
helm upgrade … --rollback-on-failure --timeout 5m | Wait, roll back if not ready (Helm 3: --atomic) |
helm upgrade … --server-side | Force Server-Side Apply on an old release |
helm rollback web 3 -n demo | Re-apply revision 3 as a new revision |
helm history web -n demo | Revisions, status, chart and app version |
helm uninstall web -n demo --keep-history | Remove resources, keep release records |
helm test web -n demo | Run pods annotated helm.sh/hook: test |
| Command | What it does |
|---|---|
helm template web ./charts/web -f dev.yaml | Render locally, no cluster |
helm install … --dry-run=server | Render + validate against the API server |
helm get values|manifest|hooks web | What the release actually used |
helm show values repo/chart | Default values of a chart |
helm dependency update ./charts/web | Fetch deps, write Chart.lock |
helm package ./charts/web / helm push x.tgz oci://reg/charts | Publish to an OCI registry |
helm lint --strict / helm registry login reg.example.com | Lint; log in (host only, no URL) |
Pipelines pass the left value as the last argument: .Values.x | default "a" = default "a" .Values.x.
| Function | Example | Note |
|---|---|---|
include + nindent | {{- include "web.labels" . | nindent 4 }} | Use instead of template (pipeable) |
toYaml | {{- toYaml .Values.resources | nindent 12 }} | Dump a values subtree |
default | {{ .Values.image.tag | default .Chart.AppVersion }} | Empty string, 0, false also count as empty |
required | {{ required "host is required" .Values.host }} | Fail render with a message |
quote / squote | value: {{ .Values.flag | quote }} | Keep strings strings ("true", "0123") |
tpl | {{ tpl .Values.extraConfig . }} | Render a template stored in values |
lookup | (lookup "v1" "Secret" .Release.Namespace "db") | Empty under helm template and Argo CD |
sha256sum | checksum/config: {{ include (print $.Template.BasePath "/cm.yaml") . | sha256sum }} | Roll pods on config change |
with / range / $ | {{- range $k, $v := .Values.env }} … {{ $.Release.Name }} | . changes inside; $ is root |
fail | {{- if and .Values.a .Values.b }}{{ fail "a and b are exclusive" }}{{ end }} | Guard invalid combinations |
The CLI talks to the API server: log in first (argocd login), or use --core to talk to Kubernetes directly.
| Command | What it does |
|---|---|
argocd app list / app get web-dev | Status, health, resources |
argocd app get web-dev --hard-refresh | Re-render, bypass the manifest cache |
argocd app diff web-dev | Live vs desired (exit 1 if different) |
argocd app sync web-dev [--prune] [--resource :Deployment:web] | Sync all or one resource |
argocd app wait web-dev --health --timeout 300 | Block until Healthy (CI gates) |
argocd app manifests web-dev [--source live] | Rendered or live manifests |
argocd app history web-dev / app rollback web-dev 3 | Rollback needs auto-sync off |
argocd app terminate-op web-dev | Stop a stuck sync operation |
| Command | What it does |
|---|---|
argocd login localhost:8080 --insecure | Log in through a port-forward |
argocd account update-password [--account alice] | Change a local password |
argocd repo add|list | Repositories and credentials |
argocd cluster add <ctx> --name prod / cluster list | Register a cluster |
argocd proj create|get|role | AppProjects and project roles |
argocd appset list|get|create | ApplicationSets |
argocd admin settings rbac can alice sync applications 'team-a/*' --policy-file policy.csv | Test an RBAC rule offline (without --policy-file it reads argocd-rbac-cm from the cluster) |
argocd admin export > backup.yaml | Back up apps, projects, settings |
Copy, then delete what you do not need. Both live in the argocd namespace.
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: web-prod
namespace: argocd
finalizers: # cascade delete
- resources-finalizer.argocd.argoproj.io
spec:
project: team-a
source:
repoURL: https://github.com/<you>/gitops-lab
targetRevision: main # branch, tag or SHA
path: charts/web
helm:
valueFiles: [values.yaml]
valuesObject: {replicaCount: 3}
destination:
server: https://kubernetes.default.svc
namespace: prod
syncPolicy:
automated: {prune: true, selfHeal: true}
syncOptions: [CreateNamespace=true, ServerSideApply=true]
retry: {limit: 5, backoff: {duration: 10s, factor: 2}}
ignoreDifferences:
- {group: apps, kind: Deployment, jsonPointers: [/spec/replicas]}apiVersion: argoproj.io/v1alpha1
kind: AppProject
metadata:
name: team-a
namespace: argocd
spec:
description: Team A services
sourceRepos:
- https://github.com/<you>/gitops-lab
destinations:
- server: https://kubernetes.default.svc
namespace: team-a
clusterResourceWhitelist: [] # no cluster-scoped objects
namespaceResourceBlacklist:
- {group: "", kind: ResourceQuota}
roles:
- name: ci
policies:
- p, proj:team-a:ci, applications, sync, team-a/*, allow
syncWindows:
- kind: deny
schedule: "0 18 * * 5" # Friday 18:00
duration: 64h
applications: ["*"]Go templates (goTemplate: true) with missingkey=error turn typos into errors instead of empty strings.
apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
name: web
namespace: argocd
spec:
goTemplate: true
goTemplateOptions: ["missingkey=error"]
generators:
- git:
repoURL: https://github.com/<you>/gitops-lab
revision: main
files:
- path: "envs/*/config.yaml" # keys become params
syncPolicy:
preserveResourcesOnDeletion: true template:
metadata:
name: 'web-{{.path.basename}}'
spec:
project: default
sources:
- repoURL: https://github.com/<you>/gitops-lab
targetRevision: main
path: charts/web
helm:
valueFiles:
- '$values/envs/{{.path.basename}}/values.yaml'
- repoURL: https://github.com/<you>/gitops-lab
targetRevision: main
ref: values # makes $values
destination:
server: '{{.server}}' # from config.yaml
namespace: '{{.path.basename}}'
syncPolicy:
automated: {prune: true, selfHeal: true}
syncOptions: [CreateNamespace=true]Other generators: list, clusters (label selector on cluster Secrets), matrix, merge, scmProvider, pullRequest.
Sync options go in spec.syncPolicy.syncOptions (whole app) or the argocd.argoproj.io/sync-options annotation (one resource).
| Option | Effect |
|---|---|
CreateNamespace=true | Create the destination namespace |
ServerSideApply=true | Apply with SSA (large CRDs, field ownership) |
PruneLast=true | Prune after all other resources are applied |
ApplyOutOfSyncOnly=true | Apply only changed resources (big apps) |
RespectIgnoreDifferences=true | Do not overwrite ignored fields on sync |
Replace=true | kubectl replace/create instead of apply |
Prune=false (annotation) | Never prune this resource |
| Annotation | Effect |
|---|---|
sync-wave: "-1" | Order within a phase (lower first) |
hook: PreSync|Sync|PostSync|SyncFail|PreDelete | Run as a hook in that phase |
hook-delete-policy: HookSucceeded|HookFailed|BeforeHookCreation | When to delete the hook |
compare-options: IgnoreExtraneous | Ignore for sync status |
sync-options: Delete=false | Keep the object when the app is deleted |
manifest-generate-paths: . | Only refresh when these paths change (monorepos) |
Labs 0–12 run free on a laptop with kind. Lab 13 is the only one that costs money.
gh auth login done (repo + PR labs).eksctl, a budget alarm.kind-gitops; second cluster kind-prod (Lab 9).https://github.com/<you>/gitops-lab: charts/web, apps/, envs/{dev,staging,prod}/, clusters/.ghcr.io/stefanprodan/podinfo (port 9898); chart web.argocd, reached with a port-forward on localhost:8080.export REPO=https://github.com/<you>/gitops-lab in every shell.brew install kind kubectl helm kustomize \
argocd kubeconform chart-testing \
yq jq gh kubeseal
brew install \
argoproj/tap/kubectl-argo-rollouts
# Lab 13 only
brew install awscli eksctl
helm version # v4.x
argocd version --client # v3.5.x
kind version
kubectl version --client
kustomize version
kubeconform -vEach lab = Build, Verify, Break it, Clean up. Commit everything you write to gitops-lab: later labs build on earlier ones (Lab 3 chart → Labs 4–12).
Goal: a two-node kind cluster and an empty config repo that every later lab commits to.
# kind-gitops.yaml
cat > kind-gitops.yaml <<'EOF'
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
name: gitops
nodes:
- role: control-plane
- role: worker
EOF
kind create cluster --config kind-gitops.yaml
kubectl config current-context # kind-gitops
kubectl get nodes -o wide # 2 nodes Ready
# helper used in every lab
cat >> ~/.zshrc <<'EOF'
alias k=kubectl
export REPO=https://github.com/<you>/gitops-lab
EOF
source ~/.zshrcgh repo create gitops-lab --public --clone
cd gitops-lab
mkdir -p charts apps clusters envs/{dev,staging,prod}
for d in apps clusters envs/dev envs/staging envs/prod; do
touch $d/.gitkeep; done
cat > README.md <<'EOF'
# gitops-lab
charts/ Helm charts (web, common)
apps/ Argo CD Application manifests and raw YAML
envs/ per-environment values (dev, staging, prod)
clusters/ bootstrap manifests per cluster
EOF
git add . && git commit -m "repo skeleton"
git push -u origin main
# record versions in the repo
{ helm version; argocd version --client --short
kubectl version --client; kind version; } > versions.txt
git add versions.txt && git commit -m versions && git pushThe most common lab mistake is a command sent to the wrong context.
kubectl get nodes shows gitops-control-plane and gitops-worker Ready.gh repo view <you>/gitops-lab --web opens the repo with the skeleton.helm version prints v4; argocd version --client prints v3.5.docker ps shows two kindest/node containers.kubectl config use-context docker-desktop (or any other) → kubectl get nodes shows other nodes. Fix: kubectl config use-context kind-gitops; add the context to your prompt.kubectl times out, kind get clusters errors. Start Docker; kind containers restart with it.--context explicitly?# keep the cluster: Labs 1-12 use it
# full reset at the very end:
kind delete clusters --all
gh repo delete <you>/gitops-lab \
--yesGoal: feel the difference between a script that pushes YAML and an agent that keeps pulling it.
# apps/podinfo-raw/podinfo.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: podinfo
spec:
replicas: 2
selector:
matchLabels: {app: podinfo}
template:
metadata:
labels: {app: podinfo}
spec:
containers:
- name: podinfo
image: ghcr.io/stefanprodan/podinfo:6.7.0
ports: [{containerPort: 9898}]
---
apiVersion: v1
kind: Service
metadata:
name: podinfo
spec:
selector: {app: podinfo}
ports: [{port: 9898}]# PUSH: what a CI job with cluster-admin would do
kubectl create namespace push-demo
kubectl -n push-demo apply -f apps/podinfo-raw/
git add apps/podinfo-raw && git commit -m podinfo && git push
# PULL: a quick Argo CD (reinstalled properly in Lab 6)
helm repo add argo https://argoproj.github.io/argo-helm
helm install argocd argo/argo-cd -n argocd \
--create-namespace --wait
kubectl -n argocd port-forward svc/argocd-server \
8080:443 >/dev/null 2>&1 &
PW=$(kubectl -n argocd get secret \
argocd-initial-admin-secret \
-o jsonpath='{.data.password}' | base64 -d)
argocd login localhost:8080 --username admin \
--password "$PW" --insecure
argocd app create podinfo --repo $REPO \
--path apps/podinfo-raw \
--dest-server https://kubernetes.default.svc \
--dest-namespace pull-demo \
--sync-policy automated --self-heal --auto-prune \
--sync-option CreateNamespace=trueMake the same manual change in both namespaces and watch what happens.
kubectl get deploy -A | grep podinfo: 2/2 in push-demo and pull-demo.argocd app get podinfo: Sync Status Synced, Health Healthy.kubectl -n push-demo scale deploy podinfo --replicas=5 → stays 5 forever; nobody knows.selfHeal); argocd app history podinfo shows nothing new because Git did not change.kubectl -n pull-demo delete deploy podinfo → recreated. Then set replicas: 3 in Git, push: only pull-demo follows (within ~3 min, or click Refresh).argocd app delete podinfo --yes
kubectl delete ns push-demo
helm uninstall argocd -n argocd
kubectl delete ns argocd
pkill -f "port-forward svc/argocd"
# keep apps/podinfo-raw in GitGoal: install, upgrade, roll back, and see exactly where Helm keeps release state.
helm repo add podinfo https://stefanprodan.github.io/podinfo
helm repo update
helm show values podinfo/podinfo | head -40
helm install web podinfo/podinfo -n demo \
--create-namespace --set replicaCount=2
helm upgrade web podinfo/podinfo -n demo \
--reuse-values --set ui.message="hello v2"
helm upgrade web podinfo/podinfo -n demo \
--reuse-values --set ui.color="#34577c"
helm history web -n demo # revisions 1-3
helm rollback web 2 -n demo # → revision 4 = copy of 2
helm history web -n demo
helm get values web -n demo # values you supplied
helm get values web -n demo --all # merged with defaults
helm get manifest web -n demo | head -30# release state = one Secret per revision
kubectl -n demo get secret -l owner=helm
# sh.helm.release.v1.web.v1 ... v4 helm.sh/release.v1
# decode: base64 (Kubernetes) -> base64 (Helm) -> gzip -> JSON
kubectl -n demo get secret sh.helm.release.v1.web.v4 \
-o jsonpath='{.data.release}' \
| base64 -d | base64 -d | gunzip \
| jq '{name, version, status: .info.status,
chart: .chart.metadata.version}'
# render without touching the cluster
helm template web podinfo/podinfo --set replicaCount=3 \
| grep -B2 -A6 "kind: Deployment"
# validate against the API server, change nothing
helm upgrade web podinfo/podinfo -n demo --reuse-values \
--set replicaCount=3 --dry-run=server | head -20
kubectl -n demo port-forward svc/web-podinfo 9898:9898 &
curl -s localhost:9898 | jq .messageHelm 4 renamed --atomic to --rollback-on-failure; the old flag still works with a warning.
helm history web -n demo shows 4 revisions; revision 4 description "Rollback to 2".curl localhost:9898 shows "hello v2" (from revision 2), not the color from 3.sh.helm.release.v1.web.v* Secrets exist; --history-max (default 10) caps them.helm upgrade web podinfo/podinfo -n demo --reuse-values --set image.tag=0.0.0-missing --rollback-on-failure --timeout 90s → pods ImagePullBackOff; after 90 s Helm rolls back; history shows failed then a new deployed.--rollback-on-failure → release status failed, broken pods stay. Recover: helm rollback web -n demo (previous revision).pending-upgrade blocks the next upgrade; fix with helm rollback.pkill -f "port-forward svc/web-podinfo"
helm uninstall web -n demo
kubectl delete ns demo
helm repo remove podinfoweb chart from scratch (1/2): metadata, values, schemaGoal: a small production-shaped chart you fully understand. Every later lab deploys it.
mkdir -p charts/web/templates
# charts/web/Chart.yaml
apiVersion: v2
name: web
description: podinfo-based web service
type: application
version: 0.1.0 # chart version: bump on every change
appVersion: "6.7.0" # default image tag
# charts/web/values.yaml
replicaCount: 2
image:
repository: ghcr.io/stefanprodan/podinfo
tag: "" # empty -> .Chart.AppVersion
service:
port: 9898
message: "hello from web"
resources:
requests: {cpu: 50m, memory: 64Mi}
limits: {memory: 128Mi}# charts/web/values.schema.json
{
"$schema": "https://json-schema.org/draft-07/schema",
"type": "object",
"required": ["replicaCount", "image", "service"],
"properties": {
"replicaCount": {"type": "integer", "minimum": 1},
"image": {
"type": "object",
"required": ["repository"],
"properties": {
"repository": {"type": "string"},
"tag": {"type": "string"}
}
},
"service": {
"type": "object",
"properties": {"port": {"type": "integer"}}
},
"message": {"type": "string"},
"resources": {"type": "object"}
}
}web chart from scratch (2/2): templatesHelpers keep names and labels identical everywhere; the checksum rolls pods when the ConfigMap changes.
{{/* charts/web/templates/_helpers.tpl */}}
{{- define "web.fullname" -}}
{{- if contains .Chart.Name .Release.Name -}}
{{- .Release.Name | trunc 63 | trimSuffix "-" -}}
{{- else -}}
{{- printf "%s-%s" .Release.Name .Chart.Name
| trunc 63 | trimSuffix "-" -}}
{{- end -}}
{{- end -}}
{{- define "web.selectorLabels" -}}
app.kubernetes.io/name: {{ .Chart.Name }}
app.kubernetes.io/instance: {{ .Release.Name }}
{{- end -}}
{{- define "web.labels" -}}
{{ include "web.selectorLabels" . }}
app.kubernetes.io/version: {{ .Chart.AppVersion | quote }}
helm.sh/chart: {{ printf "%s-%s" .Chart.Name .Chart.Version }}
{{- end -}}
{{/* templates/configmap.yaml */}}
apiVersion: v1
kind: ConfigMap
metadata:
name: {{ include "web.fullname" . }}
labels: {{- include "web.labels" . | nindent 4 }}
data:
PODINFO_UI_MESSAGE: {{ .Values.message | quote }}{{/* templates/deployment.yaml */}}
apiVersion: apps/v1
kind: Deployment
metadata:
name: {{ include "web.fullname" . }}
labels: {{- include "web.labels" . | nindent 4 }}
spec:
replicas: {{ .Values.replicaCount }}
selector:
matchLabels: {{- include "web.selectorLabels" . | nindent 6 }}
template:
metadata:
labels: {{- include "web.selectorLabels" . | nindent 8 }}
annotations:
checksum/config: {{ include (print $.Template.BasePath
"/configmap.yaml") . | sha256sum }}
spec:
containers:
- name: web
image: "{{ .Values.image.repository }}:{{
.Values.image.tag | default .Chart.AppVersion }}"
ports: [{name: http, containerPort: 9898}]
envFrom:
- configMapRef:
name: {{ include "web.fullname" . }}
resources: {{- toYaml .Values.resources | nindent 12 }}
{{/* templates/service.yaml: same metadata; spec.selector =
web.selectorLabels (nindent 4); port .Values.service.port,
targetPort: http */}}Read the error messages slowly: they name the file and the line.
helm lint charts/web --strict → 0 failures.helm template web charts/web | kubeconform -strict -summary → all valid.helm install web charts/web -n demo --create-namespace; kubectl -n demo get deploy web 2/2.message, helm upgrade web charts/web -n demo → pods roll (kubectl -n demo rollout status deploy/web) because the checksum changed.charts/web and push: Labs 4–12 use it.nindent 8 to nindent 4 → helm template fails with a YAML parse error on web/templates/deployment.yaml. helm template --debug prints the broken YAML.helm upgrade web charts/web -n demo --set replicaCount=two → "values don't meet the specifications of the schema"; --set replicaCount=0 fails minimum.| quote from the message and set message: "true" → ConfigMap rejected: data values must be strings.helm uninstall web -n demo
kubectl delete ns demo
git add charts/web
git commit -m "web chart v0.1.0"
git pushGoal: share labels through a library chart, run a migration before every upgrade, and test the release.
mkdir -p charts/common/templates
cat > charts/common/Chart.yaml <<'EOF'
apiVersion: v2
name: common
type: library # no resources of its own
version: 0.1.0
EOF
cat > charts/common/templates/_labels.tpl <<'EOF'
{{- define "common.labels" -}}
app.kubernetes.io/name: {{ .Chart.Name }}
app.kubernetes.io/instance: {{ .Release.Name }}
app.kubernetes.io/managed-by: {{ .Release.Service }}
{{- end -}}
EOF
# append to charts/web/Chart.yaml, bump version: 0.2.0
dependencies:
- name: common
version: 0.1.0
repository: file://../common
helm dependency update charts/web # charts/ + Chart.lock
# templates/tests/test-connection.yaml
apiVersion: v1
kind: Pod
metadata:
name: {{ include "web.fullname" . }}-test
annotations: {"helm.sh/hook": test}
spec:
restartPolicy: Never
containers: [{name: curl, image: "curlimages/curl:8.10.1",
args: ["-fsS", "http://{{ include "web.fullname" . }}:9898/healthz"]}]# templates/migrate-job.yaml
apiVersion: batch/v1
kind: Job
metadata:
name: {{ include "web.fullname" . }}-migrate
annotations:
"helm.sh/hook": pre-install,pre-upgrade
"helm.sh/hook-weight": "-5"
"helm.sh/hook-delete-policy": before-hook-creation,hook-succeeded
spec:
backoffLimit: 0
template:
spec:
restartPolicy: Never
containers:
- name: migrate
image: busybox:1.36
command: ["sh", "-c", "echo migrating; sleep 5"]
# local OCI registry + publish
docker run -d --name registry -p 5001:5000 registry:2
helm package charts/web -d dist/ # web-0.2.0.tgz
helm push dist/web-0.2.0.tgz oci://localhost:5001/charts \
--plain-http
helm show chart oci://localhost:5001/charts/web \
--version 0.2.0 --plain-httpOne script that CI runs on every pull request touching charts/.
# charts/web/tests/deployment_test.yaml (helm-unittest)
suite: deployment
templates:
- templates/deployment.yaml
- templates/configmap.yaml
tests:
- it: defaults the image tag to appVersion
template: templates/deployment.yaml
asserts:
- equal:
path: spec.template.spec.containers[0].image
value: ghcr.io/stefanprodan/podinfo:6.7.0
- it: honours replicaCount
template: templates/deployment.yaml
set: {replicaCount: 4}
asserts:
- equal: {path: spec.replicas, value: 4}# scripts/chart-ci.sh
#!/usr/bin/env bash
set -euo pipefail
CHART=charts/web
helm dependency build $CHART
helm lint $CHART --strict
helm template web $CHART \
| kubeconform -strict -summary -kubernetes-version 1.33.0
helm unittest $CHART
ct lint --charts $CHART --validate-maintainers=false
echo "chart OK"
# one-time setup
helm plugin install --verify=false \
https://github.com/helm-unittest/helm-unittest
# Helm 4 verifies plugin provenance; git plugins need --verify=false
chmod +x scripts/chart-ci.sh && ./scripts/chart-ci.shct lint compares against the Git default branch, so commit first. Install ct with brew install chart-testing.
Watch the order: the migration Job finishes before the Deployment changes.
helm upgrade --install web charts/web -n demo --create-namespace while kubectl -n demo get jobs,pods -w → web-migrate completes first, then is deleted (hook-succeeded).helm test web -n demo → Phase: Succeeded.helm pull oci://localhost:5001/charts/web --version 0.2.0 --plain-http downloads the package../scripts/chart-ci.sh prints "chart OK".exit 1, upgrade → "pre-upgrade hooks failed"; the Deployment is unchanged. Fix, bump version, upgrade.helm package + push 0.2.0 again → registry:2 silently overwrites it. With ECR tag immutability the push is rejected. Rule: never reuse a chart version.appVersion: "6.7.1" → the first test fails with expected/actual values.helm uninstall web -n demo
kubectl delete ns demo
docker rm -f registry
rm -rf dist
git add charts scripts
git commit -m "web 0.2.0"
git pushGoal: render once with Helm, then tweak per environment with Kustomize, and diff the environments.
mkdir -p kustomize/{base,dev,prod,helm}
helm template web charts/web --skip-tests \
> kustomize/base/web.yaml
cat > kustomize/base/kustomization.yaml <<'EOF'
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources: [web.yaml]
EOF
cat > kustomize/dev/kustomization.yaml <<'EOF'
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
namespace: dev
resources: [../base]
replicas: [{name: web, count: 1}]
EOF
cat > kustomize/prod/kustomization.yaml <<'EOF'
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
namespace: prod
resources: [../base]
replicas: [{name: web, count: 3}]
images:
- name: ghcr.io/stefanprodan/podinfo
newTag: 6.7.1
patches:
- path: limits.yaml
EOFcat > kustomize/prod/limits.yaml <<'EOF'
apiVersion: apps/v1
kind: Deployment
metadata:
name: web
spec:
template:
spec:
containers:
- name: web
resources:
limits: {memory: 256Mi}
EOF
kustomize build kustomize/dev | kubeconform -strict -summary
diff <(kustomize build kustomize/dev) \
<(kustomize build kustomize/prod)
# alternative: let Kustomize inflate the chart itself
cat > kustomize/helm/kustomization.yaml <<'EOF'
helmGlobals:
chartHome: ../../charts
helmCharts:
- name: web
releaseName: web
namespace: demo
valuesInline: {replicaCount: 2, message: "via kustomize"}
EOF
kustomize build --enable-helm kustomize/helm | head -20The diff between dev and prod should be short and boring. If it is long, the overlays are doing too much.
diff shows only: namespace, replicas, image tag, memory limit.kubectl create ns dev && kubectl apply -k kustomize/dev → kubectl -n dev get deploy web 1/1.metadata.name in limits.yaml to webapp → kustomize build fails: no resource matches the patch. Fix the name.namespace: dev from the overlay → objects have no namespace and land in your current namespace (default). Always set it in the overlay.charts/web/values.yaml → the base does not change until you re-run helm template. Lesson: render in CI, or use helmCharts.kubectl delete -k kustomize/dev
kubectl delete ns dev
git add kustomize
git commit -m "kustomize overlays"
git pushGoal: Argo CD from its Helm chart with a values file, declarative repo, project and Application.
# clusters/gitops/argocd-values.yaml
configs:
cm:
timeout.reconciliation: 180s # Git poll interval
params:
server.insecure: false
dex:
enabled: false # SSO comes in Lab 10
notifications:
enabled: true
helm repo add argo https://argoproj.github.io/argo-helm
helm upgrade --install argocd argo/argo-cd -n argocd \
--create-namespace -f clusters/gitops/argocd-values.yaml \
--wait
kubectl -n argocd get deploy,sts
kubectl -n argocd port-forward svc/argocd-server \
8080:443 >/dev/null 2>&1 &
argocd login localhost:8080 --username admin --insecure \
--password "$(kubectl -n argocd get secret \
argocd-initial-admin-secret \
-o jsonpath='{.data.password}' | base64 -d)"
argocd account update-password
kubectl apply -f clusters/gitops/bootstrap.yaml# clusters/gitops/bootstrap.yaml
apiVersion: v1
kind: Secret
metadata:
name: gitops-lab
namespace: argocd
labels: {argocd.argoproj.io/secret-type: repository}
stringData: # private repo: add username + read-only token
{type: git, url: "https://github.com/<you>/gitops-lab"}
---
apiVersion: argoproj.io/v1alpha1
kind: AppProject
metadata: {name: lab, namespace: argocd}
spec:
sourceRepos: [https://github.com/<you>/gitops-lab]
destinations: [{server: https://kubernetes.default.svc, namespace: dev}]
---
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata: {name: web-dev, namespace: argocd}
spec:
project: lab
source:
repoURL: https://github.com/<you>/gitops-lab
targetRevision: main
path: charts/web
helm: {valuesObject: {replicaCount: 1, message: "dev"}}
destination: {server: https://kubernetes.default.svc, namespace: dev}
syncPolicy: {automated: {prune: true, selfHeal: true},
syncOptions: [CreateNamespace=true]}The AppProject is what stops an Application from deploying anywhere it likes.
argocd app get web-dev → Synced, Healthy; the Helm pre-install Job ran as a PreSync hook.kubectl -n argocd get secret -l argocd.argoproj.io/secret-type=repository lists the repo.web-dev, click the Deployment → Live manifest, Diff, Events tabs.kubectl -n argocd get pods: name each pod and its job (Chapter 6).destination.namespace: prod and apply → condition "InvalidSpecError": the destination does not match any allowed destination in project lab. Fix: revert, or add the destination to the project.repoURL → ComparisonError / repository not accessible; argocd app get web-dev shows it under Conditions.path: charts/webb → "app path does not exist".# keep Argo CD and web-dev for Lab 7+
git add clusters/gitops
git commit -m "argocd bootstrap"
git push
# never commit a token in
# bootstrap.yamlGoal: see the exact order Argo CD applies things in, and what a failing hook does to a sync.
# apps/sync-demo/00-config.yaml
apiVersion: v1
kind: ConfigMap
metadata:
name: app-config
annotations: {argocd.argoproj.io/sync-wave: "-1"}
data: {MODE: demo}
---
# apps/sync-demo/10-migrate.yaml
apiVersion: batch/v1
kind: Job
metadata:
name: migrate
annotations:
argocd.argoproj.io/hook: PreSync
argocd.argoproj.io/hook-delete-policy: BeforeHookCreation
spec:
backoffLimit: 0
template:
spec:
restartPolicy: Never
containers:
- name: m
image: busybox:1.36
command: ["sh", "-c", "echo migrate; sleep 5"]
# 20-web.yaml: Deployment + Service "web" (podinfo, replicas: 1)
# with annotation argocd.argoproj.io/sync-wave: "1"
# 30-smoke.yaml: Job "smoke", hook PostSync,
# hook-delete-policy HookSucceeded, curlimages/curl
# args ["-fsS", "http://web:9898/healthz"]# metrics-server so the HPA in "Break it" can work on kind
helm repo add metrics-server \
https://kubernetes-sigs.github.io/metrics-server/
helm install metrics-server metrics-server/metrics-server \
-n kube-system --set args={--kubelet-insecure-tls}
git add apps/sync-demo && git commit -m sync-demo && git push
argocd app create sync-demo --project default --repo $REPO \
--path apps/sync-demo \
--dest-server https://kubernetes.default.svc \
--dest-namespace sync-demo \
--sync-option CreateNamespace=true
argocd app sync sync-demo
# watch the order in a second terminal
kubectl -n sync-demo get jobs,cm,deploy,pods -w
# PreSync: migrate -> wave -1: app-config
# -> wave 1: web -> PostSync: smoke
argocd app get sync-demo --show-operationTwo classic production problems, reproduced in five minutes.
argocd app get sync-demo --show-operation lists phases and waves in order.kubectl -n sync-demo get jobs: migrate stays (BeforeHookCreation); smoke is gone (HookSucceeded).exit 1, push, argocd app sync sync-demo → operation Failed, wave 1 never applied. Fix, push, sync again (argocd app terminate-op if it hangs).argocd app set sync-demo --sync-policy automated --self-heal; add an HPA (minReplicas: 2, target web) to Git → the app flips OutOfSync on spec.replicas as self-heal and the HPA fight.ignoreDifferences for /spec/replicas on Deployment web plus RespectIgnoreDifferences=true (in Git, not with kubectl patch).argocd app delete sync-demo --yes
# keep metrics-server
git add apps/sync-demo
git commit -m "sync-demo fixed"
git pushGoal: bootstrap many Applications from Git, then replace hand-written ones with a generator.
# apps/web-dev.yaml (from Lab 6, project default) and
# apps/web-staging.yaml (namespace staging, replicaCount 2)
# clusters/gitops/root.yaml
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata: {name: root, namespace: argocd}
spec:
project: default
source:
repoURL: https://github.com/<you>/gitops-lab
targetRevision: main
path: apps # top-level *.yaml only (not recursive)
destination: {server: https://kubernetes.default.svc, namespace: argocd}
syncPolicy: {automated: {prune: true, selfHeal: true}}
git add apps clusters && git commit -m root && git push
kubectl apply -f clusters/gitops/root.yaml
argocd app list # root, web-dev, web-staging
# per-environment values for the ApplicationSet
printf 'replicaCount: 1\nmessage: dev\n' > envs/dev/values.yaml
printf 'replicaCount: 2\nmessage: staging\n' > envs/staging/values.yaml
printf 'replicaCount: 3\nmessage: prod\n' > envs/prod/values.yaml# apps/web-appset.yaml (replaces web-dev / web-staging)
apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata: {name: web, namespace: argocd}
spec:
goTemplate: true
goTemplateOptions: ["missingkey=error"]
generators:
- git: {repoURL: "https://github.com/<you>/gitops-lab",
revision: main, directories: [{path: envs/*}]}
template:
metadata: {name: 'web-{{.path.basename}}'}
spec:
project: default
sources:
- repoURL: https://github.com/<you>/gitops-lab
targetRevision: main
path: charts/web
helm:
valueFiles:
- '$values/envs/{{.path.basename}}/values.yaml'
- repoURL: https://github.com/<you>/gitops-lab
targetRevision: main
ref: values
destination: {server: "https://kubernetes.default.svc",
namespace: '{{.path.basename}}'}
syncPolicy: {automated: {prune: true, selfHeal: true},
syncOptions: [CreateNamespace=true]}
syncPolicy: {preserveResourcesOnDeletion: true}
# git rm apps/web-dev.yaml apps/web-staging.yaml; add appset; pushAdding an environment is now a directory, not a YAML file per app.
argocd appset list; argocd app list → web-dev, web-staging, web-prod.envs/qa/values.yaml, push → web-qa appears within one reconcile.envs/qa, push → web-qa Application is deleted, but its Deployment stays (preserveResourcesOnDeletion).pullRequest generator (GitHub token in a Secret) creates web-pr-<number> per open PR.{{.path.basename}} to {{.path.basname}} → no apps change; kubectl -n argocd describe applicationset web shows an error condition (missingkey=error). Fix and push.apps/web-dev.yaml → two owners fight over web-dev. Remove the hand-written file first.kubectl delete applicationset web do without preserveResourcesOnDeletion?kubectl delete ns qa
# keep root and the web ApplicationSet
# for Lab 9Goal: one Argo CD (in kind-gitops) deploys prod to a second cluster; promotion is a pull request.
kind create cluster --name prod
kubectl config use-context kind-gitops # create switched it
# an identity for Argo CD inside prod
kubectl --context kind-prod -n kube-system create sa argocd-manager
kubectl --context kind-prod create clusterrolebinding \
argocd-manager --clusterrole cluster-admin \
--serviceaccount kube-system:argocd-manager
kubectl --context kind-prod -n kube-system apply -f - <<'EOF'
apiVersion: v1
kind: Secret
metadata:
name: argocd-manager-token
annotations: {kubernetes.io/service-account.name: argocd-manager}
type: kubernetes.io/service-account-token
EOF
TOKEN=$(kubectl --context kind-prod -n kube-system get secret \
argocd-manager-token -o jsonpath='{.data.token}' | base64 -d)
# register (kind nodes share the "kind" Docker network;
# insecure is for kind only: real clusters use caData / IAM)
kubectl -n argocd apply -f - <<EOF
apiVersion: v1
kind: Secret
metadata: {name: cluster-prod, labels:
{argocd.argoproj.io/secret-type: cluster, env: prod}}
stringData:
name: prod
server: https://prod-control-plane:6443
config: '{"bearerToken":"$TOKEN","tlsClientConfig":{"insecure":true}}'
EOF
argocd cluster list# one config.yaml per env: its keys become generator params
echo 'server: https://kubernetes.default.svc' > envs/dev/config.yaml
echo 'server: https://kubernetes.default.svc' > envs/staging/config.yaml
echo 'server: https://prod-control-plane:6443' > envs/prod/config.yaml
# in apps/web-appset.yaml switch the generator to files:
# - git: {repoURL: ..., revision: main,
# files: [{path: "envs/*/config.yaml"}]}
# and set destination.server: '{{.server}}'
git add envs apps && git commit -m "prod on its own cluster"
git push
# promotion = a reviewed change to one file
git switch -c promote-6.7.1
yq -i '.image.tag = "6.7.1"' envs/staging/values.yaml
git commit -am "staging: web 6.7.1" && git push -u origin HEAD
gh pr create --fill && gh pr merge --squash --delete-branch
git switch main && git pull
argocd app wait web-staging --health --timeout 300
# then the same PR for envs/prod/values.yaml
# rollback = revert the promotion commit
git revert --no-edit HEAD && git pushThe hub never needs inbound access from prod; prod never needs Git credentials.
argocd cluster list → prod with status Successful.kubectl --context kind-prod -n prod get deploy web → 3/3 replicas.argocd app get web-prod → destination https://prod-control-plane:6443.web-staging runs 6.7.0 again and git log records who did it.https://127.0.0.1:<port> → cluster connection fails (inside the hub pod 127.0.0.1 is itself). Fix: the internal name from kind get kubeconfig --name prod --internal.argocd app rollback web-prod 1 → refused: rollback cannot run while automated sync is enabled. That is the point: roll back in Git.argocd app wait) would have stopped it?kubectl -n argocd delete secret \
cluster-prod
kind delete cluster --name prod
# set envs/prod/config.yaml back to
# https://kubernetes.default.svc,
# commit and pushGoal: two teams share one Argo CD without touching each other, and secrets go to Git encrypted.
# clusters/gitops/team-values.yaml (merged into Argo CD)
configs:
cm:
accounts.alice: login
accounts.bob: login
rbac:
policy.default: "" # no access unless granted
policy.csv: |
p, role:team-a, applications, *, team-a/*, allow
p, role:team-a, projects, get, team-a, allow
p, role:team-b, applications, *, team-b/*, allow
p, role:team-b, projects, get, team-b, allow
g, alice, role:team-a
g, bob, role:team-b
helm upgrade argocd argo/argo-cd -n argocd --reuse-values \
-f clusters/gitops/team-values.yaml
kubectl create ns team-a && kubectl create ns team-b
# AppProject team-a (team-b: same with b)
apiVersion: argoproj.io/v1alpha1
kind: AppProject
metadata: {name: team-a, namespace: argocd}
spec:
sourceRepos: [https://github.com/<you>/gitops-lab]
destinations:
- {server: https://kubernetes.default.svc, namespace: team-a}
argocd account update-password --account alice \
--current-password "$ADMIN_PW" --new-password 'alice-lab-1'helm repo add sealed-secrets \
https://bitnami-labs.github.io/sealed-secrets
helm install sealed-secrets sealed-secrets/sealed-secrets \
-n kube-system
mkdir -p apps/team-a
cp apps/podinfo-raw/podinfo.yaml apps/team-a/
kubectl create secret generic db -n team-a \
--from-literal=password='s3cr3t' \
--dry-run=client -o yaml \
| kubeseal --controller-name sealed-secrets \
--controller-namespace kube-system -o yaml \
> apps/team-a/db-sealed.yaml
grep -c s3cr3t apps/team-a/db-sealed.yaml # 0
git add apps/team-a && git commit -m team-a && git push
argocd app create team-a-web --project team-a --repo $REPO \
--path apps/team-a \
--dest-server https://kubernetes.default.svc \
--dest-namespace team-a --sync-policy automated
# optional SSO: dex.config with a GitHub connector
# (callback https://localhost:8080/api/dex/callback)Log in as each user and try to cross into the other team.
argocd login localhost:8080 --username alice --insecure → argocd app list shows only team-a-*.kubectl -n team-a get secret db -o jsonpath='{.data.password}' | base64 -d → s3cr3t, decrypted in-cluster.argocd admin settings rbac can bob sync applications 'team-a/*' --namespace argocd → No.team-a with destination team-b → rejected: destination not permitted in project team-a.argocd app sync team-a-web → permission denied.db-sealed.yaml into a team-b app → the controller cannot decrypt it (strict scope binds name + namespace). Re-seal for team-b.argocd login localhost:8080 \
--username admin --insecure
argocd app delete team-a-web --yes
kubectl delete ns team-a team-b
helm uninstall sealed-secrets \
-n kube-system
# keep RBAC values for Lab 12Goal: a canary that promotes itself when metrics are good and aborts itself when they are not.
helm install argo-rollouts argo/argo-rollouts \
-n argo-rollouts --create-namespace
helm repo add prometheus-community \
https://prometheus-community.github.io/helm-charts
helm install prom prometheus-community/prometheus \
-n monitoring --create-namespace \
--set alertmanager.enabled=false \
--set prometheus-pushgateway.enabled=false
# server URL: http://prom-prometheus-server.monitoring.svc
# apps/rollout/service.yaml
apiVersion: v1
kind: Service
metadata: {name: web}
spec:
selector: {app: web}
ports: [{port: 9898}]
# load generator: 20 requests/s through the Service
kubectl create ns demo
kubectl -n demo run load --image=curlimages/curl:8.10.1 \
--command -- sh -c \
'while true; do curl -s -o /dev/null http://web:9898/; \
sleep 0.05; done'# apps/rollout/rollout.yaml
apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata: {name: web}
spec:
replicas: 5
selector: {matchLabels: {app: web}}
template:
metadata:
labels: {app: web}
annotations:
prometheus.io/scrape: "true"
prometheus.io/port: "9898"
spec:
containers:
- name: web
image: ghcr.io/stefanprodan/podinfo:6.7.0
ports: [{containerPort: 9898}]
strategy:
canary:
analysis:
templates: [{templateName: success-rate}]
startingStep: 1 # start after the first setWeight
steps:
- setWeight: 20
- pause: {duration: 2m}
- setWeight: 60
- pause: {duration: 2m}Without a traffic router, weights are approximated by the ratio of canary to stable pods.
# apps/rollout/analysis.yaml
apiVersion: argoproj.io/v1alpha1
kind: AnalysisTemplate
metadata: {name: success-rate}
spec:
metrics:
- name: success-rate
interval: 30s
failureLimit: 1
successCondition: result[0] >= 0.95
provider:
prometheus:
address: http://prom-prometheus-server.monitoring.svc
query: |
sum(rate(http_request_duration_seconds_count{
app="web", status!~"5.."}[1m]))
/
sum(rate(http_request_duration_seconds_count{
app="web"}[1m]))git add apps/rollout && git commit -m rollout && git push
argocd app create rollout-demo --repo $REPO \
--path apps/rollout \
--dest-server https://kubernetes.default.svc \
--dest-namespace demo --sync-policy automated
# a good release: change only the UI message
# env: [{name: PODINFO_UI_MESSAGE, value: "v2"}]
git commit -am "web v2" && git push
kubectl argo rollouts get rollout web -n demo --watch
# 20% -> pause -> 60% -> pause -> 100%, AnalysisRun Successful
# dashboard (optional)
kubectl argo rollouts dashboard # http://localhost:3100Argo CD shows a paused Rollout as Suspended; resume from the UI (Rollout → Resume) or kubectl argo rollouts promote web -n demo.
podinfo can return random errors: that is our "bug".
kubectl argo rollouts list rollouts -n demo → Healthy.kubectl -n demo get analysisrun → Successful, one per release.kubectl -n monitoring port-forward svc/prom-prometheus-server 9090:80) the query returns ≈ 1.0.command: ["./podinfo", "--port=9898", "--random-error=true"] to the container, push → at 20% the success rate drops below 0.95; the AnalysisRun fails; the Rollout aborts and scales the canary to 0. Argo CD shows the app Degraded.git revert the bad commit → the Rollout goes back to stable and turns Healthy.argocd app delete rollout-demo --yes
kubectl delete ns demo
helm uninstall argo-rollouts \
-n argo-rollouts
# keep Prometheus for Lab 12Goal: diagnose broken Applications with the CLI and logs only, then watch them in Prometheus.
git switch -c drill
# 1 bad Helm value (schema violation)
yq -i '.replicaCount = "two"' charts/web/values.yaml
# 2 failing hook
sed -i '' 's/sleep 5/exit 1/' apps/sync-demo/10-migrate.yaml
# 3 an object with a finalizer nobody removes
cat > apps/sync-demo/50-stuck.yaml <<'EOF'
apiVersion: v1
kind: ConfigMap
metadata:
name: stuck
finalizers: [example.com/never-removed]
EOF
cp ~/lab7-hpa.yaml apps/sync-demo/40-hpa.yaml # 4 HPA, no ignore
git add -A && git commit -m drill && git push -u origin drill
# apps that point at the broken branch
for a in web:charts/web sync:apps/sync-demo; do
argocd app create drill-${a%%:*} --repo $REPO \
--revision drill --path ${a#*:} \
--dest-server https://kubernetes.default.svc \
--dest-namespace drill --sync-option CreateNamespace=true
done
# 5 missing credential: a private repo, no Secret for it
gh repo create gitops-private --private
argocd app create drill-private \
--repo https://github.com/<you>/gitops-private --path . \
--dest-server https://kubernetes.default.svc \
--dest-namespace drill# clusters/gitops/metrics-values.yaml
controller:
metrics:
enabled: true
service:
annotations:
prometheus.io/scrape: "true"
prometheus.io/port: "8082"
server:
metrics: {enabled: true}
repoServer:
metrics: {enabled: true}
helm upgrade argocd argo/argo-cd -n argocd --reuse-values \
-f clusters/gitops/metrics-values.yaml
kubectl -n argocd port-forward \
svc/argocd-application-controller-metrics 8082:8082 &
curl -s localhost:8082/metrics | grep '^argocd_app_info' | head
# PromQL (Prometheus from Lab 11)
sum by (sync_status, health_status) (argocd_app_info)
argocd_app_info{sync_status="OutOfSync"}
histogram_quantile(0.95, sum by (le)
(rate(argocd_app_reconcile_bucket[5m])))
# your toolkit
argocd app get drill-web --hard-refresh
argocd app diff drill-sync
kubectl -n argocd logs deploy/argocd-repo-server --tail=50
kubectl -n argocd logs sts/argocd-application-controller --tail=50Evidence first, then the fix.
| Fault | Symptom | Where to look | Fix |
|---|---|---|---|
| 1 Bad value | ComparisonError | argocd app get conditions | Fix the value (schema caught it) |
| 2 Failing hook | Sync Failed at PreSync | app get --show-operation; job logs | Fix command, resync |
| 3 Finalizer | Remove 50-stuck.yaml, prune → stuck Terminating | kubectl get cm stuck -o yaml (finalizers) | kubectl patch the ConfigMap: finalizers: null |
| 4 HPA fight | OutOfSync flapping on spec.replicas | argocd app diff drill-sync | ignoreDifferences + RespectIgnoreDifferences |
| 5 No credential | Repository not accessible | argocd repo list; app conditions | Add a repo Secret with a read-only token |
for a in web sync private; do
argocd app delete drill-$a --yes; done
kubectl delete ns drill
git switch main; git branch -D drill
git push origin --delete drill
gh repo delete <you>/gitops-private --yes
pkill -f port-forwardGoal: the same repo on EKS Auto Mode, secrets from AWS Secrets Manager. Costs money: finish and destroy the same day.
export AWS_REGION=ap-northeast-1
# a small cluster (eksctl here; Chapter 13 shows Terraform)
cat > eks.yaml <<'EOF'
apiVersion: eksctl.io/v1alpha5
kind: ClusterConfig
metadata: {name: gitops-eks, region: ap-northeast-1}
autoModeConfig: {enabled: true}
EOF
eksctl create cluster -f eks.yaml # ~15-20 min
kubectl config current-context
# Argo CD: self-managed (as in Lab 6) ...
helm upgrade --install argocd argo/argo-cd -n argocd \
--create-namespace -f clusters/gitops/argocd-values.yaml
# Managed EKS Capability instead? It does not register this
# cluster for you: register it by ARN (needs an access entry)
# and target that name, not kubernetes.default.svc. This lab
# uses self-managed Argo CD; see the AWS guide for the rest.# a secret in AWS + permission for External Secrets
aws secretsmanager create-secret --name gitops-lab/db \
--secret-string '{"password":"s3cr3t"}'
eksctl create podidentityassociation --cluster gitops-eks \
--namespace external-secrets \
--service-account-name external-secrets \
--permission-policy-arns \
arn:aws:iam::aws:policy/SecretsManagerReadWrite
helm repo add external-secrets https://charts.external-secrets.io
helm install external-secrets external-secrets/external-secrets \
-n external-secrets --create-namespace
# apps/eks/store.yaml
apiVersion: external-secrets.io/v1
kind: ClusterSecretStore
metadata: {name: aws-sm}
spec:
provider:
aws: {service: SecretsManager, region: ap-northeast-1}The lab uses a broad managed policy for speed; in real use, scope it to secretsmanager:GetSecretValue on gitops-lab/*.
Everything below is committed to apps/eks/ and synced by Argo CD, not applied by hand.
# apps/eks/db-secret.yaml
apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata: {name: db, namespace: demo}
spec:
refreshInterval: 1h
secretStoreRef: {kind: ClusterSecretStore, name: aws-sm}
target: {name: db}
data:
- secretKey: password
remoteRef: {key: gitops-lab/db, property: password}
git add apps/eks && git commit -m "eks extras" && git push
argocd app create web-demo --repo $REPO --path charts/web \
--dest-server https://kubernetes.default.svc \
--dest-namespace demo --sync-policy automated \
--sync-option CreateNamespace=true
argocd app create eks-extras --repo $REPO --path apps/eks \
--dest-server https://kubernetes.default.svc \
--dest-namespace demo --sync-policy automated# apps/eks/ingress.yaml (Auto Mode built-in ALB)
apiVersion: eks.amazonaws.com/v1
kind: IngressClassParams
metadata: {name: alb}
spec: {scheme: internet-facing}
---
apiVersion: networking.k8s.io/v1
kind: IngressClass
metadata: {name: alb}
spec:
controller: eks.amazonaws.com/alb
parameters: {apiGroup: eks.amazonaws.com,
kind: IngressClassParams, name: alb}
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata: {name: web, namespace: demo}
spec:
ingressClassName: alb
rules:
- http:
paths:
- path: /
pathType: Prefix
backend: {service: {name: web, port: {number: 9898}}}The same Git repo drove kind and EKS; only envs/ and clusters/ changed.
kubectl -n demo get externalsecret db → SecretSynced, Ready True.kubectl -n demo get secret db -o jsonpath='{.data.password}' | base64 -d → s3cr3t.kubectl -n demo get ingress web → an ALB hostname; curl http://<host>/ returns podinfo JSON (wait a few minutes).refreshInterval (or annotate force-sync).SecretSyncedError with AccessDenied. Recreate it.remoteRef.property: passwd → error naming the missing key; the old Secret keeps its value.# 1) delete the Ingress first: the ALB
# belongs to it
kubectl -n demo delete ingress web
sleep 120
# 2) disable the Argo CD capability
# if you enabled it (console)
# 3) cluster, VPC, NAT gateway
eksctl delete cluster -f eks.yaml \
--wait
aws secretsmanager delete-secret \
--secret-id gitops-lab/db \
--force-delete-without-recovery
# 4) check for leftovers: ALBs,
# EBS volumes, Elastic IPsLabs 0–12 are free. Lab 13 bills by the hour from the moment eksctl create cluster starts.
eksctl create cluster.gitops-lab; it is public.--context in scripts once you have more than one cluster.kind delete clusters --all; docker rm -f registry.gitops-lab and gitops-private when done, or keep gitops-lab as your reference.eksctl delete cluster -f eks.yaml --wait; delete the secret with --force-delete-without-recovery.Short anchors; explain each in your own words.
| Term | Short meaning |
|---|---|
| GitOps | Desired state in Git, pulled and continuously reconciled by an agent |
| Reconciliation | Loop that compares desired and live state and acts on the difference |
| Drift | Live state that no longer matches Git |
| Push vs pull | CI applies to the cluster vs an in-cluster agent fetches from Git |
| Config repo | Repo holding deployable state per environment, separate from app code |
| Promotion | Moving a version from one environment to the next, as a Git change |
| Rendered manifests | Plain YAML produced from Helm/Kustomize, committed for review |
| Term | Short meaning |
|---|---|
| Chart | Helm package: templates + default values + metadata |
| Release | One installed instance of a chart, with numbered revisions |
| Values | Inputs that parameterise a chart; merged by precedence |
| Library chart | Chart with only named templates, used as a dependency |
| Helm hook | Resource run at a lifecycle point (pre-upgrade, test, …) |
| OCI chart | Chart stored as an artifact in a container registry |
| Server-Side Apply | API server merges fields and tracks owners (Helm 4 default for new installs) |
Short anchors; explain each in your own words.
| Term | Short meaning |
|---|---|
| Application | CRD: one source (or several) → one destination, plus sync policy |
| AppProject | Tenancy boundary: allowed repos, destinations, resource kinds, roles |
| ApplicationSet | Template + generators that create many Applications |
| Generator | Source of parameters: list, clusters, Git, matrix, PR, SCM |
| Sync / health status | Matches Git? / Is it working? — two separate questions |
| Sync wave | Integer order of resources within a sync phase |
| Sync hook | Resource run in PreSync, Sync, PostSync, SyncFail or PreDelete |
| Prune / self-heal | Delete what left Git / revert live changes automatically |
| Term | Short meaning |
|---|---|
| Hard refresh | Re-render manifests, bypassing the repo-server cache |
| Source Hydrator | Argo CD feature that commits rendered manifests to a branch |
| Hub-and-spoke | One Argo CD managing many clusters |
| Rollout | Argo Rollouts CRD replacing a Deployment for canary / blue-green |
| Canary | Shift a small share of traffic to the new version first |
| AnalysisTemplate | Metric checks (e.g. PromQL) with success conditions |
| AnalysisRun | One execution of an AnalysisTemplate during a rollout |
| Kargo | Promotion tool: stages, freight, verification between environments |
Versions move fast: when a slide and the docs disagree, the docs for your installed version win.
helm.sh/docs/helm.sh/blog/helm-4-released/helm.sh/docs/changelog/helm.sh/docs/chart_best_practices/kubectl.docs.kubernetes.io/references/kustomize/opengitops.devexternal-secrets.ioargo-cd.readthedocs.ioargo-cd.readthedocs.io/en/stable/operator-manual/upgrading/3.2-3.3/argo-cd.readthedocs.io/en/stable/operator-manual/upgrading/3.4-3.5/github.com/argoproj/argo-cd/releasesargo-rollouts.readthedocs.iodocs.kargo.iodocs.aws.amazon.com/eks/latest/userguide/argocd-comparison.html