Self-study deck · Terraform 1.17 · OpenTofu · AWS provider v6 · Checkov · Trivy

Terraform &
IaC A to Z

Infrastructure you can review, test and change safely: the HCL language, providers and state, refactoring with moved, import and removed blocks, modules, tests, plan risk review, policy as code with Checkov and Trivy, repository layouts per environment, and CI/CD on AWS — with a lab for every chapter.

Use ← → / Space to navigate · I contents · O overview · T theme · F fullscreen

Terraform & IaC A to Z

Contents

Click any line to jump there · press I to return to this page

Chapter 0

Orientation

Why infrastructure as code, what this deck builds, the versions it covers, and how to study it.

Mental model

Infrastructure as code turns changes into reviewed diffs

The console is fast for one change and hopeless for the hundredth. Code makes every change repeatable, reviewable and reversible.

Without IaC (ClickOps)

  • Nobody knows why a security group allows 0.0.0.0/0.
  • Dev and prod drift apart one click at a time.
  • Recreating an environment means archaeology in CloudTrail.
  • Reviews happen after the incident, not before the change.

With Terraform

  • Desired state lives in Git; terraform plan shows the exact diff against reality.
  • A pull request is the change request: code review, policy checks and tests run before apply.
  • Environments are the same modules with different inputs.
  • History, blame and rollback come from Git.

What it does not fix

  • Bad designs are now reproducible bad designs.
  • State is a new critical asset to protect (ch 4, 12).
  • A careless apply can delete production faster than a human can.
  • So: tests, plan review and policy (ch 7–9) are part of IaC, not extras.
Deck map

From a single file to a governed pipeline

Read left to right. Each box is where this deck spends its time; the bottom row is the lab repository you build.

LanguageHCL (ch 2)Providersresources, lifecycle (ch 3)StateS3 + lockfile (ch 4)Refactormoved, import (ch 5)Modulesdesign, versions (ch 6)Testsfmt, TFLint, test (ch 7)Plan risk reviewplan JSON, jq, checklist (ch 8)Policy as codeCheckov, Trivy, OPA (ch 9)Layout per environmentlive/dev, live/prod (ch 10)CI/CD, security, operationsOIDC, secrets, upgrades (ch 11–13)Lab repository tf-lab/bootstrap/ → modules/network · modules/app-bucket · modules/ci-role → live/dev/{network,app} · live/prod/{network,app}policies/ (Checkov, Rego) · scripts/plan-summary.sh · .github/workflows (plan on PR, apply on merge, nightly drift)

Highlighted boxes are the focus topics of this deck: modules, state, moved/import, plan risk review, policy as code and repository layout per environment.

Versions covered

What this deck assumes (checked Nov 2026)

Terraform ships a minor release every few months. When a slide and the official docs disagree, the docs win.

Terraform

  • 1.17 (Oct 2026): Terraform Policy GA, -minimal-refresh.
  • 1.16 (Aug 2026): action triggers with on_failure. 1.15 (Apr 2026): variables and locals in module source/version.
  • 1.14 (Nov 2025): terraform query with list blocks (.tfquery.hcl), action blocks.
  • Milestones: 1.5 import/check, 1.6 test, 1.7 mocks and removed, 1.10 ephemeral values, 1.11 write-only arguments and S3 use_lockfile.

Ecosystem

  • AWS provider v6 (6.0 Jun 2025): per-resource region, resource identity, list resources.
  • S3 backend: use_lockfile = true; dynamodb_table locking is deprecated.
  • OpenTofu 1.12 (May 2026, MPL 2.0): state encryption, dynamic prevent_destroy, enabled.
  • Terraform is BUSL 1.1 since 1.6 (Aug 2023); HashiCorp is part of IBM since Feb 2025.

Tools and services

  • Checkov and Trivy (trivy config; tfsec checks now live in Trivy).
  • HCP Terraform: Stacks GA (Sep 2025); legacy Free plan ended 31 Mar 2026.
  • Supply-chain reminder: trivy-action tags were hijacked on 19 Mar 2026 → pin CI actions by SHA.
  • Examples pin required_version = ">= 1.14" and AWS ~> 6.0.

Labs work on Terraform 1.14 or newer. Features from 1.15–1.17 are marked with their version on the slide that uses them.

Terraform and OpenTofu

Two tools, one language

OpenTofu forked from Terraform 1.5 after the license change. Most of this deck applies to both; differences are called out.

TopicTerraform (HashiCorp/IBM)OpenTofu (Linux Foundation)
LicenseBUSL 1.1 (source-available) since 1.6MPL 2.0 (open source)
CLI and registryterraform, registry.terraform.iotofu, registry.opentofu.org (mirrors most providers)
State formatCompatible across both for the versions this deck uses; migrate one direction at a timeSame; adds optional client-side state encryption (1.7+)
Own featuresterraform query/list blocks, actions, Terraform Policy, Stacks (HCP)Dynamic prevent_destroy, enabled meta-argument, early variable evaluation in source
Managed serviceHCP Terraform / Terraform EnterpriseAny TACOS (Spacelift, env0, Scalr, …) or plain CI

This deck teaches with terraform. Chapter 13 shows a migration walkthrough and what to check before switching either way.

Lab environment

What you need before Lab 0

Labs use local-only providers or free AWS resources. The only standing cost is a few KB of state in S3.

Accounts and rules

  • An AWS sandbox account (never production), AWS CLI v2, profile tf-lab, region ap-northeast-1.
  • A budget alarm before the first apply (Lab 0).
  • Only free or near-free resources: VPC, subnets, IGW, security groups, S3, IAM, SSM Standard parameters, log groups.
  • Never NAT gateways, EC2, RDS or EKS in these labs.
  • A GitHub account for Lab 11 (public repo <you>/tf-lab).

Tools

  • terraform 1.14+ (hashicorp/tap/terraform), optional tofu.
  • tflint, checkov, trivy, conftest.
  • jq, gh, git, optional infracost, terraform-docs.
tf-lab/
├── bootstrap/        # state bucket (Lab 0)
├── modules/
│   ├── network/      # VPC, subnets (Lab 6)
│   ├── app-bucket/   # S3 module (Lab 6)
│   └── ci-role/      # GitHub OIDC (Lab 11)
├── live/
│   ├── dev/
│   │   ├── network/  # 10.10.0.0/16
│   │   └── app/
│   └── prod/
│       ├── network/  # 10.20.0.0/16
│       └── app/
├── policies/         # Lab 9
├── scripts/          # Lab 8
└── .github/workflows # Lab 11
How to use this deck

A study loop that sticks

Read a part, run its lab, then answer the self-check out loud without notes.

Per part

  • Read the concept slides; redraw the diagrams from memory.
  • Type the HCL yourself and predict the plan before running it.
  • Study the "Common mistakes" slide: each one is a real outage or a surprise bill.

Per chapter

  • Do the matching lab in the Hands-on labs part.
  • Do every "Break it" step and explain the error before you fix it.
  • Commit after each lab: the Git history is your study log.

Navigation

  • ← → or Space to move, I for Contents, O for overview.
  • T toggles the theme, F fullscreen.
  • Every Contents line is a link.
  • "(ch N)" points inside this deck; GitHub Actions and OIDC details live in the CI/CD deck.

Order matters: state (ch 4) before refactoring (ch 5); modules (ch 6) before layout (ch 10); plan review and policy (ch 8–9) before CI/CD (ch 11).

Chapter 1

IaC foundations and the Terraform workflow

What "infrastructure as code" buys you, how Terraform turns configuration into API calls, and the commands you will type every day.

Chapter 1 · Part 1.1

Infrastructure as code

Describe infrastructure in versioned text files, and let a tool make reality match them.

Infrastructure as code

Why describe infrastructure as code

Clicking in a console works once. Code works the hundredth time, for the second environment, and for the person who joins next year.

Without IaC (ClickOps)

  • Nobody knows exactly how prod was built; the console history is the documentation.
  • dev and prod drift apart; "works in dev" stops meaning anything.
  • Changes are unreviewed and unreproducible; rebuilding after a disaster is guesswork.
  • Audit questions ("who opened port 22?") need CloudTrail archaeology.

With IaC

  • The repository is the source of truth: every change is a commit with an author and a review.
  • Environments are instances of the same code with different inputs.
  • A plan shows the exact effect before anything changes (ch 8).
  • Tests and policies run on every change (ch 7, ch 9); rebuilding is apply.

What it does not fix

  • Bad architecture is reproduced faithfully.
  • Manual changes still happen: you must detect and reconcile drift (ch 5, ch 11).
  • State becomes a critical asset that must be protected (ch 4, ch 12).
  • Code needs the same discipline as application code: modules, versions, tests.
Infrastructure as code

Declarative vs imperative

Terraform is declarative: you write the desired end state, and the tool computes the steps. A script is imperative: you write the steps.

# imperative: steps, and you handle every case
aws s3api create-bucket --bucket tflab-demo-123 \
  --create-bucket-configuration LocationConstraint=ap-northeast-1
aws s3api put-bucket-versioning --bucket tflab-demo-123 \
  --versioning-configuration Status=Enabled
# run it twice → BucketAlreadyOwnedByYou error
# rename the bucket → you must write the delete yourself
# declarative: the end state
resource "aws_s3_bucket" "demo" {
  bucket = "tflab-demo-123"
}

resource "aws_s3_bucket_versioning" "demo" {
  bucket = aws_s3_bucket.demo.id
  versioning_configuration {
    status = "Enabled"
  }
}
# apply twice → second run: "No changes."
# rename → plan shows destroy + create

Idempotence: applying the same configuration again changes nothing. It is what makes apply safe to rerun after a failure, and what makes drift visible: any difference in a new plan means reality moved.

Infrastructure as code

Drift, mutable and immutable infrastructure

Drift is any difference between the code, the state file and the real resources. How you change things decides how much drift you get.

Where drift comes from

  • Console or CLI changes ("just a quick fix").
  • Another tool or automation touching the same resource.
  • AWS defaults or managed updates changing attributes.
  • Partial applies that failed halfway (ch 8).

Mutable vs immutable

  • Mutable: update resources in place (resize, edit a rule). Fast, but history accumulates.
  • Immutable: replace instead of patching (new launch template, new AMI, new instance). Predictable, needs rollouts.
  • Terraform does both: in-place ~ update when the API allows, -/+ replace when an argument cannot change (ch 3).

Handling drift

  • Detect: terraform plan -refresh-only or a scheduled plan with -detailed-exitcode (ch 11).
  • Decide: code the change (it was right) or apply to revert it (it was wrong) (ch 5).
  • Prevent: read-only console roles, SCPs, tags such as ManagedBy=terraform (ch 13).
Tool landscape

IaC tools and when each fits

All of them describe desired state; they differ in language, state handling and scope.

ToolLanguageStateScopeFits when
TerraformHCL (+ JSON)State file in a backend you chooseAny provider: AWS, GCP, Azure, GitHub, Datadog, …Multi-cloud or multi-SaaS estates; huge provider and module ecosystem
OpenTofuHCL (Terraform-compatible)Same model; optional state encryptionSame providers (OpenTofu registry)You want an MPL-licensed, Linux Foundation fork (ch 13)
AWS CloudFormationYAML / JSONManaged by AWS (stacks)AWS onlyAWS-only shops that want no state to manage; StackSets for org-wide baselines
AWS CDKTypeScript, Python, Go, …Synthesizes CloudFormationAWS onlyTeams that prefer general-purpose languages and loops over HCL
PulumiTypeScript, Python, Go, …State in Pulumi Cloud or a bucketMulti-cloudInfrastructure written like application code
CrossplaneKubernetes YAML (CRDs)Kubernetes API (etcd)Multi-cloudPlatform teams exposing infra through Kubernetes and GitOps

This deck uses Terraform 1.17 with notes where OpenTofu differs. Everything in labs 0–13 also runs on OpenTofu unless a slide says otherwise.

Infrastructure as code

What belongs in Terraform, and what does not

Terraform is best at long-lived resources with an API. It is a poor fit for things that change every minute or need procedural steps.

Good fit

  • Networks: VPCs, subnets, routing, DNS zones.
  • Identity: IAM roles, policies, OIDC providers.
  • Data stores and their settings: S3, RDS, DynamoDB, KMS keys.
  • Platform resources: EKS clusters, node groups, ECR repositories.
  • SaaS configuration: GitHub repos and teams, monitors, alert routes.

Poor fit

  • Application deploys every few minutes (use CD or GitOps: GitOps deck).
  • Kubernetes workloads (Argo CD or Flux reconcile continuously).
  • Data migrations and one-off procedures (scripts, runbooks).
  • Secrets' values (they end up in state: ch 12).
  • Resources another system owns (autoscaling instances, Lambda versions created by deploys).

Rules of thumb

  • If it lives for weeks and you would want to review a change to it: Terraform.
  • If a controller already reconciles it: let the controller own it.
  • Never let two tools manage the same attribute; use ignore_changes at the boundary (ch 3).
Chapter 1 · Part 1.2

How Terraform works

Core, providers, the graph and the state: the four ideas behind every plan.

Architecture

Terraform core and providers

The core is a single binary that knows HCL, the graph and state. Everything about AWS lives in the provider, a separate plugin process.

Your machine or CI runner*.tf filesconfigurationTerraform coreparse · graph · diffStateterraform.tfstate / S3AWS providerplugin processgRPCAWS account (ap-northeast-1)AWS APIsEC2, S3, IAM, SSM …S3 buckettflab-…IAM rolearn:aws:iam::…SSM parameter/tflab/dev/…Core never calls AWS itself: the provider does.

Providers are downloaded by terraform init from a registry (registry.terraform.io by default; OpenTofu uses registry.opentofu.org), recorded with checksums in .terraform.lock.hcl (ch 3) and talk to core over the plugin protocol (gRPC).

Architecture

From configuration to API calls

Every plan and apply runs the same pipeline. Knowing the stages tells you where an error comes from.

1 · Load

  • Parse all *.tf files in the directory (order of files does not matter).
  • Load modules and providers prepared by init.
→

2 · Graph

  • One node per resource, data source, variable, output, provider.
  • Edges from references (aws_s3_bucket.demo.id) and depends_on.
→

3 · Refresh

  • Read current state of each managed resource from the API.
  • Updates the in-memory state; detects drift.
→

4 · Diff

  • Provider compares config with refreshed state.
  • Each resource: create, update, replace, delete, no-op.
→

5 · Apply

  • Walk the graph, independent nodes in parallel.
  • Write new state after each resource completes.

plan stops after step 4 and shows the diff. apply without a plan file runs 1–4 again, asks for approval, then runs step 5. Validation errors come from step 1–2; API errors (AccessDenied, InvalidParameter) only appear in step 5.

State

State: the map between code and reality

Terraform cannot find "your" bucket by name alone. The state file records which real object each resource address manages.

{
  "version": 4,
  "terraform_version": "1.17.0",
  "serial": 12,
  "lineage": "3f2c…",
  "resources": [
    {
      "mode": "managed",
      "type": "aws_ssm_parameter",
      "name": "hello",
      "instances": [
        {
          "attributes": {
            "id": "/tflab/dev/hello",
            "arn": "arn:aws:ssm:…:parameter/tflab/dev/hello",
            "value": "…"
          }
        }
      ]
    }
  ]
}

What state holds

  • Address → ID: aws_ssm_parameter.hello ↔ /tflab/dev/hello.
  • Attributes last seen, used to compute the diff and as values for references.
  • Dependencies, so deletes happen in the right order even after you remove code.
  • serial increases on every write; lineage identifies one state history.
  • Secrets in plain text when a resource has them: protect state like a password database (ch 4, ch 12).
Graph

The dependency graph and parallelism

References create edges. Terraform creates independent resources in parallel (default -parallelism=10) and dependent ones in order.

aws_vpc.mainaws_subnet.aaws_subnet.baws_internet_gatewayaws_route_table.publicroute via the gateway

How to read it

  • Arrows point from a resource to the ones that use its attributes (vpc_id = aws_vpc.main.id); the route table uses both the VPC and the gateway.
  • Both subnets and the gateway wait for the VPC, then run at the same time.
  • Destroy walks the graph in reverse: route table first, VPC last.
  • terraform graph | dot -Tsvg > graph.svg draws it (Graphviz). Since 1.7 the default output shows resources only; -type=plan shows the full plan graph.
  • Lower -parallelism when a provider hits API rate limits (ch 13).
Plan internals

Refresh, unknown values and the plan

Some values exist only after creation. Terraform plans with placeholders and resolves them during apply.

# plan output (excerpt)
  # aws_ssm_parameter.hello will be created
  + resource "aws_ssm_parameter" "hello" {
      + arn            = (known after apply)
      + id             = (known after apply)
      + name           = "/tflab/dev/hello"
      + type           = "String"
      + value          = (sensitive value)
      + version        = (known after apply)
    }

Plan: 1 to add, 0 to change, 0 to destroy.

Things to know

  • (known after apply): computed by the API (IDs, ARNs). Anything that depends on it is unknown too.
  • Unknown values cannot drive count/for_each keys: "Invalid for_each argument" (ch 2).
  • (sensitive value): hidden in output, still stored in state and plan files.
  • Refresh is part of every plan; -refresh=false skips it (faster, but blind to drift).
  • -refresh-only plans only accept reality into state, no config changes (ch 5).
Chapter 1 · Part 1.3

The core workflow

init, plan, apply, destroy — and the small commands that keep you out of trouble.

Workflow

The commands you use every day

Write → init → fmt/validate → plan → review → apply. Most mistakes come from skipping the review step.

CommandWhat it doesUseful flags
terraform initConfigure backend, download providers and modules, write the lock file-upgrade, -backend-config=…, -reconfigure, -migrate-state
terraform fmtRewrite files in canonical style-check (CI), -recursive, -diff
terraform validateCheck syntax, types and references (no API calls, no state)-json
terraform planRefresh and compute the changes-out=tfplan, -var, -var-file, -detailed-exitcode, -refresh-only, -replace=ADDR
terraform applyPlan + approve + change; or apply a saved plantfplan (no prompt), -auto-approve, -parallelism=N
terraform destroyPlan and execute deletion of everything in statealias for apply -destroy; plan -destroy to preview
terraform outputPrint root module outputs from state-json, -raw NAME
terraform showHuman or JSON view of state or a plan file-json tfplan (ch 8)
terraform consoleEvaluate expressions interactively against config and statepipe input: echo "1+1" | terraform console
init

What terraform init does

Run it first in every new working directory, and again after you change providers, modules or backend settings.

$ terraform init
Initializing the backend...
Initializing provider plugins...
- Finding hashicorp/aws versions matching "~> 6.0"...
- Finding latest version of hashicorp/random...
- Installing hashicorp/aws v6.x.y...
- Installed hashicorp/aws v6.x.y (signed by HashiCorp)
Terraform has created a lock file .terraform.lock.hcl to
record the provider selections it made above.
Terraform has been successfully initialized!

$ tree -a -L 2 .
.
├── .terraform/
│   └── providers/        # downloaded plugins
├── .terraform.lock.hcl   # COMMIT this file
├── main.tf
└── versions.tf

Details

  • Backend: connects to the state storage (local by default; S3 in ch 4). Stores backend settings in .terraform/terraform.tfstate (not your real state).
  • Providers: picks the newest version allowed by constraints and the lock file. -upgrade allows newer versions and rewrites the lock file.
  • Modules: downloads into .terraform/modules/ (ch 6).
  • Safe to rerun at any time; it never changes infrastructure.
  • .gitignore: .terraform/, *.tfstate*, *.tfplan, crash.log. Never ignore .terraform.lock.hcl.
plan and apply

Saved plans: review exactly what will run

A saved plan file pins the reviewed changes. Applying it does not re-plan, and fails if the state changed since.

terraform plan -out=tfplan        # write the plan
terraform show tfplan             # human review
terraform show -json tfplan | jq '.resource_changes | length'
terraform apply tfplan            # no prompt, no re-plan

# someone applied in between → state serial changed:
# Error: Saved plan is stale
# The given plan file can no longer be applied because
# the state was changed by another operation after the
# plan was created.

terraform plan -destroy -out=destroy.tfplan
terraform apply destroy.tfplan

Rules

  • In CI: plan on the PR, apply the same plan file after approval (ch 11).
  • A plan file is a binary archive with config, variables and planned values: it can contain secrets. Treat it like state.
  • A plan file is valid only with the same Terraform version, providers and working directory.
  • terraform apply without a file re-plans and asks yes: fine on a laptop, risky in automation (what you approve may not be what you reviewed).
  • -auto-approve belongs only in pipelines that apply a saved, reviewed plan.
Exit codes

-detailed-exitcode: plans that scripts can read

By default plan exits 0 whether or not there are changes. With -detailed-exitcode the exit code tells you what the plan found.

Exit codeMeaningTypical use
0Succeeded, no changesDrift check passes; nothing to apply
1ErrorFail the job
2Succeeded, changes presentDrift job opens an issue; PR shows "changes"
set +e
terraform plan -detailed-exitcode -out=tfplan -input=false
rc=$?
set -e
case $rc in
  0) echo "no changes" ;;
  2) echo "changes: review tfplan" ;;
  *) echo "plan failed"; exit 1 ;;
esac

Use -input=false in automation: if a variable has no value, Terraform fails instead of waiting for keyboard input forever.

Working directory

Files Terraform reads and writes

Know which files are code, which are generated, and which are secrets.

PathCreated byCommit to Git?Notes
*.tf, *.tfvars (non-secret)YouYesAll .tf files in one directory form one module
.terraform.lock.hclinitYesProvider versions and hashes; makes CI use the same providers
.terraform/initNoPlugins, modules, backend settings; recreated by init
terraform.tfstate, .backupapply (local backend)NeverContains secrets; use a remote backend (ch 4)
tfplan (any name)plan -outNoMay contain secrets; short-lived CI artifact
*.auto.tfvarsYouOnly if not secretLoaded automatically
~/.terraformrcYouNo (per user)CLI config: plugin_cache_dir, credentials, mirrors

Speed up init across many directories with a shared plugin cache: export TF_PLUGIN_CACHE_DIR="$HOME/.terraform.d/plugin-cache" (the directory must exist).

Everyday commands

fmt, validate, console and output

Small commands, used constantly. They are also the first CI gate (ch 7).

$ terraform fmt -check -recursive
main.tf                       # files that need formatting; exit 3
$ terraform fmt -recursive    # fix them

$ terraform validate
Success! The configuration is valid.

$ terraform validate
Error: Reference to undeclared resource
  on main.tf line 12, in resource "aws_ssm_parameter" "hello":
  12:   value = random_pet.nmae.id
A managed resource "random_pet" "nmae" has not been declared
in the root module.
$ terraform console
> cidrsubnet("10.10.0.0/16", 8, 1)
"10.10.1.0/24"
> random_pet.name.id          # reads current state
"select-mollusk"
> exit

$ terraform output
param_name = "/tflab/dev/hello"
$ terraform output -raw param_name
/tflab/dev/hello
$ terraform output -json | jq -r .param_name.value
/tflab/dev/hello

validate needs init first (it must know provider schemas), but no credentials and no state. That makes it cheap enough for every commit.

Common mistakes

Workflow mistakes that cost an afternoon

Each of these is a real incident pattern.

Files

  • Committing terraform.tfstate (with secrets) to Git.
  • Ignoring .terraform.lock.hcl: CI silently uses a newer provider than your laptop.
  • Two directories sharing one local state by copy-paste.
  • Keeping plan files as long-lived artifacts.

Commands

  • apply -auto-approve without a reviewed plan.
  • Reviewing a plan, then running a plain apply that re-plans against changed state.
  • Treating plan exit 0 as "no changes" in scripts (use -detailed-exitcode).
  • Skipping refresh (-refresh=false) by habit and missing drift.
  • Running destroy in the wrong directory or with the wrong profile.

Habits that prevent them

  • .gitignore from day one; remote state with locking (ch 4).
  • Print the account first: aws sts get-caller-identity.
  • Always plan -out → show → apply tfplan.
  • fmt -check + validate before every commit (pre-commit hooks, ch 7).
Self-check

Foundations: can you answer these?

Answer out loud, with the command or file name.

IaC

  • Explain idempotence and why it makes drift visible.
  • Name three sources of drift and how you detect each.
  • When would you choose CloudFormation or Crossplane over Terraform?
  • What should not be managed by Terraform?

How it works

  • What runs in the provider process, and what in core?
  • Which three things does state record besides IDs?
  • Why can for_each not use a value that is (known after apply)?
  • In what order does destroy delete a VPC and its subnets, and why?

Workflow

  • What does init -upgrade change?
  • What are the exit codes of plan -detailed-exitcode?
  • Why apply a saved plan in CI, and when does it fail as stale?
  • Which files go in .gitignore, and which must be committed?

Practice: Lab 1 in Hands-on labs — a first root module with random, local and one SSM parameter: init, saved plans, exit codes, the graph, drift and destroy.

Chapter 2

The HCL language

Blocks and types, variables and outputs, expressions and functions, count and for_each, and the values Terraform never stores.

Chapter 2 · Part 2.1

Syntax and types

Blocks, arguments and the type system that catches mistakes before any API call.

Syntax

Blocks, arguments and expressions

HCL has two constructs: blocks (type, labels, body) and arguments (name = expression). Everything else is expressions.

# block type   label 1 (type)       label 2 (local name)
resource      "aws_ssm_parameter"  "hello" {
  name  = "/tflab/dev/hello"        # argument = expression
  type  = "String"
  value = random_pet.name.id        # reference to another resource

  tags = {                          # map expression
    Owner = "platform"
  }

  lifecycle {                       # nested block (no "=")
    ignore_changes = [tags]
  }
}

/* multi-line
   comment */  // also a comment

Rules worth knowing

  • A directory is a module: all *.tf files in it are merged; file names and order do not matter.
  • Labels identify the block: address aws_ssm_parameter.hello. Local names are unique per type and module.
  • Arguments use =; nested blocks do not. Some providers accept both forms for the same setting — follow the provider docs.
  • Comments: # (preferred), //, /* */. terraform fmt aligns = signs.
  • Top-level blocks: terraform, provider, resource, data, variable, locals, output, module, moved, import, removed, check, ephemeral, action.
Types

Primitive and collection types

Every value has a type. Type constraints on variables turn bad input into a clear error at plan time.

TypeExampleNotes
string"tflab"UTF-8; templates "${x}"
number3, 0.5One numeric type for ints and floats
booltrue
list(T)["a", "b"]Ordered, index from 0
set(T)toset(["b", "a"])Unordered, unique; no index
map(T){ a = 1, b = 2 }String keys, values of one type
object({…}){ name = "x", size = 2 }Named attributes with their own types
tuple([…])["x", 2, true]Fixed length, per-position types
any—Placeholder: Terraform infers a type
nullnullAbsent value: argument uses its default

Conversion

  • Automatic when safe: "5" → 5 where a number is expected, true → "true".
  • A tuple literal ["a", "b"] converts to list(string) or set(string) by context.
  • tostring, tonumber, tolist, toset, tomap convert explicitly; convert(value, type) (1.15) converts to any type constraint.
  • type(x) in terraform console shows the inferred type.
  • Sets have no order: [for x in set : …] iterates in sorted order for strings, but never index a set.
Types

Object types with optional() attributes

One typed object input is easier to use and document than ten loose variables. optional(type, default) fills in what callers omit.

variable "bucket" {
  type = object({
    name_suffix = string
    versioning  = optional(bool, true)    # default true
    expire_days = optional(number)        # default null
    lifecycle_rules = optional(list(object({
      id     = string
      prefix = string
      days   = number
    })), [])
  })
}

# terraform.tfvars
bucket = {
  name_suffix = "logs-123"
  lifecycle_rules = [
    { id = "tmp", prefix = "tmp/", days = 7 }
  ]
}
$ echo 'var.bucket' | terraform console
{
  "expire_days" = tonumber(null)
  "lifecycle_rules" = tolist([
    {
      "days" = 7
      "id" = "tmp"
      "prefix" = "tmp/"
    },
  ])
  "name_suffix" = "logs-123"
  "versioning" = true
}

# missing required attribute:
#   Error: Invalid value for input variable
#   attribute "name_suffix" is required.
# unknown attribute (typo "versionning"):
#   rejected as well — objects are strict

optional() works in variable type constraints (Terraform 1.3+). Use it for module inputs (ch 6): callers pass only what differs from your defaults.

Syntax

Strings, heredocs and templates

Interpolation ${…} inserts values; directives %{…} add conditions and loops inside strings.

locals {
  name  = "tflab-${var.env}"
  tier  = "%{if var.env == "prod"}critical%{else}normal%{endif}"

  user_data = <<-EOT
    #!/bin/bash
    echo "env=${var.env}" > /etc/tflab
  EOT
  # <<-EOT strips the common leading indentation

  literal = "$${not_interpolated}"   # → ${not_interpolated}
}
# templates/params.tftpl
env: ${env}
%{ for k, v in params ~}
- ${k} = ${v}
%{ endfor ~}

# usage
templatefile("${path.module}/templates/params.tftpl",
  { env = var.env, params = var.params })

# result
env: dev
- greeting = hello
- owner = platform

~ trims whitespace and the newline next to a directive. Build JSON and YAML with jsonencode() / yamlencode(), never with string templates: quoting and escaping are then always correct.

Chapter 2 · Part 2.2

Variables, locals and outputs

The interface of a module: what goes in, what is computed once, and what comes out.

Variables

Declaring input variables

Variables are the parameters of a module. Declare type and description always; add validation for anything a caller can get wrong.

variable "env" {
  type        = string
  description = "Environment name (dev or prod)."

  validation {
    condition     = contains(["dev", "prod"], var.env)
    error_message = "env must be dev or prod."
  }
}

variable "vpc_cidr" {
  type    = string
  default = "10.10.0.0/16"
  validation {
    condition     = can(cidrhost(var.vpc_cidr, 0))
    error_message = "vpc_cidr must be a valid IPv4 CIDR."
  }
}

variable "db_password" {
  type      = string
  sensitive = true      # hidden in CLI output
  nullable  = false     # null is rejected
}

Arguments of a variable block

  • type: constraint (section 04). Without it, -var values arrive as strings.
  • default: makes the variable optional. No default = required.
  • description: shows up in docs and terraform-docs output.
  • validation: one or more condition + error_message; since 1.9 conditions may refer to other variables.
  • sensitive: redacts the value in plan/apply output (still in state).
  • nullable = false: passing null is an error instead of "use default".
  • ephemeral: value is never written to state or plan (section 08).
Variables

Where values come from, and which wins

Later sources override earlier ones. Knowing the order explains "why is it still using the old value?"

Order (low → high)Source
1default in the variable block
2Environment: TF_VAR_env=dev
3terraform.tfvars
4terraform.tfvars.json
5*.auto.tfvars / *.auto.tfvars.json, in lexical order of file names
6-var and -var-file on the command line, in the order given
$ terraform plan -var-file=dev.tfvars -var 'env=qa'
Error: Invalid value for variable

  on variables.tf line 1:
   1: variable "env" {
    ├────────────────
    │ var.env is "qa"

env must be dev or prod.

This was checked by the validation rule at
variables.tf:5,3-13.

# required variable and -input=false:
#   Error: No value for required variable

A practical pattern: no terraform.tfvars in shared code; one explicit -var-file=env/dev.tfvars per environment (ch 10). Then every value in a plan has an obvious origin.

Locals

Locals: name a value once

Locals are computed values inside a module. They keep naming and tagging consistent and remove repetition.

locals {
  name = "tflab-${var.env}"

  tags = {
    Project   = "tf-lab"
    Env       = var.env
    ManagedBy = "terraform"
  }

  # derived maps for for_each
  subnets = {
    for i, az in var.azs : az => cidrsubnet(var.vpc_cidr, 8, i)
  }
}

provider "aws" {
  region = "ap-northeast-1"
  default_tags {
    tags = local.tags      # applied to every taggable resource
  }
}

resource "aws_s3_bucket" "logs" {
  bucket = "${local.name}-logs-${var.account_id}"
}

Guidelines

  • Reference with local.<name> (singular), declare in locals {} (plural).
  • Use locals for derived values, not for things a caller should set (those are variables).
  • Keep them close to where they are used, or in one locals.tf if shared.
  • Provider default_tags (AWS) beats repeating tags = local.tags on every resource; per-resource tags merge on top.
  • A local that hides a complex expression deserves a comment explaining why.
Outputs

Outputs: the return values of a module

Root outputs are printed after apply and readable with terraform output; child module outputs are how modules pass data to each other.

output "param_names" {
  description = "SSM parameter names by key."
  value       = { for k, p in aws_ssm_parameter.p : k => p.name }
}

output "bucket_arn" {
  value = aws_s3_bucket.logs.arn
}

output "db_password" {
  value     = random_password.db.result
  sensitive = true      # required when the value is sensitive
}

# Apply complete! Resources: 3 added, 0 changed, 0 destroyed.
# Outputs:
# bucket_arn = "arn:aws:s3:::tflab-dev-logs-123456789012"
# db_password = <sensitive>
# param_names = {
#   "greeting" = "/tflab/dev/greeting"
# }

Good outputs

  • Output IDs and ARNs that other components need (ch 4, ch 10).
  • Shape them for consumers: a map keyed by name beats a list by index.
  • An output referencing a sensitive value must set sensitive = true, or plan fails.
  • terraform output -json is the stable way for scripts to read them.
  • Outputs are stored in state, so a "sensitive" output is still readable by anyone who can read state.
  • precondition blocks inside an output check guarantees (ch 3).
Chapter 2 · Part 2.3

Expressions and functions

References, conditionals, for expressions, splats and the built-in function library.

Expressions

References and named values

Everything a configuration can refer to has a well-defined prefix.

ReferenceMeaningExample
<type>.<name>.<attr>Managed resource attributeaws_vpc.main.id
data.<type>.<name>.<attr>Data source attributedata.aws_caller_identity.me.account_id
var.<name>Input variablevar.env
local.<name>Local valuelocal.tags
module.<name>.<output>Child module outputmodule.network.vpc_id
each.key, each.valueCurrent element in for_eacheach.value.cidr
count.indexCurrent index in countcount.index
path.module, path.root, path.cwdFilesystem paths"${path.module}/templates/x.tftpl"
terraform.workspaceCurrent CLI workspace name"default" (ch 4)
selfThe resource itself (provisioners, postconditions)self.arn
ephemeral.<type>.<name>Ephemeral resource attributeephemeral.aws_secretsmanager_secret_version.db.secret_string
Expressions

Conditionals, for expressions and splats

These three cover most "logic" you need in HCL. Try each in terraform console.

# conditional: both results must have compatible types
instance_type = var.env == "prod" ? "m7g.large" : "t4g.small"

# for over a list → list
[for s in ["a", "b"] : upper(s)]
# ["A", "B"]

# for over a map → map, with a filter
{ for k, v in { a = 1, b = 2 } : k => v * 10 if v > 1 }
# { "b" = 20 }

# group values with "..." when keys repeat
{ for u in var.users : u.team => u.name... }
# { "core" = ["ann", "bo"], "web" = ["cy"] }

# splat: one attribute from every element
[{ n = "a" }, { n = "b" }][*].n
# ["a", "b"]
aws_subnet.private[*].id         # with count
values(aws_subnet.private)[*].id # with for_each (a map)

Pitfalls

  • cond ? "a" : 1 fails: results need one type (string vs number).
  • Splat [*] does not work on a for_each resource (a map): use values(...)[*] or a for expression.
  • A for that builds a map fails on duplicate keys unless you use the grouping ... form.
  • Use if inside for to filter, not a conditional that yields null elements.
  • Anything referencing (known after apply) is unknown too — fine for arguments, not for count/for_each.
Functions

The functions you will use weekly

There are no user-defined functions in HCL; the built-in library (plus provider-defined functions since 1.8) covers it. Results below are real terraform console output.

FunctionExampleResult
mergemerge({a=1}, {a=2, b=3}){ a = 2, b = 3 } (later wins)
lookuplookup({a=1}, "b", 9)9 (default for missing key)
trytry(tonumber("x"), 0)0 (first expression without error)
cancan(cidrhost("10.0.0.0/33", 1))false (great in validation)
coalescecoalesce("", "b")"b" (first non-null, non-empty)
oneone([])null (0 or 1 element → value; more → error)
cidrsubnetcidrsubnet("10.10.0.0/16", 8, 1)"10.10.1.0/24"
cidrsubnetscidrsubnets("10.10.0.0/16", 8, 8, 4)["10.10.0.0/24", "10.10.1.0/24", "10.10.16.0/20"]
formatformat("%s-%03d", "tflab", 7)"tflab-007"
jsonencodejsonencode({a=[1,2]})"{\"a\":[1,2]}"
elementelement(["a","b"], 3)"b" (wraps around — usually a bug)
Console

Learn functions in terraform console

The console evaluates expressions against your configuration and current state. It is the fastest way to debug a for expression.

$ terraform console -var-file=dev.tfvars
> keys(var.params)
tolist([
  "greeting",
  "owner",
])
> { for k, v in var.params : upper(k) => v }
{
  "GREETING" = "hello"
  "OWNER" = "platform"
}
> var.params["missing"]
Error: Invalid index
The given key does not identify an element in this
collection value.
> try(var.params["missing"], "n/a")
"n/a"
> type(var.bucket.lifecycle_rules)
list(
    object({
        days: number,
        id: string,
        prefix: string,
    }),
)

Tips

  • Pipe for scripts and docs: echo 'cidrsubnet("10.0.0.0/16", 4, 2)' | terraform console.
  • It reads state: aws_vpc.main.id shows the real ID after apply.
  • It never changes anything, but it does hold the state lock while it runs (remote backends).
  • Pass the same -var-file you plan with, or required variables will be prompted.
  • Provider-defined functions use provider::<name>::<fn>(…), e.g. provider::aws::arn_parse(arn) with AWS provider 5.40+.
Chapter 2 · Part 2.4

count, for_each and dynamic blocks

Create many instances from one block, and keep their addresses stable.

Multiple instances

count and for_each

Both meta-arguments create several instances of one resource or module. They differ in how instances are addressed.

# count: instances are numbered
resource "aws_ssm_parameter" "by_index" {
  count = length(var.names)
  name  = "/tflab/dev/${var.names[count.index]}"
  type  = "String"
  value = "x"
}
# aws_ssm_parameter.by_index[0], [1], [2]

# for_each: instances are keyed
resource "aws_ssm_parameter" "by_key" {
  for_each = var.params          # map or set of strings
  name     = "/tflab/dev/${each.key}"
  type     = "String"
  value    = each.value
}
# aws_ssm_parameter.by_key["greeting"], ["owner"]

# a list of objects → map keyed by a unique attribute
for_each = { for s in var.subnets : s.name => s }

Choosing

  • **for_each** for anything with an identity (names, AZs, users): adding or removing one item touches only that item.
  • **count** for "N identical copies" or an on/off switch: count = var.enabled ? 1 : 0.
  • for_each accepts a map or a set of strings; convert lists with toset() or a for expression.
  • Keys must be known at plan time and not sensitive.
  • Reference: aws_x.y[0].id (count) vs aws_x.y["key"].id (for_each); one switched-on instance: one(aws_x.y[*].id).
  • Changing count → for_each later moves addresses: use moved blocks (ch 5).
Multiple instances

Why count over a list hurts: a real plan

Remove the first element of ["a", "b", "c"]. With count every later index shifts; with for_each only the removed key goes.

variable "names" {
  type    = list(string)
  default = ["a", "b", "c"]
}
resource "random_pet" "c" {           # count
  count  = length(var.names)
  prefix = var.names[count.index]
}
resource "random_pet" "f" {           # for_each
  for_each = toset(var.names)
  prefix   = each.key
}

$ terraform plan -var 'names=["b","c"]'
  # random_pet.c[0] must be replaced
      ~ prefix    = "a" -> "b" # forces replacement
  # random_pet.c[1] must be replaced
      ~ prefix    = "b" -> "c" # forces replacement
  # random_pet.c[2] will be destroyed
  # (because index [2] is out of range for count)
      - prefix    = "c" -> null
  # random_pet.f["a"] will be destroyed
  # (because key ["a"] is not in for_each map)
      - prefix    = "a" -> null

Plan: 2 to add, 0 to change, 4 to destroy.

With real resources (subnets, users, buckets) the count version replaces two healthy objects. Also note the type = list(string): without it, -var 'names=[…]' arrives as a plain string and both blocks fail.

Multiple instances

Unknown keys: "Invalid for_each argument"

Terraform must know every instance key during plan. Keys derived from values computed at apply time cannot work.

resource "random_pet" "a" {}

resource "random_pet" "b" {
  for_each = toset([random_pet.a.id])     # unknown at plan
}

$ terraform plan
Error: Invalid for_each argument

  on main.tf line 9, in resource "random_pet" "b":
   9:   for_each = toset([random_pet.a.id])
    ├────────────────
    │ random_pet.a.id is a string, known only after apply

The "for_each" set includes values derived from resource
attributes that cannot be determined until apply, ...

Fixes

  • Key by something static (names from variables) and put the computed value in the value: for_each = { for n in var.names : n => aws_x.y[n].id } is fine because the keys are known.
  • Split the apply into two components (ch 10) if the keys truly come from created resources.
  • -target to create the dependency first works as a one-off, not as a design (ch 8).
  • The same rule holds for count: the number must be known at plan time.
  • Ephemeral values cannot drive count/for_each either (section 08).
Dynamic blocks

dynamic blocks: repeat a nested block

Some resources take repeated nested blocks (lifecycle rules, ingress rules). dynamic generates them from a collection.

resource "aws_s3_bucket_lifecycle_configuration" "b" {
  count  = length(var.bucket.lifecycle_rules) > 0 ? 1 : 0
  bucket = aws_s3_bucket.b.id

  dynamic "rule" {                       # block type to generate
    for_each = var.bucket.lifecycle_rules
    content {
      id     = rule.value.id               # iterator = block name
      status = "Enabled"
      filter {
        prefix = rule.value.prefix
      }
      expiration {
        days = rule.value.days
      }
    }
  }
}

Use with restraint

  • The iterator is named after the block (rule.key, rule.value); rename with iterator = r.
  • Good for module inputs that map 1:1 to repeated blocks.
  • Bad when it hides the shape of a resource: a reader can no longer see what will be created.
  • Prefer separate resources when the provider offers them: aws_vpc_security_group_ingress_rule instead of ingress {} blocks inside aws_security_group.
  • An empty collection generates zero blocks; some resources then reject the request — hence the count guard above.
Chapter 2 · Part 2.5

Ephemeral and write-only values

Keep secrets out of state and plan files, not just out of the terminal.

Ephemeral

The problem: sensitive only hides output

Before Terraform 1.10, any secret that touched a resource argument was written to state in plain text. sensitive = true only redacts the CLI display.

sensitive

  • Redacts values in plan and apply output: (sensitive value).
  • Propagates through expressions: a value derived from a sensitive one is sensitive.
  • Still stored in state and in saved plan files, readable with terraform show -json or by anyone who can read the state bucket.
  • nonsensitive() removes the mark (use only for values you know are not secret, such as a hash).

ephemeral (1.10+)

  • Value exists only during one Terraform run; never written to state or plan files.
  • Available for: input variables, outputs of child modules, locals derived from them, and ephemeral resources.
  • May be used only where nothing is persisted: provider configuration, write-only arguments, provisioner/connection blocks, other ephemeral values.

Write-only arguments (1.11+)

  • Resource arguments named *_wo accept ephemeral values and are not stored.
  • Terraform cannot diff a value it never stored, so a companion *_wo_version number tells it when to send a new value.
  • Examples in AWS provider v6: aws_ssm_parameter.value_wo, aws_db_instance.password_wo, aws_secretsmanager_secret_version.secret_string_wo.
Ephemeral

Ephemeral variables and write-only arguments in practice

Pass a secret from the environment, write it to SSM, and keep it out of state. Validated with Terraform 1.15 and AWS provider 6.x.

variable "db_password" {
  type      = string
  sensitive = true
  ephemeral = true          # never persisted
}

variable "manage_secret" { type = bool }

resource "aws_ssm_parameter" "db_password" {
  count            = var.manage_secret ? 1 : 0
  name             = "/tflab/dev/db-password"
  type             = "SecureString"
  value_wo         = var.db_password
  value_wo_version = 1      # bump to 2 to push a new value
}

# export TF_VAR_db_password="$(openssl rand -base64 24)"
# terraform apply
# state: value_wo = null, value_wo_version = 1
# WRONG: count from an ephemeral value
resource "aws_ssm_parameter" "db_password" {
  count = var.db_password == null ? 0 : 1 ...

$ terraform validate
Error: Invalid count argument

  on main.tf line 14, in resource "aws_ssm_parameter"
  "db_password":
  14:   count = var.db_password == null ? 0 : 1

The given "count" is derived from an ephemeral value,
which means that Terraform cannot persist it between
plan/apply rounds. Use only non-ephemeral values to
specify the number of resource instances.

Ephemeral resources go one step further: ephemeral "aws_secretsmanager_secret_version" "db" { … } reads a secret at run time without storing it. Chapter 12 builds the full secret flow with these pieces.

Ephemeral

What ends up where

Use this table when deciding how a secret flows through a configuration.

MechanismCLI outputPlan fileStateTypical use
Plain valueShownStoredStoredNon-secret settings
sensitive = trueRedactedStoredStoredValues that are not secret enough to avoid state, e.g. generated names
ephemeral = true variableRedacted if also sensitiveNot storedNot storedCredentials passed in at run time
ephemeral resource—Not storedNot storedRead a secret from Secrets Manager / generate a temporary token
Write-only argument *_woRedactedNot storedStored as null; only *_wo_version keptSet a password on a resource without persisting it
Provider-managed secret (manage_master_user_password = true on RDS)——Only the secret ARNLet AWS generate and rotate it (ch 12)

OpenTofu 1.11 added ephemeral values and write-only attributes as well; OpenTofu also offers client-side state encryption (ch 4), which Terraform does not have.

Common mistakes

HCL mistakes that reach production

Most of these pass validate and only show up in a plan — or after apply.

Types and inputs

  • Untyped variables: -var values arrive as strings.
  • Loose map(any) inputs that accept typos silently; use object() with optional().
  • Secrets in terraform.tfvars committed to Git.
  • Relying on sensitive to keep a secret out of state.

Instances

  • count over a list of named things → mass replacement when the list changes.
  • for_each keys from computed values → "Invalid for_each argument".
  • element() wrapping around silently instead of an index error.
  • Deep dynamic blocks no reviewer can read.

Better defaults

  • Type and describe every variable; validate enums and CIDRs.
  • for_each over maps keyed by stable names.
  • jsonencode() for policies, never string templates.
  • Test expressions in terraform console before committing.
  • Ephemeral + write-only for secrets you must pass through Terraform.
Self-check

HCL: can you answer these?

Write the expression or block from memory, then check it in terraform console.

Types and variables

  • What does optional(number) default to, and how do you set another default?
  • List the variable value sources from lowest to highest precedence.
  • Why does an untyped variable break -var 'names=["a"]'?
  • Write a validation that accepts only valid CIDR blocks.

Expressions

  • Build a map az => cidr from a list of AZs with cidrsubnet.
  • Why does [*] fail on a for_each resource, and what do you use instead?
  • What does the ... suffix do in a for expression?
  • When do try and can differ?

Instances and secrets

  • Show what happens to addresses when you remove the first element with count vs for_each.
  • Why can a resource not use for_each = toset([random_pet.a.id])?
  • Where is a sensitive value stored? An ephemeral one?
  • What is *_wo_version for?

Practice: Lab 2 in Hands-on labs — HCL drills in terraform console, then a config with validated variables, optional() objects, for_each, a dynamic block, templatefile and shared tags.

Chapter 3

Providers, resources and the lifecycle

How Terraform talks to AWS, how resources and data sources are addressed and ordered, and how to control create, update and replace.

Chapter 3 · Part 3.1

Providers

Plugins that translate resource blocks into API calls, pinned by version and by hash.

Providers

What a provider is

Terraform core knows nothing about AWS. It builds the graph and the plan; the provider plugin knows the resource schemas and calls the APIs.

Terraform coreConfig + stateHCL, terraform.tfstateGraph, plan, applydiff, ordering, parallelismProvider plugin (process)hashicorp/awsschemas, validationCRUD functionsRead, Plan, ApplyAWS SDK for Go v2credentials, retriesgRPCAWS APIsEC2, S3, IAM, …terraform init downloads providersinto .terraform/providers/

Key facts

  • A provider is a separate binary that core starts and talks to over gRPC (plugin protocol).
  • Source address hashicorp/aws = registry.terraform.io/hashicorp/aws (hostname/namespace/type).
  • The provider defines every resource type (aws_vpc), data source, function and (1.14+) list resource and action.
  • Provider versions change behavior: defaults, new arguments, deprecations. Pin them.
  • Inspect a schema: terraform providers schema -json | jq.
Providers

required_providers and version constraints

Every root and child module declares which providers it needs. Root modules pin tight; reusable modules state the minimum they support.

# versions.tf
terraform {
  required_version = ">= 1.14"

  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 6.0"     # >= 6.0, < 7.0
    }
    random = {
      source  = "hashicorp/random"
      version = "~> 3.6"     # >= 3.6, < 4.0
    }
  }
}

# terraform version
# Terraform v1.17.0
# + provider registry.terraform.io/hashicorp/aws v6.72.0

Constraint operators

ConstraintAllows
= 6.2.0 or 6.2.0Exactly 6.2.0
>= 6.06.0 and anything newer, including 7.x
~> 6.0≥ 6.0 and < 7.0 (rightmost part may grow)
~> 6.2.0≥ 6.2.0 and < 6.3.0 (patches only)
>= 6.10, < 6.30A range; combine with commas
!= 6.15.0Exclude one known-bad release

Reusable modules: >= 6.0 (a floor, no ceiling) so callers can upgrade. Root modules: ~> 6.0 plus the lock file, which pins the exact version.

Providers

The dependency lock file: .terraform.lock.hcl

Constraints say what is allowed; the lock file records what was chosen, with checksums. Commit it.

# .terraform.lock.hcl (written by terraform init)
provider "registry.terraform.io/hashicorp/aws" {
  version     = "6.72.0"
  constraints = "~> 6.0"
  hashes = [
    "h1:mOeaj5DLQjxSyaVbYiilZh4htt9Yo2y0jngHnRucJtw=",
    "zh:2ffb6b2caaf320b61b406cff192d6e4303b64e32...",
    "zh:33cd70e1b3d0f80bfdf07480139142f81a3daec2...",
    # ... one zh: per platform package in the release
  ]
}

How it behaves

  • init installs the locked version even if newer ones match the constraint.
  • init -upgrade picks the newest allowed version and rewrites the file.
  • h1: = hash of the unpacked package for platforms you installed on; zh: = zip hashes from the registry for every platform.
  • A mismatch on download fails init: tampering or a wrong mirror.
  • In CI: terraform init -lockfile=readonly fails instead of silently changing the file.
  • Only providers are locked; modules are not (pin them with version or a Git ref, ch 6).

Laptop on darwin_arm64, CI on linux_amd64? Pre-record both: terraform providers lock -platform=darwin_arm64 -platform=linux_amd64 (Lab 3).

Providers

Configuring the AWS provider

Provider blocks belong in root modules. Credentials come from the environment, never from code.

# providers.tf (root module live/dev/network)
provider "aws" {
  region = var.region            # ap-northeast-1

  default_tags {                 # added to every taggable resource
    tags = {
      Project   = "tf-lab"
      Env       = var.env
      ManagedBy = "terraform"
    }
  }

  ignore_tags {                  # tags someone else manages
    key_prefixes = ["kubernetes.io/"]
  }

  # assume_role {                # cross-account (ch 10)
  #   role_arn     = "arn:aws:iam::<prod-acct>:role/tflab-deploy"
  #   session_name = "tf-lab"
  # }
}

Credential chain (in order)

  • Provider arguments (access_key… — avoid).
  • Environment: AWS_ACCESS_KEY_ID/AWS_SECRET_ACCESS_KEY/AWS_SESSION_TOKEN, AWS_PROFILE.
  • Shared config/credentials files, including IAM Identity Center (SSO) profiles.
  • Container credentials (ECS task role), then EC2 instance profile (IMDS).
  • In CI: OIDC web identity (ch 11) — no stored keys.
  • Check who you are: aws sts get-caller-identity before every apply.

default_tags merge with resource tags; on a key conflict the resource tag wins. Read tags_all in state to see the merged result.

Providers

Several regions or accounts: aliases and the v6 region argument

Before v6, every extra region needed its own aliased provider. AWS provider v6 adds a region argument to most resources.

# classic: an aliased provider configuration
provider "aws" {
  alias  = "use1"
  region = "us-east-1"
}

resource "aws_s3_bucket" "logs_use1" {
  provider = aws.use1          # pick the configuration
  bucket   = "tflab-logs-use1-<acct>"
}

# AWS provider v6: region per resource, same provider
resource "aws_s3_bucket" "logs_v6" {
  region = "us-east-1"         # defaults to the provider region
  bucket = "tflab-logs-v6-<acct>"
}

# modules receive aliased providers explicitly
module "dr" {
  source    = "../../modules/app-bucket"
  providers = { aws = aws.use1 }
}

Which to use

  • Other region, same account: v6 region argument; fewer provider blocks and simpler modules.
  • Other account: still an aliased provider with its own assume_role.
  • Global services (IAM, CloudFront, Route 53) have no region argument.
  • Changing region on an existing resource forces replacement: it is a different resource in a different place.
  • Upgrading v5 → v6: read the upgrade guide; several arguments were removed or renamed (ch 13).
Chapter 3 · Part 3.2

Resources and data sources

Addresses, computed attributes, data sources and the dependency graph.

Resources

Anatomy of a resource and its address

A resource block declares one or more remote objects. Its address is how state, plans, -target, moved and import refer to it.

resource "aws_subnet" "public" {     # TYPE "NAME"
  for_each = { for i, az in local.azs : az => i }   # meta-argument

  vpc_id            = aws_vpc.this.id   # argument (reference)
  availability_zone = each.key
  cidr_block        = cidrsubnet(var.vpc_cidr, 8, each.value)
}

# attributes you can read after creation:
#   aws_subnet.public["ap-northeast-1a"].id      (computed)
#   aws_subnet.public["ap-northeast-1a"].arn     (computed)
#   aws_subnet.public["ap-northeast-1a"].cidr_block

Address forms

AddressMeans
aws_vpc.thisSingle instance
aws_subnet.public["ap-northeast-1a"]One for_each instance (key)
aws_instance.web[0]One count instance (index)
aws_subnet.publicAll instances of the resource
module.network.aws_vpc.thisInside a child module
module.app["dev"].aws_s3_bucket.thisInside a for_each module
data.aws_caller_identity.currentA data source

In a shell, quote addresses with brackets: terraform state show 'aws_subnet.public["ap-northeast-1a"]'.

Resources

Arguments, attributes and "(known after apply)"

Arguments are what you set; attributes are what the provider reports. Some attributes only exist once the API has created the object.

Kinds of values

  • Required / optional arguments: set in configuration (cidr_block).
  • Computed attributes: set by the provider (id, arn, owner_id).
  • Optional + computed: you may set it; if you do not, the provider fills a default (enable_dns_support).
  • Sensitive attributes are redacted in output but still stored in state (ch 12).
  • Write-only arguments (password_wo) are never stored at all (ch 2, ch 12).
$ terraform plan
  # aws_vpc.this will be created
  + resource "aws_vpc" "this" {
      + arn                  = (known after apply)
      + cidr_block           = "10.10.0.0/16"
      + enable_dns_hostnames = true
      + id                   = (known after apply)
      + tags_all             = {
          + "Env"       = "dev"
          + "ManagedBy" = "terraform"
          + "Name"      = "tflab-dev"
          + "Project"   = "tf-lab"
        }
    }

  # aws_subnet.public["ap-northeast-1a"] will be created
  + resource "aws_subnet" "public" {
      + vpc_id = (known after apply)

Unknown values propagate: anything derived from an unknown attribute is also unknown at plan time. That limits count/for_each keys (ch 2) and is a common review blind spot (ch 8).

Data sources

Data sources: read, never manage

A data block looks something up — in AWS, in another state, or computed locally — and exposes attributes like a resource does.

data "aws_availability_zones" "available" {
  state = "available"
}

data "aws_caller_identity" "current" {}

data "aws_ssm_parameter" "vpc_id" {   # from network (ch 4)
  name = "/tflab/dev/network/vpc_id"
}

data "aws_iam_policy_document" "read" {   # local, no API call
  statement {
    actions   = ["s3:GetObject"]
    resources = ["${aws_s3_bucket.app.arn}/*"]
  }
}

locals {
  azs    = slice(data.aws_availability_zones.available.names, 0, 2)
  acct   = data.aws_caller_identity.current.account_id
  vpc_id = data.aws_ssm_parameter.vpc_id.insecure_value
}

When they are read

  • Normally during plan, so the plan shows real values.
  • If an argument depends on an unknown value (or on a resource with pending changes via depends_on), the read is deferred to apply: the plan shows <= and "(known after apply)".
  • A data source that finds nothing usually fails the run (e.g. no matching AMI); some return empty lists.
  • aws_ssm_parameter.value is always marked sensitive; use insecure_value for plain String parameters.
  • Prefer data sources over hard-coded IDs; prefer outputs/SSM over terraform_remote_state across teams (ch 4).
Dependencies

The dependency graph: references create the order

Terraform walks a graph. An expression that references another object adds an edge; independent nodes run in parallel (10 by default).

aws_vpc.thisaws_subnet.public[*]aws_internet_gateway.thisaws_security_group.webaws_route_table.publicaws_route_table_associationpublic[*]aws_vpc_sg_ingress_ruleCreate: left → right. Destroy: right → left (reverse order).terraform graph | dot -Tsvg > graph.svg (Graphviz) to see the real one.

depends_on: the exception

  • Only for hidden dependencies Terraform cannot see from references.
  • Classic case: an app that needs an IAM policy attachment to exist, but only references the role.
  • Costs: data sources that depend on it are read at apply → more "(known after apply)" in plans.
  • Prefer a real reference (e.g. pass aws_iam_role_policy_attachment.x.policy_arn).
  • On a module, depends_on makes everything in it wait: use sparingly.
Dependencies

Implicit vs explicit dependencies in code

The same ordering problem, solved badly and well.

# Hidden dependency: the Lambda needs the policy to be
# ATTACHED before it runs, but only references the role.
resource "aws_iam_role_policy_attachment" "logs" {
  role       = aws_iam_role.fn.name
  policy_arn = local.basic_execution_policy_arn
}

resource "aws_lambda_function" "fn" {
  function_name = "tflab-fn"
  role          = aws_iam_role.fn.arn   # implicit: role only
  # ...
  depends_on = [aws_iam_role_policy_attachment.logs]
}
# Unnecessary depends_on: the reference already
# creates the edge, depends_on only adds noise.
resource "aws_subnet" "a" {
  vpc_id     = aws_vpc.this.id
  depends_on = [aws_vpc.this]     # remove this
}

# Wrong tool: ordering between root modules /
# separate states. depends_on cannot cross states;
# order the applies in CI instead (ch 10, ch 11).

Rule of thumb: if you add depends_on, write a comment saying which hidden dependency it encodes. If you cannot name one, delete it.

Chapter 3 · Part 3.3

The resource lifecycle

Create, update in place, replace — and the meta-arguments that change the default behavior.

Lifecycle

Create, update in place, or replace

For each resource the provider decides whether a change can be applied in place or needs a new object.

Plan actions

SymbolActionTypical cause
+createNew block, new for_each key
~update in placeTags, description of some resources, most settings
-/+destroy, then createAn argument that "forces replacement" (ForceNew)
+/-create, then destroySame, with create_before_destroy
-destroyBlock removed, key removed, terraform destroy
<=read (data source)Data source deferred to apply
  # aws_security_group.web must be replaced
+/- resource "aws_security_group" "web" {
      ~ description = "HTTPS from the internet"
                   -> "HTTPS and HTTP" # forces replacement
      ~ id          = "sg-0a1b..." -> (known after apply)
        name_prefix = "tflab-dev-web-"
        # (6 unchanged attributes hidden)
    }

Plan: 1 to add, 0 to change, 1 to destroy.

Which arguments force replacement is provider knowledge (marked in the docs and shown as "# forces replacement"). Always read it in the plan, never assume (ch 8).

Lifecycle

create_before_destroy and prevent_destroy

Two guards for replacements: avoid downtime, and refuse to destroy what must never go away.

resource "aws_security_group" "web" {
  name_prefix = "tflab-dev-web-"   # unique names: old and
  vpc_id      = aws_vpc.this.id    # new must coexist
  description = "HTTPS from the internet"

  lifecycle {
    create_before_destroy = true
  }
}

resource "aws_s3_bucket" "data" {
  bucket = "tflab-dev-data-<acct>"

  lifecycle {
    prevent_destroy = true
  }
}

# $ terraform destroy
# Error: Instance cannot be destroyed
#   Resource aws_s3_bucket.data has lifecycle.prevent_destroy
#   set, but the plan calls for this resource to be destroyed.

Know the limits

  • create_before_destroy needs names that can coexist: use name_prefix or a random suffix, or the create fails with "already exists".
  • It propagates: resources that this one depends on get it implicitly too.
  • prevent_destroy only works while the block is in the configuration: delete the block (or the module call) and the guard disappears with it.
  • It cannot use variables in Terraform (static value). OpenTofu 1.12 allows a dynamic prevent_destroy (ch 13).
  • Real protection for data: AWS-side deletion protection, S3 versioning/Object Lock, backups, SCPs.
Lifecycle

ignore_changes and replace_triggered_by

Tell Terraform which differences to ignore, and which changes elsewhere should force a fresh object.

resource "aws_security_group" "web" {
  # ...
  lifecycle {
    # another team adds an Owner tag in the console
    ignore_changes = [tags["Owner"]]
  }
}

# replace a resource when something else changes
resource "terraform_data" "config_rev" {
  input = var.config_revision    # plain value → resource
}

resource "aws_cloudwatch_log_group" "app" {
  name = "/tflab/dev/app"
  lifecycle {
    replace_triggered_by = [terraform_data.config_rev]
  }
}

Rules

  • ignore_changes takes attribute paths (desired_capacity of an autoscaled group, a tag key), or all (rarely right: no updates at all).
  • It ignores differences on update; the value is still used on create.
  • replace_triggered_by accepts references to managed resources or their attributes, not variables — wrap values in terraform_data.
  • One-off replacement without code: terraform apply -replace='aws_security_group.web' (replaces the old taint).
  • Overusing ignore_changes hides drift: every entry needs a reason in a comment.
Lifecycle

Timeouts, terraform_data and other knobs

Long-running resources, plain values that need a lifecycle, and the CLI flags that change one apply.

resource "aws_db_instance" "main" {
  # (not in labs: costs money)
  timeouts {          # keys vary by resource
    create = "60m"
    update = "80m"
    delete = "60m"
  }
}

resource "terraform_data" "bootstrap" {
  input            = var.schema_version
  triggers_replace = [aws_vpc.this.id]
}

terraform_data

  • Built-in resource (no provider needed), replaced null_resource.
  • input → output attribute: a value that follows the resource lifecycle.
  • triggers_replace: replace when these values change.
  • Host for provisioner blocks (last resort) and a target for replace_triggered_by.

Per-run flags

  • -replace=ADDR: force replacement of one instance.
  • -target=ADDR: limit the run to some resources — emergencies only; the plan is partial (ch 8).
  • -parallelism=N: concurrent operations (default 10).
  • -refresh=false: skip reading real state — faster, blinder.
  • -lock-timeout=60s: wait for the state lock (ch 4).
Chapter 3 · Part 3.4

Conditions and checks

Assumptions written as code: validate inputs, guard resources, and keep checking after apply.

Conditions

Variable validation

Reject bad input before anything is planned. Since Terraform 1.9 a validation may also reference other variables and objects.

variable "env" {
  type    = string
  default = "dev"
  validation {
    condition     = contains(["dev", "prod"], var.env)
    error_message = "env must be dev or prod."
  }
}

variable "vpc_cidr" {
  type    = string
  default = "10.10.0.0/16"
  validation {
    condition     = can(cidrnetmask(var.vpc_cidr))
    error_message = "vpc_cidr must be a valid IPv4 CIDR."
  }
}

# $ terraform plan -var vpc_cidr=10.10.0.0/33
# Error: Invalid value for variable
#   vpc_cidr must be a valid IPv4 CIDR.

Good validations

  • Allowed values: contains([...], var.x).
  • Format: can(regex("^[a-z0-9-]+$", var.name)), can(cidrnetmask(...)).
  • Ranges: var.retention_days >= 1 && var.retention_days <= 3653.
  • Multiple validation blocks → multiple specific messages.
  • can() turns an error into false; try() returns a fallback value.
  • Error messages: say what is wrong and what is expected.
Conditions

Preconditions and postconditions

Guards on a resource, data source or output: preconditions check assumptions before the change; postconditions check guarantees after it.

locals {
  azs = slice(data.aws_availability_zones.available.names,
              0, min(2, length(
                data.aws_availability_zones.available.names)))
}

resource "aws_vpc" "this" {
  cidr_block           = var.vpc_cidr
  enable_dns_support   = true
  enable_dns_hostnames = true

  lifecycle {
    precondition {               # evaluated before plan of this resource
      condition     = length(local.azs) == 2
      error_message = "Need 2 available AZs in ${var.region}."
    }
    postcondition {              # evaluated after apply (self = result)
      condition     = self.enable_dns_hostnames
      error_message = "DNS hostnames must be enabled."
    }
  }
}

Semantics

  • A failing precondition stops the run before that object changes: an error.
  • A failing postcondition stops the run after the object changed; dependents are not touched: an error.
  • self exists only in postconditions. output blocks accept a precondition too.
  • Use them in modules to state the contract with callers: "subnets must be in this VPC", "AMI must be arm64".
  • Postconditions on data sources catch surprising lookups (wrong AMI, empty list) at plan time.
Conditions

check blocks: continuous assertions that only warn

A check block (1.5+) runs at the end of every plan and apply. A failed assertion is a warning: it never blocks a change.

check "two_public_subnets" {
  assert {
    condition     = length(aws_subnet.public) == 2
    error_message = "Expected 2 public subnets."
  }
}

# scoped data source: only visible inside the check,
# and its errors become warnings too
check "vpc_dns" {
  data "aws_vpc" "check" {
    id = aws_vpc.this.id
  }
  assert {
    condition     = data.aws_vpc.check.enable_dns_support
    error_message = "DNS support is off on ${aws_vpc.this.id}."
  }
}

# │ Warning: Check block assertion failed
# │   on main.tf line 80, in check "two_public_subnets":
# │   Expected 2 public subnets.

When to use what

  • validation: inputs, before anything else.
  • precondition / postcondition: hard guarantees around one object; failures stop the run.
  • check: health and drift signals that should not block a fix ("certificate expires in < 30 days", "endpoint returns 200").
  • HCP Terraform runs checks on a schedule (health assessments); with the CLI, a nightly plan in CI shows them (ch 11).
  • terraform test asserts are for module tests, not production runs (ch 7).
Conditions

Where each guard runs and what a failure does

Pick the weakest tool that still protects you: warnings for signals, errors for invariants.

GuardLives inEvaluatedOn failureTypical use
validationvariable blockBefore planning (as soon as the value is known)ErrorAllowed values, formats, ranges
preconditionlifecycle of resource / data, or outputBefore the object is planned or changedErrorAssumptions about inputs and other objects
postconditionlifecycle of resource / dataAfter the object is read or appliedError (dependents skipped)Guarantees about the result (self)
check / assertTop-level check blockEnd of every plan and applyWarningHealth, expiry, drift signals
assert in testsrun block in .tftest.hclterraform test onlyTest failureModule behavior (ch 7)
Policy (Checkov, Trivy, conftest, Terraform Policy)Outside the moduleIn CI on code or plan JSONPipeline failureOrganization rules (ch 9)
Common mistakes

Provider and lifecycle mistakes that bite

Each of these has deleted something real or blocked a release.

Providers

  • No lock file in Git → CI picks a newer provider than your laptop; plans differ.
  • Lock file only for darwin_arm64 → CI on linux fails or rewrites it.
  • Provider blocks inside reusable modules → the module can no longer be used with for_each or count.
  • Credentials in provider "aws" arguments or tfvars.
  • Relying on the default region of whoever runs the plan.

Lifecycle

  • create_before_destroy with a fixed name → "already exists" on replacement.
  • Believing prevent_destroy protects data after the block was removed.
  • ignore_changes = all to silence a diff nobody understood.
  • depends_on everywhere "to be safe" → slower, vaguer plans.
  • Missing a "# forces replacement" line in a long plan.

Habits that prevent them

  • Commit .terraform.lock.hcl; init -lockfile=readonly in CI.
  • Pre-lock all platforms you run on.
  • name_prefix for anything that may be replaced.
  • AWS-side deletion protection and backups for data stores.
  • Search every plan for must be replaced before approving (ch 8).
Self-check

Providers, resources and the lifecycle: can you answer these?

Answer out loud with the exact argument, symbol or command.

Providers

  • What does ~> 6.2.0 allow? And ~> 6.0?
  • What is the difference between an h1: and a zh: hash?
  • How does init behave with and without -upgrade?
  • When do you still need a provider alias in AWS provider v6?
  • Where should credentials come from, locally and in CI?

Resources and graph

  • Write the address of the ap-northeast-1c instance of aws_subnet.public inside module.network.
  • When is a data source read during apply instead of plan?
  • Why is depends_on on a module expensive?
  • In which order are resources destroyed?
  • Which plan symbol means "create the replacement first"?

Lifecycle and conditions

  • Why does create_before_destroy need name_prefix?
  • Name two ways prevent_destroy can be bypassed.
  • How do you replace a resource when a variable changes?
  • Precondition vs postcondition vs check: which ones block an apply?
  • What can a scoped data source in a check block do that a normal one cannot?

Practice: Lab 3 in Hands-on labs — build live/dev/network with provider v6 and default_tags, lock two platforms, then provoke replacements, ignored tags, prevent_destroy and a failing check.

Chapter 4

State and backends

What Terraform remembers between runs, where to keep it safely, how to inspect and repair it, and how to split it.

Chapter 4 · Part 4.1

What state is

The map between your configuration and the real objects in AWS.

State

Why Terraform needs state

Configuration says what should exist; AWS says what does exist. State is the record that binds each resource address to one real object.

Configurationaws_vpc.this (HCL)Real worldvpc-0a1b2c3d (AWS)Stateaddress → ID + attributesdesired vs actualwhich address?which object?plan = refresh state from AWS, then diff configuration against it.

What state gives Terraform

  • Identity: aws_vpc.this is vpc-0a1b2c3d. Without it Terraform would create a second VPC.
  • Attribute cache: values of every attribute after the last apply or refresh.
  • Dependencies: needed to destroy in the right order after a block is deleted from code.
  • Outputs of the root module, readable by others (terraform output, remote state).
  • Metadata: provider that manages each object, schema version, check results.
State

Inside a state file

JSON, format version 4. Read it to understand it; never edit it by hand.

{
  "version": 4,
  "terraform_version": "1.17.0",
  "serial": 12,
  "lineage": "3f9c2b1e-8d7a-4c55-9e0f-2a6b7c8d9e10",
  "outputs": {
    "vpc_id": {"value": "vpc-0a1b2c3d", "type": "string"}
  },
  "resources": [
    {
      "mode": "managed",
      "type": "aws_vpc",
      "name": "this",
      "provider": "provider[\"registry…/hashicorp/aws\"]",
      "instances": [
        {
          "schema_version": 1,
          "attributes": {
            "id": "vpc-0a1b2c3d",
            "cidr_block": "10.10.0.0/16"
          },
          "sensitive_attributes": []
        }
      ]
    }
  ]
}

Fields that matter

  • serial: increments on every write. A lower serial is an older snapshot.
  • lineage: random ID set when the state is created. Different lineage = a different state, not a newer one.
  • instances[].attributes: the full attribute set, including values marked sensitive.
  • index_key appears on count/for_each instances ("ap-northeast-1a").
  • terraform state push refuses a lower serial or another lineage unless -force (ch 4.3).
State

State holds secrets in plain text

sensitive = true hides values in CLI output. It does not encrypt them in state.

What ends up in state

  • Generated passwords (random_password.result).
  • Values of aws_ssm_parameter of type SecureString, aws_secretsmanager_secret_version.secret_string.
  • Private keys from tls_private_key, access keys from aws_iam_access_key.
  • Anything a data source read, e.g. data.aws_ssm_parameter.x.value.
  • Plan files (tfplan) contain the same values.

So protect state like a secret

  • Remote backend with encryption at rest (SSE-S3 or KMS) and TLS.
  • Bucket policy: only the CI roles and a break-glass role can read it.
  • Block public access; versioning on; access logging or CloudTrail data events.
  • Never commit terraform.tfstate or *.tfplan to Git (.gitignore).
  • OpenTofu can encrypt the state client-side (ch 4.4).

Better: keep secrets out

  • Ephemeral resources and write-only arguments (password_wo) never reach state (ch 2, ch 12).
  • Let AWS manage secrets: RDS manage_master_user_password = true.
  • Reference a secret by ARN; let the workload read it at runtime.
  • Rotate anything that was ever in a leaked state file.
State

How state changes during a run

Every plan refreshes; every apply writes. Knowing when helps you reason about locks, drift and partial failures.

1 · Lock

  • Acquire the state lock (remote backends).
  • Fails fast if someone else holds it.
→

2 · Refresh

  • Read each managed object from AWS.
  • Update the in-memory state; detect drift.
→

3 · Plan

  • Diff configuration against refreshed state.
  • Save with -out=tfplan for review.
→

4 · Apply

  • Call the provider per resource.
  • Persist updated state as resources complete and at the end (serial++).
→

5 · Unlock

  • Release the lock.
  • A crash can leave it held → force-unlock.

Partial failure: resources that succeeded are already in state, failed ones are not (or are marked tainted). Fix the cause and run plan again; never "clean up" by editing state.

State

Never edit state by hand

Every legitimate change to state has a command or a block. Hand edits break serial, lineage, schema versions and dependencies.

You want to…UseChapter
Rename or move a resource / into a modulemoved block (or terraform state mv)ch 5
Adopt an existing AWS objectimport block (or terraform import)ch 5
Stop managing an object without destroying itremoved block with destroy = false (or state rm)ch 5
Accept changes made outside Terraform into stateterraform apply -refresh-onlych 4.3
Switch a resource to another provider sourceterraform state replace-providerch 13
Recover a broken or deleted stateRestore an older S3 object version; state push as a last resortch 4.3
Read values for scriptsterraform output -json, terraform show -jsonch 4.3
Chapter 4 · Part 4.2

Remote backends and locking

Shared, durable, encrypted state with a lock so two applies never interleave.

Backends

Local state vs a remote backend

A local terraform.tfstate is fine for a tutorial and wrong for anything a team or CI touches.

Local state (default)

  • A file next to your code; one laptop owns the truth.
  • No locking across machines; no history unless you back it up.
  • Easy to commit by mistake (with secrets inside).
  • OK for bootstrap/ — the configuration that creates the state bucket (Lab 0).

Remote backend

  • State stored in S3, HCP Terraform, GCS, Azure Blob, Postgres…
  • Locking so concurrent runs fail fast instead of corrupting state.
  • History via object versioning; encryption; access control.
  • CI and humans see the same state.
  • Configured in the terraform { backend "…" {} } block of each root module.

Backend rules

  • The backend block cannot use variables or locals: values are literal or come from -backend-config.
  • One backend per root module; child modules never declare one.
  • Changing a backend requires terraform init again.
  • HCP Terraform uses a cloud block instead of backend "remote".
Backends

The S3 backend with native locking

Since Terraform 1.10, S3 can lock by itself (use_lockfile). The DynamoDB table is deprecated.

# backend.tf (live/dev/network)
terraform {
  backend "s3" {
    # bucket comes from -backend-config (one per account)
    key          = "dev/network/terraform.tfstate"
    region       = "ap-northeast-1"
    encrypt      = true
    use_lockfile = true      # default false: set it!
  }
}

# terraform init -backend-config="bucket=$TF_STATE_BUCKET"
# objects: dev/network/terraform.tfstate
#          dev/network/terraform.tfstate.tflock (while locked)

Main arguments

ArgumentMeaning
bucket, key, regionWhere the state object lives
encryptServer-side encryption of the state object
kms_key_idUse SSE-KMS with this key
use_lockfileLock with a <key>.tflock object (default false)
dynamodb_tableLegacy DynamoDB locking — deprecated
assume_role { role_arn = … }Read/write state with another role

The lock object is created with an S3 conditional write, so only one writer wins. Permissions: get/put/delete on <key>.tflock, get/put on the state key, s3:ListBucket.

Backends

Moving from DynamoDB locking to the S3 lock file

Both mechanisms can be enabled at once; a run then needs both locks. That makes the switch safe while old and new runners coexist.

# step 1: both locks (all runners on Terraform >= 1.11)
terraform {
  backend "s3" {
    bucket         = "tflab-state-<acct>"
    key            = "dev/network/terraform.tfstate"
    region         = "ap-northeast-1"
    encrypt        = true
    use_lockfile   = true
    dynamodb_table = "tflab-locks"   # deprecated, transitional
  }
}

# step 2: remove dynamodb_table once nothing older runs
# step 3: terraform init -reconfigure (state is unchanged)
# step 4: delete the DynamoDB table

Why the change

  • S3 added conditional writes (Aug 2024), so a separate lock store is no longer needed.
  • One fewer resource to bootstrap, secure and pay for.
  • Terraform docs: DynamoDB locking is deprecated and will be removed in a future minor version.
  • -reconfigure re-initializes the backend without migrating state; use it when only lock settings change.
  • OpenTofu also supports use_lockfile on its S3 backend (1.10+).
Backends

init flags for backends: partial config, migrate, reconfigure

Backend settings are resolved at init time and cached in .terraform/terraform.tfstate (the backend config, not your state).

# partial configuration: keep account-specific values out of code
cat > backend.dev.hcl <<'EOF'
bucket = "tflab-state-123456789012"
EOF
terraform init -backend-config=backend.dev.hcl
# or: -backend-config="bucket=$TF_STATE_BUCKET"

# local → S3 (or S3 key A → key B): copy the state
terraform init -migrate-state -backend-config=backend.dev.hcl
#   Do you want to copy existing state to the new backend?
#   Enter a value: yes

# non-interactive (CI) migration
terraform init -migrate-state -force-copy ...

# backend settings changed, state should NOT be copied
terraform init -reconfigure -backend-config=backend.dev.hcl

Which flag

  • none: first init, or nothing changed.
  • -migrate-state: the backend location changed and the state must come along.
  • -force-copy: answer "yes" to the copy prompts (scripts).
  • -reconfigure: ignore the cached backend settings; use the new ones as-is.
  • After migrating from local, **delete the local terraform.tfstate** (keep a backup first) so nobody applies against it.
Backends

Locks in practice: errors, timeouts and force-unlock

A lock error is a feature: someone (or something) is changing the same state right now.

$ terraform plan
│ Error: Error acquiring the state lock
│
│ Error message: ... (exact text varies by version)
│ Lock Info:
│   ID:        5b3c1a2e-7f6d-4e3b-9a1c-0d2e3f4a5b6c
│   Path:      tflab-state-<acct>/dev/network/terraform.tfstate
│   Operation: OperationTypeApply
│   Who:       alice@laptop
│   Version:   1.17.0
│   Created:   2026-11-09 03:12:45 +0000 UTC

# wait up to a minute for the lock instead of failing
terraform plan -lock-timeout=60s

# ONLY when the holder is certainly dead (crashed runner):
terraform force-unlock 5b3c1a2e-7f6d-4e3b-9a1c-0d2e3f4a5b6c

Rules

  • Check Who and Created first; ask the person or check the CI run.
  • An apply waiting at the approval prompt holds the lock.
  • Never force-unlock a lock held by a live process: two writers = lost updates.
  • -lock=false exists; do not use it on shared state.
  • In CI, serialize runs per state with a concurrency group (ch 11) so lock errors become rare.
Backends

Other backends and HCP Terraform

Same idea, different store. Pick the one your platform already secures and backs up.

BackendLockingNotes
s3use_lockfile (or legacy DynamoDB)This deck; versioning for history
cloud block (HCP Terraform / Terraform Enterprise)Built inRemote runs, UI, policies, private registry
gcsBuilt inGoogle Cloud Storage
azurermBlob leaseAzure Storage
pgAdvisory locksPostgres table; schema configurable
kubernetesLeaseSecret per state; 1 MiB size limit
# HCP Terraform: cloud block instead of backend
terraform {
  cloud {
    organization = "my-org"
    workspaces {
      name = "tflab-dev-network"
    }
  }
}
# terraform login   (stores an API token)
# terraform init    (connects the workspace)
# plans/applies then run remotely by default
Chapter 4 · Part 4.3

Inspecting and repairing state

The state subcommands, JSON views, refresh-only runs and recovery from S3 versions.

State commands

Read state: list, show, output, show -json

Read-only commands are always safe. Start every investigation with them.

$ terraform state list
data.aws_availability_zones.available
aws_internet_gateway.this
aws_route_table.public
aws_route_table_association.public["ap-northeast-1a"]
aws_route_table_association.public["ap-northeast-1c"]
aws_security_group.web
aws_subnet.private["ap-northeast-1a"]
aws_subnet.private["ap-northeast-1c"]
aws_subnet.public["ap-northeast-1a"]
aws_subnet.public["ap-northeast-1c"]
aws_vpc.this
aws_vpc_security_group_ingress_rule.https

$ terraform state list 'aws_subnet.public'   # filter by address

$ terraform state show aws_vpc.this
# aws_vpc.this:
resource "aws_vpc" "this" {
    cidr_block = "10.10.0.0/16"
    id         = "vpc-0a1b2c3d"
    ...
}
# outputs for scripts
terraform output -raw vpc_id
terraform output -json | jq -r '.public_subnet_ids.value[]'

# whole state as JSON (stable, documented format)
terraform show -json | jq -r '
  .values.root_module.resources[]
  | select(.type == "aws_subnet")
  | "\(.address)  \(.values.cidr_block)"'
# aws_subnet.private["ap-northeast-1a"]  10.10.10.0/24
# aws_subnet.public["ap-northeast-1a"]   10.10.0.0/24
# ...

# raw remote state (serial and lineage included)
terraform state pull | jq '{serial, lineage}'
# { "serial": 12, "lineage": "3f9c2b1e-..." }

Prefer terraform show -json over parsing terraform.tfstate directly: its format is documented and stable across Terraform versions.

State commands

Change state: mv, rm, push

Imperative tools for emergencies. Each writes a backup, but on a remote backend run state pull > backup.tfstate yourself first.

terraform state pull > backup-$(date +%s).tfstate

# rename without destroy/create
terraform state mv aws_security_group.web aws_security_group.app
# Move "aws_security_group.web" to "aws_security_group.app"
# Successfully moved 1 object(s).
#   → now rename the block in code too, or the next plan
#     wants to destroy .app and create .web

# forget an object (it keeps running in AWS)
terraform state rm 'aws_subnet.private["ap-northeast-1c"]'

# overwrite remote state with a file (last resort)
terraform state push restored.tfstate
#   refuses a different lineage or a higher remote serial
terraform state push -force restored.tfstate

Prefer blocks (ch 5)

  • moved instead of state mv: reviewed in a PR, applied by CI, works for every user of a module.
  • removed instead of state rm: same benefits, explicit destroy = false.
  • import blocks instead of terraform import: plannable.
  • Use the commands for one-off repairs, with the state locked by nobody else and a backup in hand.
  • Always run terraform plan right after: the target is No changes.
State commands

Drift and refresh-only runs

Drift = AWS differs from state because someone changed it outside Terraform. A normal plan proposes to revert it; refresh-only accepts it.

# someone added a tag in the console
$ terraform plan -refresh-only
Note: Objects have changed outside of Terraform

Terraform detected the following changes made outside of
Terraform since the last "terraform apply":

  # aws_vpc.this has changed
  ~ resource "aws_vpc" "this" {
      ~ tags     = {
          + "CostCenter" = "1234"
            "Name"       = "tflab-dev"
        }
        # (16 unchanged attributes hidden)
    }

# accept the real values into state (no AWS change)
terraform apply -refresh-only

# a normal plan would instead propose:
#   ~ tags = { - "CostCenter" = "1234" ... }

Deciding what to do

  • Revert (normal apply): the code is right, the console change was wrong.
  • Adopt in code: add the tag to the configuration, then plan shows no changes.
  • Accept into state only (apply -refresh-only): records reality but the next normal plan still reverts it unless code changes.
  • Ignore with ignore_changes: only when another system owns that attribute (ch 3).
  • terraform refresh is the deprecated alias of apply -refresh-only -auto-approve: avoid it.
  • Scheduled drift detection: ch 11.
State commands

Recovering a broken state from S3 versioning

With versioning on the state bucket, every write is a restorable snapshot. This is why Lab 0 turns versioning on.

B=$TF_STATE_BUCKET; K=dev/network/terraform.tfstate

# 1) list versions of the state object only
#    (the prefix also matches terraform.tfstate.tflock)
aws s3api list-object-versions --bucket $B --prefix $K \
  --query "Versions[?Key=='$K'].[VersionId,LastModified,IsLatest]" \
  --output table

# 2) inspect a candidate before restoring
aws s3api get-object --bucket $B --key $K \
  --version-id <id> /tmp/candidate.tfstate
jq '{serial, terraform_version}' /tmp/candidate.tfstate

# 3) restore = copy that version over the current object
aws s3api copy-object --bucket $B --key $K \
  --copy-source "$B/$K?versionId=<id>"

# 4) compare with reality
terraform plan     # expect: No changes

Recovery rules

  • Stop all runs on that state first (pause CI, nobody applies).
  • Pick the newest version before the bad write, not the oldest "known good".
  • Restoring old state does not change AWS: anything created after that snapshot is now unknown to Terraform → plan wants to create it again → import it (ch 5).
  • Add a lifecycle rule that expires noncurrent versions after N days, so lock-file versions and old states do not pile up.
  • Practise this before you need it (Lab 4).
Chapter 4 · Part 4.4

Splitting state and sharing outputs

One state per component and environment, and clean ways to pass values between them.

Splitting state

Blast radius: how big should one state be?

Everything in one state is planned, locked and can be broken together. Split along ownership and change frequency.

one big state (avoid)network + app + iam + dnsdev AND prod• every plan refreshes everything• one lock for every team• an app change can touch prod VPC• plans take minutes, reviews skimtf-lab: env × componentdev/networkdev/appprod/networkprod/appseparate keys, locks, plans andpermissions; app reads network outputs

Split when

  • Different environments (always).
  • Different owners or approval rules (network team vs app team).
  • Different change rates (VPC yearly, app weekly).
  • Plans get slow or reviews get skimmed.
  • Do not over-split: each cut adds an interface (outputs) and an apply order (ch 10).
Splitting state

Sharing values between states: the options

The app component needs the VPC ID and subnet IDs from network. How it gets them decides coupling and permissions.

OptionConsumer needsProsCons
terraform_remote_state data sourceRead access to the whole producer stateNo extra resources; typed outputsExposes every secret in that state; couples to backend layout and producer's Terraform version
Published values: SSM Parameter Store (tf-lab)ssm:GetParameter on /tflab/<env>/network/*Narrow permissions; readable by scripts and other tools; explicit contractProducer must write them; strings only (lists as StringList)
Provider data sources by tag or nameNormal AWS read permissionsNo dependency on the other state at allLookups can match zero or many objects; needs good tagging
HCP Terraform outputs (tfe_outputs) / StacksWorkspace access in HCP TerraformOnly outputs are sharedTied to HCP Terraform

tf-lab uses SSM parameters: network publishes /tflab/dev/network/vpc_id and /tflab/dev/network/private_subnet_ids; app reads them (Lab 4, ch 10).

Splitting state

Publishing and consuming outputs via SSM

Producer and consumer agree on parameter names, not on each other's state.

# producer: live/dev/network/ssm.tf
resource "aws_ssm_parameter" "vpc_id" {
  name  = "/tflab/${var.env}/network/vpc_id"
  type  = "String"
  value = aws_vpc.this.id
}

resource "aws_ssm_parameter" "private_subnet_ids" {
  name  = "/tflab/${var.env}/network/private_subnet_ids"
  type  = "StringList"
  value = join(",", [for s in aws_subnet.private : s.id])
}
# consumer: live/dev/app/network.tf
data "aws_ssm_parameter" "vpc_id" {
  name = "/tflab/dev/network/vpc_id"
}

data "aws_ssm_parameter" "private_subnets" {
  name = "/tflab/dev/network/private_subnet_ids"
}

locals {
  # value is always marked sensitive; insecure_value is not
  vpc_id     = data.aws_ssm_parameter.vpc_id.insecure_value
  subnet_ids = split(",",
    data.aws_ssm_parameter.private_subnets.insecure_value)
}

# the same contract via remote state (needs full read):
# data "terraform_remote_state" "network" {
#   backend = "s3"
#   config = { bucket = "tflab-state-<acct>",
#     key = "dev/network/terraform.tfstate",
#     region = "ap-northeast-1" }
# }
# data.terraform_remote_state.network.outputs.vpc_id
Workspaces

CLI workspaces: several states for one configuration

A workspace is another state under the same backend and code. Useful for short-lived copies; risky as the only separation between dev and prod.

terraform workspace list        # * default
terraform workspace new feature-x
terraform workspace select default
terraform workspace show

# in code
locals {
  name = "tflab-${terraform.workspace}"
}

# S3 keys used:
#   default   → dev/network/terraform.tfstate
#   feature-x → env:/feature-x/dev/network/terraform.tfstate
#               (workspace_key_prefix, default "env:")

Good fit

  • Temporary copies of a stack: per-branch or per-developer test environments.
  • Identical environments that differ only in a few variables, in the same account, with the same owners.

Poor fit for dev vs prod

  • Same backend and credentials for both: whoever can plan dev can apply prod.
  • The active workspace is invisible in the code: apply in the wrong one is a classic outage.
  • Prod usually needs different accounts, approvals and versions.
  • tf-lab uses directories per environment instead (ch 10). HCP Terraform "workspaces" are a different, richer concept.
OpenTofu

Client-side state encryption (OpenTofu)

OpenTofu 1.7+ can encrypt state and plan files before they reach the backend. Terraform has no equivalent; it relies on backend encryption.

# OpenTofu only — Terraform rejects this block
terraform {
  encryption {
    key_provider "aws_kms" "state" {
      kms_key_id = "alias/tflab-state"
      region     = "ap-northeast-1"
      key_spec   = "AES_256"
    }
    method "aes_gcm" "main" {
      keys = key_provider.aws_kms.state
    }
    state {
      method   = method.aes_gcm.main
      enforced = true     # refuse to write unencrypted state
    }
    plan {
      method   = method.aes_gcm.main
      enforced = true
    }
  }
}

Trade-offs

  • Protects against anyone who can read the bucket but not use the KMS key.
  • Key providers include pbkdf2 (passphrase), aws_kms, gcp_kms, OpenBao.
  • Migrating to encrypted state needs a fallback method during the switch; read the OpenTofu docs.
  • Lose the key, lose the state: back up keys and test restore.
  • Encrypted state cannot be read by Terraform or tools that parse raw state.
Common mistakes

State mistakes that cause outages

Most Terraform incidents are state incidents.

Storage and access

  • Local state on one laptop; the laptop dies.
  • terraform.tfstate committed to Git with secrets inside.
  • State bucket without versioning → no way back.
  • Everyone can read state "because it is just Terraform".
  • use_lockfile forgotten: it defaults to false, so there is no lock.

Operations

  • force-unlock while a CI apply is still running.
  • state mv without changing the code → next plan destroys and recreates.
  • apply in the wrong workspace.
  • Restoring old state and applying without reading the plan.
  • Leaving the old local terraform.tfstate after migrating.

Design

  • One state for all environments.
  • terraform_remote_state everywhere: every consumer can read every producer secret.
  • Splitting so finely that applies need a 12-step order.
  • No owner per state.
  • Fix: env × component states, SSM contracts, least-privilege bucket policy (ch 12).
Self-check

State and backends: can you answer these?

Explain with the exact field, argument or command.

Concepts

  • Name three things state gives Terraform beyond the attribute cache.
  • What is the difference between serial and lineage?
  • Does sensitive = true encrypt anything? Where do the values end up?
  • When is state written during an apply?
  • Which operations should use a block instead of a state command?

Backends and locking

  • Which S3 backend argument enables native locking, and what is its default?
  • What object appears in the bucket while a lock is held?
  • When do you use -migrate-state vs -reconfigure?
  • Why can the bucket name not come from a variable in the backend block?
  • When is force-unlock safe?

Repair and design

  • How do you restore a previous state version from S3?
  • After restoring, why might plan want to create resources that exist?
  • What does apply -refresh-only change in AWS?
  • Remote state vs SSM parameters: what permissions does each consumer need?
  • Why are CLI workspaces a poor dev/prod boundary?

Practice: Lab 4 in Hands-on labs — migrate live/dev/network to S3 with use_lockfile, watch the .tflock object, use the state commands, restore an older version, and publish outputs to SSM.

Chapter 5

Refactoring and adoption: moved, import, removed

Change addresses without destroying anything, bring existing resources under Terraform, let go of resources without deleting them, and handle drift.

Chapter 5 · Part 5.1

moved blocks

Rename, restructure and move resources into modules while the plan shows zero changes.

Refactoring

Three places a resource lives

Every refactoring tool in this chapter changes exactly one of these links. Knowing which one is the whole trick.

Code (.tf)resource addressaws_s3_bucket.reportsdesired argumentsbucket, tags, …State (S3)address → real IDreports = tflab-…last known attributesrefreshed on every planReal infrastructurethe bucket itselfuntouched by movesmoved: new addressimport: link a real objectremoved (destroy = false): forget itrefresh-only: copy reality into statemoved, import and removed are declared in code, shown in the plan and applied like any other change: reviewable, repeatable, CI-friendly.

The imperative equivalents (terraform state mv, terraform import, terraform state rm) change state immediately, with no plan and no review. Prefer the blocks; keep the commands for emergencies (ch 4).

moved blocks

Without moved: a rename is a destroy and a create

Terraform tracks objects by address. Change the address and Terraform sees one object to delete and a new one to create.

# before
resource "aws_s3_bucket" "b" {
  bucket = "tflab-dev-logs-123456789012"
}

# after: a clearer name, nothing else changed
resource "aws_s3_bucket" "logs" {
  bucket = "tflab-dev-logs-123456789012"
}
$ terraform plan
  # aws_s3_bucket.b will be destroyed
  # (because aws_s3_bucket.b is not in configuration)
  - resource "aws_s3_bucket" "b" { ... }

  # aws_s3_bucket.logs will be created
  + resource "aws_s3_bucket" "logs" { ... }

Plan: 1 to add, 0 to change, 1 to destroy.
# for a bucket with data: destroy fails (BucketNotEmpty)
# or, with force_destroy = true, deletes your data

For an S3 bucket the create would even fail: the name is still taken by the bucket being destroyed in the same run. For a database the result is data loss. Never accept a destroy/create pair caused only by a rename.

moved blocks

The fix: declare the move

A moved block tells Terraform that the object at from now lives at to. The plan updates the address in state and touches nothing in AWS.

resource "aws_s3_bucket" "logs" {
  bucket = "tflab-dev-logs-123456789012"
}

moved {
  from = aws_s3_bucket.b
  to   = aws_s3_bucket.logs
}
$ terraform plan
  # aws_s3_bucket.b has moved to aws_s3_bucket.logs
    resource "aws_s3_bucket" "logs" {
        id     = "tflab-dev-logs-123456789012"
        # (12 unchanged attributes hidden)
    }

Plan: 0 to add, 0 to change, 0 to destroy.

$ terraform apply      # records the new address
$ terraform state list
aws_s3_bucket.logs

If the plan shows a move and ~ update lines, the move worked but the code also differs from reality. Split the two: ship the pure move first (zero changes), then the real change.

moved blocks

What moved can express

Addresses can be whole resources, single instances, module calls or module instances. Since Terraform 1.8, a provider can also allow moves between resource types.

Refactoringfromto
Rename a resourceaws_vpc.mainaws_vpc.this
count → for_each (one block per instance)aws_subnet.public[0]aws_subnet.public["a"]
Single resource → count/for_eachaws_s3_bucket.logsaws_s3_bucket.logs["dev"]
Resource into a moduleaws_vpc.thismodule.network.aws_vpc.this
Rename a module call (all its resources)module.vpcmodule.network
Module call → for_each modulemodule.networkmodule.network["dev"]
Change resource type (provider must support it)null_resource.xterraform_data.x

Adding count to a single resource is an implied move (x → x[0], no block needed); adding for_each is not: without a moved block the plan destroys x and creates x["dev"]. Type changes work only where the provider implements them; otherwise use removed + import.

moved blocks

count to for_each without replacing anything

Indexes are positions: delete the first subnet and every later one shifts and gets replaced. Keys are names. Moving once to for_each ends the shifting for good (ch 2).

# before: positional
resource "aws_subnet" "public" {
  count             = 2
  vpc_id            = aws_vpc.this.id
  cidr_block        = cidrsubnet("10.10.0.0/16", 8, count.index)
  availability_zone = ["ap-northeast-1a", "ap-northeast-1c"][count.index]
}

# after: keyed by AZ suffix
locals {
  public_subnets = { a = "10.10.0.0/24", c = "10.10.1.0/24" }
}

resource "aws_subnet" "public" {
  for_each          = local.public_subnets
  vpc_id            = aws_vpc.this.id
  cidr_block        = each.value
  availability_zone = "ap-northeast-1${each.key}"
}
moved {
  from = aws_subnet.public[0]
  to   = aws_subnet.public["a"]
}

moved {
  from = aws_subnet.public[1]
  to   = aws_subnet.public["c"]
}

$ terraform plan
  # aws_subnet.public[0] has moved to aws_subnet.public["a"]
  # aws_subnet.public[1] has moved to aws_subnet.public["c"]
Plan: 0 to add, 0 to change, 0 to destroy.

# also move every resource that was indexed the same
# way (route table associations, NACL associations…)

Check that each key maps to the instance with the same arguments: [0] was 10.10.0.0/24 in 1a, so it must become "a". A wrong mapping shows up as -/+ forces replacement on cidr_block.

moved blocks

Rules, scope and lifetime

Moves are evaluated before planning, in order, and only inside one configuration and one state.

Scope rules

  • A moved block in a module may only refer to addresses inside that module (and its children).
  • The root module can move objects into, out of and between child modules.
  • Moves never cross state files: two root modules = two states. Use removed + import for that (section 20).
  • The to address must exist in configuration; the from address must not.

Chains and history

  • Moves can chain: a → b and later b → c. Keep both blocks; Terraform follows the chain.
  • Anyone still at an old address (another environment, a module consumer) needs the old blocks.
  • Leave moved blocks in shared modules for at least one major version (ch 6).
  • In a root module, delete them once every state that uses the code has been applied.

Versus terraform state mv

  • state mv edits state now: no plan, no review, no record in Git.
  • With a remote backend it rewrites the shared state while others may be planning.
  • Every environment needs the same commands typed again.
  • Use it only when blocks cannot express the move, and back up state first (state pull).
Chapter 5 · Part 5.2

import blocks

Bring an existing resource under Terraform through a reviewed plan instead of a one-off command.

import blocks

Adopting a resource: import block + resource block

Since Terraform 1.5, import is a block: it says which real object (id or identity) should be bound to which address (to). The plan shows the import before anything happens.

import {
  to = aws_s3_bucket.reports
  id = "tflab-clickops-reports-123456789012"
}

resource "aws_s3_bucket" "reports" {
  bucket = "tflab-clickops-reports-123456789012"
}
$ terraform plan
  # aws_s3_bucket.reports will be imported
    resource "aws_s3_bucket" "reports" {
        arn    = "arn:aws:s3:::tflab-clickops-reports-…"
        bucket = "tflab-clickops-reports-123456789012"
        ...
    }

Plan: 1 to import, 0 to add, 0 to change, 0 to destroy.

$ terraform apply
Apply complete! Resources: 1 imported, 0 added,
0 changed, 0 destroyed.

Goal of every import: "1 to import, 0 to change". If the plan also shows ~ update, your resource block does not match reality yet; Terraform would change the real resource on apply. Fix the code until only the import remains (unless the change is intended).

import blocks

id, identity and for_each

The import id format is provider- and resource-specific (see "Import" at the bottom of each registry page). Resource identity (Terraform 1.12+) replaces the opaque string with named attributes.

# identity: named, provider-defined attributes
import {
  to = aws_s3_bucket.reports
  identity = {
    bucket = "tflab-clickops-reports-123456789012"
    # optional: account_id, region
  }
}

# many objects at once (Terraform 1.7+)
locals {
  params = {
    feature_flag = "/tflab/dev/clickops/feature-flag"
    owner        = "/tflab/dev/clickops/owner"
  }
}

import {
  for_each = local.params
  to       = aws_ssm_parameter.clickops[each.key]
  id       = each.value
}

Rules worth knowing

  • id and identity are mutually exclusive in one block.
  • Identity keys come from the provider: aws_s3_bucket needs bucket; aws_ssm_parameter needs name; aws_vpc needs id (account_id, region optional).
  • id/identity may use variables and locals, but must be known at plan time.
  • Import blocks are allowed only in the root module; to can point into a child module (module.network.aws_vpc.this).
  • Use provider = aws.other to import through an aliased provider.
  • After the apply, the block is a no-op: delete it or keep it as a record.
import blocks

Let Terraform write the first draft: -generate-config-out

With an import block and no matching resource block, terraform plan -generate-config-out=FILE writes HCL from the real object's attributes.

# only the import block exists
$ terraform plan -generate-config-out=generated.tf
...
Terraform has generated configuration and written it to
generated.tf. Please review the configuration and edit
it as necessary before adding it to version control.

Plan: 1 to import, 0 to add, 0 to change, 0 to destroy.

# generated.tf (excerpt)
# __generated__ by Terraform
resource "aws_s3_bucket" "reports" {
  bucket              = "tflab-clickops-reports-123456789012"
  bucket_prefix       = null
  force_destroy       = false
  object_lock_enabled = false
  tags                = { Owner = "clickops" }
}

Then edit it like any code

  • The file must not exist yet; the flag never overwrites.
  • The help text still marks the flag experimental: always review the output.
  • Remove null and default-valued arguments; replace literals with references (aws_vpc.this.id, local.tags).
  • Sensitive attributes are not written out: fill them in yourself (or use write-only arguments, ch 2).
  • Some generated arguments conflict with each other; plan tells you which to delete.
  • Rename the resource, move it into a module later with moved.
  • Plan again until it says only "1 to import".
import blocks

The import command, and when you still need it

terraform import ADDRESS ID predates the block. It writes state immediately; the resource block must already exist.

# imperative: state changes now, no plan
terraform import aws_s3_bucket.reports \
  tflab-clickops-reports-123456789012

terraform import 'aws_ssm_parameter.clickops["owner"]' \
  /tflab/dev/clickops/owner      # quote the address

terraform plan     # check for drift afterwards

Block vs command

import blockterraform import
Shown in a plan before it happensYesNo
Reviewed in a pull request, runs in CIYesOnly by hand
Many objects (for_each)YesOne per call
Can generate configurationYes (-generate-config-out)No
Works in older Terraform (< 1.5)NoYes

With the command, a typo in the ID or address lands straight in shared state. If you must use it, terraform state pull > backup.tfstate first.

Chapter 5 · Part 5.3

Finding resources to adopt

Search an account for unmanaged resources with list blocks and terraform query, then import them in bulk.

terraform query

List blocks: searching instead of guessing IDs

Terraform 1.14 added list blocks in *.tfquery.hcl files. terraform query asks the provider for existing objects of a type and prints their identities.

# search.tfquery.hcl   (next to your .tf files)
list "aws_s3_bucket" "all" {
  provider = aws
  limit    = 50          # default: 100 per list block
}

list "aws_ssm_parameter" "all" {
  provider         = aws
  include_resource = true   # full attributes, slower
  config {
    region = "ap-northeast-1"
  }
}

How it works

  • A list block has a type and a name, like a resource: list.aws_s3_bucket.all.
  • provider is required; config { } holds provider-specific filters (for aws_s3_bucket only region; aws_instance also accepts filter blocks).
  • By default only identities come back; include_resource = true also fetches attributes.
  • Only types the provider implements as list resources can be queried. Check: terraform providers schema -json | jq '.provider_schemas[].list_resource_schemas | keys'.
  • terraform validate -query also validates the .tfquery.hcl files.

In AWS provider 6.7x, list resources exist for S3 buckets, SSM parameters, VPCs, subnets, security groups, IAM roles, CloudWatch log groups and a few hundred more types. The list grows with each release.

terraform query

From search results to import blocks

terraform query -generate-config-out=FILE writes an import block (with identity) and a resource block for every result.

$ terraform query
list.aws_s3_bucket.all   bucket=tflab-state-123456789012 …
list.aws_s3_bucket.all   bucket=tflab-clickops-reports-… …
list.aws_ssm_parameter.all   name=/tflab/dev/clickops/…  …
# (exact columns depend on the provider)

$ terraform query -json | jq …        # machine-readable

$ terraform query -generate-config-out=found.tf
# found.tf: import { to = … identity = { … } }
#           resource "aws_s3_bucket" "all_0" { … }

Turning results into code

  • Delete everything you do not want to manage (other teams' buckets, the state bucket itself).
  • Rename generated labels (all_0) to meaningful names before the first apply: renaming later needs moved.
  • Trim and refactor the resource blocks exactly as with plan -generate-config-out.
  • Move the kept blocks into the right root module (one per component, ch 10).
  • Run terraform plan: expect only "N to import".
  • Terraform 1.17 can also evaluate Terraform Policy during a query (-policies, ch 9).
Adoption

Adopting a ClickOps account, step by step

Bulk import is easy; deciding ownership is the real work. Treat adoption as a small project, one component at a time.

1 · Inventory

  • terraform query per type, or AWS Resource Explorer / Config.
  • Tag owners; mark what is dead.
→

2 · Decide

  • Manage, delete or leave alone?
  • Which component and state owns it (ch 10)?
→

3 · Generate

  • -generate-config-out.
  • Trim, rename, use references and modules.
→

4 · Plan

  • Only "N to import, 0 to change".
  • Review in a pull request.
→

5 · Apply + lock down

  • Apply through CI.
  • Remove console write access (ch 13).

Import first, refactor second. Mixing "adopt" and "improve" in one plan hides real changes among dozens of imports. Ship the import with zero changes, then make the improvements visible in their own plans.

Adoption

Things that bite during imports

Most surprises come from resources that AWS models as one object but the provider splits into several.

Split resources

  • An S3 bucket's versioning, encryption, policy, public access block and lifecycle are separate resources: import each (aws_s3_bucket_versioning, …).
  • Security group rules: aws_vpc_security_group_ingress_rule objects are imported one per rule ID.
  • IAM roles: inline policies and attachments are separate resources too.

Defaults and drift

  • Arguments you omit can still show as changes if the provider sets defaults differently than the console did.
  • default_tags adds tags to every adopted resource: expect ~ tags_all updates on the first plan, which is usually fine and desired.
  • Some attributes cannot be read back (passwords): ignore_changes or write-only arguments.

Safety net

  • Add lifecycle { prevent_destroy = true } to precious adopted resources (ch 3).
  • Import into a new state first; move between states later if needed.
  • Never import the same object into two states: two owners will fight forever.
  • Keep the import blocks in the PR history as documentation.
Chapter 5 · Part 5.4

removed blocks and drift

Stop managing a resource without deleting it, and decide what to do when reality changes behind your back.

removed blocks

Forget a resource, keep the object

Since Terraform 1.7, a removed block drops an object from state through a normal plan. With destroy = false the real resource stays untouched.

# 1. delete the resource block from the code
# 2. add:
removed {
  from = aws_s3_bucket.legacy

  lifecycle {
    destroy = false
  }
}
$ terraform plan
 # aws_s3_bucket.legacy will no longer be managed by
 # Terraform, but will not be destroyed
 # (destroy = false is set in the configuration)

Plan: 0 to add, 0 to change, 0 to destroy.

Warning: Some objects will no longer be managed by Terraform
If you apply this plan, Terraform will discard its tracking
information for the following objects, but it will not
delete them:
 - aws_s3_bucket.legacy

Without a removed block, deleting the resource block means destroy. destroy = true (or omitting the lifecycle) makes the intent explicit and lets you attach destroy-time provisioners. from names a whole resource or module, never a single instance key.

removed blocks

Moving a resource between states

moved cannot cross state files. To hand a resource from one root module to another, forget it in the old one and import it in the new one.

# live/dev/network  (old owner)
removed {
  from = aws_security_group.app
  lifecycle {
    destroy = false
  }
}

# live/dev/app       (new owner)
import {
  to = aws_security_group.app
  id = "sg-0abc1234def567890"
}

resource "aws_security_group" "app" {
  name   = "tflab-dev-app"
  vpc_id = data.aws_ssm_parameter.vpc_id.insecure_value
}

Order of operations

  • 1 · Plan both: old shows "no longer managed", new shows "1 to import". Review together.
  • 2 · Apply the new owner first: now both states know the object, but only for a moment.
  • 3 · Apply the old owner: it forgets the object.
  • Never the other way round with destroy = true by mistake: plan text is the safeguard.
  • Fix references: anything in the old state that used the resource must read it from SSM or remote state now (ch 4).
Drift

Detecting drift with refresh-only plans

A refresh-only plan compares state with reality and proposes to update state only. It never changes infrastructure.

# someone adds a tag in the console
$ terraform plan -refresh-only

Note: Objects have changed outside of Terraform

Terraform detected the following changes made outside of
Terraform since the last "terraform apply":

  # aws_s3_bucket.reports has changed
  ~ resource "aws_s3_bucket" "reports" {
      ~ tags     = {
          + "Owner" = "alice"
        }
        # (11 unchanged attributes hidden)
    }

This is a refresh-only plan, so Terraform will not take
any actions to undo these. …

Three different questions

  • plan -refresh-only: what changed in reality since the last apply?
  • plan: what will Terraform change to make reality match the code (refresh included)?
  • apply -refresh-only: accept reality into state without touching AWS.
  • terraform refresh is a deprecated alias of apply -refresh-only -auto-approve: no review, avoid it.
  • plan -detailed-exitcode returns 2 when there are changes: the basis of nightly drift jobs (ch 11).
Drift

Reconciling drift: code it or revert it

Drift is a decision, not a bug report. Either the manual change was right and the code must learn it, or it was wrong and the next apply must undo it.

Code it (the change was right)

  • Update the resource block so a normal plan shows No changes.
  • Ship it through a PR: the review records why the change happened.
  • For attributes another system owns (autoscaling desired count, tags added by a tool): lifecycle { ignore_changes = [...] } (ch 3).

Revert it (the change was wrong)

  • Run a normal plan: it proposes to undo the manual change.
  • Apply it through the usual pipeline, not from a laptop.
  • Find out who changed it and why: CloudTrail, then fix the process (read-only console, ch 13).

Accept into state only

  • apply -refresh-only updates state, nothing else.
  • Useful when outputs or dependents must see the new value before code catches up.
  • Rarely the end of the story: the next normal plan will still revert the drift unless the code changes.

Do not "fix" drift by running apply blindly: a plan that undoes a colleague's emergency change at 3 am can cause the next incident. Read the plan, then decide.

Common mistakes

Refactoring mistakes that destroy things

Each of these turns a harmless rename or adoption into an outage.

Moves

  • Renaming a resource or module call without a moved block → destroy + create.
  • Mapping count indexes to the wrong for_each keys → -/+ replacements.
  • Deleting old moved blocks from a shared module → consumers on older versions see destroys.
  • Mixing a move and real changes in one plan, so the move hides them.

Imports

  • Applying an import plan that also shows ~ update: the import "fixes" reality to match sloppy code.
  • Committing -generate-config-out output unreviewed, full of literals and nulls.
  • Importing one object into two states → two owners overwrite each other.
  • Forgetting the split resources (bucket versioning, policies, SG rules).

Removal and drift

  • Deleting a resource block to "stop managing" it → Terraform destroys it.
  • terraform refresh or apply -refresh-only without reading the diff.
  • Applying old code over an emergency console fix.
  • Cross-state moves applied in the wrong order.
Self-check

Refactoring and adoption: can you answer these?

Explain each with the exact block, flag or plan line.

moved

  • What does the plan print for a pure rename with a moved block?
  • How do you turn aws_subnet.public[0] into aws_subnet.public["a"] safely?
  • Why can a moved block not move an object into another root module?
  • When may you delete moved blocks from a shared module?

import and query

  • What are the two ways to identify the object in an import block, and can you use both?
  • What does plan -generate-config-out refuse to do if the file exists?
  • Which file extension holds list blocks, and what does include_resource change?
  • Why should an import plan say "0 to change"?

removed and drift

  • What happens if you delete a resource block without a removed block?
  • In which order do you apply a cross-state move, and why?
  • What is the difference between plan, plan -refresh-only and apply -refresh-only?
  • Name two ways to resolve drift.

Practice: Lab 5 in Hands-on labs — adopt a ClickOps bucket and SSM parameters with import blocks and terraform query, rename with moved, let go with removed, then detect drift with a refresh-only plan.

Chapter 6

Modules

Package infrastructure into reusable building blocks with a small, typed interface, compose them per environment, and version them so upgrades are deliberate.

Chapter 6 · Part 6.1

Module basics

Every directory of .tf files is a module; a module block calls one with inputs and reads its outputs.

Module basics

Root modules and child modules

A module is a directory of .tf files. The directory you run Terraform in is the root module; every module block calls a child module.

tf-lab/
├── modules/                 # reusable child modules
│   ├── network/
│   │   ├── main.tf          # resources
│   │   ├── variables.tf     # inputs
│   │   ├── outputs.tf       # outputs
│   │   ├── versions.tf      # required_providers
│   │   ├── README.md
│   │   ├── examples/basic/  # a runnable root module
│   │   └── tests/           # *.tftest.hcl (ch 7)
│   └── app-bucket/
└── live/                    # root modules: one state each
    └── dev/
        ├── network/         # backend, provider, module "network"
        └── app/

What belongs where

  • Child module: resources, variable, output, required_providers. No backend, ideally no provider blocks.
  • Root module: backend, provider configuration, module calls, environment values.
  • File names are convention only: Terraform loads every *.tf in the directory.
  • Each root module has its own state; child modules live inside the caller's state.
  • Addresses gain a prefix: module.network.aws_vpc.this.
Module basics

Calling a module

The module block passes inputs as arguments and exposes outputs as module.<name>.<output>. Changing source or version requires terraform init.

# live/dev/network/main.tf
module "network" {
  source = "../../../modules/network"

  name = "tflab-dev"
  cidr = "10.10.0.0/16"
  public_subnets = {
    a = "10.10.0.0/24"
    c = "10.10.1.0/24"
  }
  private_subnets = {
    a = "10.10.10.0/24"
    c = "10.10.11.0/24"
  }
}

output "vpc_id" {
  value = module.network.vpc_id
}

Meta-arguments on module blocks

  • source (required), version (registry sources only).
  • count / for_each: several instances, module.app["dev"].
  • providers: which provider configurations the module uses.
  • depends_on: whole-module ordering. Avoid it: it makes every data source inside wait and can turn known values into "known after apply".
  • Outputs of a child are invisible to the outside world unless the root re-exports them.
Module basics

Module sources

The source string decides where terraform init downloads the module from. Local paths are used in place; everything else is copied into .terraform/modules/.

SourceExampleVersioning
Local path../../modules/networkSame commit as the caller
Public registryterraform-aws-modules/vpc/awsversion = "~> 6.7"
Private registryapp.terraform.io/acme/network/awsversion = "1.4.0"
GitHub shorthandgithub.com/acme/tf-modules//network?ref=network-v1.4.0
Generic Gitgit::https://example.com/tf.git//modules/network?ref=v1.4.0Tag, branch or commit SHA
Git over SSHgit::ssh://git@github.com/acme/tf.git//network?ref=…Same
S3 buckets3::https://s3-ap-northeast-1.amazonaws.com/bucket/network.zipObject key / version

// separates the repository from a subdirectory. version works only with registry sources; for Git, pin with ?ref= — ideally a tag you never move, or a full commit SHA for strict supply-chain hygiene (ch 12).

Module basics

Inputs and outputs: the module interface

Variables are the parameters, outputs are the return values. Typed objects with optional() keep calls short without losing validation.

# modules/network/variables.tf
variable "name" {
  type = string        # name prefix, e.g. tflab-dev
}
variable "cidr" {
  type = string
  validation {
    condition = (
      can(cidrnetmask(var.cidr)) && endswith(var.cidr, "/16")
    )
    error_message = "cidr must be an IPv4 /16."
  }
}
variable "public_subnets" {
  type = map(string)   # AZ suffix => CIDR
}
variable "flow_logs" {   # typed object, optional fields
  type = object({
    enabled        = optional(bool, false)
    retention_days = optional(number, 14)
  })
  default = {}
}
# modules/network/outputs.tf
output "vpc_id" {
  description = "ID of the VPC"
  value       = aws_vpc.this.id
}

output "public_subnet_ids" {
  description = "AZ suffix => subnet ID"
  value       = { for k, s in aws_subnet.public : k => s.id }
}

output "private_subnet_ids" {
  value = { for k, s in aws_subnet.private : k => s.id }
}

# Terraform 1.15+: declare the output type
output "subnet_count" {
  type  = number
  value = length(aws_subnet.public) + length(aws_subnet.private)
}

Return maps keyed by something stable (AZ suffix) rather than lists: callers can then for_each over them without index shifts.

Module basics

Providers inside modules

A child module declares which providers it needs; the caller decides which configurations it gets. With AWS provider v6, many multi-region cases need no aliases at all.

# modules/app-bucket/versions.tf
terraform {
  required_version = ">= 1.14"
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = ">= 6.0"     # a floor, not a pin
    }
  }
}
# a module that needs two configurations
terraform {
  required_providers {
    aws = {
      source                = "hashicorp/aws"
      configuration_aliases = [aws.replica]
    }
  }
}

# caller
module "logs" {
  source    = "../../modules/replicated-bucket"
  providers = { aws = aws, aws.replica = aws.osaka }
}

Guidelines

  • Modules state minimum versions (>=); root modules pin with ~> and the lock file (ch 3).
  • Never put provider blocks in a shared module: such modules cannot be used with count, for_each or depends_on, and removing the module call breaks destroy.
  • Without providers, the child inherits the default (un-aliased) configurations.
  • AWS provider v6: most resources accept region = "ap-northeast-3", so a module can take a region input instead of an aliased provider.
  • default_tags set in the root provider apply to every resource in every child module.
Chapter 6 · Part 6.2

Designing good modules

Small interfaces, sane defaults, and modules that encode decisions instead of wrapping resources one to one.

Module design

What makes a module worth having

A good module encodes your organization's decisions (encryption on, public access off, tags, naming) behind a small interface.

Good signs

  • One purpose you can say in a sentence: "a private, encrypted, versioned S3 bucket for an app".
  • Fewer inputs than the resources it creates have arguments.
  • Secure and compliant by default: passing no options gives the right answer.
  • Outputs everything a caller might wire to: IDs, ARNs, names.
  • Has an examples/ directory and tests (ch 7).

Bad signs

  • A thin wrapper around one resource that re-exposes every argument as a variable.
  • A "god module" that builds VPC, cluster, databases and DNS in one call: huge blast radius.
  • Boolean flags that switch between unrelated designs (create_vpc, use_existing_vpc, legacy_mode).
  • Wrappers of wrappers: three layers to change a tag.
  • Reading global state inside (data sources for "the" VPC) instead of taking inputs.

Rule of thumb

  • Write the resources in a root module first.
  • Extract a module the second time you need the same pattern, with moved blocks (ch 5).
  • Start private (inside the repo); publish and version only when other teams depend on it.
  • Prefer composition (several small modules wired in the root) over one configurable mega-module.
Module design

modules/app-bucket: decisions built in

The caller chooses a name and a few knobs. Encryption, versioning, public access block and tags are not optional.

# modules/app-bucket/main.tf
resource "aws_s3_bucket" "this" {
  bucket        = "tflab-${var.env}-${var.name}"
  force_destroy = var.force_destroy
}
resource "aws_s3_bucket_public_access_block" "this" {
  bucket                  = aws_s3_bucket.this.id
  block_public_acls       = true
  block_public_policy     = true
  ignore_public_acls      = true
  restrict_public_buckets = true
}
resource "aws_s3_bucket_versioning" "this" {
  bucket = aws_s3_bucket.this.id
  versioning_configuration {
    status = var.versioning ? "Enabled" : "Suspended"
  }
}
resource "aws_s3_bucket_server_side_encryption_configuration" "this" {
  bucket = aws_s3_bucket.this.id
  rule {
    apply_server_side_encryption_by_default {
      sse_algorithm     = var.kms_key_arn == null ? "AES256" : "aws:kms"
      kms_master_key_id = var.kms_key_arn
    }
  }
}
# modules/app-bucket/variables.tf
variable "env" {
  type = string
}
variable "name" { # bucket = tflab-<env>-<name>
  type = string
  validation {
    condition     = can(regex("^[a-z0-9-]{3,40}$", var.name))
    error_message = "name: 3-40 chars, a-z 0-9 and -."
  }
}

variable "versioning" {
  type    = bool
  default = true
}
variable "kms_key_arn" {
  type    = string
  default = null       # null → SSE-S3 (AES256)
}
variable "force_destroy" {
  type    = bool
  default = false      # labs may set true
}
# outputs.tf
output "bucket_name" { value = aws_s3_bucket.this.bucket }
output "bucket_arn" { value = aws_s3_bucket.this.arn }
Module design

Interface rules that age well

The interface is a contract with every caller. Changing it is a breaking change; design it so most changes are additive.

Inputs

  • Required inputs only for what has no sensible default (name, CIDR).
  • Group related settings in an object({...}) with optional(type, default) fields: new fields stay backward-compatible.
  • Add validation for formats and ranges; error messages should say how to fix it.
  • nullable = false when null would be meaningless.
  • Mark secrets sensitive = true (or ephemeral = true, ch 2).

Outputs

  • Output IDs and ARNs of everything a caller may reference.
  • Prefer maps keyed by stable names over lists.
  • Do not output whole resource objects: the interface then changes with every provider release.
  • Document each output with description.
  • Terraform 1.15+: declare type on outputs to catch accidental shape changes.

Evolution

  • Rename an input: add the new one, mark the old one deprecated = "Use x instead." (1.15+), remove it in the next major version.
  • Rename internal resources: ship moved blocks in the module.
  • Never change a default that alters existing infrastructure in a minor version.
  • Keep a CHANGELOG; every breaking change bumps the major version.
Module design

Deprecating an input without breaking callers

Terraform 1.15 added deprecated to variable and output blocks: callers get a warning, not an error, while they migrate.

# modules/app-bucket/variables.tf (v1.3.0)
variable "enable_versioning" {
  type       = bool
  default    = null
  deprecated = "Use versioning instead; removed in v2.0.0."
}

variable "versioning" {
  type    = bool
  default = true
}

locals {
  versioning = coalesce(var.enable_versioning, var.versioning)
}
# a caller still on the old name
$ terraform validate
Warning: Deprecated variable got a value

  on main.tf line 4, in module "logs":
   4:   enable_versioning = true

Use versioning instead; removed in v2.0.0.

Success! The configuration is valid, but there were
some validation warnings as shown above.

Referencing a deprecated output prints "Deprecated value used". Older Terraform versions reject the deprecated argument, so a module that uses it needs required_version = ">= 1.15".

Chapter 6 · Part 6.3

Composition and reuse

Wire small modules together in root modules, use public modules carefully, and stamp out copies with for_each.

Composition

Composing network and app across two states

Each root module is one component with one state. Components talk through published values (SSM parameters here), not through each other's internals.

live/dev/network (state: dev/network)module "network"modules/networkaws_ssm_parameter ×N/tflab/dev/network/*VPC, subnets, IGWroute tablesoutputslive/dev/app (state: dev/app)module "assets"modules/app-bucketdata "aws_ssm_parameter"vpc_idaws_security_grouptflab-dev-apppublished in SSM Parameter Store

Why SSM instead of terraform_remote_state: the app needs read access to one parameter, not to the whole network state (which may contain secrets). Chapter 4 compares the options; chapter 10 covers ordering between components.

Composition

The app root module

The root module wires inputs from the environment, published values and module outputs together. It contains almost no resource logic of its own.

# live/dev/app/main.tf
data "aws_ssm_parameter" "vpc_id" {
  name = "/tflab/dev/network/vpc_id"
}

module "assets" {
  source = "../../../modules/app-bucket"

  env           = "dev"
  name          = "assets-${local.acct}" # S3 names are global
  force_destroy = true          # lab only
}

data "aws_caller_identity" "me" {}
locals {
  acct = data.aws_caller_identity.me.account_id
}

resource "aws_security_group" "app" {
  name        = "tflab-dev-app"
  description = "App tier (no rules yet)"
  vpc_id      = data.aws_ssm_parameter.vpc_id.insecure_value
}

output "bucket" { value = module.assets.bucket_name }

Reading it

  • Two data sources, one module call, one resource: the shape of most root modules.
  • data.aws_ssm_parameter…insecure_value is known at plan time (read during plan), so the security group plan is complete.
  • The bucket name includes the account ID: S3 names are global.
  • Same code for prod; only names and the SSM path change (ch 10 removes even that duplication).
  • Dependencies are implicit: Terraform reads SSM before planning the security group.
Composition

Many instances of a module with for_each

A module call supports for_each like a resource. Each instance gets its own address prefix and its own resources.

locals {
  buckets = {
    uploads = { versioning = true }
    exports = { versioning = false }
  }
}

module "bucket" {
  source   = "../../../modules/app-bucket"
  for_each = local.buckets

  env        = "dev"
  name       = "${each.key}-${local.acct}"
  versioning = each.value.versioning
}

output "bucket_arns" {
  value = { for k, m in module.bucket : k => m.bucket_arn }
}
$ terraform plan
  # module.bucket["exports"].aws_s3_bucket.this
  #   will be created
  # module.bucket["exports"].aws_s3_bucket_versioning.this
  #   will be created
  ...
  # module.bucket["uploads"].aws_s3_bucket.this
  #   will be created
Plan: 8 to add, 0 to change, 0 to destroy.

# turning a single call into for_each later:
moved {
  from = module.assets
  to   = module.bucket["assets"]
}

The keys become part of every address inside the module. Choose them like resource names: stable and meaningful (uploads), never list indexes or values that may change.

Composition

Public registry modules: use, but verify

Community modules such as terraform-aws-modules/vpc/aws (6.7.x in Sep 2026, for AWS provider v6) save weeks. They are also third-party code with broad permissions over your account.

module "vpc" {
  source  = "terraform-aws-modules/vpc/aws"
  version = "~> 6.7"      # allow 6.x patches/minors, not 7.0

  name = "tflab-dev"
  cidr = "10.10.0.0/16"

  azs             = ["ap-northeast-1a", "ap-northeast-1c"]
  public_subnets  = ["10.10.0.0/24", "10.10.1.0/24"]
  private_subnets = ["10.10.10.0/24", "10.10.11.0/24"]

  enable_nat_gateway = false   # NAT costs money; labs never create one
}

Checklist before adopting one

  • Read the source of the version you pin, not only the README.
  • Check maintenance: recent releases, open issues, provider v6 support.
  • Pin a version; let Renovate/Dependabot propose upgrades as PRs (ch 11).
  • Read the upgrade guide for every major version: they often need moved blocks or state moves.
  • Large modules expose 100+ inputs: wrap them in a thin opinionated module of your own only if it removes decisions, not just renames them.
  • Run Checkov/Trivy on the plan, not only on your code: module defaults matter (ch 9).
Composition

Variables in module source and version (Terraform 1.15)

Module sources are resolved by terraform init, before normal evaluation. Terraform 1.15 allows const variables (and locals built from them) there.

variable "modules_ref" {
  type    = string
  default = "app-bucket-v1.1.0"
  const   = true      # value must be known at init
}

locals {
  repo = "git::https://github.com/<you>/tf-lab.git"
}

module "assets" {
  source = "${local.repo}//modules/app-bucket?ref=${var.modules_ref}"
  env    = "dev"
  name   = "assets-123456789012"
}

# terraform init -var modules_ref=app-bucket-v1.2.0

What it is for, and the limits

  • One place to bump a module version for every call in a root module.
  • Without const = true, init fails: "Only literal values and const variables can be evaluated during init."
  • Locals work when they are built only from literals and const variables.
  • Changing the value means terraform init again, like any source change.
  • Requires Terraform 1.15+; older versions (and tools that parse HCL statically) still expect literals.
  • Do not use it to pick different code per environment: environments should differ in inputs, not modules (ch 10).
Chapter 6 · Part 6.4

Versioning and publishing

Semantic versions, Git tags or a registry, changelogs and upgrade paths that never surprise a caller.

Versioning

Semantic versioning for modules

A version number is a promise about the plan a caller will see after upgrading.

Change in the moduleBumpWhat the caller's plan shows
Bug fix, documentation, new optional outputPatch 1.4.1No changes
New optional input with a default that keeps today's behaviorMinor 1.5.0No changes
Internal rename shipped with moved blocksMinorMoves only, 0 to change
New resource created by default (e.g. a log group)Minor, announce it+ creates only
Removed or renamed input/output, new required inputMajor 2.0.0Errors until the caller changes code
Anything that replaces or destroys existing resourcesMajor + upgrade guide-/+ or -: must be planned
Raising the minimum Terraform or provider versionMajor (or minor with notice)init may fail on old versions

Callers constrain with ~> 1.4 (any 1.x ≥ 1.4) for registry modules, or pin a tag with ?ref=. In a monorepo with local paths, there is no version: every commit is a release, and tests must catch breaking changes (ch 7).

Versioning

Publishing from Git: tags per module

In a repository with several modules, prefix tags with the module name so each module has its own release line.

# release app-bucket 1.1.0
git switch main && git pull
git log --oneline -- modules/app-bucket   # what changed?
git tag -a app-bucket-v1.1.0 \
  -m "app-bucket: add kms_key_arn input"
git push origin app-bucket-v1.1.0

# consume it (GitHub shorthand = git over HTTPS)
module "assets" {
  source = "github.com/<you>/tf-lab//modules/app-bucket?ref=app-bucket-v1.1.0"
  env    = "prod"
  name   = "assets-123456789012"
}

# strictest: a full commit SHA (tags can be moved)
#   ?ref=4f1c2a9e0b…

Monorepo vs module repos

  • One repo, local paths (tf-lab): simplest; a change affects every environment on its next plan, which is why plans run per environment (ch 10–11).
  • One repo, tagged module refs: environments upgrade independently (dev on v1.2, prod on v1.1).
  • One repo per module (terraform-aws-network): required by the public registry naming rules; most overhead.
  • Private registry (HCP Terraform, or any registry implementing the module registry protocol): gives real version constraints and discovery.
Versioning

Promotion and upgrade workflow

Treat a module upgrade like an application release: change dev first, read the plan, then promote the exact same version.

Release

  • Merge to main.
  • Tests pass (ch 7).
  • Tag app-bucket-v1.2.0, CHANGELOG entry.
→

Dev

  • Bump ref in live/dev/app.
  • terraform init -upgrade.
  • Plan: expected diff only.
→

Review

  • Plan reviewed in PR (ch 8).
  • Apply via CI.
  • Watch for a day.
→

Prod

  • Same tag in live/prod/app.
  • Plan should match dev's shape.
  • Apply with approval.

terraform init -upgrade re-downloads modules (and providers within constraints). Without it, a changed ?ref= is still fetched on init, but a floating constraint like ~> 1.4 keeps the version already installed in .terraform/modules.

Versioning

A module README that answers the real questions

Generate the reference part with terraform-docs; write the decisions part by hand.

# modules/app-bucket/README.md
## app-bucket
Private, encrypted, versioned S3 bucket for one app.

### Decisions (not configurable)
- Public access fully blocked
- Encryption: SSE-S3, or SSE-KMS when kms_key_arn is set
- Versioning on by default

### Usage
module "uploads" {
  source = "../../modules/app-bucket"
  env    = "dev"
  name   = "uploads-123456789012"
}

<!-- BEGIN_TF_DOCS -->
(inputs, outputs, providers: generated)
<!-- END_TF_DOCS -->

Keep it true automatically

  • terraform-docs markdown table --output-file README.md --output-mode inject modules/app-bucket fills the marked block.
  • Run it in pre-commit or CI and fail on a diff, so docs never drift from code.
  • examples/basic is a root module that CI can plan (or test) on every change.
  • Document upgrade steps for every major version next to the CHANGELOG.
Common mistakes

Module mistakes that hurt later

Most module pain shows up months later, during an upgrade or a second environment.

Design

  • One-resource wrappers that re-export every argument.
  • A mega-module for a whole environment: every change plans everything.
  • provider blocks inside shared modules.
  • Lists of subnets as outputs; callers index them and break on reorder.
  • Data sources that look up "the VPC" by tag inside the module.

Versioning

  • Git sources without ?ref=: every init gets the latest default branch.
  • Moving a tag after release; consumers get different code under the same version.
  • Internal renames released without moved blocks → destroys for every caller.
  • Changing a default in a minor release that replaces resources.

Usage

  • Forgetting terraform init after changing source or version.
  • Using module depends_on to "fix" ordering, making plans show (known after apply) everywhere.
  • Public modules adopted without reading their source or upgrade guides.
  • version on a Git source (only registry sources support it).
Self-check

Modules: can you answer these?

Answer with the exact block, argument or command.

Basics

  • What makes a directory a root module rather than a child module?
  • Which source types support version, and how do you pin a Git source?
  • What does // mean in a module source?
  • Why should shared modules not contain provider blocks?

Design

  • Why return maps keyed by AZ suffix instead of lists?
  • How do optional() object attributes help evolve an interface?
  • How do you rename an input without breaking callers (1.15+)?
  • What are two signs that a module should not exist?

Versions and reuse

  • Which module changes require a major version?
  • What does terraform init -upgrade change?
  • Why must the module source variable be const?
  • How do you turn module.assets into module.bucket["assets"] with no changes?

Practice: Lab 6 in Hands-on labs — extract modules/network and modules/app-bucket from the flat Lab 3 code with moved blocks (zero-change plan), build live/dev/app from SSM inputs, then tag a module version and consume it with ?ref=.

Chapter 7

Testing Terraform

Catch mistakes before a plan reaches a reviewer: formatting and validation, TFLint, terraform test with mocks, and the slower tests that touch real infrastructure.

Chapter 7 · Part 7.1

Static checks

Seconds-fast checks that need no cloud credentials: run them on every save and every pull request.

Testing pyramid

What each layer of Terraform testing catches

Cheap checks run often and catch most mistakes; expensive checks run rarely and catch what only real cloud APIs reveal.

Integrationreal apply in a sandboxterraform testplan asserts, mocks, expect_failuresStatic checksfmt, validate, TFLint, scanners (ch 9)slower,costlier,more real

Where each runs

  • Static (seconds): pre-commit hook and every PR. No credentials, no state.
  • **terraform test with command = plan and mocks** (seconds): every PR. Provider schema only; no API calls.
  • **terraform test with command = apply** against real AWS (minutes): nightly or on module release, in a sandbox account.
  • Plan review and policy (ch 8–9): every PR, on the real plan of each environment.
  • Rule of thumb: every bug you find in review becomes a test or a policy, so it is caught one layer lower next time.
Static checks

fmt and validate: the zero-cost gate

fmt makes diffs readable; validate proves the configuration is internally consistent. Neither talks to AWS.

# formatting: fail CI if anything is not canonical
terraform fmt -check -recursive -diff
# exit 0 = formatted, 3 = files need formatting

# fix locally
terraform fmt -recursive

# validate needs providers, not a backend:
terraform init -backend=false
terraform validate
# Success! The configuration is valid.

# machine-readable result for CI
terraform validate -json | jq '.valid, .error_count'

validate: checks and blind spots

  • Checks: syntax and block structure; references to undeclared variables, resources, locals.
  • Checks: argument names, types and required arguments against the provider schema.
  • Misses: values only AWS knows (does the AMI exist? is the bucket name free?).
  • Misses: variable validation with values from tfvars (runs at plan time).
  • Misses: whether an instance type is real (→ TFLint) and security posture (→ Checkov, Trivy, ch 9).
TFLint

TFLint: the linter that knows AWS

TFLint adds rules that validate cannot know: invalid instance types, deprecated syntax, unused declarations, missing version constraints.

# .tflint.hcl (repo root)
plugin "terraform" {
  enabled = true
  preset  = "recommended"
}

plugin "aws" {
  enabled = true
  version = "0.50.0"   # pin; check the latest release
  source  = "github.com/terraform-linters/tflint-ruleset-aws"
}
tflint --init            # downloads the aws plugin
tflint --recursive       # every directory with .tf files

# 3 issue(s) found:
# Warning: `unused` variable has no type
#          (terraform_typed_variables)
# Warning: [Fixable] variable "unused" is declared
#          but not used (terraform_unused_declarations)
# Error: "t3.mircro" is an invalid value as
#        instance_type (aws_instance_invalid_type)

tflint --format compact   # one line per issue (CI logs)
tflint --fix              # auto-fix the [Fixable] ones
tflint --minimum-failure-severity=error   # warnings pass

Exit codes: 0 = no issues; 2 = issues found (warnings included unless you raise --minimum-failure-severity). The terraform ruleset is bundled; the aws ruleset is a plugin you pin and download with --init.

TFLint

Rules worth knowing

The recommended terraform preset plus the AWS ruleset cover most of what reviewers otherwise comment on by hand.

RuleRulesetCatches
terraform_required_versionterraformRoot module without required_version
terraform_required_providersterraformProvider used without a version constraint
terraform_typed_variablesterraformvariable without type
terraform_unused_declarationsterraformVariables, locals, data sources nobody references
terraform_deprecated_interpolationterraform"${var.x}" where var.x is enough
aws_instance_invalid_typeawsTypos in instance_type (t3.mircro)
aws_*_invalid_* (many)awsEnum-like arguments with impossible values, checked against AWS API models
Deep checking (deep_check = true)awsCalls AWS APIs (needs credentials) to check e.g. that an AMI or IAM profile exists

Disable a rule for one line with a comment above it: # tflint-ignore: aws_instance_invalid_type. Prefer fixing; when you must ignore, say why in the same comment.

Pre-commit

Run the static checks before every commit

The same commands in a pre-commit hook and in CI: developers see failures in seconds, CI enforces them for everyone.

# .pre-commit-config.yaml
repos:
  - repo: https://github.com/antonbabenko/pre-commit-terraform
    rev: <latest tag>          # pin a release tag or SHA
    hooks:
      - id: terraform_fmt
      - id: terraform_validate
      - id: terraform_tflint
        args: ["--args=--config=__GIT_WORKING_DIR__/.tflint.hcl"]
      - id: terraform_docs     # keeps README inputs/outputs in sync
      - id: terraform_trivy    # IaC scan (ch 9)

# install once per clone
pipx install pre-commit
pre-commit install
pre-commit run --all-files

Keep it fast

  • Hooks run only on changed files; --all-files in CI.
  • terraform_validate runs init -backend=false per directory: cache providers with TF_PLUGIN_CACHE_DIR.
  • Put slow tests (terraform test with real apply) in CI only, never in the hook.
  • The hook is a convenience; CI is the gate. Developers can skip hooks with --no-verify.
Chapter 7 · Part 7.2

terraform test

The built-in test framework (1.6+): run plans or applies of a module and assert on the result, with mocked providers when you want no cloud at all.

terraform test

Anatomy of a test file

Test files end in .tftest.hcl and live in tests/ (or next to the module). Each run block plans or applies the module and checks assertions.

# modules/app-bucket/tests/unit.tftest.hcl

variables {                 # defaults for every run block
  env  = "dev"
  name = "logs"
}

run "name_follows_convention" {
  command = plan            # default is apply

  assert {
    condition     = aws_s3_bucket.this.bucket == "tflab-dev-logs"
    error_message = "bucket name must be tflab-<env>-<name>"
  }
}

run "rejects_uppercase_name" {
  command = plan
  variables {
    name = "Logs"           # overrides the file-level value
  }
  expect_failures = [var.name]   # validation MUST fail
}

How it runs

  • terraform test runs every *.tftest.hcl in the module dir and tests/.
  • Run blocks execute in file order and share state by default: a later run sees what an earlier apply created.
  • After the file, Terraform destroys everything the runs applied, in reverse order.
  • State is in memory: your backend and real state are never touched.

Assertions can reference

  • Resources and data sources of the module: aws_s3_bucket.this.bucket.
  • Outputs, variables and earlier runs: output.bucket_arn, var.env, run.setup.vpc_id.
  • After a plan, values only known after apply cannot be asserted (unless mocked, next slides).
terraform test

Running tests and reading the result

A failing assertion shows the condition, the actual vs expected values, and your error message.

terraform init -backend=false
terraform test

# tests/unit.tftest.hcl... in progress
#   run "name_follows_convention"... pass
#   run "versioning_on_by_default"... pass
#   run "rejects_uppercase_name"... pass
#   run "rejects_unknown_env"... pass
#   run "apply_with_mocks"... pass
# tests/unit.tftest.hcl... tearing down
# tests/unit.tftest.hcl... pass
#
# Success! 5 passed, 0 failed.

# useful flags
terraform test -filter=tests/unit.tftest.hcl
terraform test -verbose          # print plan/state per run
terraform test -junit-xml=test.xml   # CI test report
terraform test -var 'env=prod'
#   run "subnet_plan"... fail
#
# Error: Test assertion failed
#
#   on tests/network.tftest.hcl line 17,
#   in run "subnet_plan":
#   17:  condition = aws_subnet.private[
#          "ap-northeast-1a"].cidr_block
#          == "10.10.10.0/24"
#     ├────────────────
#     │ Diff:
#     │ --- actual
#     │ +++ expected
#     │ - "10.10.20.0/24"
#     │ + "10.10.10.0/24"
#
# private subnet in 1a must be 10.10.10.0/24
#
# Failure! 2 passed, 1 failed.

Exit code is non-zero when any run fails: CI can gate on it directly. Output above is from Terraform 1.15 with the AWS provider 6.x and mocks.

terraform test

Run block reference

Everything a run block can set. Most tests need only command, variables and assert.

Attribute / blockPurpose
command = plan | applyPlan only (fast, nothing created) or apply (default)
variables { … }Inputs for this run; override file-level variables
assert { condition, error_message }One or more checks; all are evaluated and reported
expect_failures = [var.x, resource.y, check.z]The run passes only if these checkable objects fail (validation, pre/postconditions, check blocks)
module { source = "./tests/setup" }Run a helper module instead of the module under test (create prerequisites, then reference run.<name>.<output>)
plan_options { mode, refresh, replace, target }Same as CLI flags: refresh-only mode, -refresh=false, -replace, -target
providers = { aws = aws.tokyo }Pass specific provider configurations to the run
state_key = "x"Choose which in-memory state the run uses (1.9+): runs with the same key share state
parallel = trueLet independent run blocks execute concurrently

A test { parallel = true } block at the top of a file sets the default for all its runs; a run can override it with parallel = false.

Mocks

mock_provider: tests with no cloud and no credentials

A mocked provider keeps the real schema (types, required arguments) but never calls an API. Computed attributes get fake values you can control.

mock_provider "aws" {
  # optional: fixed values for computed attributes
  mock_resource "aws_s3_bucket" {
    defaults = {
      arn = "arn:aws:s3:::tflab-dev-logs"
    }
  }
  mock_data "aws_caller_identity" {
    defaults = { account_id = "123456789012" }
  }
}

run "apply_with_mocks" {
  # command = apply (default) against the MOCK:
  # nothing is created in AWS
  assert {
    condition     = output.bucket_arn == "arn:aws:s3:::tflab-dev-logs"
    error_message = "arn output comes from the mocked bucket"
  }
}

What mocks give you

  • apply runs in seconds and costs nothing: computed values (id, arn) become known.
  • Without defaults, Terraform generates values of the right type (random strings for id).
  • Provider still needs terraform init (it reads the schema), but no credentials.
  • Good for: naming, tagging, for_each keys, outputs, conditional resources, validation.

What mocks cannot prove

  • That AWS accepts the request (bucket names globally unique, IAM policy syntax, quotas).
  • Eventual-consistency and ordering problems.
  • Provider bugs or diffs after apply (perpetual diffs).
  • → keep a few real apply tests in a sandbox (section 27).
Mocks

Overrides: pin one resource, data source or module

override_* blocks replace the result of one specific object. They work with real or mocked providers, at file level or inside one run.

mock_provider "aws" {}

run "ids_from_overrides" {
  override_resource {
    target = aws_vpc.this
    values = { id = "vpc-0123456789abcdef0" }
  }

  assert {
    condition     = output.vpc_id == "vpc-0123456789abcdef0"
    error_message = "vpc_id output must come from aws_vpc.this"
  }
  assert {
    condition = alltrue([for s in aws_subnet.private :
                  s.vpc_id == "vpc-0123456789abcdef0"])
    error_message = "every private subnet must be in the VPC"
  }
}

# also available:
# override_data   { target = data.aws_availability_zones.this
#                   values = { names = ["ap-northeast-1a"] } }
# override_module { target  = module.network
#                   outputs = { vpc_id = "vpc-123" } }

override_during

  • By default mocked and overridden values are applied at apply time; during a plan they stay "known after apply".
  • override_during = plan (on mock_provider or an override_* block) makes them known at plan time, so command = plan runs can assert on IDs.
  • Without it, asserting on output.vpc_id after a plan fails because the value is unknown.

Rules of thumb

  • Mock at the boundary: other teams' modules, data sources that read live accounts, slow resources.
  • Do not mock the thing you are testing.
  • Prefer override_data over real data sources in unit tests: tests must not depend on what exists in an account.
terraform test

Plan tests vs apply tests

Choose per run block. Most unit tests should be plans; a few apply tests prove the module really works.

command = plancommand = apply + mockscommand = apply real
SpeedSecondsSecondsMinutes (real API calls)
CredentialsNone with mocks; read-only with real providerNoneSandbox account role
Cost / riskNoneNoneCreates and destroys real resources
Computed values (id, arn)Unknown (unless override_during = plan)Known (mocked)Known (real)
ProvesLogic: names, counts, conditions, validationWiring between resources and outputsAWS accepts it; no perpetual diff
Run onEvery PREvery PRNightly / before a module release

After a real apply test, check the cleanup: if a destroy fails, Terraform prints the objects it could not remove. A sandbox account with a budget alarm and tag-based clean-up is the safety net.

terraform test

Setup modules and multi-step tests

A run can apply a helper module to create prerequisites; later runs read its outputs through run.<name>.

# modules/app-bucket/tests/integration.tftest.hcl
# REAL apply in the sandbox account (nightly job)

provider "aws" {
  region = "ap-northeast-1"
}

run "setup" {
  module { source = "./tests/setup" }  # random suffix
}

run "create_bucket" {
  variables {
    env  = "dev"
    name = "it-${run.setup.suffix}"   # globally unique
  }
  assert {
    condition = startswith(output.bucket_name,
                           "tflab-dev-it-")
    error_message = "unexpected bucket name"
  }
}
# after the file: both runs destroyed, newest first
# modules/app-bucket/tests/setup/main.tf
terraform {
  required_providers {
    random = { source = "hashicorp/random" }
  }
}

resource "random_id" "suffix" {
  byte_length = 3
}

output "suffix" {
  value = random_id.suffix.hex   # e.g. "a1b2c3"
}

Real-apply tests need credentials and a unique name per run (S3 bucket names are global). Never point them at a shared or production account.

Chapter 7 · Part 7.3

Beyond unit tests

Integration tests, contract tests, upgrade tests and cost checks: the slower layer that catches what mocks cannot.

Integration

Integration tests in a sandbox account

Apply the module for real, check the result through the AWS API, destroy. Slow, so run them on a schedule and before module releases.

Design

  • A dedicated sandbox account, not dev: tests may leave debris when they fail.
  • A CI role that can create only what the modules need, with a permission boundary.
  • Unique names per run (random_id, run ID) and a Project=tf-lab-test tag.
  • A scheduled janitor (e.g. aws-nuke with a tag filter) for leftovers.
  • Budget alarm on the account.

What to check

  • The apply succeeds and a second plan is empty (no perpetual diff): terraform plan -detailed-exitcode → exit 0.
  • Properties only AWS can confirm: bucket encryption really on (aws s3api get-bucket-encryption), public access blocked.
  • Outputs are usable by consumers (contract, next slide).
  • Destroy works: no prevent_destroy or dependency surprises.

Tools

  • terraform test with command = apply and a real provider: native, HCL only.
  • Terratest (Go library from Gruntwork): calls terraform and the AWS SDK; good for HTTP checks, retries, complex assertions.
  • Kitchen-Terraform and similar Ruby tools: legacy; prefer the first two.
  • Whatever you choose: same tests locally and in CI.
Terratest

Terratest in one slide

A Go test that applies the module, asserts with the AWS SDK and always destroys. Use it when HCL assertions are not enough.

// test/app_bucket_test.go
package test

import (
	"strings"
	"testing"

	"github.com/gruntwork-io/terratest/modules/aws"
	"github.com/gruntwork-io/terratest/modules/random"
	"github.com/gruntwork-io/terratest/modules/terraform"
)

func TestAppBucket(t *testing.T) {
	t.Parallel()
	// UniqueId can contain upper case; S3 names cannot
	name := "it-" + strings.ToLower(random.UniqueId())
	opts := &terraform.Options{
		TerraformDir: "../modules/app-bucket/examples/basic",
		Vars:         map[string]any{"env": "dev", "name": name},
	}
	defer terraform.Destroy(t, opts) // always clean up
	terraform.InitAndApply(t, opts)

	bucket := terraform.Output(t, opts, "bucket_name")
	aws.AssertS3BucketExists(t, "ap-northeast-1", bucket)
}

Notes

  • defer terraform.Destroy runs even when an assertion fails.
  • Run with go test -v -timeout 30m ./test/; the default Go test timeout (10 min) is too short for most infrastructure.
  • Test an examples/ root module, not the module directly: it also documents usage.
  • Terratest has helpers for AWS, Kubernetes, HTTP retries, SSH.
Contracts

Contract tests: protect what consumers rely on

A module's outputs and inputs are an API. Contract tests fail when a change would break callers, before a new version is released.

# modules/network/tests/contract.tftest.hcl
mock_provider "aws" {
  override_during = plan
  mock_data "aws_availability_zones" {
    defaults = { names = ["ap-northeast-1a", "ap-northeast-1c"] }
  }
}

variables {
  env  = "dev"
  cidr = "10.10.0.0/16"
}

run "outputs_contract" {
  command = plan

  assert {
    condition     = can(regex("^vpc-", output.vpc_id))
    error_message = "vpc_id must stay a VPC ID"
  }
  assert {
    condition     = length(output.private_subnet_ids) == 2
    error_message = "callers expect one private subnet per AZ"
  }
}

What a contract covers

  • Output names and types (renaming vpc_id breaks every caller).
  • Shapes: list vs map, number of elements.
  • Required inputs: adding a required variable is a breaking change → new major version (ch 6).
  • Resource addresses callers may reference with moved blocks or imports.

Upgrade tests

  • Apply version N in the sandbox, then switch the source to N+1 and plan.
  • Expected: no replacements of stateful resources; moved blocks cover renamed addresses.
  • Same idea for provider upgrades (ch 13): plan with the new provider against existing state.
  • Automate it for modules many teams use.
Cost

Cost estimation as a check

Infracost reads Terraform code or a plan and prices it from public price lists. Useful in PRs, not a bill.

# one-time: free API key
infracost auth login

# price a directory
infracost breakdown --path live/dev/network

# show the cost difference a PR introduces
infracost breakdown --path . \
  --format json --out-file base.json   # on main
infracost diff --path . --compare-to base.json \
  --format json --out-file diff.json   # on the PR branch

# in CI: post the diff as a PR comment
infracost comment github --path diff.json ...

Use it well

  • Catches the classic surprises: a NAT gateway per AZ, a large instance type, provisioned IOPS, an extra load balancer.
  • Usage-based costs (S3 requests, data transfer, Lambda invocations) need a usage file; otherwise they show as 0.
  • Treat the number as an estimate for review, not a guarantee.
  • For tf-lab the estimate should be ~0 USD/month: a non-zero line usually means someone added a NAT gateway or EC2.

Other checks that fit here

  • Policy on cost: a conftest/Terraform Policy rule that denies aws_nat_gateway in dev (ch 9).
  • Tags: every resource has Project and Env (ch 9) so the bill can be split.
  • Budgets: AWS Budgets alarm per account; the real backstop.
Common mistakes

Testing mistakes

Each of these lets a broken module reach main looking green.

Tests that prove nothing

  • Asserting on values copied from the code (bucket == var.name) instead of the rule (tflab-<env>-<name>).
  • Plan tests asserting on computed values: they fail or are skipped unless mocked with override_during = plan.
  • expect_failures on the wrong object: the run passes for a different failure.
  • Only happy paths: no test for the validation that protects prod.

Tests that hurt

  • Real-apply tests in the dev account → debris, name clashes, quota hits.
  • Fixed names in real tests (tflab-dev-test) → parallel runs collide; S3 names are global.
  • Integration tests on every PR → 20-minute pipelines and people skipping them.
  • Ignoring a failed destroy at the end of a run.

Habits that work

  • fmt -check, validate, tflint and plan-mode terraform test with mocks on every PR (< 1 min).
  • Real applies nightly in a sandbox with a janitor and a budget.
  • One test per bug found in review.
  • Contract tests on every module output other teams use.
Self-check

Testing: can you answer these?

Answer out loud with the exact command, block or attribute.

Static

  • What exit code does terraform fmt -check return when files need formatting?
  • Why does terraform validate not catch instance_type = "t3.mircro"?
  • How do you pin the TFLint AWS ruleset, and what does tflint --init do?
  • Which TFLint exit code fails CI on warnings, and how do you change that?

terraform test

  • What is the default command of a run block?
  • When are resources created by apply runs destroyed?
  • How do you assert that a variable validation rejects an input?
  • Why does a plan-mode run fail when asserting on output.vpc_id, and how does override_during = plan fix it?
  • What does run.setup.suffix refer to?

Beyond

  • Name three things only a real apply test can prove.
  • What makes a module change "breaking" for callers?
  • Why should integration tests run in a separate sandbox account?
  • What does Infracost not price unless you give it usage data?

Practice: Lab 7 in Hands-on labs — TFLint with the AWS ruleset, then terraform test for modules/app-bucket and modules/network with plan asserts, expect_failures and mocked providers, all without touching AWS.

Chapter 8

Plan risk review

The plan is the last point where a mistake is free. Read it like a diff of production, turn it into data, know which changes are dangerous, and apply exactly what was reviewed.

Chapter 8 · Part 8.1

Reading a plan

Symbols, annotations and the summary line: what Terraform is telling you it will do, object by object.

Reading a plan

The symbols, and why order matters

Every object in a plan gets one action. The two replacement symbols differ in order: which copy exists during the change.

SymbolActionJSON change.actionsMeaning
+create["create"]New object
~update in-place["update"]Same object, changed arguments (API update call)
-destroy["delete"]Object removed from AWS and state
-/+destroy then create["delete","create"]Replacement; downtime / data loss window
+/-create then destroy["create","delete"]Replacement with create_before_destroy (ch 3)
<=read["read"]Data source read during apply (inputs unknown at plan)
.forget["forget"]removed block with destroy = false: leaves state only (ch 5)
(none)no-op / moved["no-op"] + previous_addressmoved block: address changes, object does not

Read in this order

  • 1. The summary line at the bottom: any destroy at all?
  • 2. Every -, -/+, +/-: which object, and why (the # forces replacement comment).
  • 3. ~ on IAM, security groups, KMS, bucket policies: access changes.
  • 4. +: expected? names, tags, sizes.
  • 5. The drift section ("changed outside of Terraform") at the top.
  • 6. Output changes other components read.
Reading a plan

A real plan, annotated

Output from Terraform 1.15 on a small configuration. The comment line above each object says what and why.

  # terraform_data.cbd must be replaced
+/- resource "terraform_data" "cbd" {
      ~ id               = "dc3f…" -> (known after apply)
      ~ triggers_replace = "a" -> "b"
    }

  # terraform_data.forgot will no longer be managed by
  # Terraform, but will not be destroyed
 . resource "terraform_data" "forgot" { … }

  # terraform_data.gone will be destroyed
  # (because terraform_data.gone is not in configuration)
  - resource "terraform_data" "gone" {
      - input  = "x" -> null
    }

  # terraform_data.old_name has moved to
  # terraform_data.new_name
    resource "terraform_data" "new_name" { … }

  # terraform_data.upd will be updated in-place
  ~ resource "terraform_data" "upd" {
      ~ input  = "v1" -> "v2"
      ~ output = "v1" -> (known after apply)
    }

Plan: 3 to add, 1 to change, 3 to destroy.

Annotations to spot

  • must be replaced + an attribute line ending in # forces replacement (provider marks the argument as ForceNew).
  • is not in configuration: someone deleted code, or a for_each key disappeared.
  • has moved to: refactor, no infrastructure change.
  • will be read during apply: a data source depends on unknown values.
  • (known after apply): computed by AWS; not a change by itself.

The summary line counts

  • Replacements count in both "add" and "destroy": 1 replace = 1 to add + 1 to destroy.
  • "3 to destroy" above = 2 replacements + 1 delete.
  • Imports show as N to import; forgotten objects are listed in a warning below the plan.
Reading a plan

Replacements in AWS plans

For real providers the plan names the argument that forces the replacement. That line is the one to discuss in review.

  # module.bucket.aws_s3_bucket.this must be replaced
-/+ resource "aws_s3_bucket" "this" {
      ~ arn    = "arn:aws:s3:::tflab-dev-assets"
                 -> (known after apply)
      ~ bucket = "tflab-dev-assets" -> "tflab-dev-static"
                 # forces replacement
      ~ id     = "tflab-dev-assets" -> (known after apply)
        # (… unchanged attributes hidden)
    }

  # module.network.aws_vpc.this must be replaced
-/+ resource "aws_vpc" "this" {
      ~ cidr_block = "10.10.0.0/16" -> "10.30.0.0/16"
                     # forces replacement
        …
    }

Arguments that force replacement (examples)

  • aws_s3_bucket.bucket (the name).
  • aws_vpc.cidr_block; aws_subnet.cidr_block, availability_zone, vpc_id.
  • aws_security_group.name, description, vpc_id.
  • aws_iam_role.name; aws_lambda_function.function_name.
  • aws_db_instance.identifier, engine; aws_dynamodb_table.name, hash_key.

How to know in advance

  • The provider docs mark these arguments "Forces new resource".
  • Renaming in code ≠ renaming in AWS: most names are immutable.
  • A replaced subnet replaces everything inside it that pins subnet_id.
Reading a plan

Drift, outputs and sensitive values

Three plan sections people scroll past: what changed outside Terraform, what other components will read, and what Terraform hides.

Note: Objects have changed outside of Terraform

Terraform detected the following changes made outside
of Terraform since the last "terraform apply" which may
have affected this plan:

  # aws_security_group.web has changed
  ~ resource "aws_security_group" "web" {
      ~ ingress = [
          + { cidr_blocks = ["0.0.0.0/0"], from_port = 22, … },
        ]
    }

Unless you have made equivalent changes to your
configuration, or ignored the relevant attributes using
ignore_changes, the following plan may include actions
to undo or respond to these changes.

Changes to Outputs:
  ~ vpc_id = "vpc-0a…" -> (known after apply)
  + bucket = "tflab-dev-static"

  ~ password = (sensitive value)

Drift section

  • Shown when refresh found real objects different from state.
  • If the code does not match the drift, the plan reverts it: someone's console fix (or an attacker's SSH rule) disappears.
  • Decide: adopt it in code, or let the plan revert it, but never ignore it (ch 5).

Outputs and sensitive

  • Output changes break consumers that read them via SSM or remote state (ch 10).
  • (sensitive value) hides display only; the plan file and its JSON still contain the value (ch 12).
Chapter 8 · Part 8.2

Plan as data

A saved plan converted to JSON: filter it with jq, summarize it for reviewers, and gate on it in CI.

Plan JSON

From plan file to JSON

terraform show -json turns a saved plan into a documented JSON format. Policies (ch 9) and scripts read this, never the text output.

terraform plan -out=tfplan        # binary plan file
terraform show -json tfplan > tfplan.json

jq 'keys' tfplan.json
# [ "applyable", "complete", "configuration",
#   "errored", "format_version", "planned_values",
#   "prior_state", "resource_changes",
#   "terraform_version", "timestamp" ]
# (+ "resource_drift", "output_changes",
#    "relevant_attributes", "checks" when present)

jq -r '.format_version, .terraform_version' tfplan.json
# 1.2
# 1.15.5

Top-level keys you will use

KeyContains
resource_changes[]One entry per resource instance: the planned action
resource_drift[]Changes found by refresh (outside Terraform)
output_changesPer-output actions, before, after
planned_valuesFull expected state after apply
prior_stateState after refresh, before the change
configurationThe parsed configuration (expressions, references)
errored / applyablePlan failed / plan can be applied

The JSON format is versioned by format_version (minor bumps add fields, a major bump would break readers). Scripts should ignore unknown keys.

Plan JSON

One resource_changes entry

Everything a reviewer or a policy needs about one object is in this structure.

{
  "address": "terraform_data.rep",
  "previous_address": null,
  "mode": "managed",
  "type": "terraform_data",
  "name": "rep",
  "provider_name": "terraform.io/builtin/terraform",
  "change": {
    "actions": ["delete", "create"],
    "before": { "triggers_replace": "a", "id": "c698…" },
    "after":  { "triggers_replace": "b" },
    "after_unknown": { "id": true, "output": true },
    "before_sensitive": {},
    "after_sensitive": {},
    "replace_paths": [["triggers_replace"]]
  },
  "action_reason": "replace_because_cannot_update"
}

Fields to filter on

  • type; change.actions (always an array: replace = two actions).
  • change.replace_paths: argument(s) that forced the replacement.
  • previous_address (moved) and change.importing.id (import blocks).
  • before/after are not redacted: *_sensitive only marks values. Protect plan JSON like state (ch 12).

action_reason values seen

ReasonWhen
replace_because_cannot_updateForceNew argument changed
replace_by_request-replace=ADDR
replace_by_triggersreplace_triggered_by
delete_because_no_resource_configCode removed
Plan JSON

jq recipes for reviewers

Copy these into your toolbox. All were run against a real Terraform 1.15 plan.

P=tfplan.json

# every non-trivial change, one line each
jq -r '.resource_changes[]
  | select(.change.actions != ["no-op"])
  | "\(.change.actions | join(","))\t\(.address)"' $P
# create,delete   terraform_data.cbd
# forget          terraform_data.forgot
# delete          terraform_data.gone
# create          terraform_data.new
# delete,create   terraform_data.rep
# update          terraform_data.upd

# count by action
jq -r '[.resource_changes[].change.actions | join(",")]
  | group_by(.) | map("\(.[0]): \(length)") | .[]' $P

# anything that deletes (delete or replace)
jq -r '.resource_changes[]
  | select(.change.actions | index("delete"))
  | .address' $P
# why is it replaced?
jq -c '.resource_changes[]
  | select(.change.replace_paths)
  | {address, paths: .change.replace_paths,
     reason: .action_reason}' $P
# {"address":"terraform_data.rep",
#  "paths":[["triggers_replace"]],
#  "reason":"replace_because_cannot_update"}

# IAM and security group changes (access review)
jq -r '.resource_changes[]
  | select(.type | test("^aws_(iam_|security_group)"))
  | select(.change.actions != ["no-op"])
  | .address' $P

# moved objects
jq -r '.resource_changes[]
  | select(.previous_address)
  | "\(.previous_address) -> \(.address)"' $P

# does this plan change anything? (CI)
jq -e '[.resource_changes[]
  | select(.change.actions != ["no-op"])] | length > 0' $P
Plan JSON

A plan summary script for every PR

scripts/plan-summary.sh (Lab 8) prints a reviewer-friendly table and fails when a protected resource type would be deleted or replaced.

#!/usr/bin/env bash
# usage: scripts/plan-summary.sh tfplan.json
set -euo pipefail
plan=${1:?usage: plan-summary.sh <plan.json>}
re=${PROTECTED_TYPES:-'^aws_(s3_bucket|vpc|subnet|kms_key|iam_role)$'}

jq -r '
  {"create": "CREATE", "update": "UPDATE",
   "delete": "DELETE", "forget": "FORGET",
   "delete,create": "REPLACE(-/+)",
   "create,delete": "REPLACE(+/-)"} as $label
  | .resource_changes[] as $r
  | ($r.change.actions | join(",")) as $a
  | ($label[$a] // (if $r.previous_address
                    then "MOVE" else empty end))
  | [., $r.address, ($r.action_reason // "")]
  | @tsv' "$plan" | column -t -s $'\t'
blocked=$(jq -r --arg re "$re" '
  .resource_changes[]
  | select((.change.actions | index("delete"))
           and (.type | test($re)))
  | .address' "$plan")
if [[ -n "$blocked" && "${ALLOW_DESTROY:-}" != "true" ]]; then
  echo "BLOCKED: deletes/replaces protected:" >&2
  echo "$blocked" >&2
  exit 1
fi
echo "OK: protected-type gate passed"
# REPLACE(+/-)  terraform_data.cbd
# FORGET        terraform_data.forgot
# DELETE        terraform_data.gone
# CREATE        terraform_data.new
# MOVE          terraform_data.new_name
# REPLACE(-/+)  terraform_data.rep
# UPDATE        terraform_data.upd
# OK: protected-type gate passed

BLOCKED is not "never": it means "needs a decision". In CI a reviewer adds the PR label allow-destroy, which the pipeline turns into ALLOW_DESTROY=true (Lab 8 runs it by hand).

Plan JSON

Exit codes and machine-readable plans in CI

CI needs three answers from a plan: did it fail, does it change anything, and what exactly. Use exit codes for the first two, JSON for the third.

terraform plan -input=false -lock-timeout=5m \
  -detailed-exitcode -out=tfplan
rc=$?
# 0 = success, no changes
# 1 = error
# 2 = success, there are changes
case $rc in
  0) echo "no changes" ;;
  2) terraform show -json tfplan > tfplan.json
     scripts/plan-summary.sh tfplan.json ;;
  *) exit 1 ;;
esac

# human text for the PR comment (no colors)
terraform show -no-color tfplan > tfplan.txt

CI rules

  • Always -input=false: a prompt hangs the job.
  • -detailed-exitcode is the drift detector of ch 11: exit 2 on a schedule = drift.
  • Store tfplan as an artifact for the apply job; the JSON and text are for humans and policies.
  • Plan files contain secrets: short artifact retention, restricted access (ch 12).

Streaming JSON

  • terraform plan -json prints machine-readable log events (one JSON object per line), not the plan format above.
  • Use it for progress parsing; use show -json on a saved plan for review and policy.
Chapter 8 · Part 8.3

Risk patterns

The handful of change types that cause most Terraform outages, and a checklist that makes reviewers look at them every time.

Risk patterns

Severity levels for a plan

Classify every plan before reading the details. The level decides who must approve and how the apply is run.

LevelPlan containsExamplesProcess
LowOnly creates, tag changes, no-ops, movesNew SSM parameter, extra tag, moved blockOne reviewer; apply from the pipeline
MediumIn-place updates of non-data resourcesLambda memory, SG rule description, log retentionOne reviewer who knows the service
HighAccess changes, deletes of stateless objects, replacements with +/-IAM policy, SG ingress, route table, replace a launch templateOwner + security for IAM/SG; apply in a change window
CriticalDelete or -/+ replace of stateful or shared objectsS3 bucket with data, RDS, KMS key, VPC/subnet, IAM role used by other systemsExplicit override, backup/snapshot first, rollback plan, two approvers

Automate the classification with the plan JSON (type + actions) and let CI put the level in the PR comment. Humans then spend their attention on High and Critical.

Risk patterns

Stateful and shared resources

Replacing these loses data or breaks other systems, and Terraform cannot undo it.

Data stores

  • S3 buckets: replacement = new empty bucket; old one fails to delete if not empty (or is emptied with force_destroy = true).
  • RDS / Aurora: identifier or engine change → new instance; final snapshot only if skip_final_snapshot = false.
  • DynamoDB: name/key schema change → new, empty table.
  • EBS / EFS volumes, ElastiCache clusters.

Keys and identities

  • KMS keys: delete schedules key deletion (7–30 days); data encrypted with it becomes unreadable afterwards.
  • IAM roles: a recreated role gets a new unique ID; resource and trust policies that named the old role stop matching it, even with the same ARN, until they are re-saved.
  • Secrets: replacement changes the ARN suffix that consumers may pin.

Shared plumbing

  • VPC, subnets: replacing them replaces everything inside (ENIs, instances, endpoints).
  • Route tables, NAT, TGW attachments: a short window with no route = an outage.
  • DNS records and certificates: TTLs and validation make "recreate" slow.
  • Security groups referenced by other SGs: replacement breaks references.

Guards: lifecycle { prevent_destroy = true } on the resource (ch 3), protected types in the plan gate (this chapter), deletion protection in AWS itself (deletion_protection = true on RDS), S3 versioning and backups.

Risk patterns

Changes that look harmless

Small diffs in the code that produce large plans. Recognize them in the PR before running the plan.

Change in codeWhat the plan doesSafer way
Remove an item from a list used with countShifts every later index: updates or replaces items after itUse for_each with stable keys (ch 2); moved blocks to migrate
Rename a resource or module labelDestroy + createmoved block (ch 5)
Change a for_each key format (e.g. AZ name → index)Every instance is replacedKeep keys stable, or moved per key
Edit a value that feeds name or cidr_blockReplacement (ForceNew)Check provider docs; create new + migrate
for_each over values known only after applyPlan error: "Invalid for_each argument"Use keys known at plan (variables, static maps)
Bump a provider major versionNew defaults, renamed attributes, sometimes replacementsUpgrade guide + a plan with no other changes (ch 13)
Move code between root modulesDestroy in one state, create in the otherremoved + import blocks, or a state mv plan (ch 5)
Risk patterns

-target, refresh and other sharp tools

Flags that change what the plan covers. Each is legitimate in an incident and dangerous as a habit.

-target=ADDR

  • Plans only that object and its dependencies.
  • Terraform warns: "Resource targeting is in effect" and the plan is incomplete.
  • Leaves the rest of the state stale: the next full plan may show surprising diffs.
  • Use for recovery only, then run a full plan right after.

-refresh=false and -refresh-only

  • -refresh=false: faster, but plans against stale state; drift is invisible.
  • -refresh-only (plan or apply): updates state to match reality, changes no infrastructure. Review it like any plan.
  • -minimal-refresh (1.17): refreshes only what the plan needs; read its docs before using it in CI.

-replace=ADDR

  • Forces a replacement of a healthy object (action_reason = replace_by_request).
  • Replaces the old terraform taint workflow; shows in the plan, so it is reviewable.
  • Combine with create_before_destroy to avoid a gap.
Risk patterns

The reviewer checklist

Paste it into the PR template. Each line takes seconds with the plan summary in front of you.

Scope

  • Plan is from the right environment and component (live/<env>/<component>).
  • Plan is fresh (same commit as the PR head) and complete (no -target).
  • Summary line matches the intent of the PR description.
  • No drift section, or the drift is explained.

Danger

  • Any -, -/+, +/-: which argument forces it, is data lost, is there a backup?
  • IAM, SG, KMS, bucket policy diffs read line by line: who gains access?
  • Protected-type gate passed, or an approved override with a reason.
  • Output changes: which consumers read them?

Hygiene

  • Tags present (Project, Env, ManagedBy).
  • Policy checks green (Checkov, Trivy, conftest, ch 9).
  • Cost diff reasonable (Infracost, ch 7).
  • Provider/module version bumps are in their own PR.
Chapter 8 · Part 8.4

Safe apply

Apply exactly what was reviewed, handle locks and partial failures, and know what "rollback" means in Terraform.

Safe apply

Apply the saved plan, nothing else

The reviewed artifact is the plan file. terraform apply tfplan executes it without re-planning; terraform apply alone plans again.

PR openedcommit abc123plan -out=tfplansame commitsummary + policyJSON gates (ch 9)human reviewchecklistmergeapprovedapply tfplanexact actionsartifact: the binary plan file (encrypted, short retention)state changed since plan?→ "Saved plan is stale": re-plan, re-review

If anything changed in between (another apply, a lock, a drift fix), Terraform refuses the saved plan instead of applying a different change. That refusal is a feature: re-plan and review again.

Safe apply

Stale plans, locks and timeouts

Two applies on the same state cannot run at once (ch 4). The tools below decide what happens when they collide.

# a plan saved before someone else applied:
terraform apply a.tfplan
# Error: Saved plan is stale
#
# The given plan file can no longer be applied
# because the state was changed by another
# operation after the plan was created.

# wait for a lock instead of failing at once
terraform plan  -lock-timeout=5m -out=tfplan
terraform apply -lock-timeout=5m tfplan

# a lock left by a killed job (check first!)
terraform force-unlock <LOCK_ID>

In CI

  • concurrency group per state (one apply per live/<env>/<component> at a time, ch 11).
  • -lock-timeout long enough for a normal plan of another job.
  • Never -lock=false against shared state.

force-unlock

  • Only after confirming no process holds the lock (the lock info shows who and when).
  • Unlocking under a running apply can corrupt state: the S3 versioning of the state bucket is the recovery path (ch 4).
Safe apply

Partial failures and recovery

An apply is not a transaction. When one resource fails, the ones already created stay, and state records them.

What happens

  • Terraform applies the graph in parallel (default 10).
  • On an error it stops starting new operations, waits for running ones, writes state, exits 1.
  • State now reflects reality: successful objects are recorded; the failed one may be tainted (created but not ready).
  • A new plan shows only the remaining work and the replacement of tainted objects.

Recovery steps

  • Read the error; fix the cause (quota, permission, name clash, eventual consistency).
  • Run a fresh plan; review it like a new change.
  • Retrying blindly may replace a tainted resource that is actually fine: terraform untaint ADDR if you verified it.
  • If state was not written (crash, killed runner): errored.tfstate in the working directory → terraform state push errored.tfstate after review.

Rollback = forward

  • There is no terraform rollback.
  • Rollback means: git revert the change, plan, review, apply. It goes through the same gates.
  • Data deleted by a replacement is not coming back from Terraform: backups, snapshots, S3 versioning.
  • Write the rollback plan before applying Critical changes.
Safe apply

Approvals and break-glass

Normal changes go through the pipeline. Emergencies need a documented path that is still auditable.

Normal path

  • Apply only from CI with the apply role (gha-tf-apply), after merge (ch 11).
  • GitHub environment prod with required reviewers gates the apply job.
  • Humans have read-only roles in prod; no local applies.
  • Every apply links to a PR, a plan and a reviewer.

Break-glass

  • A separate role, assumed only during incidents, with MFA and alerting on use (CloudTrail → alarm).
  • Fix in the console or a local apply only if the pipeline cannot.
  • Write down every manual change during the incident.
  • Afterwards: reconcile with a PR (code it, or plan to revert it) and -refresh-only review.

Signals of a healthy process

  • Drift plans (ch 11) are empty most days.
  • No force-unlock without an incident ticket.
  • Critical plans are rare and always have a rollback note.
  • Plan summaries in PRs are read: reviewers ask about replacements.
Common mistakes

Plan review mistakes

All of these have caused real outages; most were visible in the plan.

Reading

  • Only reading the summary line: "1 to destroy" was the production database.
  • Missing # forces replacement on a renamed bucket or role.
  • Ignoring the drift section: the apply reverts an emergency console fix.
  • Reviewing a plan from another environment or an old commit.

Applying

  • terraform apply (re-plan) instead of terraform apply tfplan.
  • -auto-approve on a laptop against prod.
  • -target as a routine, leaving state stale.
  • force-unlock while another apply was still running.

Fixes

  • Plan summary + protected-type gate in every PR (Lab 8).
  • Severity level drives approvals; Critical needs backup + rollback note.
  • Saved plan as the only apply input; CI is the only applier.
  • prevent_destroy and AWS deletion protection on stateful resources.
Self-check

Plan risk review: can you answer these?

Use the exact symbol, JSON field or flag in each answer.

Reading

  • What is the difference between -/+ and +/-, and which lifecycle setting produces the second?
  • How does one replacement show up in the summary line?
  • Where does the plan say which argument forced a replacement?
  • What does . mean in front of a resource in a plan?

JSON

  • Which command produces plan JSON, and why not terraform plan -json?
  • Write a jq filter that lists every address whose actions include delete.
  • What are replace_paths and action_reason?
  • Why must plan JSON be protected like state?

Applying

  • What does "Saved plan is stale" protect you from?
  • Name two dangers of -target and of -refresh=false.
  • What does rollback mean in Terraform?
  • What happens to state after a partial apply failure?

Practice: Lab 8 in Hands-on labs — write scripts/plan-summary.sh, trigger a bucket rename and a CIDR change, watch the gate block them, then apply only a reviewed, saved plan.

Chapter 9

Policy as code: Checkov, Trivy, OPA, Terraform Policy

Turn security and team rules into checks that run on every change: static scans of HCL, policies over the plan, and how to handle findings without drowning in them.

Chapter 9 · Part 9.1

Why policy as code

Guardrails that review every change the same way, before anything reaches AWS.

Why policy as code

Rules a reviewer forgets, a check never does

Code review catches design problems. Machines are better at "every bucket blocks public access" on the 400th pull request.

What policies encode

  • Security: no public S3, encryption on, no 0.0.0.0/0 on SSH, IAM without *:*.
  • Compliance: required tags, approved regions, log retention.
  • Cost: no NAT gateways in dev, allowed instance families.
  • Safety: no deletes of stateful resources without an explicit exception.

Why as code

  • Versioned and reviewed like the infrastructure itself.
  • Same result on a laptop, in CI and in HCP Terraform.
  • A failing check explains itself: ID, resource, file and line.
  • Exceptions are explicit, justified and searchable (#checkov:skip=…:reason).

What it does not replace

  • Design review: a policy cannot tell a good module boundary from a bad one.
  • Runtime controls: SCPs, AWS Config, IAM boundaries still catch ClickOps (ch 13).
  • Plan review (ch 8): policies flag known patterns, humans judge intent.
Where checks run

Shift left, but keep a gate at the end

The earlier a check runs, the cheaper the fix. The later it runs, the more it knows (resolved values, the actual plan).

Editor / pre-commitfmt, tflint, trivyPull requeststatic scan of HCLPlancheckov / conftest on JSONApply gateTerraform Policy, SentinelRuntimeSCP, AWS ConfigKnows: only the codeKnows: resolved values, tags_all,actions (create, delete, replace)Knows: real resources,including ClickOpsFeedback: secondsFeedback: after the factLab 9 wires the middle three into make policy; Lab 11 runs the same target in CI (ch 11).

Run the same tools with the same config everywhere. A check that only runs in CI produces "works on my machine" arguments; a check that only runs locally is optional.

Static vs plan scanning

Two inputs, different blind spots

Scanning HCL is fast and points at a file and line. Scanning the plan JSON sees what Terraform will actually do.

Static (HCL directory)Plan (terraform show -json tfplan)
Needs credentials / initNoYes: a real plan against the backend and provider
Variables and default_tagsPartly resolved; provider default_tags not mergedFully resolved: tags_all has every tag
References between resourcesResolved by the scanner from codeIDs are (known after apply) on create
Actions (create, delete, replace)UnknownExact: resource_changes[].change.actions
Points to file and lineYesOnly with enrichment (--repo-root-for-plan-enrichment in Checkov)
Best forMisconfigurations inside modulesTags, deletes, values that depend on variables

Tested in Lab 9: a "required tags" check fails on HCL when tags come from provider default_tags, and passes on the plan. Trivy on plan JSON reports a missing public access block that the HCL scan correctly links to the bucket.

Findings lifecycle

Fix, suppress with a reason, or baseline — never ignore silently

A policy program dies when findings pile up and everybody learns to click past red.

1 · Fix

  • Default answer. Most findings are one argument (block_public_acls = true).
  • Fix in the module, not in each root module: one change protects every user.

2 · Suppress with justification

  • Inline, next to the resource, with a reason a reviewer can challenge.
  • Checkov: #checkov:skip=CKV_AWS_144:single-region lab data.
  • Trivy: #trivy:ignore:AWS-0089 above the block, or an ID in .trivyignore.
  • Review suppressions like code: CODEOWNERS on policy files (ch 13).

3 · Baseline

  • Adopting a scanner on an old repo: record today's findings, fail only on new ones.
  • Checkov: --create-baseline writes .checkov.baseline; --baseline applies it.
  • Burn the baseline down over time; never add to it.

Gate on severity where the tool has it: trivy config --severity HIGH,CRITICAL --exit-code 1. Checkov OSS built-in checks report no severity without a Prisma Cloud API key, so gate with --check/--skip-check lists instead.

Chapter 9 · Part 9.2

Checkov

Graph-aware static analysis with 1000+ built-in checks, custom YAML/Python checks, and plan scanning.

Checkov

Running Checkov on a module

Checkov builds a graph of resources and their connections, so it knows a public access block belongs to a bucket.

python3 -m pip install checkov      # or: brew install checkov
checkov --version                    # 3.x

checkov -d modules/app-bucket --compact --quiet

terraform scan results:

Passed checks: 10, Failed checks: 5, Skipped checks: 0

Check: CKV2_AWS_62: "Ensure S3 buckets should have
                     event notifications enabled"
	FAILED for resource: aws_s3_bucket.this
Check: CKV2_AWS_61: "Ensure that an S3 bucket has a
                     lifecycle configuration"
	FAILED for resource: aws_s3_bucket.this
Check: CKV_AWS_18: "Ensure the S3 bucket has access
                    logging enabled"
	FAILED for resource: aws_s3_bucket.this
Check: CKV_AWS_144: "Ensure that S3 bucket has
                     cross-region replication enabled"
	FAILED for resource: aws_s3_bucket.this
Check: CKV_AWS_145: "Ensure that S3 buckets are encrypted
                     with KMS by default"
	FAILED for resource: aws_s3_bucket.this

Reading the result

  • CKV_AWS_*: checks on one resource's attributes.
  • CKV2_AWS_*: graph checks across connected resources (bucket ↔ public access block, VPC ↔ flow log).
  • --quiet shows only failures; --compact hides code snippets.
  • Exit code 1 on any failure, 0 when clean; --soft-fail always returns 0.
  • Not every failure is a bug: replication and event notifications are rarely needed in a lab. That is what suppressions are for.
  • Output above is from Checkov 3.4 on the tf-lab app-bucket module (Lab 9).
Checkov

The network module and what the checks mean

Same scan on modules/network. Each finding is a design decision to make, not a box to tick.

CheckFindingtf-lab decision
CKV_AWS_130Subnets assign public IPs by defaultKeep for public subnets only; suppress with reason there
CKV2_AWS_11VPC flow logging not enabledFix: flow logs to a CloudWatch log group with short retention
CKV2_AWS_12Default security group does not restrict all trafficFix: aws_default_security_group with no rules
CKV_AWS_18S3 access logging (app-bucket)Suppress in the lab; production: server access logs or CloudTrail data events
CKV_AWS_145S3 not encrypted with KMS by defaultLab: SSE-S3 is enough (suppress); production: SSE-KMS with a CMK
CKV_AWS_144, CKV2_AWS_62Replication, event notificationsNot needed: skip in config with a comment

Write the decision down where the next engineer sees it: the suppression comment, not a wiki page.

Checkov

Suppressions: inline and in a config file

Inline skips belong to one resource; config-level skips switch a check off for the whole scan.

# modules/app-bucket/main.tf
resource "aws_s3_bucket" "this" {
  #checkov:skip=CKV_AWS_144:single-region lab data
  #checkov:skip=CKV_AWS_18:audited via CloudTrail
  bucket = var.name
}
# .checkov.yaml  (picked up from the current directory)
framework:
  - terraform
skip-check:
  - CKV2_AWS_62   # S3 event notifications: not used in tf-lab
  - CKV2_AWS_61   # lifecycle rules: buckets are short-lived
compact: true
quiet: true

Rules

  • Inline format: #checkov:skip=<ID>:<reason> inside the resource block.
  • Skipped checks are counted (Skipped checks: 1) and listed with the reason, so they stay visible.
  • Config file keys are the long CLI flag names without --; --config-file points to another path.
  • Prefer inline skips: they move with the code and get reviewed in the same diff.
  • Skip a check globally only when it never applies to your estate.
  • --skip-path excludes directories (e.g. examples/), not individual findings.
Checkov

A custom check in YAML: required tags

YAML checks need no Python: conditions on attributes, connections and resource types. This one requires three tags.

# policies/checkov/required_tags.yaml
metadata:
  id: "CKV2_TFLAB_1"
  name: "Ensure taggable resources carry Project, Env and ManagedBy tags"
  category: "GENERAL_SECURITY"
  severity: "MEDIUM"
definition:
  and:
    - cond_type: attribute
      resource_types: [aws_s3_bucket, aws_vpc, aws_subnet, aws_internet_gateway]
      attribute: tags_all.Project
      operator: exists
    - cond_type: attribute
      resource_types: [aws_s3_bucket, aws_vpc, aws_subnet, aws_internet_gateway]
      attribute: tags_all.Env
      operator: exists
    - cond_type: attribute
      resource_types: [aws_s3_bucket, aws_vpc, aws_subnet, aws_internet_gateway]
      attribute: tags_all.ManagedBy
      operator: exists

How it is used

  • Load with --external-checks-dir policies/checkov; select with --check CKV2_TFLAB_1.
  • tags_all = resource tags merged with provider default_tags. It only exists in the plan.
  • On HCL the check fails for every resource that relies on default_tags — run it on the plan JSON.
  • Other operators: equals, not_equals, contains, within, regex_match, is_true, …; cond_type: connection for graph rules.
  • Python checks (BaseResourceCheck) cover logic YAML cannot express.
Checkov

Scanning the plan

Give Checkov the JSON of a saved plan. Resolved variables, tags_all and module calls are all there.

cd live/dev/app
terraform plan -out=tfplan
terraform show -json tfplan > tfplan.json

checkov -f tfplan.json \
  --external-checks-dir ../../../policies/checkov \
  --check CKV2_TFLAB_1 --compact --quiet

terraform_plan scan results:

Passed checks: 5, Failed checks: 0, Skipped checks: 0

# all built-in checks on the plan, findings mapped
# back to files so inline skips are honoured
checkov -f tfplan.json \
  --repo-root-for-plan-enrichment ../../.. \
  --compact --quiet

Notes

  • The framework is detected as terraform_plan; force it with --framework terraform_plan.
  • Without enrichment, plan findings have no file/line and inline #checkov:skip comments are not seen.
  • tfplan.json contains every resolved value, including secrets: treat it like state (ch 12).
  • Plan scans cover only what changes or exists in this root module — run them per component.
  • Same plan JSON feeds Lab 8's plan-summary.sh and conftest (section 35).
Checkov

Checkov in CI: outputs and exit codes

One command, machine-readable output, and a clear pass/fail.

# several outputs in one run
checkov -d modules \
  -o cli -o sarif --output-file-path console,results.sarif

# adopt on a legacy repo: fail only on NEW findings
checkov -d modules --create-baseline   # writes modules/.checkov.baseline
checkov -d modules --baseline modules/.checkov.baseline

# report but never fail (first weeks of adoption)
checkov -d modules --soft-fail

Formats and gates

  • -o values include cli, json, junitxml, sarif, github_failed_only.
  • SARIF uploads to GitHub code scanning; JUnit shows in most CI test tabs.
  • --soft-fail-on / --hard-fail-on take check IDs or severities (severities need the platform API key for built-in checks).
  • Pin the Checkov version in CI (pip install checkov==3.x.y or a pinned image digest): new releases add checks and can break a green build.
Chapter 9 · Part 9.3

Trivy for IaC

One scanner for images, filesystems and IaC; trivy config carries the former tfsec checks.

Trivy

trivy config on Terraform

tfsec's checks now live in Trivy; tfsec itself is no longer developed separately. IDs look like AWS-0089 (older output and docs show AVD-AWS-0089).

trivy --version                  # 0.7x
trivy config modules

app-bucket/main.tf (terraform)
Tests: 2 (SUCCESSES: 0, FAILURES: 2)
Failures: 2 (LOW: 1, MEDIUM: 0, HIGH: 1, CRITICAL: 0)

AWS-0089 (LOW): Bucket has logging disabled
AWS-0132 (HIGH): Bucket does not encrypt data with
                 a customer managed key.

network/main.tf (terraform)
Tests: 3 (SUCCESSES: 0, FAILURES: 3)
Failures: 3 (LOW: 0, MEDIUM: 1, HIGH: 2, CRITICAL: 0)

AWS-0164 (HIGH): Subnet associates public IP address.
   via network/main.tf:6-11 (aws_subnet.public[0])
AWS-0164 (HIGH): Subnet associates public IP address.
   via network/main.tf:6-11 (aws_subnet.public[1])
AWS-0178 (MEDIUM): VPC does not have VPC Flow Logs enabled.

Compared with Checkov

  • Every finding has a severity (LOW → CRITICAL) in the open-source tool.
  • Fewer, more focused checks: Trivy flagged 5 issues where Checkov flagged 9 on the same modules.
  • Same idea, different IDs: AWS-0164 ≈ CKV_AWS_130, AWS-0178 ≈ CKV2_AWS_11.
  • Running both is common: overlapping findings are cheap, gaps are not.
  • Checks ship inside the binary and are updated from a registry at run time; --skip-check-update uses the bundled set (offline, reproducible).
Trivy

Gating and ignoring findings

Fail the build on what matters, and record every exception next to the code or in one reviewed file.

# CI gate: only HIGH and CRITICAL fail the build
trivy config --severity HIGH,CRITICAL --exit-code 1 modules

# inline: the comment goes directly above the block
#trivy:ignore:AWS-0089 -- access is audited with CloudTrail data events
resource "aws_s3_bucket" "this" {
  bucket = var.name
}

# .trivyignore in the directory you run trivy from
# (read automatically; --ignorefile points elsewhere)
AWS-0132   # lab buckets use SSE-S3; prod module uses a CMK

# pass variables so conditionals and counts resolve
trivy config --tf-vars live/dev/app/dev.tfvars modules

# SARIF for GitHub code scanning
trivy config -f sarif -o trivy.sarif modules

Details that bite

  • .trivyignore is picked up from the working directory: running from another directory silently drops it.
  • Text after the ID in #trivy:ignore:<ID> is free text — use it for the reason.
  • --exit-code 1 is needed: by default Trivy exits 0 even with failures.
  • --tf-exclude-downloaded-modules skips findings inside registry modules you cannot change.
  • --skip-dirs excludes paths such as examples/.
Trivy

Plan scanning and its false positives

Trivy also reads plan JSON (terraformplan-json) and binary plan files. Unknown IDs at plan time can break the link between resources.

terraform show -json tfplan > tfplan.json
trivy config tfplan.json

main.tf (terraformplan)
Failures: 11 (LOW: 2, MEDIUM: 2, HIGH: 7, CRITICAL: 0)

AWS-0086 (HIGH): No public access block so not blocking public acls
AWS-0087 (HIGH): No public access block so not blocking public policies
AWS-0090 (MEDIUM): Bucket does not have versioning enabled
AWS-0091 (HIGH): No public access block so not blocking public acls
AWS-0093 (HIGH): No public access block so not restricting public buckets
AWS-0094 (LOW): Bucket does not have a corresponding public access block.
...

Why

  • On create, aws_s3_bucket_public_access_block.bucket is (known after apply) in the plan.
  • The scanner cannot tell which bucket the block (or versioning resource) belongs to, so it reports them as missing.
  • The HCL scan of the same code passes these checks: there the reference aws_s3_bucket.this.id is visible.
  • Rule of thumb: scan HCL with Trivy; use plan scans for things only the plan knows (tags, actions, resolved values).
  • Output reproduced in Lab 9 with Trivy 0.73 and AWS provider 6.x.
Trivy

Custom checks in Rego

Trivy evaluates Rego checks against a normalized cloud model (input.aws.s3.buckets, …), so one check works for Terraform, CloudFormation and plan files.

# policies/trivy/bucket_prefix.rego
# METADATA
# title: S3 bucket names must start with tflab-
# custom:
#   id: TFLAB-0001
#   severity: MEDIUM
#   input:
#     selector:
#       - type: cloud
#         subtypes:
#           - service: s3
#             provider: aws
package user.tflab.bucket_prefix

import rego.v1

deny contains res if {
	some bucket in input.aws.s3.buckets
	not startswith(bucket.name.value, "tflab-")
	msg := sprintf("bucket %q does not start with tflab-",
		[bucket.name.value])
	res := result.new(msg, bucket.name)
}
trivy config \
  --config-check policies/trivy \
  --check-namespaces user \
  modules

TFLAB-0001 (MEDIUM): bucket "random-name"
  does not start with tflab-

Notes

  • The # METADATA block gives ID, severity and which inputs the check applies to.
  • result.new(msg, attr) attaches the finding to the attribute's file and line.
  • Packages must match --check-namespaces (here user).
  • Tested with Trivy 0.73 (OPA v1 syntax: contains, if, in).
Trivy

The scanner is part of your supply chain too

On 19 Mar 2026, attackers (the TeamPCP campaign) force-pushed most version tags of aquasecurity/trivy-action and setup-trivy to malicious commits that stole CI secrets.

What happened

  • Pipelines using aquasecurity/trivy-action@<tag> ran attacker code with the job's secrets and tokens.
  • Affected Trivy releases were reported as v0.69.4–v0.69.6 (19–23 Mar 2026); check Aqua's advisory for the authoritative list.
  • Stolen cloud keys were used downstream; a scanner had become the attack path.

Controls for Terraform pipelines

  • Pin every action by full commit SHA, never by tag (the CI/CD deck shows how).
  • Pin scanner versions (binary checksum, image digest, checkov==x.y.z).
  • Scan jobs need no cloud credentials: run them before the OIDC step, in a separate job.
  • Give the plan role read-only access (ch 11–12), so a compromised step cannot apply.

Ask of every tool in CI

  • Who publishes it and how are releases signed?
  • What secrets can it see in this job?
  • Would we notice if its version changed?

The same reasoning applies to providers and modules: lock file hashes and pinned module versions (ch 12).

Chapter 9 · Part 9.4

OPA/conftest and Terraform Policy

Your own rules over the plan JSON, and HashiCorp's native policy framework.

conftest

Rego over the plan: the data you get

conftest runs OPA Rego policies against structured files. For Terraform, the input is terraform show -json tfplan.

{
  "format_version": "1.2",
  "terraform_version": "1.17.0",
  "resource_changes": [
    {
      "address": "module.bucket.aws_s3_bucket.this",
      "module_address": "module.bucket",
      "type": "aws_s3_bucket",
      "name": "this",
      "change": {
        "actions": ["delete"],
        "before": { "bucket": "tflab-dev-app-…", "...": "..." },
        "after": null
      }
    },
    {
      "address": "module.bucket.aws_s3_bucket_public_access_block.this",
      "type": "aws_s3_bucket_public_access_block",
      "change": {
        "actions": ["update"],
        "after": { "block_public_acls": false, "...": "..." }
      }
    }
  ]
}

Fields policies use

  • resource_changes[].type, .address, .change.actions.
  • Actions: ["create"], ["update"], ["delete"], ["no-op"], ["read"], and replacement as ["delete","create"] or ["create","delete"].
  • .change.before / .change.after: full attribute values (null on create/delete side).
  • .change.after_unknown: which values are known only after apply.
  • configuration and prior_state are also present for advanced rules.
  • A check for "delete" in actions catches deletes and replacements.
conftest

A deny policy for S3 buckets

Two rules: never delete or replace buckets or their public access block, and never weaken a public access block setting.

# policies/opa/s3.rego
package main

import rego.v1

protected := {"aws_s3_bucket", "aws_s3_bucket_public_access_block"}

pab_settings := [
	"block_public_acls", "block_public_policy",
	"ignore_public_acls", "restrict_public_buckets",
]

deny contains msg if {
	some rc in input.resource_changes
	rc.type in protected
	"delete" in rc.change.actions
	msg := sprintf("%s: %v is not allowed for %s",
		[rc.address, rc.change.actions, rc.type])
}

deny contains msg if {
	some rc in input.resource_changes
	rc.type == "aws_s3_bucket_public_access_block"
	some setting in pab_settings
	rc.change.after[setting] != true
	msg := sprintf("%s: %s must be true", [rc.address, setting])
}

How it reads

  • package main is the namespace conftest evaluates by default.
  • deny rules fail the run; warn rules print but pass; violation rules can return structured results.
  • Each deny contains msg if { … } adds one message per matching resource.
  • On delete, change.after is null, so the second rule simply does not match; the first one already fails.
  • Rego v1 syntax (contains, if, in): current conftest releases are built on OPA v1; import rego.v1 keeps it valid on older ones.
conftest

Running and testing the policy

The policy is code, so it gets unit tests with small fake plans.

conftest test --policy policies/opa tfplan.json

FAIL - tfplan.json - main - module.bucket.aws_s3_bucket.this:
       ["delete"] is not allowed for aws_s3_bucket

# summary: N tests, N passed, N warnings,
#          N failures, N exceptions

# exit code: 1 on any failure → usable as a CI gate
echo $?    # 1
# policies/opa/s3_test.rego
package main

import rego.v1

test_bucket_delete_denied if {
	count(deny) == 1 with input as {"resource_changes": [{
		"address": "aws_s3_bucket.x",
		"type": "aws_s3_bucket",
		"change": {"actions": ["delete"], "after": null},
	}]}
}

# run the tests (no plan needed)
#   conftest verify --policy policies/opa

Exceptions

  • Sometimes a bucket must go. Make the exception explicit and reviewable, e.g. an allowlist of addresses in a data file (--data policies/opa/exceptions.json) changed in the same PR.
  • Avoid "disable the check for this run" switches in CI: they become the default.
  • --all-namespaces runs every package, not just main.
  • -o json, -o junit, -o github change the output format.
Terraform Policy

Terraform Policy: native, HCL-based (GA in 1.17)

HashiCorp's newer policy framework: policies written in HCL and evaluated by Terraform itself during plan/apply, locally or in HCP Terraform.

# policies/tfpolicy/s3.policy.hcl — shape only;
# check the Terraform Policy reference for current syntax
policy {
  required_providers {
    aws = { source = "hashicorp/aws" }
  }
}

resource_policy "aws_s3_bucket_public_access_block" "all_blocked" {
  enforcement_level = "mandatory"

  enforce {
    condition     = attrs.block_public_acls == true
    error_message = "Public ACLs must be blocked."
  }
  enforce {
    condition     = attrs.restrict_public_buckets == true
    error_message = "Public bucket access must be restricted."
  }
}

What we know (Nov 2026)

  • Terraform 1.17.0 made Terraform Policy generally available: -policies on plan, apply and init no longer needs -allow-experimental-features.
  • Policies are HCL blocks such as resource_policy, provider_policy and module_policy, each with one or more enforce { condition, error_message } blocks; a file needs a policy { required_providers { … } } block.
  • Enforcement levels include mandatory and mandatory_overridable.
  • The standalone tfpolicy CLI validates and tests policies (tfpolicy validate, tfpolicy test).
  • HCP Terraform and TFE run it next to Sentinel and OPA policy sets.

New and moving fast: treat the snippet as an illustration and copy real policies from developer.hashicorp.com/terraform/policy. The labs use Checkov, Trivy and conftest.

Choosing

Which tool for which rule

Most teams end up with two or three layers. Pick by where the rule needs its data, not by tool loyalty.

RuleBest toolWhy
Secure defaults inside modules (encryption, public access, flow logs)Trivy and/or Checkov on HCLFast, file and line, no credentials
Required tags with default_tagsCheckov YAML on plan, or conftestOnly the plan has tags_all
No deletes/replacements of stateful resourcesconftest on plan JSONNeeds change.actions
Org-specific naming, regions, instance typesCheckov YAML, Trivy Rego, or conftestWhichever your team already reads
Blocking applies in HCP Terraform / TFETerraform Policy, Sentinel, OPA policy setsEnforced by the platform, not by CI scripts
Anything done outside TerraformSCPs, AWS Config, IAM boundariesPolicies on code cannot see ClickOps

tf-lab uses Trivy + Checkov on HCL, Checkov + conftest on the plan, all behind one make policy target (Lab 9) that CI runs unchanged (Lab 11).

Common mistakes

Policy mistakes that make checks worthless

Each one turns a gate into noise or into a false sense of safety.

Coverage

  • Running the tag check on HCL only: everything fails, so someone adds a global skip.
  • Scanning only live/ and never modules/ (or the reverse).
  • Plan scans of one component treated as "the estate is compliant".
  • Trusting plan-scan findings about missing links (unknown IDs) without checking the HCL.

Exceptions

  • --soft-fail left on after the adoption phase.
  • Global skip-check used for one resource's exception.
  • Skip comments without a reason, or .trivyignore nobody reviews.
  • Running Trivy from a different directory and losing .trivyignore silently.
  • Baselines that grow instead of shrinking.

Pipeline hygiene

  • Scanner actions referenced by tag (Mar 2026: trivy-action tags were hijacked).
  • Unpinned scanner versions: new checks break unrelated PRs.
  • Scan job holding AWS credentials it does not need.
  • Policies without tests: a typo in Rego denies nothing, forever.
Self-check

Policy as code: can you answer these?

Answer with the exact flag, field or ID where one exists.

Concepts

  • Name two things only the plan JSON knows and HCL does not.
  • Why does a required-tags check fail on HCL when tags come from default_tags?
  • What is the difference between CKV_AWS_* and CKV2_AWS_* checks?
  • Why can Trivy on plan JSON report a missing public access block that exists?
  • When is a baseline the right tool?

Checkov and Trivy

  • Write an inline Checkov skip with a reason.
  • Which flag maps plan findings back to files and line numbers?
  • Which Trivy flags make HIGH/CRITICAL findings fail the build?
  • Where must .trivyignore live to be picked up?
  • What happened to trivy-action on 19 Mar 2026, and which control would have prevented it?

conftest and Terraform Policy

  • Which plan field tells you a resource will be replaced?
  • Write a Rego rule that denies deleting any aws_s3_bucket.
  • How do you unit-test a conftest policy?
  • Which Terraform version made Terraform Policy GA, and what changed for -policies?
  • Where would you enforce a rule that must also apply to ClickOps?

Practice: Lab 9 in Hands-on labs — scan modules and plans with Checkov and Trivy, fix or justify every finding, add a required-tags check and an S3 delete policy, and wire it all into make policy.

Chapter 10

Repository layout per environment

Where code, state and environments live: directories vs workspaces, promoting the same module from dev to prod, wiring components together, and when to reach for Terragrunt, Terramate or Stacks.

Chapter 10 · Part 10.1

Layout options

Three common ways to express environments, and the one tf-lab uses.

Layout options

Three ways to model environments

Every layout answers the same questions: where is the code, where is the state, and how different may dev and prod be?

Directory per environmentCLI workspacesOne root + tfvars per env
Shapelive/dev/network, live/prod/networkOne directory, terraform workspace select prodOne directory, -var-file=prod.tfvars + -backend-config=prod.hcl
StateSeparate key per directorySame backend, key prefix env:/<workspace>/Separate key via backend config
Visible in the repo?Yes: ls live/ shows every environmentNo: the current workspace is local CLI statePartly: tfvars files, but the active one is a flag
Different shape per envEasy (prod has extra resources)Only via terraform.workspace conditionalsOnly via variables and count
Credentials per envNatural: one role per directoryHard: same provider config for allPossible via variables
RiskSome duplication (thin root modules)Applying to the wrong workspaceForgetting a flag, mixing state and tfvars

HashiCorp's own guidance: CLI workspaces are not a good fit for environments that need separate credentials or access control. Use them for short-lived copies (a feature branch test), not for dev/prod.

Layout options

The tf-lab layout: thin roots over shared modules

Reusable logic lives in modules/. Each directory under live/ is a small root module: backend, provider, one module call, outputs.

tf-lab/
├── bootstrap/                 # state bucket (Lab 0), local state
├── modules/
│   ├── network/               # VPC, subnets, IGW (no NAT)
│   ├── app-bucket/            # S3 + encryption + PAB
│   └── ci-role/               # GitHub OIDC role (Lab 11)
├── live/
│   ├── dev/
│   │   ├── network/  backend.tf  main.tf  terraform.tfvars
│   │   └── app/      backend.tf  main.tf  terraform.tfvars
│   └── prod/
│       ├── network/  backend.tf  main.tf  terraform.tfvars
│       └── app/      backend.tf  main.tf  terraform.tfvars
├── policies/                  # checkov, trivy, opa (ch 9)
├── scripts/                   # plan-summary.sh (Lab 8)
└── Makefile                   # fmt, test, policy, plan-all

state keys in tflab-state-<account_id>:
  dev/network/terraform.tfstate    prod/network/terraform.tfstate
  dev/app/terraform.tfstate        prod/app/terraform.tfstate

Why this shape

  • Blast radius: a broken app plan cannot touch the network state.
  • Speed: each plan refreshes only its own resources.
  • Ownership: CODEOWNERS per directory (network team vs app team).
  • Credentials: dev and prod directories can use different roles or accounts.
  • Readability: the tree *is* the inventory; no hidden workspace state.
  • Duplication is limited to ~30 lines per root, and the lines differ on purpose (env name, CIDR, key).
Layout options

Monorepo or separate modules repository?

Both work. The question is how a module change reaches production.

Monorepo (tf-lab)

  • live/ refers to modules by relative path: source = "../../../modules/network".
  • A module change and its use land in one PR; dev and prod see it at the same commit.
  • Promotion = apply dev first, then prod, from the same commit (ch 11 pipeline order).
  • Simple for one team; CI must detect which roots a module change affects.

Modules repo + live repo

  • Modules are versioned with Git tags or a private registry.
  • source = "git::https://github.com/acme/tf-modules.git//network?ref=network-v1.4.0".
  • Promotion = bump ref in dev, then in prod: an explicit, reviewable diff.
  • Better for many consumers; more release work (changelogs, semver, ch 6).

Hybrid

  • Start as a monorepo; extract a module when other repos need it.
  • Registry modules from outside (terraform-aws-modules/vpc/aws) are always pinned with version.
  • Terraform 1.15+ accepts variables and locals in module source/version; keep them constant per environment so plans stay reviewable.
Layout options

Why not CLI workspaces for dev and prod?

Workspaces are real and useful — for the right job.

cd live/app
terraform workspace new prod
terraform workspace select prod
terraform plan -var-file=prod.tfvars

# state lands at (S3 backend):
#   s3://tflab-state-<acct>/env:/prod/app/terraform.tfstate
# the selected workspace is stored locally in
#   .terraform/environment

# in code
locals {
  env = terraform.workspace     # "default", "prod", ...
  instance_count = terraform.workspace == "prod" ? 3 : 1
}

Problems for long-lived environments

  • Nothing in the repo shows which workspaces exist or what is in them.
  • The active workspace is per-machine: an apply in the wrong one is one forgotten command away.
  • One provider configuration → the same credentials for dev and prod.
  • Environment differences turn into terraform.workspace == conditionals across the code.

Good uses: a disposable copy per pull request or developer (pr-123), destroyed afterwards. HCP Terraform "workspaces" are a different concept: one state + settings, closer to one live/ directory here.

Layout options

The layout as a diagram

Two environments, two components each, four states, one module source.

modules/ (shared code)networkVPC, subnets, IGWapp-bucketS3 + PAB + SSEci-roleLab 11live/dev (10.10.0.0/16)dev/networkkey dev/network/…dev/appkey dev/app/…live/prod (10.20.0.0/16)prod/networkkey prod/network/…prod/appkey prod/app/…SSMSSMone state per box; network applies first

Arrows from modules/ to prod are omitted for clarity: prod calls the same module sources. Blue arrows are data flowing through SSM parameters (section 38).

Chapter 10 · Part 10.2

Environments and promotion

Same code, different inputs: what may differ between dev and prod, and how a change moves forward.

Environments

A thin root module per environment

Only the inputs differ. The module call is identical, so prod runs exactly the code dev tested.

# live/prod/network/main.tf
terraform {
  required_version = ">= 1.14"
  required_providers {
    aws = { source = "hashicorp/aws", version = "~> 6.0" }
  }
}

provider "aws" {
  region = "ap-northeast-1"
  default_tags {
    tags = { Project = "tf-lab", Env = var.env, ManagedBy = "terraform" }
  }
}

variable "env"      { type = string }
variable "vpc_cidr" { type = string }

module "network" {
  source = "../../../modules/network"
  env    = var.env
  cidr   = var.vpc_cidr
}
# live/prod/network/terraform.tfvars
# (auto-loaded: plain `terraform plan` in CI
#  needs no -var-file flag)
env      = "prod"
vpc_cidr = "10.20.0.0/16"

# live/prod/network/backend.tf
terraform {
  backend "s3" {
    key          = "prod/network/terraform.tfstate"
    region       = "ap-northeast-1"
    encrypt      = true
    use_lockfile = true
    # bucket comes from -backend-config: it
    # contains the account ID and backend
    # blocks cannot use variables
  }
}

terraform init \
  -backend-config="bucket=$TF_STATE_BUCKET"
terraform plan -out=tfplan
Environments

What may differ between environments

If dev and prod differ in shape, dev stops predicting prod. Differ in size and safety settings, not in architecture.

OK to differ (inputs)

  • CIDR ranges, names, tags (Env).
  • Sizes and counts: instance class, replicas, retention days.
  • Protection: deletion_protection, longer backups in prod.
  • Accounts and roles: prod in its own account (recommended).

Should not differ (code)

  • Which modules and module versions are used (after promotion).
  • Which resources exist (no "prod-only" WAF that dev never exercises).
  • Provider versions: the lock file is per root — keep them aligned (ch 13).
  • Policies: the same make policy gate for every environment.

Make differences visible

  • Diff the inputs: diff live/{dev,prod}/network/terraform.tfvars.
  • Keep defaults in the module; tfvars hold only what changes.
  • Validate inputs (validation blocks, ch 2) so prod cannot get a dev-sized value by mistake.
Environments

Promotion: the same change, dev first

A change is proven in dev, then the identical code is applied to prod. How "identical" is guaranteed depends on the layout.

PR mergedcommit abc123plan + apply devnetwork, then appverify devtests, smoke checksplan prodreview: same diff?apply prodapproval (ch 11)Monorepo: promotion = run the prod roots at the same commit abc123 after dev succeeded.Modules repo: promotion = a PR that bumps ?ref=network-v1.4.0 in live/prod (dev bumped earlier).Either way: the prod plan should show the same kinds of changes as the dev plan, with prod values.Red flag: the prod plan contains changes the dev plan did not (drift, or the environments already differ).

Avoid changing prod first. If an emergency forces it, make the change in code (never in the console), apply prod, then bring dev to the same commit the same day.

Environments

Account per environment

Separate AWS accounts give the strongest isolation: a dev role cannot touch prod, even by mistake.

# live/prod/network/main.tf — prod in its own account
provider "aws" {
  region = "ap-northeast-1"
  assume_role {
    role_arn     = "arn:aws:iam::<prod-acct>:role/tflab-deployer"
    session_name = "tf-prod-network"
  }
  allowed_account_ids = ["<prod-acct>"]   # refuse anything else
  default_tags {
    tags = { Project = "tf-lab", Env = "prod", ManagedBy = "terraform" }
  }
}

Notes

  • allowed_account_ids makes Terraform error out if the credentials point at another account — cheap insurance.
  • State can stay in one central state account (bucket policy grants each deployer role its prefix) or live in each account.
  • CI uses one OIDC role per environment (ch 11); humans get read-only in prod.
  • tf-lab keeps dev and prod in one sandbox account to stay simple and free; the layout does not change when you split accounts — only provider blocks do.
Chapter 10 · Part 10.3

Dependencies between components

How the app component learns the VPC ID without sharing a state file.

Dependencies

Three ways to pass outputs between states

The app needs vpc_id from network. Choose how tightly the two states are coupled.

OptionHowProsCons
SSM Parameter Store (tf-lab)network writes /tflab/dev/network/vpc_id; app reads it with data "aws_ssm_parameter"Explicit contract; readers need only ssm:GetParameter; usable by non-Terraform toolsOne more resource per output; values are strings
terraform_remote_stateapp reads network's state file from S3No extra resources; typed outputsReader needs read access to the whole state (all attributes, secrets)
Data source lookupdata "aws_vpc" filtered by tagsNo coupling to Terraform at allFragile if tags are not unique; implicit contract
OrchestratorTerragrunt dependency, Stacks component outputsTyped, ordered, mocked in planAnother tool or platform (section 39)

Whatever you choose, treat published outputs as an API: add new ones freely, rename or remove them only with a migration (like module outputs, ch 6).

Dependencies

Publishing and consuming through SSM

The producer owns the parameter names; the consumer depends only on those names.

# live/dev/network/outputs.tf — producer
resource "aws_ssm_parameter" "vpc_id" {
  name  = "/tflab/${var.env}/network/vpc_id"
  type  = "String"
  value = module.network.vpc_id
}

resource "aws_ssm_parameter" "private_subnet_ids" {
  name  = "/tflab/${var.env}/network/private_subnet_ids"
  type  = "StringList"
  value = join(",", module.network.private_subnet_ids)
}
# live/dev/app/main.tf — consumer
data "aws_ssm_parameter" "vpc_id" {
  name = "/tflab/${var.env}/network/vpc_id"
}

data "aws_ssm_parameter" "private_subnet_ids" {
  name = "/tflab/${var.env}/network/private_subnet_ids"
}

locals {
  # insecure_value: non-sensitive copy for String/StringList
  # (value is always marked sensitive)
  vpc_id             = data.aws_ssm_parameter.vpc_id.insecure_value
  private_subnet_ids = split(",",
    data.aws_ssm_parameter.private_subnet_ids.insecure_value)
}

# app plan fails (ParameterNotFound) until network
# is applied: the dependency order made visible

Standard-tier SSM parameters are free. Do not put secrets here as String; secrets belong in Secrets Manager or SecureString and stay out of state where possible (ch 12).

Dependencies

terraform_remote_state and why tf-lab avoids it

It works and is common. Know the trade-off before you depend on it.

data "terraform_remote_state" "network" {
  backend = "s3"
  config = {
    bucket = "tflab-state-<account_id>"
    key    = "dev/network/terraform.tfstate"
    region = "ap-northeast-1"
  }
}

locals {
  vpc_id = data.terraform_remote_state.network.outputs.vpc_id
}

Trade-offs

  • Only root-module outputs are readable, but the reader downloads the whole state file — every attribute of every resource, including any secrets.
  • The reader's IAM role needs s3:GetObject on the producer's state key: state access spreads across teams.
  • The consumer is coupled to the producer's backend type and key layout; moving state breaks readers.
  • Fine inside one team and one account; prefer an explicit published contract (SSM, outputs API) across teams.
Dependencies

Ordering applies and destroys

Separate states mean Terraform no longer orders things for you. Your pipeline or script has to.

Apply order

  • Producers before consumers: network → app.
  • Within an environment: dev fully before prod.
  • Encode the order once (Makefile, CI matrix with needs:, Terragrunt, Stacks) — not in people's heads.
  • A consumer plan may show (known after apply) or fail until the producer is applied: plan in order too.

Destroy order

  • Reverse: app → network. Destroying network first fails (dependent ENIs, security groups) or orphans the app.
  • Before deleting a published output, check who reads it (search the repo for the parameter name).
  • Use prevent_destroy on resources that other states depend on (ch 3).

Cross-component changes

  • Renaming an output: add the new one, migrate consumers, then remove the old one — three PRs, never one.
  • Avoid cycles: if network needs something from app, the split is wrong; move the resource.
  • Changes that touch both components in one PR need the order spelled out in the PR description.
Dependencies

A dependency-ordered plan script

Plain shell is enough for a handful of components. Lab 10 builds this as make plan-all.

#!/usr/bin/env bash
# scripts/plan-all.sh ENV — plan every component in order
set -euo pipefail
ENV=${1:?usage: plan-all.sh dev|prod}
ORDER=(network app)          # producers first

for c in "${ORDER[@]}"; do
  dir="live/$ENV/$c"
  echo "== $dir"
  terraform -chdir="$dir" init -input=false \
    -backend-config="bucket=$TF_STATE_BUCKET" >/dev/null
  set +e
  terraform -chdir="$dir" plan -input=false \
    -out=tfplan -detailed-exitcode
  rc=$?
  set -e
  case $rc in
    0) echo "   no changes" ;;
    2) echo "   changes pending" ;;
    *) echo "   plan FAILED"; exit $rc ;;
  esac
done

Details

  • -chdir runs Terraform in another directory without cd; plan files are written relative to that directory.
  • -detailed-exitcode: 0 = no changes, 1 = error, 2 = changes present (ch 1). set -e must not treat 2 as failure.
  • Plan files stay next to each root (live/dev/app/tfplan) for review and for Lab 8's summary script.
  • If app reads SSM values that network has not applied yet, its plan fails — the script stops at the first error on purpose.
Chapter 10 · Part 10.4

Orchestration tools

Terragrunt, Terramate and HCP Terraform Stacks — what they add and what they cost.

Terragrunt

Terragrunt: DRY backends and explicit dependencies

A thin wrapper around Terraform/OpenTofu. Terragrunt 1.0 (Mar 2026) froze its CLI and HCL syntax for the 1.x line.

# live/root.hcl — shared by every unit
remote_state {
  backend = "s3"
  generate = {
    path      = "backend.tf"
    if_exists = "overwrite_terragrunt"
  }
  config = {
    bucket       = "tflab-state-${get_aws_account_id()}"
    key          = "${path_relative_to_include()}/terraform.tfstate"
    region       = "ap-northeast-1"
    encrypt      = true
    use_lockfile = true
  }
}
# live/dev/app/terragrunt.hcl — one unit
include "root" {
  path = find_in_parent_folders("root.hcl")
}

terraform {
  source = "../../../modules//app-bucket"
}

dependency "network" {
  config_path = "../network"
  mock_outputs = { vpc_id = "vpc-00000000" }
  mock_outputs_allowed_terraform_commands = ["plan", "validate"]
}

inputs = {
  env    = "dev"
  vpc_id = dependency.network.outputs.vpc_id
}
# from live/dev, in dependency order:
#   terragrunt run --all -- plan

The state key becomes dev/app/terraform.tfstate — the same layout as plain tf-lab, so you can adopt or drop Terragrunt without moving state. Verify remote_state keys such as use_lockfile against the Terragrunt version you pin.

Orchestration

Terramate and HCP Terraform Stacks

Two other answers to "many roots, one change".

Terramate

  • Open-source CLI that treats each root as a stack (a stack {} block, usually in stack.tm.hcl).
  • Change detection: run only stacks changed in this Git diff (including changed local modules).
  • Orchestrates order between stacks and can generate shared HCL (backend, providers).
  • Leaves your Terraform code plain; adoption is incremental.

HCP Terraform Stacks (GA Sep 2025)

  • Configuration in *.tfcomponent.hcl (components = module calls) and *.tfdeploy.hcl (deployments = environments, regions, accounts).
  • Beta used .tfstack.hcl; GA renamed it to .tfcomponent.hcl.
  • Commands live in the main CLI as terraform stacks … (Terraform 1.13+); runs happen in HCP Terraform.
  • Components pass outputs to each other inside one stack, with ordering handled for you.
  • Stacks resources count toward HCP Terraform billing (resources under management).

What they all solve

  • Repeated backend and provider blocks.
  • Ordering and passing outputs between states.
  • Running only what changed.
  • Fan-out to many environments or regions.
Orchestration

Do you need an orchestrator?

Add a tool when the pain is real, not because the repo might grow.

SignalPlain Terraform + Makefile/CIAdd Terragrunt / Terramate / Stacks
Number of root modulesUp to a few dozenHundreds, many near-identical
Environments / regions / accounts2–4 environments, one regionMany accounts × regions (fan-out)
Backend/provider duplicationA few lines per root, rarely changedCopy-paste drift between roots is a real problem
Ordering between statesOne or two chains (network → app)A deep graph that people get wrong
Team skills and toolingEveryone knows plain TerraformTeam willing to learn and pin another tool
PlatformSelf-run CI (GitHub Actions)Already on HCP Terraform → Stacks is native

tf-lab stays plain on purpose: four roots, one chain, a Makefile. Lab 10 shows the same layout in Terragrunt as a read-only comparison so you can judge the trade-off yourself.

Common mistakes

Layout mistakes that hurt later

Most of these are cheap to avoid on day one and expensive to undo once state exists.

Structure

  • One giant root for everything: every plan touches everything, every apply risks everything.
  • Workspaces for dev/prod with one set of credentials.
  • Environment logic as terraform.workspace == "prod" conditionals across modules.
  • Prod-only resources that dev never exercises.

Dependencies

  • terraform_remote_state across teams: everyone can read everyone's state.
  • Removing or renaming a published output in the same PR as its consumers.
  • Applying app before network in a fresh environment, or destroying network first.
  • Cycles between components ("network needs the app's security group").

Promotion

  • Hotfixing prod first and forgetting dev.
  • Modules pinned to a branch (?ref=main) instead of a tag.
  • Lock files diverging between dev and prod roots.
  • Copying a root to make a new env without changing the backend key — two environments, one state.
Self-check

Layout: can you answer these?

Explain each with a path, a key or a command.

Layouts

  • List three ways to model environments and one risk of each.
  • Where does the S3 backend store state for workspace prod?
  • Why is a thin root module per environment worth the duplication?
  • When are CLI workspaces the right tool?
  • What does allowed_account_ids protect against?

Promotion and dependencies

  • How is "same module version" guaranteed in a monorepo? In a modules repo?
  • What may differ between dev and prod, and what should not?
  • Compare SSM parameters and terraform_remote_state for passing vpc_id.
  • In which order do you apply and destroy network and app, and why?
  • How do you rename a published output safely?

Tools

  • What does -detailed-exitcode return for "changes present"?
  • Write a Terragrunt dependency block with mock outputs for plan.
  • What command plans every Terragrunt unit in dependency order?
  • What are .tfcomponent.hcl and .tfdeploy.hcl for?
  • Name two signals that justify adding an orchestrator.

Practice: Lab 10 in Hands-on labs — build live/prod/{network,app} from the same modules as dev, give each root its own backend key and tfvars, and plan all components in dependency order with one command.

Chapter 11

CI/CD for Terraform

Every change goes through a pull request: checks and a plan per component on the PR, a human reviews the plan, and the pipeline applies exactly that plan after merge.

Chapter 11 · Part 11.1

Pipeline design

What runs on a pull request, what runs after merge, and which plan gets applied.

Pipeline design

Why Terraform belongs in a pipeline

Running terraform apply from laptops works for one person. With a team it produces surprises nobody reviewed.

Laptop applies

  • Whatever is in the local checkout gets applied, including uncommitted edits.
  • Different Terraform and provider versions per person.
  • Long-lived admin credentials on many machines.
  • No record of who applied what, or which plan they saw.

Pipeline applies

  • Only code merged to main reaches real infrastructure.
  • One pinned Terraform version, one lock file.
  • Short-lived credentials from OIDC (the CI/CD deck, ch 4).
  • The plan is reviewed in the PR and the applied plan is logged.

Still needed outside CI

  • Bootstrap: the state bucket and the CI roles themselves (Lab 0, Lab 11).
  • Break-glass: a documented manual path for outages (ch 8).
  • Imports and state surgery, done by a person and reviewed like code (ch 5).
Pipeline design

The flow of one change

Two triggers, two credentials. The pull request proves the change is safe to apply; the merge applies it.

pull_request → role gha-tf-planChecksfmt, validate, tflintTests + policytest, checkov, trivyPlan per componentchanged dirs onlyPR commentplan summaryReviewcode + planpush to main → gha-tf-plan, then gha-tf-applyPlan againsaved tfplandev applyenvironment devApprovalenvironment prodprod applysame tfplanmergeThe approver reads the plan from this run; apply uses that file.

"Apply the reviewed plan" means: the plan the approver saw is the file that terraform apply tfplan executes. If anything changed in between, Terraform refuses the stale plan.

Pipeline design

What runs where

Cheap, credential-free checks first; anything that needs AWS runs only after them.

StageCommandsAWS accessFails the PR when
Formatterraform fmt -check -recursivenoneany file is not formatted
Validateterraform init -backend=false + terraform validatenonesyntax or type errors
Linttflint --recursivenonedeprecated syntax, invalid instance types, unused declarations
Unit teststerraform test in each module (mocks, ch 7)nonean assertion fails
Policymake policy: Checkov, Trivy, conftest (ch 9)none (static) / plan JSONa HIGH finding without a justified skip
Planterraform plan -out=tfplan per changed componentread-only + state lockplan errors; risky changes are flagged for review (ch 8)
Applyterraform apply tfplan after mergewrite, per environmentapply error → fix forward with a new PR

Plan-based policy (conftest on tfplan.json, ch 9) runs after the plan step, in the same job, because it needs the plan file.

Pipeline design

Plan only what changed

tf-lab has one root module per environment and component. A PR that touches live/dev/app should not lock or plan live/prod/network.

# which root modules does this change touch?
BASE=$(git merge-base origin/main HEAD)
git diff --name-only "$BASE" HEAD > changed.txt

if grep -q '^modules/' changed.txt; then
  # a shared module changed: plan every component
  ls -d live/dev/* live/prod/* > dirs.txt
else
  grep -oE '^live/(dev|prod)/[^/]+' changed.txt \
    | sort -u > dirs.txt
fi

# JSON matrix for GitHub Actions
jq -R -s -c 'split("\n") | map(select(length > 0))
  | map({dir: ., env: (split("/")[1]),
         name: (split("/")[1:] | join("-"))})' dirs.txt
# [{"dir":"live/dev/app","env":"dev","name":"dev-app"}]

Rules of thumb

  • A module change can affect every caller: plan them all (or only callers of that module, if you track the graph).
  • Run components of one environment in parallel, but give each state its own concurrency group (section 41).
  • live/shared/ci (the CI roles) stays out of the matrix: CI must not be able to rewrite its own permissions.
  • Orchestrators such as Terragrunt and Terramate compute this for you (ch 10).
  • Order matters when one component reads another's outputs: apply network before app (ch 10).
Pipeline design

Saved plans: what they guarantee

A saved plan is a contract: "these exact actions against this exact state". Terraform enforces the second half.

Guarantees

  • terraform apply tfplan performs only the actions in the file; no new diff is computed.
  • If the state changed since the plan (another apply), Terraform stops: Saved plan is stale.
  • Applying needs the same configuration, Terraform version and provider versions (same lock file).

Handle with care

  • A plan file contains sensitive values in clear text (variables, attributes).
  • Store it as a short-lived CI artifact (retention-days: 1), never commit it, never attach it to a public PR.
  • Ephemeral variables are not saved in the plan; pass them again at apply.
  • Plans age: reality may drift after the plan even if state does not. Apply soon after approval.

Alternatives

  • Re-plan and auto-apply on merge: simpler, but what runs may differ from what was reviewed.
  • Atlantis and HCP Terraform keep the plan server-side and apply it on command (section 42).
  • This deck: plan on main after merge, approve, apply that file in the same workflow run.
Chapter 11 · Part 11.2

GitHub Actions with OIDC

Two roles, three jobs, one concurrency group per state.

GitHub Actions

Two roles, trusted by different token subjects

The OIDC exchange is covered in the CI/CD deck (ch 4). Here the point is splitting who may plan from who may apply.

tf-lab CI roles

RoleTrusted sub (repo created after 15 Jul 2026)Permissions
gha-tf-planrepo:<you>@<oid>/tf-lab@<rid>:pull_request and …:ref:refs/heads/mainReadOnlyAccess + state read + lock-file write
gha-tf-apply…:environment:dev and …:environment:prod onlywhat the components create + state read/write

Why it works

  • A PR job, even from a malicious branch, can only read and plan.
  • Only jobs that run in a GitHub environment get write credentials, and prod has required reviewers.
  • Repos created, renamed or transferred after 15 Jul 2026 get the immutable sub format with numeric IDs (CI/CD deck, ch 4).
  • Fork PRs get no OIDC token at all: skip the plan job for them.

One apply role per environment (or one AWS account per environment with its own role) is stronger still; the lab uses one role and relies on environment protection to keep the setup small.

GitHub Actions

State permissions for the plan role

With use_lockfile = true, even a plan writes: it creates and deletes <key>.tflock next to the state object.

{
  "Version": "2012-10-17",
  "Statement": [
    { "Sid": "ListStateBucket",
      "Effect": "Allow",
      "Action": "s3:ListBucket",
      "Resource": "arn:aws:s3:::tflab-state-<acct>" },
    { "Sid": "ReadState",
      "Effect": "Allow",
      "Action": "s3:GetObject",
      "Resource": "arn:aws:s3:::tflab-state-<acct>/*" },
    { "Sid": "LockFiles",
      "Effect": "Allow",
      "Action": ["s3:GetObject", "s3:PutObject",
                 "s3:DeleteObject"],
      "Resource":
        "arn:aws:s3:::tflab-state-<acct>/*.tflock" }
  ]
}

Notes

  • The S3 backend docs list s3:GetObject, s3:PutObject and s3:DeleteObject on the lock file when use_lockfile is on.
  • s3:DeleteObject is never needed on the state file itself: Terraform does not delete it.
  • The apply role additionally needs s3:PutObject on */terraform.tfstate.
  • With a KMS key on the bucket, add kms:Decrypt (plan) and kms:Encrypt/kms:GenerateDataKey (apply).
  • Planning without the lock (-lock=false) avoids the write but lets a plan race an apply: keep the lock.
GitHub Actions

The PR workflow: detect and check

One workflow file handles both triggers. Credential-free checks run before anything touches AWS.

# .github/workflows/terraform.yml
name: terraform
on:
  pull_request: {paths: ["live/**", "modules/**", "policies/**"]}
  push: {branches: [main], paths: ["live/**", "modules/**"]}
permissions:
  contents: read
  id-token: write        # OIDC
  pull-requests: write   # plan comments
jobs:
  detect:
    runs-on: ubuntu-latest
    outputs:
      dirs: ${{ steps.d.outputs.dirs }}
    steps:
      - uses: actions/checkout@<sha>
        with: {fetch-depth: 0}
      - id: d
        run: echo "dirs=$(scripts/changed-dirs.sh)" >> "$GITHUB_OUTPUT"
  checks:
    if: github.event_name == 'pull_request'
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@<sha>
      - uses: hashicorp/setup-terraform@<sha>
        with: {terraform_version: 1.17.0}
      - uses: terraform-linters/setup-tflint@<sha>
      - run: terraform fmt -check -recursive
      - run: tflint --init && tflint --recursive
      - name: module tests (mocked, no AWS)
        run: |
          for m in modules/*/; do
            (cd "$m" && terraform init -backend=false \
               && terraform test)
          done
      - run: make policy     # Lab 9

scripts/changed-dirs.sh is the diff script from section 40; on push it compares github.event.before with github.sha instead of the merge base.

GitHub Actions

The plan job: one matrix entry per component

Same job for PRs and for main. On a PR it comments; on main it uploads the plan for the apply job.

  plan:
    needs: detect
    if: >-
      needs.detect.outputs.dirs != '[]' &&
      (github.event_name == 'push' ||
       github.event.pull_request.head.repo.full_name
         == github.repository)
    runs-on: ubuntu-latest
    strategy:
      fail-fast: false
      matrix:
        c: ${{ fromJSON(needs.detect.outputs.dirs) }}
    concurrency:            # one run per state at a time
      group: tf-${{ matrix.c.dir }}
      cancel-in-progress: false
    defaults: {run: {working-directory: "${{ matrix.c.dir }}"}}
    steps:
      - uses: actions/checkout@<sha>
      - uses: hashicorp/setup-terraform@<sha>
        with: {terraform_version: 1.17.0, terraform_wrapper: false}
      - uses: aws-actions/configure-aws-credentials@<sha>
        with:
          role-to-assume: arn:aws:iam::<acct>:role/gha-tf-plan
          aws-region: ap-northeast-1
      - run: terraform init -input=false -lockfile=readonly
               -backend-config="bucket=tflab-state-<acct>"
      - run: terraform plan -input=false -out=tfplan
      - run: terraform show -no-color tfplan > plan.txt
      - name: comment on the PR
        if: github.event_name == 'pull_request'
        env: {GH_TOKEN: "${{ github.token }}"}
        run: |
          { echo "### ${{ matrix.c.dir }}"; echo '```'
            head -c 60000 plan.txt; echo '```'; } > c.md
          gh pr comment ${{ github.event.number }} -F c.md
      - if: github.event_name == 'push'
        uses: actions/upload-artifact@<sha>
        with:
          name: tfplan-${{ matrix.c.name }}-${{ github.run_attempt }}
          retention-days: 1
          path: ${{ matrix.c.dir }}/tfplan

terraform_wrapper: false: real exit codes and plain stdout. -lockfile=readonly: fail if the lock file would change (ch 12). PR comments max out at 65,536 characters.

GitHub Actions

The apply job: environment gate, then the saved plan

The job waits for environment approval before it gets credentials. Then it applies the file uploaded by the plan job of the same run.

  apply:
    needs: [detect, plan]
    if: github.event_name == 'push'
    runs-on: ubuntu-latest
    strategy:
      fail-fast: false
      matrix:
        c: ${{ fromJSON(needs.detect.outputs.dirs) }}
    environment: ${{ matrix.c.env }}   # dev | prod
    concurrency:
      group: tf-${{ matrix.c.dir }}
      cancel-in-progress: false
    defaults: {run: {working-directory: "${{ matrix.c.dir }}"}}
    steps:
      - uses: actions/checkout@<sha>
      - uses: hashicorp/setup-terraform@<sha>
        with:
          terraform_version: 1.17.0
          terraform_wrapper: false
      - uses: aws-actions/configure-aws-credentials@<sha>
        with:
          role-to-assume: arn:aws:iam::<acct>:role/gha-tf-apply
          aws-region: ap-northeast-1
          role-session-name: apply-${{ github.run_id }}
      - uses: actions/download-artifact@<sha>
        with:
          name: tfplan-${{ matrix.c.name }}-${{ github.run_attempt }}
          path: ${{ matrix.c.dir }}
      - run: terraform init -input=false -lockfile=readonly
               -backend-config="bucket=tflab-state-<acct>"
      - run: terraform show -no-color tfplan   # what we apply
      - run: terraform apply -input=false tfplan

# GitHub → Settings → Environments:
#   dev : no reviewers, branch main only
#   prod: required reviewers, branch main only,
#         prevent self-review

The sub of this job ends with environment:dev or environment:prod, so only these jobs can assume gha-tf-apply. The prod approver reads the plan in the plan job log of the same run before approving.

GitHub Actions

Ordering, re-runs and failures

A matrix applies components in parallel. That is fine within one layer and wrong across layers.

Ordering components

  • app reads network outputs from SSM (ch 10): if both change, network must apply first.
  • Option 1: one job per layer (apply-network, then apply-app with needs:).
  • Option 2: keep cross-layer changes in separate PRs (simple, common).
  • Option 3: an orchestrator that knows the graph (Terragrunt run --all, Terramate, Stacks).

When apply fails

  • Some resources may already exist: the state records what succeeded.
  • Do not re-run the same job: its plan is now stale and will be refused.
  • Fix forward: push a commit, get a new plan, review, apply.
  • If the lock stays behind after a cancelled job, terraform force-unlock <ID> by a person, after checking nothing is running.

Concurrency details

  • concurrency with cancel-in-progress: false queues runs; GitHub keeps at most one pending run per group and cancels older pending ones.
  • The S3 lock file is the real safety net: a second apply fails to get the lock.
  • Plan jobs share the group with apply jobs so a plan never reads half-applied state.
Chapter 11 · Part 11.3

Other runners

Atlantis, HCP Terraform and hosted platforms do the same job with different trade-offs.

Other runners

Atlantis: plan and apply from PR comments

A self-hosted server that receives PR webhooks, runs plan for changed projects and applies when someone comments atlantis apply.

# atlantis.yaml (repo root)
version: 3
projects:
  - name: dev-network
    dir: live/dev/network
    autoplan:
      when_modified: ["*.tf", "../../../modules/network/**"]
  - name: prod-network
    dir: live/prod/network
    apply_requirements: [approved, mergeable]

# in a PR comment:
#   atlantis plan -p prod-network
#   atlantis apply -p prod-network

Trade-offs

  • Plans are stored on the server and applied as-is; the PR shows the plan as a comment.
  • Locks a project (directory + workspace) per PR, so two PRs cannot plan the same state.
  • Applies before merge by default; apply_requirements adds approval and mergeability checks (the server must allow repo-level overrides).
  • You run and secure the server: it holds powerful cloud credentials, so isolate it and restrict who can comment.
  • Good fit for many root modules and teams who want PR-centric workflows without writing YAML pipelines.
Other runners

HCP Terraform and hosted platforms

A managed backend plus remote runs. The cloud block replaces the S3 backend.

terraform {
  cloud {
    organization = "tf-lab-org"
    workspaces {
      name = "dev-network"
      # or: tags = ["tf-lab"]  (select many)
    }
  }
}

# terraform login: API token for the CLI
# terraform plan: runs remotely,
#   logs stream to your terminal

Workflows

  • VCS-driven: the workspace watches a repo; PRs get speculative plans, merges trigger runs.
  • CLI-driven: terraform plan/apply on a laptop or CI, executed remotely.
  • API-driven: your pipeline uploads configuration versions.
  • Built in: state, locking, run history, policy checks, private registry, dynamic provider credentials (OIDC to AWS).

Before you choose

  • Free tier (since 31 Mar 2026): up to 500 managed resources and one concurrent run; paid tiers bill per managed resource.
  • Stacks (GA Sep 2025) orchestrate many components and deployments (ch 10).
  • Alternatives with similar models: Spacelift, env0, Scalr.
  • Self-managed GitHub Actions: no per-resource cost, but you build state, locks, approvals and audit yourself.
Other runners

Choosing a runner

All of them run the same Terraform. The differences are where plans live, who holds credentials, and what you have to operate.

GitHub Actions (this deck)AtlantisHCP Terraform
Where plans liveCI artifacts (you manage retention)Atlantis server diskHCP run history
CredentialsOIDC roles per jobServer role or per-project rolesDynamic provider credentials or variables
Apply timingAfter merge (environment approval)Before merge by defaultAfter merge (VCS) or on confirm
LockingS3 lock file + concurrencyProject locks per PR + backend lockWorkspace lock
PolicyCheckov, Trivy, conftest stepsBuilt-in conftest policy checksSentinel, OPA, Terraform Policy
You operateWorkflow YAMLA server with cloud credentialsNothing (SaaS); pay per resource

Pick one runner per state. Two systems applying the same state is how locks get broken and drift appears.

Chapter 11 · Part 11.4

Drift and scheduled runs

Find out that reality changed before the next PR does.

Drift

Two kinds of "difference"

A scheduled plan answers "would applying main change anything?". A refresh-only plan answers "did reality move away from state?".

terraform plan -detailed-exitcode

  • Compares **code on main** with refreshed reality.
  • Exit 0: no changes, 1: error, 2: changes present.
  • Catches console edits and merged-but-not-applied code.
  • Best default for a nightly job.

terraform plan -refresh-only -detailed-exitcode

  • Compares state with reality; ignores the code.
  • Shows only what changed outside Terraform.
  • Useful when main legitimately contains unapplied changes.
  • Accepting it (apply -refresh-only) updates state, not resources.

After drift is found

  • Someone changed it on purpose → codify it in a PR (ch 5).
  • Someone changed it by mistake → apply main to revert.
  • A provider or service default changed → ignore_changes only with a written reason (ch 3).
  • Prevent repeats: read-only console roles, SCPs (ch 13).
Drift

A nightly drift job that opens an issue

Runs with the read-only plan role on main. Exit code 2 becomes one issue per component, not one per night.

# .github/workflows/drift.yml
name: drift
on: {schedule: [{cron: "0 21 * * *"}], workflow_dispatch: {}}
permissions: {contents: read, id-token: write, issues: write}
jobs:
  drift:
    runs-on: ubuntu-latest
    strategy:
      fail-fast: false
      matrix:
        dir: [live/dev/network, live/dev/app,
              live/prod/network, live/prod/app]
    concurrency: {group: "tf-${{ matrix.dir }}",
                  cancel-in-progress: false}
    steps:
      - uses: actions/checkout@<sha>
      - uses: hashicorp/setup-terraform@<sha>
        with: {terraform_version: 1.17.0, terraform_wrapper: false}
      - uses: aws-actions/configure-aws-credentials@<sha>
        with: {role-to-assume: "arn:aws:iam::<acct>:role/gha-tf-plan",
               aws-region: ap-northeast-1}   # cron 21:00 UTC = 06:00 JST
      - name: plan
        id: p
        working-directory: ${{ matrix.dir }}
        run: |
          terraform init -input=false -lockfile=readonly \
            -backend-config="bucket=tflab-state-<acct>"
          set +e; terraform plan -input=false -no-color \
            -detailed-exitcode > plan.txt; code=$?; set -e
          echo "code=$code" >> "$GITHUB_OUTPUT"
          [ "$code" -ne 1 ]          # 1 = real error
      - name: open or update an issue
        if: steps.p.outputs.code == '2'
        env: {GH_TOKEN: "${{ github.token }}"}
        run: |
          t="Drift: ${{ matrix.dir }}"
          n=$(gh issue list --state open --search "$t in:title" \
                --json number --jq '.[0].number')
          { echo '```'; tail -c 60000 ${{ matrix.dir }}/plan.txt
            echo '```'; } > body.md
          if [ -n "$n" ]; then gh issue comment "$n" -F body.md
          else gh issue create -t "$t" -F body.md; fi

Use the same concurrency key as the plan/apply workflow for each state. GitHub disables scheduled workflows in public repos after 60 days without activity.

Drift

Keeping versions fresh with update bots

Pinned versions are safe until they are old. Bots turn upgrades into ordinary reviewed PRs.

# .github/dependabot.yml
version: 2
updates:
  - package-ecosystem: terraform
    directories: ["/live/*/*", "/modules/*"]
    schedule: {interval: weekly}
    groups:
      aws-provider:
        patterns: ["hashicorp/aws"]
  - package-ecosystem: github-actions
    directory: "/"
    schedule: {interval: weekly}

What to expect

  • Dependabot's terraform ecosystem updates provider and module versions; Renovate's terraform manager also updates required_version and can maintain the lock file.
  • Each update PR runs the full pipeline: a provider upgrade shows up as a plan diff you can review.
  • Group the AWS provider into one PR per week; read its changelog for "breaking" and "deprecated" before merging.
  • Major versions (AWS provider v5 → v6) deserve their own planned upgrade (ch 13), not an auto-merge.
  • Pin GitHub Actions by SHA; the bot keeps the SHAs current (the Mar 2026 trivy-action incident, ch 12).
Common mistakes

CI/CD mistakes that hurt later

Each of these has caused a production outage or a credential leak somewhere.

Pipeline design

  • Auto-applying on PR open: an unreviewed branch reaches production.
  • Re-planning at apply time and calling it "the reviewed plan".
  • Planning every component on every PR: slow, noisy, and locks states unrelated to the change.
  • Running -target in CI to "speed things up" (ch 8).

Credentials and artifacts

  • One admin role for plan and apply, trusted for repo:org/tf-lab:*.
  • Plan files uploaded with default 90-day retention, or posted in a public PR.
  • Secrets passed as TF_VAR_ in logs with set -x.
  • CI role allowed to modify its own IAM policy (live/shared/ci in the matrix).

Operations

  • Two runners (Atlantis and Actions) applying the same state.
  • No concurrency groups: plan reads state mid-apply.
  • Drift job that fails silently because exit code 2 is treated as an error.
  • Lock file not committed: CI downloads a different provider than reviewers tested.
Self-check

CI/CD for Terraform: can you answer these?

Answer with the exact flag, permission or setting.

Design

  • Which checks can run without AWS credentials?
  • What does "apply the reviewed plan" mean, and what makes a saved plan stale?
  • Why does a plan job need s3:PutObject with use_lockfile?
  • How do you decide which root modules a PR must plan?
  • Why must live/shared/ci stay out of the pipeline?

GitHub Actions

  • Which sub values does gha-tf-plan trust, and which does gha-tf-apply trust?
  • What does terraform_wrapper: false change?
  • Why does the plan job skip fork PRs?
  • How long should a plan artifact live, and why?
  • What happens to a second run in the same concurrency group?

Drift and runners

  • What do exit codes 0, 1 and 2 mean with -detailed-exitcode?
  • When is -refresh-only the better drift check?
  • When does Atlantis apply, relative to merge?
  • Name two limits of the HCP Terraform Free tier.
  • Which upgrades should never be auto-merged?

Practice: Lab 11 in Hands-on labs — push tf-lab to GitHub, create the OIDC roles with modules/ci-role, and build the PR plan, gated apply and nightly drift workflows.

Chapter 12

Security and secrets

Who can change infrastructure, where secrets end up, and how to trust the providers and modules Terraform downloads.

Chapter 12 · Part 12.1

Credentials and least privilege

Terraform runs with whatever power you give it. Give it as little as the plan or apply needs.

Credentials

Terraform is a privileged automation

The identity that runs apply can usually create IAM roles. Whoever controls that identity controls the account.

What a stolen apply role can do

  • Create an IAM user or role with admin rights: persistence.
  • Read and rewrite the state: hide resources, change outputs other stacks consume.
  • Delete stateful resources (buckets, databases) in one apply.
  • Exfiltrate secrets from state, SSM and Secrets Manager.

Where the credentials come from

  • Laptops: SSO profiles (aws sso login), never long-lived access keys.
  • CI: OIDC federation, one role per purpose (ch 11; CI/CD deck, ch 4).
  • Other accounts: assume_role in the provider block from a central identity.
  • HCP Terraform: dynamic provider credentials (OIDC from the workspace).

Least-privilege layers

  • Split plan (read) and apply (write) roles.
  • One role per environment or, better, one AWS account per environment.
  • Permission boundaries on roles that Terraform creates.
  • SCPs in AWS Organizations as the final guard (AWS Security deck).
Credentials

Who can do what in tf-lab

Read the arrows as "may assume" or "may write". Everything else is denied by the bucket policy (Lab 12).

IdentitiesYou (SSO admin)laptop, break-glassPR / main jobsGitHub OIDCdev / prod jobsGitHub environmentsgha-tf-planReadOnlyAccessgha-tf-applyscoped writeAWS account (ap-northeast-1)tflab-state-<acct>state + .tflock objectsVPC, S3, SSM, logsdev and prod componentslive/shared/ciOIDC provider + rolesIAMadmin onlyread + lockread/writecreate/updateyou apply live/shared/ci by handPlan reads everything, changes nothing.Apply never touches IAM in the lab.CI never applies live/shared/ci.

gha-tf-apply must not be able to edit gha-tf-apply. Keeping the CI roles in a separately applied root module (live/shared/ci) is what makes that true.

Credentials

Provider credentials: assume_role per account

One identity logs in, Terraform assumes a narrow role in the target account. The role ARN is configuration; the secret material is not.

provider "aws" {
  region = "ap-northeast-1"

  assume_role {
    role_arn     = "arn:aws:iam::${var.account_id}:role/tflab-deployer"
    session_name = "tf-${var.env}"
    # optional: external_id, duration, tags,
    #           policy_arns to narrow the session
  }

  allowed_account_ids = [var.account_id]  # wrong profile? stop.

  default_tags {
    tags = {
      Project   = "tf-lab"
      Env       = var.env
      ManagedBy = "terraform"
    }
  }
}

Why each line matters

  • assume_role keeps the base credentials (SSO or OIDC) out of the configuration and gives CloudTrail a session name.
  • allowed_account_ids makes Terraform refuse to run if your credentials point at another account: cheap insurance against applying dev to prod.
  • Never put access_key / secret_key in provider blocks or .tfvars; use the default credential chain.
  • For several accounts, use provider aliases (ch 3) or one root module per account (ch 10).
  • The role ARN is not secret; committing it is fine.
Credentials

Protecting the state bucket

State is the most sensitive file Terraform produces: resource IDs, outputs, and any secret that went through a normal argument.

Bucket settings (Lab 0)

  • Versioning on: every apply leaves a recoverable previous version (ch 4).
  • Block Public Access on; Object Ownership "bucket owner enforced" (ACLs disabled).
  • Encryption: SSE-S3 by default; SSE-KMS with a customer managed key when you need key-policy control and CloudTrail records of decrypts.
  • Optional: lifecycle rule to expire non-current versions after N days.

Bucket policy (Lab 12)

  • Deny any request without TLS (aws:SecureTransport = false).
  • Deny object access to every principal except the admin role and the two CI roles (aws:PrincipalArn).
  • Keep s3:DeleteObject on *.tfstate away from CI: Terraform never deletes state objects.
  • Test it with a third role before relying on it.

Lock-out safety

  • A Deny on aws:PrincipalArn also applies to you if you get the ARN wrong.
  • SSO role ARNs contain a path (/aws-reserved/sso.amazonaws.com/…): copy the role ARN, not the session ARN.
  • If you lock yourself out, the account root user can still delete the bucket policy.
  • Apply the policy from bootstrap/, never from a CI-applied root module.
Credentials

Roles that Terraform creates: permission boundaries

An apply role that may call iam:CreateRole can create a role more powerful than itself. A permission boundary caps what any created role can do.

{
  "Sid": "CreateRolesOnlyWithBoundary",
  "Effect": "Allow",
  "Action": ["iam:CreateRole", "iam:PutRolePolicy",
             "iam:AttachRolePolicy"],
  "Resource": "arn:aws:iam::<acct>:role/tflab-*",
  "Condition": {
    "StringEquals": {
      "iam:PermissionsBoundary":
        "arn:aws:iam::<acct>:policy/tflab-boundary"
    }
  }
}

How it fits together

  • The apply role may create roles only under the tflab-* name and only with permissions_boundary = tflab-boundary.
  • Effective permissions of a created role = its policies ∩ the boundary.
  • Also deny iam:DeleteRolePermissionsBoundary and iam:CreatePolicyVersion on the boundary policy itself.
  • In HCL: resource "aws_iam_role" "x" { permissions_boundary = local.boundary_arn … }.
  • The labs avoid this complexity: gha-tf-apply gets no IAM write permissions at all.
Credentials

Auditing who changed what

Every Terraform API call is a CloudTrail event signed by the role session. Good session names turn CloudTrail into a deployment log.

# which CI run created or changed things today?
aws cloudtrail lookup-events \
  --lookup-attributes \
    AttributeKey=Username,AttributeValue=apply-1234567890 \
  --max-results 20 \
  --query 'Events[].[EventTime,EventName]' --output table

# who assumed the apply role, and from which run?
aws cloudtrail lookup-events \
  --lookup-attributes \
    AttributeKey=EventName,AttributeValue=AssumeRoleWithWebIdentity \
  --query 'Events[].CloudTrailEvent' --output text \
  | jq -r 'select(.requestParameters.roleArn
             | endswith("role/gha-tf-apply"))
           | [.eventTime, .requestParameters.roleSessionName]
           | @tsv'

What makes this work

  • role-session-name: apply-${{ github.run_id }} in the workflow (ch 11) puts the run ID into every event.
  • For assumed roles, the CloudTrail Username is the session name.
  • lookup-events covers 90 days of management events in one Region; use a trail to S3 or CloudTrail Lake for longer retention and queries across accounts.
  • The state bucket's versions plus S3 server access logs or CloudTrail data events show who wrote each state version.
  • Tag resources with ManagedBy = "terraform" (default_tags): an untagged resource in a managed VPC is a ClickOps signal.
Chapter 12 · Part 12.2

Secrets in Terraform

Where a password ends up depends on which kind of argument carries it.

Secrets

Where secrets leak

Terraform shows (sensitive value) on screen. That is a display setting, not protection.

PlaceContains secrets?Notes
Configuration (.tf, .tfvars in Git)Only if you put them thereNever commit secrets; use variables, data sources or ephemeral resources
CLI output and CI logsMasked for sensitive valuesTF_LOG=TRACE can print request bodies; never enable it in CI
Saved plan file (tfplan)Yes, in clear textIncludes variable values and planned attributes; treat it like the secret itself (ch 11)
State (local or S3)Yes, in clear textEvery attribute of every resource and data source, including sensitive ones
terraform show -json, output -jsonYesJSON output prints sensitive values unredacted
Ephemeral values / write-only argumentsNoNever written to plan or state (Terraform 1.10 / 1.11+)

Rule: if a secret must exist in Terraform at all, it should flow through an ephemeral resource or variable into a write-only argument, or be generated and stored by the service itself.

Secrets

sensitive vs ephemeral

Two different promises. sensitive hides a value from output; ephemeral keeps it out of plan and state files.

variable "db_password" {
  type      = string
  sensitive = true      # masked in CLI output
}                       # ...but stored in plan + state
                        #    wherever it is used

variable "api_token" {
  type      = string
  ephemeral = true      # never written to plan or state
}                       # (Terraform 1.10+)

output "token_for_child" {
  value     = var.api_token
  ephemeral = true      # allowed in child modules only;
}                       # root module outputs cannot be
                        # ephemeral

Rules for ephemeral values

  • They may only be used in ephemeral contexts: other ephemeral variables/outputs/resources, provider blocks, provisioner and connection blocks, and write-only arguments.
  • Using one in a normal argument is an error, which is exactly the point.
  • Ephemeral variables are not saved in a plan file: pass them again when you apply the saved plan.
  • ephemeralasnull(value) turns an ephemeral value into null where a normal value is required.
  • sensitive is still useful for non-secret-but-private values such as internal hostnames.
Secrets

Ephemeral resources

An ephemeral block reads or generates a value for this run only. Nothing is stored; it is fetched or generated again in every plan and apply.

# generate a password for this run only
ephemeral "random_password" "db" {   # hashicorp/random 3.7+
  length  = 24
  special = true
}

# read an existing secret without storing it
ephemeral "aws_secretsmanager_secret_version" "api" {
  secret_id = "tflab/api-token"
}

# use them only in ephemeral contexts, e.g. a provider
provider "github" {
  token = jsondecode(
    ephemeral.aws_secretsmanager_secret_version.api
      .secret_string)["token"]
}

Good to know

  • Ephemeral resources have an "open" and optional "renew"/"close" lifecycle, so providers can hand out short-lived credentials (for example a Vault lease) and revoke them after the run.
  • AWS provider examples: aws_secretsmanager_secret_version, aws_ssm_parameter, aws_kms_secrets, aws_eks_cluster_auth, aws_lambda_invocation.
  • Because an ephemeral random_password changes on every run, pair it with a write-only argument and an explicit *_wo_version (next slide).
  • Needs Terraform 1.10+ (and a provider version that implements the ephemeral resource; pin it).
Secrets

Write-only arguments: set it, never store it

A write-only argument is sent to the provider during apply and then forgotten. Its companion *_wo_version decides when to send it again.

ephemeral "random_password" "app" {
  length = 32
}

resource "aws_ssm_parameter" "app_secret" {
  name             = "/tflab/dev/app/secret"
  type             = "SecureString"
  value_wo         = ephemeral.random_password.app.result
  value_wo_version = 1    # bump to 2 to rotate
}

resource "aws_secretsmanager_secret_version" "db" {
  secret_id                = aws_secretsmanager_secret.db.id
  secret_string_wo         = ephemeral.random_password.app.result
  secret_string_wo_version = 1
}

resource "aws_db_instance" "main" {
  # ...
  password_wo         = ephemeral.random_password.app.result
  password_wo_version = 1
}

How it behaves

  • Terraform 1.11+ and a provider that implements the argument (AWS provider: password_wo, secret_string_wo, value_wo, master_password_wo, auth_token_wo, …).
  • State stores value_wo = null and value_wo_version = 1. Plans show (write-only attribute).
  • Changing only value_wo produces no diff: Terraform cannot compare a value it never kept. Bump *_wo_version to send it again.
  • Downstream consumers read the secret from SSM or Secrets Manager at runtime, not from Terraform outputs.
  • Importing a resource with a write-only argument: set *_wo_version to match what you want, then expect one update.
Secrets

Finding secrets that are already there

Before migrating, find out what is exposed today: in the repository, in current state, and in old state versions.

# 1) the repository (and its history)
gitleaks git .                  # or: trufflehog git file://.

# 2) attributes the provider marks as sensitive,
#    in the current state of one root module
terraform show -json | jq -r '
  .values.root_module.resources[]
  | select(.sensitive_values | length > 0)
  | "\(.address): \(.sensitive_values | keys | join(", "))"'
# random_password.app: result
# aws_ssm_parameter.app_secret: value

# 3) how many old copies exist in S3
aws s3api list-object-versions --bucket $TF_STATE_BUCKET \
  --prefix dev/app/terraform.tfstate \
  --query 'length(Versions)'

Reading the results

  • sensitive_values lists the attributes that are sensitive (from the provider schema or sensitive = true); their values sit in state.
  • Child module resources live under .values.root_module.child_modules[]: extend the filter for a full audit.
  • Every old version of the state object holds the values that were current then.
  • Fix order: stop new leaks (write-only, ephemeral), rotate the secret, then decide whether to purge old versions.
  • Checkov also flags hard-coded secrets in .tf files (its secrets framework).
Secrets

Choosing a pattern

From best to worst. The best secret is one Terraform never sees.

PatternSecret in state?ExampleUse when
Service-managed secretNoRDS manage_master_user_password = true (RDS stores and rotates it in Secrets Manager)The service supports it
Ephemeral + write-onlyNoephemeral "random_password" → password_wo / value_woTerraform must set the initial value
Reference by ARN/name onlyNoPass the secret ARN to ECS/Lambda; the workload reads it at runtimeConsumers can read Secrets Manager/SSM themselves
Normal random_password + normal argumentYespassword = random_password.db.resultLegacy code; migrate it
Secret in .tfvars or TF_VAR_Yes, and often in Git or CI logsdb_password = "hunter2"Never

Migrating a legacy secret: switch to the write-only argument, apply (the old value is removed from state on the next write), then rotate the secret, because the old value still sits in old state versions in S3.

Chapter 12 · Part 12.3

Supply chain

Providers and modules are code that runs with your cloud credentials.

Supply chain

What Terraform downloads and runs

A provider is an executable that receives your AWS credentials. A module is code that decides what gets created.

Providers

  • Downloaded by terraform init from the registry (registry.terraform.io/hashicorp/aws), a mirror or a cache.
  • Release SHA256SUMS files are signed; Terraform checks the signature and the package hash.
  • Registry tiers: official (HashiCorp), partner, community: know who maintains each provider you use.
  • Versions and hashes are pinned by .terraform.lock.hcl (next slide).

Modules

  • Registry modules: pinned by version; not recorded in the lock file.
  • Git modules: ?ref= a tag can move; ?ref=<full commit SHA> cannot.
  • Read a public module before you depend on it: what IAM does it create? which providers does it require?
  • Prefer a few well-maintained modules (terraform-aws-modules) or your own (ch 6).

CI tools

  • setup-terraform, tflint, checkov, trivy, conftest all run with the plan role's credentials.
  • 19 Mar 2026: tags of aquasecurity/trivy-action and setup-trivy were force-pushed to malicious commits that stole CI secrets (TeamPCP campaign).
  • Pinned SHAs were not affected. Pin actions by full SHA and tool binaries by version + checksum.
Supply chain

The dependency lock file

.terraform.lock.hcl pins the exact provider version and the hashes Terraform will accept. Commit it.

provider "registry.terraform.io/hashicorp/aws" {
  version     = "6.72.0"
  constraints = "~> 6.0"
  hashes = [
    "h1:Pq3...=",      # hash of the unpacked package
    "zh:0a1...",       # hashes of the release zips,
    "zh:1f2...",       #   one per platform, from the
    # ...              #   signed SHA256SUMS
  ]
}

# record h1: hashes for every platform you run on
terraform providers lock \
  -platform=darwin_arm64 \
  -platform=linux_amd64 \
  -platform=linux_arm64

# CI: fail instead of silently updating the file
terraform init -lockfile=readonly

What it gives you

  • Everyone and CI install the same provider build; a different package fails init with a checksum error.
  • init records h1: for the platform it ran on plus zh: hashes from the registry's signed checksums.
  • Installs from a plugin cache or mirror can only be checked against h1: hashes, so record them for all platforms with providers lock.
  • Upgrades are explicit: terraform init -upgrade rewrites versions and hashes, and the diff shows up in review.
  • Version in example output is illustrative; yours depends on when you run init.
Supply chain

Caches, mirrors and private registries

Speed up init and control what can be installed, without giving up hash verification.

# ~/.terraformrc  (or TF_CLI_CONFIG_FILE)
plugin_cache_dir = "$HOME/.terraform.d/plugin-cache"

provider_installation {
  network_mirror {
    url     = "https://tf-mirror.example.com/providers/"
    include = ["registry.terraform.io/hashicorp/*"]
  }
  direct {
    exclude = ["registry.terraform.io/hashicorp/*"]
  }
}

# fill a mirror directory from your lock file
terraform providers mirror -platform=linux_amd64 ./mirror

Options

  • Plugin cache (plugin_cache_dir or TF_PLUGIN_CACHE_DIR): one download per version on a machine or runner; still verified against the lock file.
  • Network mirror: an HTTPS server with the provider mirror protocol; useful for air-gapped or allow-listed environments.
  • Filesystem mirror: a local directory, e.g. baked into a runner image.
  • Private registry (HCP Terraform, Artifactory, others): publish internal modules and providers with versions.
  • Combined with -lockfile=readonly, only reviewed versions and hashes can be installed.
Common mistakes

Security mistakes in Terraform code and pipelines

Each one turns a reviewable change into an unreviewable risk.

Credentials

  • Access keys in provider blocks, .tfvars or repo secrets.
  • The CI apply role can edit its own trust policy.
  • No allowed_account_ids: a dev profile applies to prod.
  • One admin role for plan and apply.

Secrets

  • Believing sensitive = true keeps a value out of state.
  • Passwords in .tfvars committed "just for the sandbox".
  • Changing value_wo and expecting an update without bumping value_wo_version.
  • Plan files kept as CI artifacts for 90 days.
  • Forgetting old state versions in S3 still hold the migrated secret.

Supply chain

  • Lock file not committed, or regenerated on every CI run.
  • Git modules pinned to a branch or a movable tag.
  • CI actions pinned by tag (@v1, @0.35.0).
  • Community providers adopted without reading who maintains them.
Self-check

Security and secrets: can you answer these?

Answer with the exact argument, permission or command.

Credentials

  • What can an attacker do with a stolen apply role that can call iam:CreateRole?
  • What does allowed_account_ids protect against?
  • Which bucket-policy condition key matches the calling role, and how do SSO role ARNs differ?
  • How does a permission boundary limit roles that Terraform creates?

Secrets

  • Where does a sensitive variable end up?
  • Where may an ephemeral value be used?
  • Why does changing value_wo alone produce no plan diff?
  • What does state contain for a write-only argument?
  • Which RDS argument avoids handling the master password at all?

Supply chain

  • What do h1: and zh: hashes cover?
  • Why run terraform providers lock -platform=…?
  • Which dependencies does the lock file not pin, and how do you pin them?
  • What does -lockfile=readonly do in CI?
  • What made the Mar 2026 trivy-action compromise possible?

Practice: Lab 12 in Hands-on labs — see a password in state, move it to an ephemeral value and a write-only argument, lock down the state bucket, and record lock-file hashes for three platforms.

Chapter 13

Operating IaC at scale

Keeping a growing estate healthy: routine upgrades, fast plans on big states, governance that scales with teams, and a clear view of Terraform vs OpenTofu.

Chapter 13 · Part 13.1

Upgrades

Terraform core, providers and modules all move every month. Small, regular upgrades are cheap; a three-year jump is a project.

Upgrades

Three things that version independently

Each has its own changelog, its own constraint in code, and its own failure mode when you skip it for too long.

Terraform core

  • Constraint: required_version in the terraform block.
  • Minor releases every few months (1.14 Nov 2025 → 1.17 Oct 2026).
  • v1.x compatibility promise: state written by one 1.x release is readable by other 1.x releases — but config using newer features needs the newer core.
  • Pin the exact binary in CI (setup-terraform terraform_version) and locally (tenv, tfenv, .terraform-version).

Providers

  • Constraint: required_providers (version = "~> 6.0").
  • The exact version and hashes live in .terraform.lock.hcl — commit it.
  • AWS provider: weekly minors, a major every 1–2 years (v5 2023, v6 Jun 2025) with an upgrade guide.
  • terraform init -upgrade is the only command that moves the lock file to newer versions.

Modules

  • Constraint: version on registry modules, ?ref= on Git sources.
  • Not in the lock file: an unpinned module changes on the next init.
  • Major versions of modules often rename resources → they should ship moved blocks (ch 5, ch 6).
  • Upgrade one root at a time, starting with dev.

Rule of thumb: never combine a core upgrade, a provider major and a module major in one PR. Three small plans are reviewable; one combined plan is not.

Upgrades

A constraint strategy that keeps upgrades boring

Constraints say what is allowed; the lock file says what is used. Upgrades become a deliberate PR that changes the lock file.

# versions.tf in every root module
terraform {
  required_version = ">= 1.14, < 2.0"
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 6.0"     # any 6.x, never 7.0
    }
  }
}

# reusable modules: express what they NEED,
# not what you happen to run
terraform {
  required_version = ">= 1.11"
  required_providers {
    aws = { source = "hashicorp/aws", version = ">= 6.0" }
  }
}

Constraint operators

ConstraintAllows
= 6.20.0 / 6.20.0Exactly that version
~> 6.20>= 6.20, < 7.0 (rightmost component may grow)
~> 6.20.0>= 6.20.0, < 6.21.0 (patches only)
>= 6.0, < 7.0Explicit range, same as ~> 6.0
!= 6.31.0Exclude a known-bad release

Root modules pin tightly (~> + lock file); shared modules use lower bounds (>=) so callers can choose. A module that pins = 6.20.0 blocks every caller from upgrading.

Upgrades

The upgrade workflow, one root at a time

Same loop for core, providers and modules. The plan is the test: an upgrade that should be a no-op must produce "No changes".

Readchangelog + upgrade guideBumpconstraint / versioninit -upgradelock file changesvalidateremoved argumentsplanNo changes, no warningsPRlock diff + planUnexpected diff?Defaults changed → set the old value explicitly, or accept itAttribute renamed / removed → update code, re-planResource replaced → stop: read the guide, use moved orlifecycle, never apply a surprise replacementThen: dev → apply → soak → prod (same lock file)Automate the boring partRenovate / Dependabot open one PR per provider bumpCI runs plan for every affected root (ch 11)Lock file regenerated for all CI and laptop platformsMajors are grouped and labelled "needs reading"

Commit the lock file for every platform you run on: terraform providers lock -platform=linux_amd64 -platform=darwin_arm64. Otherwise a lock file holding only one platform’s h1: hashes (for example from a filesystem mirror, without the registry’s zh: hashes) fails on CI with checksum errors.

Upgrades

Worked example: AWS provider 5.x → 6.x (1/2)

v6.0 (GA Jun 2025) is a major release: read the "Version 6 Upgrade Guide" in the Registry before touching the constraint.

What v6 changed (examples)

  • Enhanced region support: most resources and data sources accept a region argument, so one provider configuration can manage several Regions without aliases.
  • Some long-deprecated resources and attributes were removed (for example the OpsWorks Stacks resources).
  • Several attributes changed type or allowed values (for example ElastiCache auto_minor_version_upgrade).
  • New deprecations: data.aws_region.current.name → data.aws_region.current.region.
# 0) start from a clean plan on 5.x
terraform plan            # No changes.

# 1) bump the constraint
#    version = "~> 5.0"  →  "~> 6.0"

# 2) download v6, rewrite the lock file
terraform init -upgrade
git diff .terraform.lock.hcl
#  -  version = "5.100.0"
#  +  version = "6.x.y"

# 3) removed arguments fail validate;
#    deprecated attributes warn in plan
terraform validate
terraform plan
# Warning: Deprecated attribute
#   The attribute "name" is deprecated. …

Upgrade the last 5.x minor first (~> 5.0 + init -upgrade) and fix every deprecation warning there: most v6 removals were warned about in late 5.x.

Upgrades

Worked example: AWS provider 5.x → 6.x (2/2)

Fix the code, then make the plan prove nothing real changes. Only then roll forward environment by environment.

# 4) fix the deprecation
- region = data.aws_region.current.name
+ region = data.aws_region.current.region

# 5) the real test
terraform plan -out=tfplan
# Plan: 0 to add, 0 to change, 0 to destroy.

# if a diff appears, classify it:
#   ~ update in-place  → default changed?
#   -/+ replace        → STOP, read the guide
terraform show -json tfplan | jq -r '
  .resource_changes[]
  | select(.change.actions != ["no-op"])
  | "\(.change.actions|join("/")) \(.address)"'

# 6) dev: apply; prod: same lock file, later
terraform apply tfplan

Optional after the upgrade

  • Collapse per-Region provider aliases into the new region argument — but as a separate PR.
  • Common practice: run a -refresh-only apply first, so state records the provider's new attributes before you refactor.
  • Watch for state upgrades: the first apply with v6 rewrites resource schemas in state; older v5 binaries may then refuse that state.
  • Keep a state backup: S3 versioning (Lab 0) already gives you one per write.
Upgrades

Deprecations are a backlog, not noise

A warning today is an error after the next major. Track them like any other debt.

Where they show up

  • terraform plan (and validate for some cases): "Deprecated attribute / argument / resource".
  • Backend config: the S3 backend warns on dynamodb_table → move to use_lockfile = true (ch 4).
  • Core: removed CLI flags or language features are listed in the release notes and upgrade guides.
  • Modules: variable descriptions or check blocks that warn about inputs being retired.

How to handle them

  • Fail CI on new warnings: count warning diagnostics in terraform plan -json output (one JSON message per line).
  • Fix them in the same minor line where they appear.
  • For your own modules: deprecate an input with a validation or check warning for one minor, remove it in the next major.
terraform plan -json \
 | jq -r 'select(.type=="diagnostic"
     and .diagnostic.severity=="warning")
   | .diagnostic.summary' | sort | uniq -c
#   2 Deprecated attribute
#   1 Argument is deprecated
Upgrades

Module upgrades: let moved blocks carry the refactor

A module author who renames resources must ship moved blocks. Callers then upgrade with a no-op plan instead of a destroy/create.

# modules/network v2.0.0 — inside the MODULE
# v1 used count, v2 uses for_each by AZ
moved {
  from = aws_subnet.public[0]
  to   = aws_subnet.public["ap-northeast-1a"]
}
moved {
  from = aws_subnet.public[1]
  to   = aws_subnet.public["ap-northeast-1c"]
}

# caller: bump the ref, nothing else
module "network" {
  source = "git::https://github.com/<you>/tf-modules.git//network?ref=v2.0.0"
  ...
}
# plan:
#  module.network.aws_subnet.public[0] has moved to
#  module.network.aws_subnet.public["ap-northeast-1a"]
#  Plan: 0 to add, 0 to change, 0 to destroy.

Rules for module majors

  • Keep moved blocks for at least one major (callers skip versions).
  • Write the upgrade path in CHANGELOG.md: removed inputs, new required inputs, expected plan.
  • Test it: a terraform test run that applies v1 and plans v2 is the strongest proof (ch 7).
  • If a refactor cannot be expressed with moved (e.g. a resource type changed), document removed + import steps (ch 5).
Chapter 13 · Part 13.2

Performance and large estates

When plan takes ten minutes, people stop reading it. Measure where the time goes, then fix the cause.

Performance

Where plan time goes

A plan is: load config and providers, refresh every resource in state, walk the graph, diff. On large states, refresh dominates.

Typical costs

  • Refresh: at least one read API call per resource; some resources need several (S3 buckets read many sub-APIs).
  • Provider startup: each provider is a plugin process; dozens of aliases multiply it.
  • Graph walk: limited by -parallelism (default 10 concurrent operations).
  • Data sources: read on every plan.
  • API throttling: retries with backoff can turn seconds into minutes.

Measure before tuning

  • time terraform plan with and without refresh (Lab 13).
  • terraform state list | wc -l: resources per state.
  • TF_LOG=DEBUG TF_LOG_PATH=plan.log terraform plan — look for throttling and retry messages.
  • Count data sources: grep -c "^data " *.tf.
  • In CI, keep the timing of every plan as a metric.

Usual fixes, in order

  • Split the state (smaller blast radius and faster plans, ch 4 and ch 10).
  • Replace data-source lookups with values passed in (SSM parameter, outputs).
  • Cache providers in CI (TF_PLUGIN_CACHE_DIR).
  • Use a smarter refresh mode for PR plans (next slide).
  • Tune -parallelism only after the above — higher values increase throttling.
Performance

Refresh modes compared

Every option trades plan speed against how much drift you can see. Pick per use case, not once for the whole team.

ModeWhat is refreshedSees drift on…Use for
terraform plan (default)Every resource in stateEverythingProd applies, nightly drift job
plan -minimal-refresh (1.17)Only resources with proposed changes; others use cached stateOnly resources you are changingFast PR plans on large states
plan -refresh=falseNothingNothing — plans against stale stateEmergencies, debugging config
plan -refresh-onlyEverything, but proposes only state updatesEverything (that is the point)Reviewing and accepting drift (ch 5)
plan -target=ADDRTarget and its dependenciesOnly the targeted partRecovery only; never routine (ch 8)

-minimal-refresh is new and opt-in. A PR plan with it can say "No changes" while an untouched resource has drifted — keep a full-refresh plan before apply in prod and in the scheduled drift job.

Performance

Provider and API tuning

Most "Terraform is slow" reports on AWS are throttling or plugin downloads.

provider "aws" {
  region = "ap-northeast-1"

  # SDK retries: "standard" (default) or "adaptive"
  # (client-side rate limiting when throttled)
  retry_mode  = "adaptive"
  max_retries = 25         # the default; raise if throttled

  default_tags {
    tags = { Project = "tf-lab", ManagedBy = "terraform" }
  }
}

# CI: reuse downloaded providers between jobs
export TF_PLUGIN_CACHE_DIR="$HOME/.terraform.d/plugin-cache"
mkdir -p "$TF_PLUGIN_CACHE_DIR"
# cache that directory keyed on .terraform.lock.hcl

Also worth knowing

  • terraform providers mirror ./mirror downloads providers for air-gapped or rate-limited CI; point the CLI config at it with provider_installation { filesystem_mirror { … } }.
  • Fewer provider aliases: with AWS provider v6, per-resource region can replace one alias per Region.
  • Big for_each over many accounts or Regions in one root is a design smell: split by account/Region.
  • HCP Terraform and other runners have their own concurrency limits per organization.
Performance

Big states: when and how to split

Split by lifecycle and ownership, not by resource type. A good split makes plans fast and reviews small.

Signs a state is too big

  • Plans take minutes; people use -target to avoid waiting.
  • Unrelated teams are blocked by each other's locks.
  • A small change shows dozens of unrelated diffs (drift you never clean up).
  • Hundreds of resources from different lifecycles (network vs app vs data).

How to split safely

  • Create the new root module and backend key.
  • Move resources with removed { lifecycle { destroy = false } } in the old root and import blocks in the new one (ch 5).
  • Replace references with SSM parameters or remote-state outputs (ch 4).
  • Both plans must show only "forget" and "import" — no create, no destroy.
  • Alternative: terraform state mv -state-out on pulled copies (older, error-prone).

Target sizes (guidance, not rules)

  • One state per environment × component (tf-lab: dev/network, dev/app, …).
  • A few hundred resources per state is usually comfortable on AWS.
  • Things that change together and are owned by one team belong together.
  • Shared foundations (network, DNS) change rarely: keep them separate from apps.
Chapter 13 · Part 13.3

Team workflows and governance

Guardrails that let many teams change infrastructure without one central bottleneck.

Governance

Ownership: who may change what

Git already has the tools: code owners, rulesets and required checks. Map them onto the repository layout (ch 10).

# .github/CODEOWNERS
/modules/network/      @<org>/platform-network
/modules/ci-role/      @<org>/platform-security
/live/prod/            @<org>/platform-leads
/live/*/app/           @<org>/team-shop
/policies/             @<org>/platform-security
*.tftest.hcl           @<org>/platform

# ruleset on main (GitHub):
#  - require PR, 1–2 approvals, code owner review
#  - required checks: fmt, validate, tflint,
#    test, checkov, trivy, conftest, plan
#  - no force push, no bypass except break-glass

Principles

  • Every root module has exactly one owning team.
  • Prod roots need a second reviewer from the owning team.
  • Policy files are owned by security, not by the teams they constrain.
  • The apply identity (gha-tf-apply) is used only by the pipeline, gated by the prod environment (ch 11).
  • Humans get read-only console roles in prod; changes go through PRs.
Governance

Tags, cost allocation and cost checks

If it is not tagged, nobody owns it and nobody pays for it. Enforce tags in code, in policy and in the account.

In code

  • Provider default_tags: Project, Env, ManagedBy, Owner applied to every taggable resource.
  • Module inputs for extra tags, merged with merge(var.tags, {…}).
  • tags_all in state shows the effective tags.

In policy and accounts

  • Checkov/conftest: deny resources without Owner (ch 9).
  • AWS Organizations tag policies standardize keys and values.
  • Activate the tags as cost allocation tags in Billing so Cost Explorer can group by them.
  • SCPs can deny creation of untagged resources for selected services.

Cost before apply

  • Infracost turns a plan into a monthly cost diff and comments it on the PR.
  • Flag expensive resource types in policy (NAT gateways, large instances) — tf-lab forbids them entirely.
  • Budgets per account with alerts (Lab 0).
Governance

Stopping ClickOps without stopping incidents

Console changes create drift and invisible risk, but you still need a way to act fast when production is down.

Prevent

  • Read-only console roles by default in prod.
  • SCPs that deny write actions to anyone except the pipeline roles and break-glass role.
  • Nightly drift job (ch 11) opens an issue for every diff.
  • Tag ManagedBy=terraform: anyone touching such a resource in the console knows it will be reverted.

Break-glass

  • A separate role with MFA, short sessions, alerting on every use (CloudTrail → alarm).
  • Written procedure: who may assume it, what to record.
  • After the incident: reconcile within a day — code the change (or revert it) and confirm plan shows "No changes".

Reconcile drift

  • Want the change? Update code, plan shows no diff, merge.
  • Do not want it? apply reverts it (review the plan first).
  • Want Terraform to stop managing an attribute? ignore_changes — with a comment saying why (ch 3).
  • Want to accept state as-is? apply -refresh-only (ch 5).
Governance

Documentation that stays true

Generate what can be generated; write down only decisions and procedures.

brew install terraform-docs

# README.md in each module gets an injected section
# between the markers:
#   <!-- BEGIN_TF_DOCS -->
#   <!-- END_TF_DOCS -->
terraform-docs markdown table \
  --output-file README.md --output-mode inject \
  modules/network

# CI: fail if docs are stale
terraform-docs markdown table \
  --output-file README.md --output-mode inject \
  modules/network
git diff --exit-code modules/network/README.md

What to write by hand

  • ADRs for decisions: layout, state split, why not workspaces, Terraform vs OpenTofu.
  • Runbooks: state recovery from S3 versions, force-unlock, break-glass, provider upgrade.
  • Module READMEs: purpose, example call, upgrade notes per major.
  • **examples/** folders that CI actually plans, so examples cannot rot.
Chapter 13 · Part 13.4

Terraform vs OpenTofu and the road ahead

Two compatible-but-diverging tools. Know the differences before a migration decides for you.

Terraform vs OpenTofu

Same roots, different paths since 2023

OpenTofu forked from Terraform 1.5 (MPL 2.0) when Terraform moved to BUSL 1.1. Configurations and state are largely compatible; new features diverge.

TopicTerraform (HashiCorp, IBM)OpenTofu (Linux Foundation)
LicenseBUSL 1.1 (1.6+)MPL 2.0
Latest (Nov 2026)1.17 (Oct 2026)1.12 line (1.12.0 May 2026)
Registryregistry.terraform.ioregistry.opentofu.org (mirrors most providers and modules)
Only in TerraformHCP Terraform integration, Stacks, Terraform Policy (GA 1.17), action blocks, list blocks + terraform query (1.14)—
Only in OpenTofu—State and plan encryption (1.7), early evaluation of variables in backends and module sources (1.8), provider for_each and -exclude (1.9), enabled meta-argument (1.11), dynamic prevent_destroy (1.12)
In both (different versions)Ephemeral values (1.10), write-only arguments (1.11)Ephemeral values and write-only attributes (1.11)

Feature parity is a moving target: check both changelogs before you rely on a feature list, including this one.

Terraform vs OpenTofu

Migration walkthrough: try it without risk

The OpenTofu docs publish a migration guide per Terraform version. The safe pattern: back up, try on a copy, compare plans, then switch.

# 0) prerequisites: a clean plan, a state backup
terraform plan                       # No changes.
terraform state pull > backup.tfstate

# 1) install OpenTofu
brew install opentofu
tofu version

# 2) init with OpenTofu (same backend block works)
tofu init
# lock file gains registry.opentofu.org entries:
#   provider "registry.opentofu.org/hashicorp/aws"

# 3) the real test
tofu plan                            # expect: No changes.

# 4) small, harmless change through tofu apply
#    (e.g. a tag in dev), then decide

Watch out for

  • Version gap: if a newer Terraform wrote the state or the config uses Terraform-only features (action, .tfquery.hcl, policies, Stacks), OpenTofu may refuse it. Follow the guide for your exact Terraform version.
  • Lock file: both tools edit .terraform.lock.hcl with different registry hostnames — do not let both write it in the same branch.
  • Shared state: once tofu apply writes a state, go back with care (the guide has a rollback section).
  • CI: swap setup-terraform for opentofu/setup-opentofu, pinned by SHA (CI/CD deck).
  • Commercial runners: check which tool and version your runner supports.

Lab 13 runs step 2–3 against a local copy of the state with a separate TF_DATA_DIR, so your real backend and .terraform/ are never touched.

Terraform vs OpenTofu

Choosing, with criteria instead of opinions

Both are production-grade. The decision is about license, platform, features you need, and the cost of switching later.

Stay on Terraform when

  • You use or plan to use HCP Terraform / Terraform Enterprise, Stacks or Terraform Policy.
  • You want terraform query + bulk import, or action blocks, today.
  • Your organization is fine with BUSL for internal use (it restricts offering competing products).

Choose OpenTofu when

  • An open-source license (MPL 2.0) is a requirement.
  • You want built-in state encryption or the enabled / dynamic prevent_destroy features.
  • Your platform vendor (Spacelift, env0, Scalr, Harness, …) supports it well.

Either way

  • Keep configs in the common subset where cheap: it keeps the door open.
  • Pin the tool version in CI and locally (tenv manages both).
  • Write the decision as an ADR with a review date.
  • Never mix tools on the same state without a deliberate migration.
Road ahead

What changed recently — and how to keep up

Terraform 1.14–1.17 added several new top-level concepts. Learn what problem each solves before adopting it.

FeatureVersionProblem it solves
List blocks (.tfquery.hcl) + terraform query1.14Find existing resources and generate import + resource blocks in bulk (ch 5)
action blocks + terraform apply -invoke1.14Provider operations outside CRUD (invoke a Lambda, invalidate a CDN cache)
Action triggers with on_failure (halt, taint, continue)1.16Control what happens when a triggered action fails
Variables and locals in module source / version1.15One place to set module versions per environment
Terraform Policy (-policies), GA1.17HCL-based policy checks in the CLI and HCP Terraform (ch 9)
-minimal-refresh1.17Faster plans on large states (refreshes only changed resources)
Stacks (HCP Terraform)GA Sep 2025Deploy the same components across many environments/accounts

Keeping up: read the release notes of every minor (github.com/hashicorp/terraform/releases), the AWS provider CHANGELOG, and the OpenTofu blog. Try new features in a scratch root, never first in prod.

Common mistakes

Operating mistakes that hurt later

None of these fail on day one. All of them fail on the worst possible day.

Upgrades

  • Running init -upgrade "to fix an error" and committing whatever lock file it produced.
  • A provider major and a module major in one PR — an unreadable plan.
  • Lock file committed with only darwin_arm64 h1: hashes and no zh: lines: Linux CI fails.
  • Ignoring deprecation warnings until the next major removes the attribute.
  • Shared modules pinning exact provider versions.

Scale

  • Routine -target because full plans are slow.
  • -refresh=false or -minimal-refresh in the drift job — drift never shows.
  • Raising -parallelism until AWS throttles every call.
  • One giant state shared by many teams and locks.

Governance and tooling

  • Console fixes during incidents never reconciled → the next apply reverts them.
  • Untagged resources: no owner, no cost allocation.
  • Switching to OpenTofu by running tofu apply on prod state "to see if it works".
  • Both Terraform and OpenTofu writing the same lock file on one branch.
Self-check

Operating IaC: can you answer these?

Answer with the exact command, flag or file.

Upgrades

  • Which file records the exact provider version, and which command moves it forward?
  • What does ~> 6.20 allow? And ~> 6.20.0?
  • What should a no-op provider upgrade plan show, and what do you do if it shows -/+?
  • How does a module author make a v2 refactor safe for callers?
  • How do you make the lock file work on Linux CI and macOS laptops?

Scale

  • Name four causes of slow plans and how to measure each.
  • What does -minimal-refresh skip, and when must you not use it?
  • How do you move resources from one state to another with no destroy?
  • What do retry_mode = "adaptive" and TF_PLUGIN_CACHE_DIR change?

Governance and OpenTofu

  • How do CODEOWNERS, rulesets and environments combine for prod roots?
  • What is a break-glass role, and what must happen after it is used?
  • Name three features only in Terraform and three only in OpenTofu.
  • How can you test OpenTofu on a root without touching its real backend?

Practice: Lab 13 in Hands-on labs — a provider upgrade with a deprecation fix, refresh-mode timings and drift, OpenTofu on a state copy, terraform-docs, and the full teardown.

Capstone

tf-lab, end to end

One repository, two environments, a pipeline that plans every PR and applies only what was reviewed, with policies and tests as gates.

GitHub: <you>/tf-labmodules/network · app-bucket · ci-rolelive/dev|prod × network|appPR checksfmt · validate · tflint · test · checkov · trivy · conftestplan per rootcomment on PRmerge → applysaved plan, prod gatenightly: plan -detailed-exitcode → issue on driftIdentity (OIDC)gha-tf-planread-only + state readgha-tf-applyenvironments dev, prodAWS ap-northeast-1 (sandbox)S3 tflab-state-<account_id>versioned · encrypted · use_lockfiledevVPC 10.10.0.0/16 + app bucketprodVPC 10.20.0.0/16 + app bucketSSM parametersnetwork outputs → app roots

Build it after Lab 12, from a fresh clone: if the README and the pipeline are enough to recreate everything, the capstone is done. Lab 13 tears it down.

Capstone

Build order and acceptance checklist

Done means every box is ticked and proven by a command or a pipeline run, not by a screenshot.

Build order

  • 1 · bootstrap/ state bucket: versioned, encrypted, prevent_destroy (Lab 0).
  • 2 · network and app-bucket modules with typed inputs and tests (Labs 2, 6, 7).
  • 3 · live/dev and live/prod roots, one state key each, SSM outputs (Labs 4, 10).
  • 4 · One import and one moved refactor, both no-destroy plans (Lab 5).
  • 5 · Plan summary, Checkov, Trivy and conftest gates (Labs 8, 9).
  • 6 · OIDC pipeline: PR plan comment, gated apply, nightly drift job (Lab 11).
  • 7 · No secret in state where write-only or ephemeral options exist (Lab 12).

Acceptance

  • terraform plan is "No changes" in every root on main.
  • A PR deleting an S3 bucket is blocked by policy.
  • Prod apply needs approval and applies the reviewed plan.
  • No AWS keys in GitHub; CloudTrail shows gha-tf-* sessions.
  • A console tag added at night appears as an issue the next morning.

Stretch goals

  • Add a third environment with only a new live/ folder and tfvars.
  • Express one guardrail in Terraform Policy (1.17) next to its conftest version.
  • Run the same roots with OpenTofu in a branch and compare plans.
  • Adopt hand-made resources with terraform query.
Cheat sheet

Terraform CLI: the everyday commands

The plan file is the contract: plan with -out, review, apply exactly that file.

# set up and check
terraform init -backend-config="bucket=$TF_STATE_BUCKET"
terraform init -upgrade          # newer providers within constraints
terraform init -reconfigure      # backend changed, do not migrate
terraform init -migrate-state    # backend changed, copy state
terraform fmt -recursive -check  # CI; drop -check to rewrite
terraform validate

# plan and apply
terraform plan -out=tfplan
terraform plan -detailed-exitcode  # 0 none, 1 error, 2 changes
terraform plan -refresh-only       # drift only
terraform plan -replace=aws_s3_bucket.logs
terraform apply tfplan             # exactly the reviewed plan
terraform destroy

# inspect
terraform show tfplan
terraform show -json tfplan > plan.json
terraform output -json
terraform console
# state (read first, change last)
terraform state list
terraform state show 'module.network.aws_vpc.this'
terraform state pull > backup.tfstate
terraform state mv  SRC DST        # prefer moved blocks
terraform state rm  ADDR           # prefer removed blocks
terraform force-unlock LOCK_ID     # only if the holder is dead

# providers and graph
terraform providers
terraform providers lock \
  -platform=linux_amd64 -platform=darwin_arm64
terraform graph | dot -Tsvg > graph.svg

# tests and generation
terraform test
terraform test -filter=tests/defaults.tftest.hcl
terraform plan -generate-config-out=generated.tf
terraform query                    # 1.14+, .tfquery.hcl

# useful environment variables
TF_LOG=DEBUG TF_LOG_PATH=tf.log    # debug logging
TF_VAR_env=dev                     # sets var.env
TF_PLUGIN_CACHE_DIR=~/.terraform.d/plugin-cache
TF_IN_AUTOMATION=1                 # quieter CI output
Cheat sheet

HCL: expressions and functions

Try any of these in terraform console before using them in code.

# types and inputs
variable "subnets" {
  type = map(object({
    cidr   = string
    public = optional(bool, false)   # default
  }))
}
variable "env" {
  type = string
  validation {
    condition     = contains(["dev", "prod"], var.env)
    error_message = "env must be dev or prod."
  }
}

# for expressions and splat
[for k, s in var.subnets : s.cidr if s.public]
{for k, s in var.subnets : k => s.cidr}
aws_subnet.public[*].id          # count resources
values(aws_subnet.private)[*].id # for_each resources

# conditional and string template
local.is_prod ? 3 : 1
"tflab-${var.env}-%{if local.is_prod}ha%{else}single%{endif}"

# for_each from a list of names
for_each = toset(["a", "b"])

Functions you will use weekly

CallResult
cidrsubnet("10.10.0.0/16", 8, 1)"10.10.1.0/24"
merge({a=1}, {b=2}){a=1, b=2} (later keys win)
lookup({a="1"}, "b", "none")"none"
try(var.cfg.timeout, 30)30 if the attribute is missing
can(regex("^tflab-", var.name))true / false
coalesce("", "b")"b" (first non-null, non-empty)
one(["x"])"x"; null for []; error if > 1
zipmap(["a","b"], [1,2]){a=1, b=2}
flatten([[1],[2,3]])[1, 2, 3]
format("%s-%s", "tflab", "dev")"tflab-dev"
jsonencode(...) / templatefile(path, vars)Policies and user data from data, not string concatenation
Cheat sheet

Refactor and adoption blocks

Blocks are reviewed in a PR and shown in the plan; state commands are not. Prefer blocks.

# rename / count → for_each / into a module
moved {
  from = aws_s3_bucket.logs
  to   = module.logs.aws_s3_bucket.this
}

# adopt an existing resource
import {
  to = aws_s3_bucket.legacy
  id = "tflab-legacy-assets"
}
# many at once
import {
  for_each = var.legacy_buckets       # map: key => name
  to       = aws_s3_bucket.legacy[each.key]
  id       = each.value
}
# write the resource blocks for you, then review them
#   terraform plan -generate-config-out=generated.tf

# stop managing, keep the real resource
removed {
  from = aws_s3_bucket.old
  lifecycle {
    destroy = false
  }
}
# find.tfquery.hcl — search existing resources (1.14+)
# (only types with list support; config is
#  provider-specific: see the provider docs)
list "aws_instance" "dev" {
  provider = aws
  limit    = 50          # default 100
  config {
    filter {
      name   = "tag:Env"
      values = ["dev"]
    }
  }
}

terraform query
terraform query -generate-config-out=generated.tf
# → import + resource blocks to review and commit

# plan annotations to look for
#  # aws_s3_bucket.a has moved to aws_s3_bucket.b
#  # aws_s3_bucket.legacy will be imported
#  # aws_s3_bucket.old will no longer be managed
#      by Terraform
# Plan: 1 to import, 0 to add, 0 to change,
#       0 to destroy.
Cheat sheet

Plan JSON with jq

Machine-read the plan for reviews and gates: actions tells you what happens to each resource.

terraform plan -out=tfplan
terraform show -json tfplan > plan.json

# every resource that will be deleted (incl. replacements)
jq -r '.resource_changes[]
  | select(.change.actions | index("delete"))
  | .address' plan.json

# count changes by action
jq -r '[.resource_changes[].change.actions | join("/")]
  | group_by(.) | map({key: .[0], value: length})
  | from_entries' plan.json
# { "create": 2, "no-op": 14, "update": 1,
#   "delete/create": 1 }

# why a replacement happens
jq -r '.resource_changes[]
  | select(.action_reason != null)
  | "\(.address): \(.action_reason)"' plan.json
# aws_s3_bucket.app: replace_because_cannot_update

change.actions values

ActionsMeaning
["no-op"]Nothing to do
["create"]New resource
["read"]Data source read during apply
["update"]Update in place
["delete", "create"]Replace, destroy first (-/+)
["create", "delete"]Replace, create first (+/-, create_before_destroy)
["delete"]Destroy
["forget"]Drop from state, keep the real resource (removed with destroy = false)

Plan files and plan JSON contain sensitive values in clear text. Treat them like the state: short-lived CI artifacts, never committed.

Cheat sheet

Checkov, Trivy and conftest

Static scans on HCL catch most mistakes fast; scans on plan JSON see resolved values and modules.

# Checkov — HCL and plan
checkov -d . --framework terraform
checkov -f plan.json --framework terraform_plan
checkov -d . --check CKV_AWS_18,CKV_AWS_21
checkov -d . --skip-check CKV_AWS_144
checkov -d . --external-checks-dir policies/checkov
checkov -d . -o sarif --output-file-path reports/
checkov -d . --create-baseline   # accept today's state
checkov -d . --baseline .checkov.baseline
# inline, with a reason:
#   #checkov:skip=CKV_AWS_144:single-region lab data

# Trivy — IaC misconfigurations
trivy config .
trivy config --severity HIGH,CRITICAL --exit-code 1 .
trivy config --tf-vars live/dev/app/dev.tfvars .
trivy config --ignorefile .trivyignore .
# inline:  #trivy:ignore:AVD-AWS-0089
# pin the binary and action by version/SHA:
# v0.69.4–0.69.6 were malicious (Mar 2026)
# conftest — Rego over plan JSON
conftest test plan.json -p policies/rego

# policies/rego/s3.rego
package main

import rego.v1

deny contains msg if {
  rc := input.resource_changes[_]
  rc.type == "aws_s3_bucket"
  "delete" in rc.change.actions
  msg := sprintf("%s would be deleted", [rc.address])
}

warn contains msg if {
  rc := input.resource_changes[_]
  "create" in rc.change.actions
  not rc.change.after.tags.Owner
  msg := sprintf("%s has no Owner tag", [rc.address])
}

# exit code 1 on any deny → CI fails

Inline tags may be empty when the tag comes from provider default_tags; check tags_all in the plan when your policy is about effective tags.

Cheat sheet

S3 backend and a CI plan job

One bucket, one key per root, S3-native locking. CI authenticates with OIDC and never stores keys.

# backend.tf (bucket passed at init: partial config)
terraform {
  backend "s3" {
    key          = "dev/network/terraform.tfstate"
    region       = "ap-northeast-1"
    encrypt      = true
    use_lockfile = true   # lock object next to the state
    # dynamodb_table = "…"  deprecated
  }
}

terraform init -input=false \
  -backend-config="bucket=$TF_STATE_BUCKET"

# share outputs without remote state (ch 4, ch 10)
resource "aws_ssm_parameter" "vpc_id" {
  name  = "/tflab/dev/network/vpc_id"
  type  = "String"
  value = aws_vpc.this.id
}
# .github/workflows/plan.yml (excerpt)
permissions: {contents: read, id-token: write}
jobs:
  plan:
    runs-on: ubuntu-latest
    defaults: {run: {working-directory: live/dev/network}}
    steps:
      - uses: actions/checkout@<sha>
      - uses: hashicorp/setup-terraform@<sha>
        with: {terraform_version: 1.17.0,
               terraform_wrapper: false}
      - uses: aws-actions/configure-aws-credentials@<sha>
        with:
          role-to-assume: arn:aws:iam::<acct>:role/gha-tf-plan
          aws-region: ap-northeast-1
      - run: terraform init -input=false
               -backend-config="bucket=tflab-state-<acct>"
      - run: terraform plan -input=false -lock-timeout=60s
               -detailed-exitcode -out=tfplan
        # exit 2 = changes: handle it, do not fail the job

<sha> = the full commit SHA of the action's release (CI/CD deck). The full pipeline with PR comments, gated apply and drift detection is in ch 11 and Lab 11.

Lab setup

Before you start

All labs run from your laptop against an AWS sandbox account in ap-northeast-1; Lab 11 adds a public GitHub repo. Resources are free or near-free.

Accounts and conventions

  • AWS sandbox account; profile tf-lab; export AWS_PROFILE=tf-lab AWS_REGION=ap-northeast-1.
  • Budget alarm before the first apply (Lab 0).
  • Prefix tflab; tags Project=tf-lab, Env, ManagedBy=terraform through provider default_tags.
  • State bucket tflab-state-<account_id>; one key per component: <env>/<component>/terraform.tfstate; use_lockfile = true.
  • Free resources only: VPC, subnets, IGW, security groups, S3, IAM, SSM Standard parameters. Never NAT gateways, EC2, RDS, EKS.
  • Each lab = Build, Verify, Break it, Clean up. Later labs extend earlier ones; commit after each lab.
brew tap hashicorp/tap
brew install hashicorp/tap/terraform \
  tflint checkov trivy conftest jq gh awscli
brew install opentofu terraform-docs infracost  # optional

terraform version   # Terraform v1.14+ (examples: 1.17)
tflint --version
checkov --version
trivy --version
conftest --version
aws --version       # aws-cli/2.x

# a profile for the sandbox account (SSO or keys)
aws configure sso --profile tf-lab   # or: aws configure
export AWS_PROFILE=tf-lab AWS_REGION=ap-northeast-1
aws sts get-caller-identity
# { "Account": "123456789012", "Arn": "arn:aws:...", ... }

export ACCT=$(aws sts get-caller-identity \
  --query Account --output text)
export TF_STATE_BUCKET=tflab-state-$ACCT
Lab 0 · Build (1/3)

Setup: repository, budget alarm and identity check

Goal: a Git repo with a safe .gitignore, a budget alarm before anything bills, and proof that the CLI talks to the sandbox account.

mkdir tf-lab && cd tf-lab && git init
cat > .gitignore <<'EOF'
# provider binaries and module cache
.terraform/
# state and its backups: never in Git
*.tfstate
*.tfstate.*
# saved plans can contain secrets
*.tfplan
tfplan
tfplan.json
crash.log
crash.*.log
# local overrides
override.tf
override.tf.json
*_override.tf
*_override.tf.json
.terraformrc
terraform.rc
EOF
# commit the lock file, NOT .terraform/
git add .gitignore && git commit -m "gitignore"

aws sts get-caller-identity --query Account --output text
# 123456789012  ← must be the SANDBOX account
# budget: e-mail when actual spend passes 80% of $10
aws budgets create-budget --account-id $ACCT --budget \
 '{"BudgetName":"tf-lab","TimeUnit":"MONTHLY",
   "BudgetType":"COST",
   "BudgetLimit":{"Amount":"10","Unit":"USD"}}' \
 --notifications-with-subscribers '[{"Notification":{
   "NotificationType":"ACTUAL",
   "ComparisonOperator":"GREATER_THAN",
   "Threshold":80},
   "Subscribers":[{"SubscriptionType":"EMAIL",
   "Address":"you@example.com"}]}]'

aws budgets describe-budgets --account-id $ACCT \
  --query 'Budgets[].BudgetName'
# [ "tf-lab" ]

# shared provider plugin cache: faster init everywhere
mkdir -p ~/.terraform.d/plugin-cache
cat >> ~/.terraformrc <<'EOF'
plugin_cache_dir = "$HOME/.terraform.d/plugin-cache"
EOF

ACCT and TF_STATE_BUCKET come from the Lab setup slide. Put the export lines in your shell profile so every lab terminal has them.

Lab 0 · Build (2/3)

Setup: the state bucket (bootstrap/)

Goal: one versioned, encrypted, private bucket for every state file. It starts with local state: the bucket does not exist yet.

mkdir bootstrap && cd bootstrap && cat > main.tf <<'EOF'
terraform {
  required_version = ">= 1.14"
  required_providers {
    aws = { source = "hashicorp/aws", version = "~> 6.0" }
  }
}
provider "aws" {
  region = "ap-northeast-1"
  default_tags {
    tags = { Project = "tf-lab", Env = "shared", ManagedBy = "terraform" }
  }
}

data "aws_caller_identity" "current" {}
resource "aws_s3_bucket" "state" {
  bucket = "tflab-state-${data.aws_caller_identity.current.account_id}"
  lifecycle {
    prevent_destroy = true # the one bucket you never lose
  }
}
cat >> main.tf <<'EOF'

resource "aws_s3_bucket_versioning" "state" {
  bucket = aws_s3_bucket.state.id
  versioning_configuration {
    status = "Enabled" # every state write is a version
  }
}

resource "aws_s3_bucket_server_side_encryption_configuration" "state" {
  bucket = aws_s3_bucket.state.id
  rule {
    apply_server_side_encryption_by_default {
      sse_algorithm = "AES256"
    }
  }
}

resource "aws_s3_bucket_public_access_block" "state" {
  bucket                  = aws_s3_bucket.state.id
  block_public_acls       = true
  block_public_policy     = true
  ignore_public_acls      = true
  restrict_public_buckets = true
}

output "state_bucket" {
  value = aws_s3_bucket.state.bucket
}
EOF
Lab 0 · Build (3/3)

Setup: apply, then move bootstrap's own state into the bucket

Goal: create the bucket with local state, then migrate that state into the bucket so no state file lives only on your laptop.

terraform init
# Initializing provider plugins...
# - Installing hashicorp/aws v6.x.y...
# Terraform has been successfully initialized!

terraform fmt -check && terraform validate
terraform plan -out=tfplan
# Plan: 4 to add, 0 to change, 0 to destroy.
terraform apply tfplan
# Outputs:
# state_bucket = "tflab-state-123456789012"

aws s3api get-bucket-versioning --bucket $TF_STATE_BUCKET
# { "Status": "Enabled" }
aws s3api get-public-access-block \
  --bucket $TF_STATE_BUCKET \
  --query 'PublicAccessBlockConfiguration'
# all four flags: true
# partial backend config: the bucket name is passed at
# init time (backend blocks cannot use variables)
cat > backend.tf <<'EOF'
terraform {
  backend "s3" {
    key          = "bootstrap/terraform.tfstate"
    region       = "ap-northeast-1"
    encrypt      = true
    use_lockfile = true # S3-native lock, no DynamoDB
  }
}
EOF
terraform init -migrate-state \
  -backend-config="bucket=$TF_STATE_BUCKET"
# Do you want to copy existing state to the new backend?
#   Enter a value: yes

aws s3 ls s3://$TF_STATE_BUCKET/bootstrap/
# ... terraform.tfstate
rm -f terraform.tfstate terraform.tfstate.backup tfplan
terraform plan     # No changes. (now read from S3)

cd .. && git add bootstrap && git commit -m "bootstrap"
# includes .terraform.lock.hcl

Every later root module uses the same pattern: a backend "s3" block with its own key and use_lockfile = true, initialized with -backend-config="bucket=$TF_STATE_BUCKET".

Lab 0 · Verify, break it, clean up

A protected home for every state file

Everything in later labs stores its state here. Make sure it is versioned, private and hard to delete.

Verify

  • terraform plan in bootstrap/ says No changes and reads state from S3.
  • aws s3api list-object-versions --bucket $TF_STATE_BUCKET --prefix bootstrap/ shows one version.
  • Console → S3 → bucket → Properties: versioning on, default encryption SSE-S3; Permissions: all public access blocked.
  • git status is clean; git ls-files lists bootstrap/.terraform.lock.hcl but no .tfstate.
  • The budget appears in Billing → Budgets.

Break it

  • Protected: terraform destroy in bootstrap/ → "Instance cannot be destroyed … lifecycle.prevent_destroy". Good.
  • Lock: run terraform plan in two terminals at once → the second fails at once with "Error acquiring the state lock" (the default -lock-timeout is 0s; -lock-timeout=60s makes it wait).
  • No partial config: terraform init -reconfigure without -backend-config → Terraform asks for bucket interactively. Why does that make CI hang without -input=false?
# keep the bucket until the very
# end of the deck: every lab
# stores its state here.

# final teardown (Lab 13):
#  1. destroy every live/* root
#  2. bootstrap: migrate its state
#     back to local, remove
#     prevent_destroy, apply
#  3. empty the bucket, including
#     all object versions
#  4. destroy bootstrap, then
#     delete the budget
Lab 1 · Build (1/3)

A first root module: random, local and one SSM parameter

Goal: the full workflow on a tiny configuration with local state. One free AWS resource (a Standard-tier SSM parameter) makes the plan real.

cd tf-lab && mkdir -p labs/01-hello && cd labs/01-hello
aws sts get-caller-identity --query Account --output text
# must print your sandbox account ($ACCT)

cat > versions.tf <<'EOF'
terraform {
  required_version = ">= 1.14"
  required_providers {
    aws    = { source = "hashicorp/aws", version = "~> 6.0" }
    random = { source = "hashicorp/random", version = "~> 3.6" }
    local  = { source = "hashicorp/local", version = "~> 2.5" }
  }
}

provider "aws" {
  region = "ap-northeast-1"
  default_tags {
    tags = { Project = "tf-lab", Env = "dev", ManagedBy = "terraform" }
  }
}
EOF
cat > main.tf <<'EOF'
resource "random_pet" "name" {
  length = 2
}

resource "local_file" "hello" {
  filename = "${path.module}/out/hello.txt"
  content  = "hello, ${random_pet.name.id}\n"
}

resource "aws_ssm_parameter" "hello" {
  name  = "/tflab/dev/hello"
  type  = "String"
  value = random_pet.name.id
}

output "pet" {
  value = random_pet.name.id
}

output "param_name" {
  value = aws_ssm_parameter.hello.name
}
EOF

terraform init
terraform fmt -check && terraform validate
# Success! The configuration is valid.
Lab 1 · Build (2/3)

Saved plans, apply and exit codes

Goal: plan to a file, review it, apply exactly that plan, then prove the second plan is empty with -detailed-exitcode.

terraform plan -out=tfplan
# Plan: 3 to add, 0 to change, 0 to destroy.
terraform show tfplan | less          # review
terraform apply tfplan
# Apply complete! Resources: 3 added, 0 changed, 0 destroyed.
# Outputs:
# param_name = "/tflab/dev/hello"
# pet = "select-mollusk"              (yours differs)

cat out/hello.txt
aws ssm get-parameter --name /tflab/dev/hello \
  --query Parameter.Value --output text

# idempotence: a second plan finds nothing
terraform plan -detailed-exitcode -input=false
echo "exit=$?"
# No changes. Your infrastructure matches the configuration.
# exit=0
# what is in state?
terraform state list
# aws_ssm_parameter.hello
# local_file.hello
# random_pet.name
terraform state show aws_ssm_parameter.hello | head

# the dependency graph (needs Graphviz: brew install graphviz)
terraform graph | dot -Tsvg > graph.svg && open graph.svg
# random_pet.name ← local_file.hello
# random_pet.name ← aws_ssm_parameter.hello

# outputs for scripts
terraform output -raw param_name
terraform output -json | jq -r .pet.value

# .gitignore for every Terraform directory in tf-lab
cat >> ../../.gitignore <<'EOF'
.terraform/
*.tfstate
*.tfstate.*
*.tfplan
tfplan
crash.log
labs/*/out/
EOF
Lab 1 · Build (3/3)

Drift and replacement

Goal: change reality behind Terraform's back, watch the plan detect it, then trigger a replacement and read the -/+ symbols.

# 1) drift in AWS: edit the parameter outside Terraform
aws ssm put-parameter --name /tflab/dev/hello \
  --value "changed-by-hand" --overwrite

terraform plan -refresh-only
# ~ value = (sensitive value)   ... "changed outside of Terraform"
terraform plan -detailed-exitcode; echo "exit=$?"
# ~ update in-place  → exit=2
terraform apply          # type yes: value restored

# 2) drift locally: delete the generated file
rm out/hello.txt
terraform plan
# local_file.hello will be created
# 3) replacement: a new pet name forces new dependants
terraform plan -replace=random_pet.name
# random_pet.name must be replaced  (-/+)
# aws_ssm_parameter.hello will be updated in-place (~)
# local_file.hello must be replaced (content forces it)
terraform apply -replace=random_pet.name

# 4) destroy (preview first)
terraform plan -destroy
# Plan: 0 to add, 0 to change, 3 to destroy.
terraform destroy
aws ssm get-parameter --name /tflab/dev/hello
# ParameterNotFound

The SSM value is shown as (sensitive value) because the AWS provider marks it sensitive in the schema: hidden in output, still stored in terraform.tfstate (ch 2, section 08).

Lab 1 · Verify, break it, clean up

One directory, three providers, a reviewed plan

You have now used every command of the core workflow on real infrastructure.

Verify

  • terraform plan -detailed-exitcode returns 0 right after apply and 2 after the manual put-parameter.
  • .terraform.lock.hcl lists hashicorp/aws, random and local with h1: and zh: hashes; commit it.
  • graph.svg shows both resources depending on random_pet.name.
  • terraform.tfstate exists locally and is ignored by Git (git status does not list it).

Break it

  • Stale plan: plan -out=tfplan, then apply once without the file, then apply tfplan → "Saved plan is stale".
  • Wrong profile: AWS_PROFILE=default terraform plan in another account → Terraform wants to create everything again (the state does not match that account). Always check sts get-caller-identity.
  • Typo: rename random_pet.name to random_pet.nmae in the reference only → validate fails with "Reference to undeclared resource".
  • No init: move .terraform/ away and run plan → "Required plugins are not installed"; terraform init restores the exact versions from the lock file.
# after destroy nothing in AWS
# remains; remove local files
cd tf-lab/labs/01-hello
rm -rf .terraform out tfplan \
  terraform.tfstate*
# keep versions.tf, main.tf and
# .terraform.lock.hcl in Git
git add . ../../.gitignore
git commit -m "lab 1: hello"
Lab 2 · Build (1/4)

HCL drills in terraform console

Goal: build intuition for types and expressions before writing resources. Predict each result, then press Enter.

mkdir -p tf-lab/labs/02-hcl && cd tf-lab/labs/02-hcl
terraform console          # works in an empty directory

> cidrsubnet("10.10.0.0/16", 8, 1)
"10.10.1.0/24"
> [for i, az in ["a", "c", "d"] :
    cidrsubnet("10.10.0.0/16", 8, i)]
["10.10.0.0/24", "10.10.1.0/24", "10.10.2.0/24"]
> { for k, v in { a = 1, b = 2 } : k => v * 10 if v > 1 }
{ "b" = 20 }
> merge({ a = 1 }, { a = 2, b = 3 })
{ "a" = 2, "b" = 3 }
> try(tonumber("x"), 0)
0
> can(regex("^[a-z0-9-]+$", "Bad_Key"))
false
> toset(["b", "a", "b"])
toset(["a", "b"])
> type(tomap({ a = 1 }))
map(number)
> format("%s-%03d", "tflab", 7)
"tflab-007"
> exit
# grouping with ... (keys repeat)
> { for u in [
    { team = "core", name = "ann" },
    { team = "core", name = "bo" },
    { team = "web",  name = "cy" } ] : u.team => u.name... }
{ "core" = ["ann", "bo"], "web" = ["cy"] }

# splat
> [{ n = "a" }, { n = "b" }][*].n
["a", "b"]

# string directive
> "%{if 1 > 0}yes%{else}no%{endif}"
"yes"

# your turn (answers in ch 2):
#  1. all pairs of ["dev","prod"] x ["a","b"]
#  2. a map az => cidr for ["a","c","d"]
#  3. first non-empty of "", null, "x"
#  4. why does this fail?
#     element(["a","b"], 3)  … it does not! why is
#     that dangerous?

Console output is shown condensed here; real output prints one element per line.

Lab 2 · Build (2/4)

A configuration that uses the whole language

Goal: typed, validated inputs and optional() objects. Copy versions.tf from Lab 1 and set default_tags { tags = local.tags }.

cat > variables.tf <<'EOF'
variable "env" {
  type = string
  validation {
    condition     = contains(["dev", "prod"], var.env)
    error_message = "env must be dev or prod."
  }
}
variable "params" {
  type    = map(string)
  default = {}
  validation {
    condition = alltrue([for k in keys(var.params) :
    can(regex("^[a-z0-9-]+$", k))])
    error_message = "Parameter keys may contain only a-z, 0-9 and '-'."
  }
}
variable "bucket" {
  type = object({
    name_suffix = string
    versioning  = optional(bool, true)
    lifecycle_rules = optional(list(object({
      id     = string
      prefix = string
      days   = number
    })), [])
  })
}
EOF
cat > dev.tfvars <<'EOF'
env = "dev"
params = {
  greeting = "hello"
  owner    = "platform"
}
bucket = {
  name_suffix = "lab2"
  lifecycle_rules = [
    { id = "tmp", prefix = "tmp/", days = 7 },
    { id = "logs", prefix = "logs/", days = 30 },
  ]
}
EOF

mkdir -p templates && cat > templates/summary.tftpl <<'EOF'
# tf-lab ${env}
bucket: ${bucket}
%{ for k, name in params ~}
- ${k}: ${name}
%{ endfor ~}
EOF

# check the inputs before writing resources
# (console needs init once versions.tf exists)
terraform init
echo 'var.bucket' | terraform console -var-file=dev.tfvars
# versioning = true filled in by optional(bool, true)
Lab 2 · Build (3/4)

Resources: for_each, a dynamic block and shared tags

Goal: parameters keyed by name, a bucket named from locals, and lifecycle rules generated from the optional() list.

cat > main.tf <<'EOF'
data "aws_caller_identity" "me" {}

locals {
  name = "tflab-${var.env}"
  tags = { Project = "tf-lab", Env = var.env, ManagedBy = "terraform" }
}

resource "aws_ssm_parameter" "p" {
  for_each = var.params
  name     = "/tflab/${var.env}/${each.key}"
  type     = "String"
  value    = each.value
}

resource "aws_s3_bucket" "b" {
  bucket = join("-", [local.name, var.bucket.name_suffix,
  data.aws_caller_identity.me.account_id])
  force_destroy = true # lab only
}
EOF
cat >> main.tf <<'EOF'

resource "aws_s3_bucket_versioning" "b" {
  bucket = aws_s3_bucket.b.id
  versioning_configuration {
    status = var.bucket.versioning ? "Enabled" : "Suspended"
  }
}

resource "aws_s3_bucket_lifecycle_configuration" "b" {
  count  = length(var.bucket.lifecycle_rules) > 0 ? 1 : 0
  bucket = aws_s3_bucket.b.id

  dynamic "rule" {
    for_each = var.bucket.lifecycle_rules
    content {
      id     = rule.value.id
      status = "Enabled"
      filter { prefix = rule.value.prefix }
      expiration { days = rule.value.days }
    }
  }
}
EOF

New S3 buckets already block public access and encrypt with SSE-S3 by default; the shared app-bucket module (ch 6) still sets both explicitly so the intent is reviewable and scanners pass (ch 9).

Lab 2 · Build (4/4)

templatefile, outputs, apply

Goal: render a summary from resource attributes, apply a saved plan and read the instance addresses.

cat >> main.tf <<'EOF'

resource "local_file" "summary" {
  filename = "${path.module}/out/summary.md"
  content = templatefile("${path.module}/templates/summary.tftpl", {
    env    = var.env
    bucket = aws_s3_bucket.b.bucket
    params = { for k, p in aws_ssm_parameter.p : k => p.name }
  })
}

output "param_names" {
  value = { for k, p in aws_ssm_parameter.p : k => p.name }
}
EOF

terraform init
terraform fmt -check -recursive && terraform validate
terraform plan -var-file=dev.tfvars -out=tfplan
# + aws_ssm_parameter.p["greeting"]
# + aws_ssm_parameter.p["owner"]
# + aws_s3_bucket.b
# + aws_s3_bucket_versioning.b
# + aws_s3_bucket_lifecycle_configuration.b[0]
#     (2 rule blocks: tmp, logs)
# + local_file.summary
# Plan: 6 to add, 0 to change, 0 to destroy.

terraform apply tfplan
cat out/summary.md
# # tf-lab dev
# bucket: tflab-dev-lab2-123456789012
# - greeting: /tflab/dev/greeting
# - owner: /tflab/dev/owner
Lab 2 · Verify, break it, clean up

Inputs that defend themselves, instances that stay stable

Every input mistake now fails at plan time with your own error message.

Verify

  • terraform output param_names shows greeting and owner mapped to /tflab/dev/....
  • aws s3api get-bucket-lifecycle-configuration --bucket <name> returns rules tmp and logs.
  • out/summary.md lists the bucket and both parameters.
  • The bucket has tags Project, Env, ManagedBy without a tags argument on the resource (provider default_tags).

Break it

  • Validation: -var 'env=qa' → "env must be dev or prod."; a params key Bad_Key → your regex message.
  • Stable keys: remove greeting from dev.tfvars → plan destroys only aws_ssm_parameter.p["greeting"]. Rewrite p with count over keys(var.params) and repeat: predict the replacements first.
  • Empty dynamic: set lifecycle_rules = [] → the count guard removes the lifecycle resource instead of sending an empty rule list.
  • Unknown keys: set for_each = toset([aws_s3_bucket.b.id]) on a new resource → "Invalid for_each argument".
cd tf-lab/labs/02-hcl
terraform destroy \
  -var-file=dev.tfvars
# Destroy complete!
#   Resources: 6 destroyed.
rm -rf .terraform out tfplan \
  terraform.tfstate*
git add . && git commit \
  -m "lab 2: hcl"
Lab 3 · Build (1/4)

live/dev/network: versions, provider and inputs

Goal: the dev network root module with AWS provider v6, default_tags and validated inputs. Local state for now; Lab 4 moves it to S3. Needs Lab 0.

cd tf-lab && mkdir -p live/dev/network
cd live/dev/network

cat > versions.tf <<'EOF'
terraform {
  required_version = ">= 1.14"
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 6.0"
    }
  }
}
EOF

cat > providers.tf <<'EOF'
provider "aws" {
  region = var.region
  default_tags {
    tags = {
      Project   = "tf-lab"
      Env       = var.env
      ManagedBy = "terraform"
    }
  }
}
EOF
cat > variables.tf <<'EOF'
variable "region" {
  type    = string
  default = "ap-northeast-1"
}

variable "env" {
  type    = string
  default = "dev"
  validation {
    condition     = contains(["dev", "prod"], var.env)
    error_message = "env must be dev or prod."
  }
}

variable "vpc_cidr" {
  type    = string
  default = "10.10.0.0/16"
  validation {
    condition     = can(cidrnetmask(var.vpc_cidr))
    error_message = "vpc_cidr must be a valid IPv4 CIDR."
  }
}
EOF
Lab 3 · Build (2/4)

Lock two platforms, then the VPC with guards

Goal: a lock file that works on your laptop and in Linux CI, and a VPC protected by a precondition and a postcondition.

terraform init
# - Installing hashicorp/aws v6.x.y...
# Terraform has created a lock file .terraform.lock.hcl

terraform providers lock \
  -platform=darwin_arm64 -platform=linux_amd64
grep -A3 'hashicorp/aws' .terraform.lock.hcl
#   version     = "6.x.y"
#   constraints = "~> 6.0"
#   hashes = [ "h1:…", "h1:…", "zh:…", … ]

git add versions.tf providers.tf variables.tf \
  .terraform.lock.hcl
git commit -m "network: provider and inputs"

# .gitignore (Lab 0) must exclude:
#   .terraform/  *.tfstate  *.tfstate.*  *.tfplan
cat > main.tf <<'EOF'
data "aws_availability_zones" "available" {
  state = "available"
}

locals {
  name    = "tflab-${var.env}"
  az_list = data.aws_availability_zones.available.names
  azs     = slice(local.az_list, 0, min(2, length(local.az_list)))
}

resource "aws_vpc" "this" {
  cidr_block           = var.vpc_cidr
  enable_dns_support   = true
  enable_dns_hostnames = true
  tags                 = { Name = local.name }
  lifecycle {
    precondition {
      condition     = length(local.azs) == 2
      error_message = "Need 2 available AZs in ${var.region}."
    }
    postcondition {
      condition     = self.enable_dns_hostnames
      error_message = "DNS hostnames must be enabled."
    }
  }
}
EOF
Lab 3 · Build (3/4)

Subnets, routing and a check block

Goal: 2 public and 2 private subnets (no NAT gateway), a public route table, and a check that only warns.

cat >> main.tf <<'EOF'

resource "aws_subnet" "public" {
  for_each                = { for i, az in local.azs : az => i }
  vpc_id                  = aws_vpc.this.id
  availability_zone       = each.key
  cidr_block              = cidrsubnet(var.vpc_cidr, 8, each.value)
  map_public_ip_on_launch = true
  tags                    = { Name = "${local.name}-public-${each.key}" }
}

resource "aws_subnet" "private" {
  for_each          = { for i, az in local.azs : az => i }
  vpc_id            = aws_vpc.this.id
  availability_zone = each.key
  cidr_block        = cidrsubnet(var.vpc_cidr, 8, each.value + 10)
  tags              = { Name = "${local.name}-private-${each.key}" }
}

check "two_public_subnets" {
  assert {
    condition     = length(aws_subnet.public) == 2
    error_message = "Expected 2 public subnets."
  }
}
EOF
cat >> main.tf <<'EOF'

resource "aws_internet_gateway" "this" {
  vpc_id = aws_vpc.this.id
  tags   = { Name = local.name }
}

resource "aws_route_table" "public" {
  vpc_id = aws_vpc.this.id
  route {
    cidr_block = "0.0.0.0/0"
    gateway_id = aws_internet_gateway.this.id
  }
  tags = { Name = "${local.name}-public" }
}

resource "aws_route_table_association" "public" {
  for_each       = aws_subnet.public
  subnet_id      = each.value.id
  route_table_id = aws_route_table.public.id
}
EOF
terraform fmt -check && terraform validate
terraform plan -out=tfplan
# Plan: 9 to add, 0 to change, 0 to destroy.
terraform apply tfplan

The for_each keys are the AZ names your account returns, usually ap-northeast-1a and ap-northeast-1c in Tokyo.

Lab 3 · Build (4/4)

Lifecycle experiments on a security group

Goal: watch replacement order, an ignored tag, prevent_destroy and a failing check change the plan.

cat > sg.tf <<'EOF'
resource "aws_security_group" "web" {
  name_prefix = "${local.name}-web-"
  description = "HTTPS from the internet"
  vpc_id      = aws_vpc.this.id
  lifecycle {
    create_before_destroy = true
    ignore_changes        = [tags["Owner"]]
  }
}

resource "aws_vpc_security_group_ingress_rule" "https" {
  security_group_id = aws_security_group.web.id
  cidr_ipv4         = "0.0.0.0/0"
  from_port         = 443
  to_port           = 443
  ip_protocol       = "tcp"
}
EOF
terraform apply                  # 2 to add

# A) description forces replacement
sed -i.bak 's/from the internet/(public)/' sg.tf
terraform plan
# aws_security_group.web must be replaced
# +/- resource "aws_security_group" "web"
# Plan: 2 to add, 0 to change, 2 to destroy.
terraform apply      # new SG first, old one last
# B) an ignored tag vs a managed one
SG=$(echo 'aws_security_group.web.id' \
  | terraform console | tr -d '"')
aws ec2 create-tags --resources $SG \
  --tags Key=Owner,Value=alice
terraform plan       # Owner ignored in tags; if tags_all
                     # still differs, see provider ignore_tags
aws ec2 create-tags --resources $SG \
  --tags Key=Team,Value=web
terraform plan       # ~ tags: - "Team" (reverts)
terraform apply

# C) in aws_vpc.this lifecycle add:
#      prevent_destroy = true
terraform plan -destroy
# Error: Instance cannot be destroyed

# D) make the check fail: == 2 → == 3 in main.tf
terraform plan
# Warning: Check block assertion failed
#   Expected 2 public subnets.
# (the plan still succeeds)

# revert C and D, then:
terraform plan       # No changes
Lab 3 · Verify, break it, clean up

Providers pinned, guards proven

You have watched each lifecycle rule change a plan. Keep the network: Lab 4 moves its state to S3.

Verify

  • terraform version lists hashicorp/aws v6.x; the lock file has h1: and zh: hashes.
  • aws ec2 describe-vpcs --filters Name=tag:Project,Values=tf-lab shows Env=dev, ManagedBy=terraform from default_tags.
  • terraform state list shows 11 managed resources (VPC, 4 subnets, IGW, route table, 2 associations, SG, ingress rule) plus the data source.
  • After reverting C and D: terraform plan → No changes.

Break it

  • Fixed name: replace name_prefix with name = "${local.name}-web", apply, then change the description → the replacement fails: that group name already exists in the VPC.
  • Bad input: terraform plan -var vpc_cidr=10.10.0.0/33 → validation error before any API call.
  • Region per resource (plan only): add region = "us-east-1" to aws_internet_gateway.this → plan shows -/+. Applying would delete the working gateway, then fail (no such VPC in us-east-1). Remove the line.
  • Lock drift: terraform init -upgrade → if the lock file changes, review the diff before committing.
# keep everything for Lab 4

# housekeeping
rm -f *.bak tfplan

# full teardown (only at the very
# end of the deck, after Lab 4
# moved the state to S3):
# terraform destroy
Lab 4 · Build (1/4)

Move live/dev/network to the S3 backend

Goal: the Lab 3 state moves from a local file into tflab-state-<acct> under dev/network/terraform.tfstate, with native S3 locking. Needs Labs 0 and 3.

cd tf-lab/live/dev/network
echo "$TF_STATE_BUCKET"          # tflab-state-<acct> (Lab 0)

cat > backend.tf <<'EOF'
terraform {
  backend "s3" {
    # bucket: from -backend-config (no variables here)
    key          = "dev/network/terraform.tfstate"
    region       = "ap-northeast-1"
    encrypt      = true
    use_lockfile = true
  }
}
EOF

cp terraform.tfstate /tmp/network-local-backup.tfstate
terraform init -migrate-state \
  -backend-config="bucket=$TF_STATE_BUCKET"
# ... Do you want to copy existing state to the new
#     backend?  Enter a value: yes
# Successfully configured the backend "s3"!
# state is now remote
aws s3 ls s3://$TF_STATE_BUCKET/dev/network/
#  ... terraform.tfstate

terraform plan            # No changes (same state)

# remove the stale local copy so nobody applies with it
rm -f terraform.tfstate terraform.tfstate.backup

# record the bucket for teammates and CI
echo "bucket = \"$TF_STATE_BUCKET\"" > backend.hcl
echo 'backend.hcl' >> ../../../.gitignore   # account-specific

# next time, on any machine:
#   terraform init -backend-config=backend.hcl

Without -migrate-state Terraform asks anyway; with -reconfigure it would start with an empty state in S3 and plan to create everything again.

Lab 4 · Build (2/4)

Watch the lock

Goal: see the .tflock object appear while an apply waits for approval, and a second run fail fast.

# terminal A: start an apply and leave it at the prompt
terraform apply -replace=aws_security_group.web
#   Do you want to perform these actions?
#     Enter a value:            <- do not answer yet
# (-replace forces a change, so the prompt appears)

# terminal B: the lock object exists now
aws s3 ls s3://$TF_STATE_BUCKET/dev/network/
#  ... terraform.tfstate
#  ... terraform.tfstate.tflock
aws s3 cp \
  s3://$TF_STATE_BUCKET/dev/network/terraform.tfstate.tflock - \
  | jq '{ID, Operation, Who, Created}'
# terminal B: a second run cannot take the lock
terraform plan
# Error: Error acquiring the state lock
#   Lock Info:
#     ID:        <uuid>
#     Operation: OperationTypeApply
#     Who:       you@laptop

terraform plan -lock-timeout=2m &  # waits instead

# terminal A: answer "no"
#   Apply cancelled.
# → lock released, the waiting plan in B continues

aws s3 ls s3://$TF_STATE_BUCKET/dev/network/
#  ... terraform.tfstate      (.tflock is gone)

In a versioned bucket every lock leaves noncurrent versions and delete markers behind. Check for a rule that expires noncurrent versions (aws s3api get-bucket-lifecycle-configuration --bucket $TF_STATE_BUCKET); if there is none, add one to bootstrap/ (e.g. 90 days).

Lab 4 · Build (3/4)

State commands and a recovery from S3 versioning

Goal: rename with state mv (and see why code must follow), then break the state on purpose and restore the previous version.

terraform state list | wc -l          # 12 (11 + data)
terraform state show aws_vpc.this | head -5
terraform state pull | jq '{serial, lineage}'

# rename web → app: state AND code
terraform state mv aws_security_group.web \
  aws_security_group.app
terraform plan
#   - destroy aws_security_group.app
#   + create  aws_security_group.web   ← code not renamed!
sed -i.bak 's/aws_security_group" "web"/aws_security_group" "app"/;
  s/aws_security_group.web.id/aws_security_group.app.id/' sg.tf
terraform plan                       # No changes
# (ch 5: a moved block does this reviewably)

# break it: forget the internet gateway
terraform state rm aws_internet_gateway.this
terraform plan
#   + aws_internet_gateway.this will be created
#   ~ aws_route_table.public ...
#   (applying would try to attach a 2nd gateway)
B=$TF_STATE_BUCKET; K=dev/network/terraform.tfstate
aws s3api list-object-versions --bucket $B --prefix $K \
  --query "Versions[?Key=='$K'].[VersionId,LastModified,IsLatest]" \
  --output table
# pick the version just BEFORE the state rm (2nd newest)
V=<version-id>

aws s3api get-object --bucket $B --key $K \
  --version-id $V /tmp/candidate.tfstate >/dev/null
jq '[.resources[] | select(.type=="aws_internet_gateway")]
    | length' /tmp/candidate.tfstate        # 1

aws s3api copy-object --bucket $B --key $K \
  --copy-source "$B/$K?versionId=$V" >/dev/null
terraform plan                       # No changes

state rm and state mv each wrote a new version. If you restore a version from before the state mv, plan will want to rename back: pick the version right before the mistake.

Lab 4 · Build (4/4)

Publish network outputs to SSM for other components

Goal: a narrow, explicit contract: live/dev/app (ch 6, ch 10) reads two SSM parameters instead of the whole network state.

cat > outputs.tf <<'EOF'
output "vpc_id" {
  value = aws_vpc.this.id
}
output "public_subnet_ids" {
  value = [for s in aws_subnet.public : s.id]
}
output "private_subnet_ids" {
  value = [for s in aws_subnet.private : s.id]
}
EOF

cat > ssm.tf <<'EOF'
resource "aws_ssm_parameter" "vpc_id" {
  name  = "/tflab/${var.env}/network/vpc_id"
  type  = "String"
  value = aws_vpc.this.id
}

resource "aws_ssm_parameter" "private_subnet_ids" {
  name  = "/tflab/${var.env}/network/private_subnet_ids"
  type  = "StringList"
  value = join(",", [for s in aws_subnet.private : s.id])
}
EOF
terraform apply      # 2 to add (outputs change too)
terraform output
# private_subnet_ids = ["subnet-0aa…", "subnet-0bb…"]
# public_subnet_ids  = ["subnet-0cc…", "subnet-0dd…"]
# vpc_id             = "vpc-0a1b2c3d…"

aws ssm get-parameters-by-path --path /tflab/dev/network \
  --query 'Parameters[].[Name,Type,Value]' --output table
# /tflab/dev/network/private_subnet_ids  StringList  subnet-…,subnet-…
# /tflab/dev/network/vpc_id              String      vpc-0a1b…

# a consumer reads them like this (live/dev/app, later):
#   data "aws_ssm_parameter" "vpc_id" {
#     name = "/tflab/dev/network/vpc_id"
#   }
#   local.vpc_id = data.aws_ssm_parameter.vpc_id.insecure_value

git add backend.tf outputs.tf ssm.tf sg.tf
git commit -m "network: S3 backend, outputs, SSM contract" 

Standard-tier SSM parameters are free. Use SecureString only for secrets — and then keep them out of Terraform state where you can (ch 12).

Lab 4 · Verify, break it, clean up

Remote, locked, recoverable — and shared through a contract

The network state now lives in S3 with a lock, you have restored it once, and other components can consume it without reading it.

Verify

  • aws s3 ls s3://$TF_STATE_BUCKET/dev/network/ shows only terraform.tfstate when idle.
  • No terraform.tfstate left in live/dev/network/; terraform plan → No changes.
  • terraform state pull | jq .serial grew with each write; lineage never changed.
  • aws ssm get-parameter --name /tflab/dev/network/vpc_id --query Parameter.Value equals terraform output -raw vpc_id.

Break it

  • No lock: set use_lockfile = false, terraform init -reconfigure -backend-config=backend.hcl, and run two plans at once: both start. Restore true and reconfigure again.
  • Wrong flag: change key to dev/network2/terraform.tfstate and run init -reconfigure → plan wants to create 13 resources (empty state). Do not apply; restore the key, init -reconfigure.
  • force-unlock a live run: start terraform apply, then force-unlock <ID> from another terminal, and explain what could happen if both runs wrote. Then cancel A.
# keep network + state for later labs

rm -f sg.tf.bak /tmp/candidate.tfstate
rm -f /tmp/network-local-backup.tfstate

# full teardown (end of the deck):
# terraform destroy
#   (removes VPC, subnets, SG, SSM
#    parameters; the state object
#    stays, versioned, in S3)
Lab 5 · Build (1/4)

Adopt ClickOps resources: create the mess, then search for it

Goal: resources created by hand, found again with terraform query. A new root module live/dev/adopt with its own state key. Needs Lab 0 (ACCT, TF_STATE_BUCKET) and Lab 3 (files copied from network).

# 1) "ClickOps": made by hand, outside Terraform
B=tflab-clickops-$ACCT; aws s3api create-bucket --bucket $B \
  --create-bucket-configuration LocationConstraint=ap-northeast-1
aws ssm put-parameter --type String \
  --name /tflab/dev/clickops/feature-flag --value on
aws ssm put-parameter --type String \
  --name /tflab/dev/clickops/owner --value alice
# 2) a root module for the adoption
mkdir -p tf-lab/live/dev/adopt && cd tf-lab/live/dev/adopt
cp ../network/versions.tf ../network/providers.tf \
   ../network/variables.tf .
cat > backend.tf <<'EOF'
terraform {
  backend "s3" {
    key          = "dev/adopt/terraform.tfstate"
    region       = "ap-northeast-1"
    encrypt      = true
    use_lockfile = true
  }
}
EOF
terraform init -backend-config="bucket=$TF_STATE_BUCKET"  # S3
# 3) search the account (Terraform 1.14+)
cat > search.tfquery.hcl <<'EOF'
list "aws_s3_bucket" "all" {
  provider = aws
}
list "aws_ssm_parameter" "all" {
  provider = aws
  config {
    region = "ap-northeast-1"
  }
}
EOF
terraform validate -query
terraform query
# list.aws_s3_bucket.all      … tflab-clickops-123456789012 …
# list.aws_s3_bucket.all      … tflab-state-123456789012 …
# list.aws_ssm_parameter.all  … /tflab/dev/clickops/owner …
# (one line per object; columns depend on the provider)
# which types can be listed at all?
terraform providers schema -json | jq -r \
  '.provider_schemas[].list_resource_schemas | keys[]' \
  | grep -E 'aws_(s3_bucket|ssm_parameter)$'

list blocks return identities of every matching object: here also your state bucket and anything else in the account. Choosing what to adopt is your job, not Terraform's.

Lab 5 · Build (2/4)

Import blocks and a generated first draft

Goal: an import plan that says "3 to import" and changes nothing except the tags that default_tags adds on purpose.

cat > imports.tf <<EOF
import {
  to = aws_s3_bucket.reports
  identity = {
    bucket = "tflab-clickops-$ACCT"   # shell expands ACCT
  }
}

locals {
  clickops = {
    feature_flag = "/tflab/dev/clickops/feature-flag"
    owner        = "/tflab/dev/clickops/owner"
  }
}

import {
  for_each = local.clickops
  to       = aws_ssm_parameter.clickops[each.key]
  id       = each.value
}
EOF
# the SSM resource by hand (it uses for_each)
cat > ssm.tf <<'EOF'
resource "aws_ssm_parameter" "clickops" {
  for_each = local.clickops
  name     = each.value
  type     = "String"
  value    = each.key == "owner" ? "alice" : "on"
}
EOF
# let Terraform draft the bucket (single import)
terraform plan -generate-config-out=generated.tf
# Terraform has generated configuration and written it
# to generated.tf. Please review the configuration …
# review: delete null/default arguments, keep `bucket`
mv generated.tf bucket.tf && vi bucket.tf
terraform plan -out=tfplan
#   # aws_s3_bucket.reports will be updated in-place
#   # (imported from "tflab-clickops-123456789012")
#   ~ tags_all = { + "Env" = "dev" + "ManagedBy" = … }
# Plan: 3 to import, 0 to add, 3 to change, 0 to destroy.
terraform apply tfplan

The 3 changes are only tags_all (from default_tags). Any other ~ line means the code disagrees with reality: fix the code first. Public access block, encryption and versioning are separate resources: unmanaged unless imported too.

Lab 5 · Build (3/4)

Refactor with moved: count to for_each

Goal: replace positional count addresses with named keys, with a plan that shows moves and nothing else.

cat > extra.tf <<'EOF'
locals {
  extra_names = [
    "/tflab/dev/adopt/color",
    "/tflab/dev/adopt/size",
  ]
}

resource "aws_ssm_parameter" "extra" {   # positional
  count = length(local.extra_names)
  name  = local.extra_names[count.index]
  type  = "String"
  value = "v${count.index}"
}
EOF
terraform apply          # 2 to add
terraform state list | grep extra
# aws_ssm_parameter.extra[0]
# aws_ssm_parameter.extra[1]
# keyed version + one moved block per instance
cat > extra.tf <<'EOF'
locals {
  extra = {
    color = { name = "/tflab/dev/adopt/color", value = "v0" }
    size  = { name = "/tflab/dev/adopt/size", value = "v1" }
  }
}

resource "aws_ssm_parameter" "extra" {
  for_each = local.extra
  name     = each.value.name
  type     = "String"
  value    = each.value.value
}

moved {
  from = aws_ssm_parameter.extra[0]
  to   = aws_ssm_parameter.extra["color"]
}
moved {
  from = aws_ssm_parameter.extra[1]
  to   = aws_ssm_parameter.extra["size"]
}
EOF
terraform plan
# aws_ssm_parameter.extra[0] has moved to …extra["color"]
# aws_ssm_parameter.extra[1] has moved to …extra["size"]
# Plan: 0 to add, 0 to change, 0 to destroy.
terraform apply
Lab 5 · Build (4/4)

Let go with removed, then catch drift

Goal: hand the bucket back without deleting it, then see how a console change shows up in a refresh-only plan.

# A) stop managing the bucket, keep the bucket
rm bucket.tf
cat > removed.tf <<'EOF'
removed {
  from = aws_s3_bucket.reports
  lifecycle {
    destroy = false
  }
}
EOF
# the import block for the bucket must go as well:
# edit imports.tf, delete the first import { } block
terraform plan
#  # aws_s3_bucket.reports will no longer be managed by
#  # Terraform, but will not be destroyed
# Plan: 0 to add, 0 to change, 0 to destroy.
# Warning: Some objects will no longer be managed …
terraform apply
aws s3api head-bucket --bucket tflab-clickops-$ACCT \
  && echo still-there   # the bucket survived
# B) drift: someone changes a managed parameter
aws ssm put-parameter --overwrite \
  --name /tflab/dev/clickops/owner --value bob

terraform plan -refresh-only
# Note: Objects have changed outside of Terraform
#   # aws_ssm_parameter.clickops["owner"] has changed
#   ~ resource "aws_ssm_parameter" "clickops" {
#       ~ value   = (sensitive value)
#       ~ version = 1 -> 2
# This is a refresh-only plan, so Terraform will not
# take any actions to undo these. …

terraform plan
#   ~ value = (sensitive value)
# Plan: 0 to add, 1 to change, 0 to destroy.
# → decide: revert (apply this) or code it
#   (change "alice" to "bob" in ssm.tf)

terraform plan -detailed-exitcode; echo "exit=$?"
# exit=2 → changes present (what CI drift jobs use)

value of aws_ssm_parameter is sensitive, so plans show (sensitive value) instead of alice/bob. You still see that it changed, and the parameter version makes it obvious.

Lab 5 · Verify, break it, clean up

Adopted, renamed, released, and drift caught

Every step should have produced a plan you could read and approve before anything changed.

Verify

  • terraform state list shows aws_ssm_parameter.clickops["feature_flag"], ["owner"] and aws_ssm_parameter.extra["color"], ["size"]; no aws_s3_bucket.reports.
  • The S3 state object dev/adopt/terraform.tfstate exists in $TF_STATE_BUCKET.
  • The ClickOps bucket still exists after the removed apply.
  • Imported parameters now carry Project, Env and ManagedBy tags.
  • After you code or revert the drift, terraform plan says No changes.

Break it

  • No moved block: in step 3, plan once with the moved blocks commented out → 2 to add, 2 to destroy. Do not apply.
  • Wrong ID: change an import id to /tflab/dev/clickops/nope → "Cannot import non-existent remote object".
  • Delete without removed: move extra.tf away → plan wants to destroy both extra parameters. Do not apply; restore the file.
  • Think: if ssm.tf had said value = "off" in step 2, the plan would show the import and ~ value. What would applying it do to the real parameter?
cd tf-lab/live/dev/adopt
terraform destroy
#   4 parameters destroyed
# the bucket is no longer managed:
aws s3 rb s3://tflab-clickops-$ACCT \
  --force
rm -f search.tfquery.hcl tfplan
git add . && git commit \
  -m "lab5: adoption and refactor"
# the state object stays in S3
# until final teardown (Lab 13)
Lab 6 · Build (1/4)

Extract modules/network from the flat Lab 3 code

Goal: move the resources into a child module without recreating anything. Step 1: make the module out of the existing files. Needs Labs 3–4 (live/dev/network on the S3 backend).

cd tf-lab/live/dev/network && terraform plan  # "No changes"
mkdir -p ../../../modules/network
git mv main.tf sg.tf ../../../modules/network/
cd ../../../modules/network
sed -i.bak -e 's/ in ${var.region}//' \
  -e 's/var\.vpc_cidr/var.cidr/g' main.tf && rm main.tf.bak
cat > variables.tf <<'EOF'
variable "env" {
  type = string
  validation {
    condition     = contains(["dev", "prod"], var.env)
    error_message = "env must be dev or prod."
  }
}
variable "cidr" {
  type = string
  validation {
    condition     = can(cidrnetmask(var.cidr)) && endswith(var.cidr, "/16")
    error_message = "cidr must be an IPv4 /16."
  }
}
EOF
cat > outputs.tf <<'EOF'
output "vpc_id" { value = aws_vpc.this.id }
output "app_sg_id" { value = aws_security_group.app.id }
output "public_subnet_ids" {
  value = { for k, s in aws_subnet.public : k => s.id }
}
output "private_subnet_ids" {
  value = { for k, s in aws_subnet.private : k => s.id }
}
EOF
cat > versions.tf <<'EOF'
terraform {
  required_version = ">= 1.14"
  required_providers {
    aws = { source = "hashicorp/aws", version = ">= 6.0" }
  }
}
EOF
terraform fmt -check   # names stay as in Lab 3:
# every move is X → module.network.X

Keeping the resource names unchanged inside the module is what makes the moves mechanical. Rename resources later, in a separate change, with their own moved blocks inside the module.

Lab 6 · Build (2/4)

Call the module and generate the moved blocks

Goal: the root module calls module.network; the plan shows only "has moved to" lines and 0 to add, 0 to change, 0 to destroy.

cd ../../live/dev/network
cat > main.tf <<'EOF'
module "network" {
  source = "../../../modules/network"
  env    = var.env
  cidr   = var.vpc_cidr
}
EOF

# fix references left in the root (Lab 4's SSM
# parameters and outputs still point at aws_vpc.this …)
grep -n 'aws_vpc\.\|aws_subnet\.\|aws_security_group\.' *.tf
# → aws_vpc.this.id  becomes  module.network.vpc_id
#   [for s in aws_subnet.public : s.id] becomes
#   [for id in module.network.public_subnet_ids : id]

terraform init             # installs the local module
terraform plan | tail -1
# Plan: 11 to add, 0 to change, 11 to destroy.  ← NO!
# one moved block per resource (all instances move)
terraform state list \
  | grep -v -e '^data\.' -e '^aws_ssm_parameter\.' \
  | sed 's/\[.*//' | sort -u \
  | while read a; do
      printf 'moved {\n  from = %s\n  to   = module.network.%s\n}\n' \
        "$a" "$a"
    done > moved.tf
cat moved.tf
# moved {
#   from = aws_internet_gateway.this
#   to   = module.network.aws_internet_gateway.this
# }  … one per resource

terraform plan
# # aws_vpc.this has moved to module.network.aws_vpc.this
# # aws_subnet.public["ap-northeast-1a"] has moved to
# #   module.network.aws_subnet.public["ap-northeast-1a"]
# ...
# Plan: 0 to add, 0 to change, 0 to destroy.
terraform apply
cd ../../.. && git add . && git commit -m "network module"

The SSM parameters stay in the root (they publish the module's outputs), so they are excluded from the moves. If the plan still shows ~ lines, a reference in the root changed a value: compare it with the old expression.

Lab 6 · Build (3/4)

modules/app-bucket: an opinionated bucket

Goal: a second module with decisions built in (private, encrypted, versioned), validated input names, and outputs for callers.

mkdir -p modules/app-bucket && cd modules/app-bucket
cp ../network/versions.tf . && cat > main.tf <<'EOF'
resource "aws_s3_bucket" "this" {
  bucket        = "tflab-${var.env}-${var.name}"
  force_destroy = var.force_destroy
}
resource "aws_s3_bucket_public_access_block" "this" {
  bucket                  = aws_s3_bucket.this.id
  block_public_acls       = true
  block_public_policy     = true
  ignore_public_acls      = true
  restrict_public_buckets = true
}
resource "aws_s3_bucket_versioning" "this" {
  bucket = aws_s3_bucket.this.id
  versioning_configuration {
    status = "Enabled" # a decision, not an input
  }
}
EOF
cat > outputs.tf <<'EOF'
output "bucket_name" { value = aws_s3_bucket.this.bucket }
output "bucket_arn" { value = aws_s3_bucket.this.arn }
EOF
cat >> main.tf <<'EOF'
resource "aws_s3_bucket_server_side_encryption_configuration" "this" {
  bucket = aws_s3_bucket.this.id
  rule {
    apply_server_side_encryption_by_default {
      sse_algorithm = "AES256"
    }
  }
}
EOF
cat > variables.tf <<'EOF'
variable "env" {
  type = string
}
variable "name" { # bucket = tflab-<env>-<name>
  type = string
  validation {
    condition     = can(regex("^[a-z0-9-]{3,40}$", var.name))
    error_message = "name: 3-40 chars, a-z 0-9 and -."
  }
}
variable "force_destroy" {
  type    = bool
  default = false
}
EOF

terraform fmt -check && cd ../..
Lab 6 · Build (4/4)

live/dev/app from SSM inputs, then pin the module to a Git tag

Goal: a second component that reads the network through SSM, and a module consumed by version instead of by path.

mkdir -p live/dev/app && cd live/dev/app
cp ../network/{versions,providers,variables}.tf .
sed 's#dev/network/#dev/app/#' ../network/backend.tf \
  > backend.tf        # key = dev/app/terraform.tfstate
cat > main.tf <<'EOF'
data "aws_caller_identity" "me" {}
data "aws_ssm_parameter" "vpc_id" {
  name = "/tflab/dev/network/vpc_id" # Lab 4's name
}
module "assets" { # S3 names are global: add ACCT
  source        = "../../../modules/app-bucket"
  env           = "dev"
  name          = "assets-${data.aws_caller_identity.me.account_id}"
  force_destroy = true # lab only
}
resource "aws_security_group" "app" {
  name        = "tflab-dev-app"
  description = "App tier"
  vpc_id      = data.aws_ssm_parameter.vpc_id.insecure_value
}
output "bucket_name" { value = module.assets.bucket_name }
EOF
terraform init -backend-config="bucket=$TF_STATE_BUCKET"
terraform apply      # Plan: 5 to add, 0 to change, 0 to destroy.
# release the module: commit + annotated tag
cd ../../.. && git add . && git commit -m "app-bucket"
git tag -a app-bucket-v1.0.0 -m "app-bucket 1.0.0"

# consume the tag instead of the path
cd live/dev/app
REPO=$(git rev-parse --show-toplevel)
SRC="git::file://$REPO//modules/app-bucket?ref=app-bucket-v1.0.0"
sed -i.bak "s#\"../../../modules/app-bucket\"#\"$SRC\"#" main.tf
rm main.tf.bak && grep source main.tf
terraform init
# Downloading git::file:///…/tf-lab?ref=app-bucket-v1.0.0
#   for assets...
terraform plan       # No changes: same code, new source

# main moves on; dev stays on v1.0.0 until you bump it
cat >> ../../../modules/app-bucket/outputs.tf <<'EOF'

output "bucket_domain_name" {
  value = aws_s3_bucket.this.bucket_regional_domain_name
}
EOF
terraform init && terraform plan   # still No changes

git::file:// works for a local repository. After Lab 11 pushes the repo, the source becomes github.com/<you>/tf-lab//modules/app-bucket?ref=app-bucket-v1.0.0. Commit the new output, tag app-bucket-v1.1.0, bump ref, terraform init: only then does dev see it.

Lab 6 · Verify, break it, clean up

Modules extracted with zero changes; versions under control

The proof of a good refactor is a boring plan.

Verify

  • terraform state list in live/dev/network shows only module.network.*, data.* and the SSM parameters.
  • The apply in step 2 reports 0 added, 0 changed, 0 destroyed; the VPC ID in the console did not change.
  • live/dev/app state has module.assets.aws_s3_bucket.this (bucket tflab-dev-assets-<ACCT>) and aws_security_group.app in the dev VPC.
  • .terraform/modules/modules.json in live/dev/app shows the git::file://…?ref=app-bucket-v1.0.0 source.
  • git tag -l lists app-bucket-v1.0.0.

Break it

  • No moves: rename moved.tf to moved.tf.off and plan → every network resource is destroyed and recreated. Restore it.
  • Wrong mapping: edit one block to to = module.network.aws_subnet.private for the public subnets → Terraform reports an error instead of guessing. Read it, restore.
  • Bad input: set name = "Assets" (upper case) → the module's validation rejects it at plan time; set the network cidr to a /20 → rejected too.
  • Moving tag: git tag -f app-bucket-v1.0.0 on a newer commit → dev silently gets new code on the next init. Why do some teams pin full commit SHAs?
# keep network and app for Labs 7+
# (both cost nothing)

# keep moved.tf: it documents the
# refactor; delete it once every
# environment that uses this code
# has applied it (prod: Lab 10)

# app bucket (force_destroy = true):
# cd live/dev/app
# terraform destroy
git status   # clean
Lab 7 · Build (1/3)

TFLint with the AWS ruleset

Goal: one lint config for the whole repo, run recursively, with the AWS ruleset pinned. No AWS credentials needed.

cd tf-lab
cat > .tflint.hcl <<'EOF'
plugin "terraform" {
  enabled = true
  preset  = "recommended"
}

plugin "aws" {
  enabled = true
  version = "0.50.0"   # latest release when written: pin yours
  source  = "github.com/terraform-linters/tflint-ruleset-aws"
}
EOF

tflint --init                 # installs the aws plugin
tflint --recursive \
  --config "$PWD/.tflint.hcl" # every dir with .tf files
echo "exit=$?"                # 0 = clean, 2 = issues

# plant two mistakes to see the rules fire
cat > modules/app-bucket/lintbait.tf <<'EOF'
variable "unused" {}
EOF
tflint --chdir=modules/app-bucket \
  --config "$PWD/.tflint.hcl" --format compact
# 2 issue(s) found:
# modules/app-bucket/lintbait.tf:1:1: Warning -
#   `unused` variable has no type
#   (terraform_typed_variables)
# modules/app-bucket/lintbait.tf:1:1: Warning -
#   variable "unused" is declared but not used
#   (terraform_unused_declarations)
echo "exit=$?"                 # 2

# warnings only → still fail? decide per repo:
tflint --chdir=modules/app-bucket \
  --config "$PWD/.tflint.hcl" \
  --minimum-failure-severity=error; echo "exit=$?"  # 0

rm modules/app-bucket/lintbait.tf

# fix what the clean run reported (typical first-run
# findings: missing `type`, unused variables, missing
# required_version), then re-run until exit=0
git add .tflint.hcl && git commit -m "lint: tflint + aws ruleset" 

Check the latest tflint-ruleset-aws release on GitHub before pinning. --config "$PWD/.tflint.hcl" makes the root config apply when TFLint changes into subdirectories.

Lab 7 · Build (2/3)

terraform test for modules/app-bucket

Goal: plan-mode asserts, validation tests with expect_failures, and one apply against a mocked provider. Seconds, no credentials.

# assumes Lab 6: vars env (dev|prod), name; bucket
# tflab-<env>-<name>; resources "this"; output bucket_arn
cd modules/app-bucket && mkdir -p tests
cat > tests/unit.tftest.hcl <<'EOF'
mock_provider "aws" {
  mock_resource "aws_s3_bucket" {
    defaults = { arn = "arn:aws:s3:::tflab-dev-logs" }
  }
}
variables {
  env  = "dev"
  name = "logs"
}
run "name_follows_convention" {
  command = plan
  assert {
    condition     = aws_s3_bucket.this.bucket == "tflab-dev-logs"
    error_message = "bucket name must be tflab-<env>-<name>"
  }
}
# (continued on the right)
run "versioning_on_by_default" {
  command = plan
  assert {
    condition = (aws_s3_bucket_versioning.this
      .versioning_configuration[0].status == "Enabled")
    error_message = "versioning must default to Enabled"
  }
}
run "rejects_uppercase_name" {
  command = plan
  variables { name = "Logs" }
  expect_failures = [var.name]
}
run "rejects_unknown_env" {
  command = plan
  variables { env = "staging" }
  expect_failures = [var.env]
}
run "apply_with_mocks" {   # apply against the MOCK only
  assert {
    condition     = output.bucket_arn == "arn:aws:s3:::tflab-dev-logs"
    error_message = "arn output comes from the mocked bucket"
  }
}
EOF
unset AWS_PROFILE        # prove no credentials are used
terraform init -backend=false && terraform test
# … Success! 5 passed, 0 failed.
Lab 7 · Build (3/3)

terraform test for modules/network, then CI

Goal: assert subnet math at plan time, reject bad CIDRs, wire IDs with overrides, and run every module's tests in one command.

cd ../network && mkdir -p tests
cat > tests/network.tftest.hcl <<'EOF'
mock_provider "aws" {
  mock_data "aws_availability_zones" {
    defaults = { names = ["ap-northeast-1a", "ap-northeast-1c"] }
  }  # else names = []
}
variables {
  env  = "dev"
  cidr = "10.10.0.0/16"
}
run "subnet_plan" {
  command = plan
  assert {
    condition = (length(aws_subnet.public) == 2
                 && length(aws_subnet.private) == 2)
    error_message = "expected 2 public and 2 private subnets"
  }
  assert {
    condition = (aws_subnet.private["ap-northeast-1a"]
                 .cidr_block == "10.10.10.0/24")
    error_message = "private subnet in 1a must be 10.10.10.0/24"
  }
}

run "rejects_bad_cidr" {
  command   = plan
  variables { cidr = "10.10.0.0/24" }
  expect_failures = [var.cidr]
}
cat >> tests/network.tftest.hcl <<'EOF'

run "ids_from_overrides" {
  override_resource {
    target = aws_vpc.this
    values = { id = "vpc-0123456789abcdef0" }
  }
  assert {
    condition = alltrue([for s in aws_subnet.private :
                  s.vpc_id == "vpc-0123456789abcdef0"])
    error_message = "every private subnet must be in the VPC"
  }
}
EOF
terraform init -backend=false && terraform test
# … 3 passed, 0 failed.

# all modules at once (from the repo root)
cd ../..
for m in modules/*/; do
  (cd "$m" && terraform init -backend=false -input=false \
     >/dev/null && terraform test -junit-xml=test.xml) \
  || exit 1
done
printf 'test.xml\n' >> .gitignore
git add modules .gitignore
git commit -m "tests: unit tests with mocked aws provider" 
Lab 7 · Verify, break it, clean up

Green tests that would turn red

A test suite is only useful if it fails when the module breaks. Break the module on purpose and watch which test catches it.

Verify

  • tflint --recursive exits 0 on the whole repo.
  • terraform test passes in every module with AWS_PROFILE unset.
  • test.xml files exist (JUnit format, for CI test reports in ch 11).
  • terraform test -verbose prints the planned values for each run.

Break it

  • Subnet math: change i + 10 to i + 20 in modules/network → subnet_plan fails with a Diff (10.10.20.0/24 vs 10.10.10.0/24).
  • Weak validation: delete the regex in app-bucket → rejects_uppercase_name fails: the run expected a failure that did not happen.
  • Unknown at plan: add a plan-mode assert on output.bucket_arn → "Unknown condition value"; fix it with override_during = plan on the mock.
  • Lint: set required_version aside → TFLint reports terraform_required_version.
# nothing was created in AWS:
# mocks only. Revert the breaks:
git checkout -- modules

# keep the tests and .tflint.hcl;
# Lab 11 runs them in CI

# provider cache (optional):
export TF_PLUGIN_CACHE_DIR=\
"$HOME/.terraform.d/plugin-cache"
mkdir -p "$TF_PLUGIN_CACHE_DIR"
Lab 8 · Build (1/3)

A plan summary with a protected-type gate

Goal: scripts/plan-summary.sh turns any saved plan into a short table and exits 1 when a protected resource type would be deleted or replaced.

cd tf-lab && mkdir -p scripts
# no-op/read map to nothing (skipped), except a no-op
# with previous_address = a moved object
cat > scripts/plan-summary.sh <<'EOF'
#!/usr/bin/env bash
# usage: scripts/plan-summary.sh tfplan.json
set -euo pipefail
plan=${1:?usage: plan-summary.sh <plan.json>}
re=${PROTECTED_TYPES:-'^aws_(s3_bucket|vpc|subnet|kms_key|iam_role)$'}

jq -r '
  {"create": "CREATE", "update": "UPDATE",
   "delete": "DELETE", "forget": "FORGET",
   "delete,create": "REPLACE(-/+)",
   "create,delete": "REPLACE(+/-)"} as $label
  | .resource_changes[] as $r
  | ($r.change.actions | join(",")) as $a
  | ($label[$a] // (if $r.previous_address
                    then "MOVE" else empty end))
  | [., $r.address, ($r.action_reason // "")]
  | @tsv' "$plan" | column -t -s $'\t'
EOF
cat >> scripts/plan-summary.sh <<'EOF'

blocked=$(jq -r --arg re "$re" '
  .resource_changes[]
  | select((.change.actions | index("delete"))
           and (.type | test($re)))
  | .address' "$plan")
if [[ -n "$blocked" && "${ALLOW_DESTROY:-}" != "true" ]]; then
  echo "BLOCKED: deletes/replaces protected:" >&2
  echo "$blocked" >&2
  exit 1
fi
echo "OK: protected-type gate passed"
EOF
chmod +x scripts/plan-summary.sh

# baseline: no changes expected in dev/app
cd live/dev/app
terraform plan -input=false -out=tfplan -detailed-exitcode
echo "plan exit=$?"                 # 0 = nothing to do
terraform show -json tfplan > tfplan.json
../../../scripts/plan-summary.sh tfplan.json
# OK: protected-type gate passed   (empty table)

printf 'tfplan\ntfplan.json\n' >> ../../../.gitignore
Lab 8 · Build (2/3)

Two risky changes, caught before apply

Goal: see a bucket rename and a CIDR change as reviewers should see them. Neither plan is applied.

# 1) rename the bucket: in live/dev/app change the
#    module input (Lab 6): "assets-<ACCT>" -> "static-<ACCT>"
terraform plan -input=false -out=tfplan
#   # module.assets.aws_s3_bucket.this must be replaced
# -/+ resource "aws_s3_bucket" "this" {
#   ~ bucket = "tflab-dev-assets-<ACCT>"
#           -> "tflab-dev-static-<ACCT>"
#              # forces replacement
# Plan: 4 to add, 0 to change, 4 to destroy.
#  (bucket + 3 companions that take `bucket`)

terraform show -json tfplan > tfplan.json
../../../scripts/plan-summary.sh tfplan.json
echo "gate exit=$?"
# REPLACE(-/+)  module.assets.aws_s3_bucket.this
#               replace_because_cannot_update
# REPLACE(-/+)  module.assets.aws_s3_bucket_versioning.this …
# BLOCKED: deletes/replaces protected:
# module.assets.aws_s3_bucket.this
# gate exit=1
jq -c '.resource_changes[] | select(.change.replace_paths)
  | {address, paths: .change.replace_paths}' tfplan.json

git checkout -- .                  # undo the rename
# 2) change the dev VPC CIDR 10.10.0.0/16 -> 10.30.0.0/16
cd ../network                      # live/dev/network
F=$(grep -l '10.10.0.0/16' *.tf *.tfvars 2>/dev/null)
sed -i.bak 's#10.10.0.0/16#10.30.0.0/16#' $F && rm -f *.bak
terraform plan -input=false -out=tfplan
terraform show -json tfplan > tfplan.json
../../../scripts/plan-summary.sh tfplan.json
# REPLACE(-/+)  module.network.aws_vpc.this
# REPLACE(-/+)  module.network.aws_subnet.private["ap-northeast-1a"]
# … every subnet, route table association, IGW attachment
# UPDATE        aws_ssm_parameter.vpc_id   (new value)
# BLOCKED: deletes/replaces protected:
# module.network.aws_vpc.this
# module.network.aws_subnet.private["ap-northeast-1a"] …

# severity: Critical (ch 8) — everything in the VPC
# would be replaced, and dev/app reads these IDs
git checkout -- .                  # undo the CIDR change

# count actions for the PR description
jq -r '[.resource_changes[].change.actions | join(",")]
  | group_by(.) | map("\(.[0]): \(length)") | .[]' \
  tfplan.json

Addresses depend on your Lab 6 module names (module.assets, module.network); the shape of the output does not. Exact counts depend on what your network module contains.

Lab 8 · Build (3/3)

A safe change, applied only from the saved plan

Goal: review a low-risk plan, apply exactly that plan, and watch Terraform refuse a plan that went stale.

cd ../app                          # live/dev/app
# low-risk change: one more default tag in the provider
#   default_tags { tags = { …, Owner = "<you>" } }
terraform plan -input=false -out=tfplan
terraform show -json tfplan > tfplan.json
../../../scripts/plan-summary.sh tfplan.json
# UPDATE  module.assets.aws_s3_bucket.this   (tags_all)
# UPDATE  aws_ssm_parameter.…                 (tags_all)
# OK: protected-type gate passed

# review the human-readable plan once more
terraform show -no-color tfplan | less

# apply EXACTLY the reviewed plan (no re-plan, no prompt)
terraform apply -input=false tfplan
# Apply complete! Resources: 0 added, N changed,
# 0 destroyed.
# a stale plan: make two plans, apply the second
#   (change Owner to another value first)
terraform plan -input=false -out=a.tfplan
terraform plan -input=false -out=b.tfplan
terraform apply -input=false b.tfplan
terraform apply -input=false a.tfplan
# Error: Saved plan is stale
#
# The given plan file can no longer be applied
# because the state was changed by another
# operation after the plan was created.

# the escape hatch for a reviewed deletion:
ALLOW_DESTROY=true \
  ../../../scripts/plan-summary.sh tfplan.json
# (Lab 11 sets it from a PR label "allow-destroy")

rm -f tfplan tfplan.json a.tfplan b.tfplan
git add ../../../scripts ../../../.gitignore .
git commit -m "plan-summary gate; Owner tag in dev/app" 
Lab 8 · Verify, break it, clean up

The gate blocks what reviewers must see

The script is only as good as its list of protected types and the habit of reading its output. Break both on purpose.

Verify

  • Baseline plan -detailed-exitcode exits 0 in live/dev/app and live/dev/network after the clean-up steps.
  • The rename plan shows # forces replacement on bucket, and the gate exits 1.
  • The CIDR plan replaces the VPC and every subnet; the gate lists them.
  • The safe tag change was applied from tfplan; a new plan is empty.
  • a.tfplan was refused as stale.

Break it

  • Weak list: PROTECTED_TYPES="^aws_kms_key$" on the rename plan → exit 0 with a bucket replacement in it. The list is policy: review changes to it.
  • Re-plan apply: run terraform apply (no file) after editing a tag → it plans again and asks; with -auto-approve nobody reviewed that plan.
  • Target: terraform plan -target=module.assets → "Resource targeting is in effect": the summary looks small because the plan is incomplete.
  • force_destroy: set force_destroy = true on the bucket and re-run the rename plan: the gate still blocks it, but an apply would delete the objects.
# nothing risky was applied.
# make sure both roots are clean:
for c in app network; do
  (cd live/dev/$c &&
   terraform plan -input=false \
     -detailed-exitcode >/dev/null;
   echo "$c exit=$?")   # expect 0
done

# remove the Owner tag again if
# you do not want it: edit, plan,
# summary, apply tfplan.
rm -f live/dev/*/tfplan*
Lab 9 · Build (1/5)

Static scans: Checkov and Trivy on the modules

Goal: scan modules/ with both tools and list every finding. Needs Labs 6–8. Your list may be longer if your modules have more resources.

cd tf-lab
pipx install checkov          # Checkov 3.x (Python)
brew install trivy conftest
checkov --version; trivy --version; conftest --version

mkdir -p policies/checkov policies/opa

checkov -d modules --compact --quiet
# Check: CKV2_AWS_11: "Ensure VPC flow logging is enabled …"
#     FAILED for resource: aws_vpc.this
# Check: CKV2_AWS_12: "Ensure the default security group of
#     every VPC restricts all traffic"
# Check: CKV_AWS_130: "Ensure VPC subnets do not assign
#     public IP by default"  (aws_subnet.public[…])
# Check: CKV_AWS_18 / CKV_AWS_144 / CKV_AWS_145 /
#     CKV2_AWS_61 / CKV2_AWS_62   (aws_s3_bucket.this)
# … plus anything your own modules add
trivy config modules
# AWS-0089 (LOW)   Bucket has logging disabled
# AWS-0132 (HIGH)  … customer managed key
# AWS-0164 (HIGH)  Subnet associates public IP address
# AWS-0178 (MEDIUM) VPC does not have VPC Flow Logs

# triage each finding:
#   real risk in OUR use?      → fix in the module
#   accepted for this module?  → inline skip + reason
#   never applies to tf-lab?   → global skip, reviewed
#
# rule: every exception says WHY, in the diff
# where a reviewer will see it
Lab 9 · Build (2/5)

Fix or justify every finding

Goal: one real fix, inline suppressions with reasons for accepted risks, and two small reviewed files for global decisions.

# 1) FIX: lock down the default security group (free)
cat >> modules/network/main.tf <<'EOF'

# no ingress/egress blocks = all rules removed
resource "aws_default_security_group" "this" {
  vpc_id = aws_vpc.this.id
}
EOF

# 2) SUPPRESS WITH A REASON, next to the resource:
#    modules/network/main.tf, inside aws_subnet.public
#      #checkov:skip=CKV_AWS_130:public subnets by design
#    and directly ABOVE resource "aws_subnet" "public"
#      #trivy:ignore:AWS-0164 -- public subnets by design
#
#    modules/app-bucket/main.tf, inside aws_s3_bucket.this
#      #checkov:skip=CKV_AWS_18:lab data, no access logs
#      #checkov:skip=CKV_AWS_145:SSE-S3 is enough for lab
#    and above it
#      #trivy:ignore:AWS-0089 -- lab data, no access logs
# 3) GLOBAL decisions in one reviewed file each
cat > policies/checkov.yaml <<'EOF'
skip-check:
  - CKV_AWS_144   # replication: single-region lab
  - CKV2_AWS_61   # lifecycle rules: short-lived lab
  - CKV2_AWS_62   # event notifications: unused
  - CKV2_AWS_11   # flow logs: cost; enable in prod
compact: true
quiet: true
EOF
cat > policies/.trivyignore <<'EOF'
AWS-0132   # SSE-S3 in the lab; prod uses a CMK
AWS-0178   # flow logs: same decision as CKV2_AWS_11
EOF

Run terraform fmt -recursive and the module tests from Lab 7 after editing: suppression comments change nothing, but the default security group resource removes the default egress rule.

Lab 9 · Build (3/5)

Plan check 1: required tags with a custom Checkov check

Goal: every taggable resource carries Project, Env and ManagedBy — checked on the plan, where default_tags are merged into tags_all.

cat > policies/checkov/required_tags.yaml <<'EOF'
metadata:
  id: "CKV2_TFLAB_1"
  name: "Ensure taggable resources carry Project, Env and ManagedBy tags"
  category: "GENERAL_SECURITY"
  severity: "MEDIUM"
definition:
  and:
    - cond_type: attribute
      resource_types: &taggable
        [aws_s3_bucket, aws_vpc, aws_subnet, aws_internet_gateway]
      attribute: tags_all.Project
      operator: exists
    - cond_type: attribute
      resource_types: *taggable
      attribute: tags_all.Env
      operator: exists
    - cond_type: attribute
      resource_types: *taggable
      attribute: tags_all.ManagedBy
      operator: exists
EOF
cd live/dev/app
terraform plan -out=tfplan && terraform show -json tfplan > tfplan.json
checkov -f tfplan.json --external-checks-dir ../../../policies/checkov \
  --check CKV2_TFLAB_1 --compact --quiet
# terraform_plan scan results:
# Passed checks: 1, Failed checks: 0, Skipped checks: 0
cd ../../..
# a resource without ManagedBy would show:
#   Check: CKV2_TFLAB_1: "Ensure taggable resources …"
#     FAILED for resource: module.network.aws_vpc.this

YAML anchors (&taggable, *taggable) keep the three resource lists identical. The pass count depends on what your live/dev/app plan contains.

Lab 9 · Build (4/5)

Plan check 2: no S3 deletes or weakened public access (conftest)

Goal: a Rego policy over the plan JSON, with unit tests that run without any plan.

cat > policies/opa/s3.rego <<'EOF'
package main

import rego.v1

protected := {"aws_s3_bucket", "aws_s3_bucket_public_access_block"}

pab_settings := [
	"block_public_acls", "block_public_policy",
	"ignore_public_acls", "restrict_public_buckets",
]

deny contains msg if {
	some rc in input.resource_changes
	rc.type in protected
	"delete" in rc.change.actions
	msg := sprintf("%s: %v is not allowed for %s",
		[rc.address, rc.change.actions, rc.type])
}

deny contains msg if {
	some rc in input.resource_changes
	rc.type == "aws_s3_bucket_public_access_block"
	some setting in pab_settings
	rc.change.after[setting] != true
	msg := sprintf("%s: %s must be true", [rc.address, setting])
}
EOF
cat > policies/opa/s3_test.rego <<'EOF'
package main

import rego.v1

test_bucket_delete_denied if {
	count(deny) == 1 with input as {"resource_changes": [{
		"address": "aws_s3_bucket.x", "type": "aws_s3_bucket",
		"change": {"actions": ["delete"], "after": null}}]}
}

test_create_allowed if {
	count(deny) == 0 with input as {"resource_changes": [{
		"address": "aws_s3_bucket.x", "type": "aws_s3_bucket",
		"change": {"actions": ["create"], "after": {}}}]}
}
EOF
conftest verify --policy policies/opa      # 2 tests pass
conftest test --policy policies/opa live/dev/app/tfplan.json
# 0 failures on a normal plan
Lab 9 · Build (5/5)

One target: make policy

Goal: a single command that runs every static and plan check, locally and later in CI (Lab 11). Any failure stops it with a non-zero exit code.

cat >> Makefile <<'EOF'
PLAN_DIRS ?= live/dev/network live/dev/app

.PHONY: policy policy-static policy-plan
policy: policy-static policy-plan

policy-static:
	checkov -d modules --config-file policies/checkov.yaml
	trivy config --severity HIGH,CRITICAL --exit-code 1 \
	  --ignorefile policies/.trivyignore modules
	conftest verify --policy policies/opa

policy-plan:
	@for d in $(PLAN_DIRS); do \
	  test -f $$d/tfplan.json || { \
	    echo "missing $$d/tfplan.json: plan first"; exit 1; }; \
	  echo "== $$d"; \
	  checkov -f $$d/tfplan.json --compact --quiet \
	    --external-checks-dir policies/checkov \
	    --check CKV2_TFLAB_1 || exit 1; \
	  conftest test --policy policies/opa \
	    $$d/tfplan.json || exit 1; \
	done
EOF
# plans for both dev roots (Lab 8 style)
for d in live/dev/network live/dev/app; do
  terraform -chdir=$d plan -out=tfplan >/dev/null
  terraform -chdir=$d show -json tfplan > $d/tfplan.json
done

make policy
# checkov: Failed checks: 0, Skipped checks: N
# trivy:   no HIGH/CRITICAL findings left
# conftest verify: all tests pass
# == live/dev/network … 0 failures
# == live/dev/app     … 0 failures
echo $?        # 0

git add Makefile policies modules
git commit -m "policy: checkov, trivy, conftest gates"

Recipe lines start with a tab (spaces → missing separator); $$d passes $d to the shell.

Lab 9 · Verify, break it, clean up

Every finding fixed or justified, every bad plan blocked

The gate is only useful if it fails for the right reasons. Prove it with plans that must be rejected.

Verify

  • make policy exits 0 on the current code.
  • checkov -d modules lists each suppression under "Skipped checks" with its reason.
  • trivy config modules (no ignore file) still shows AWS-0132; with --ignorefile policies/.trivyignore it is gone.
  • Running the tag check on HCL (checkov -d live/dev/app --external-checks-dir policies/checkov --check CKV2_TFLAB_1) fails — explain why (ch 9).
  • git grep -n "checkov:skip\|trivy:ignore" gives the full list of exceptions to review.

Break it

  • Replace a bucket: change the bucket name input in live/dev/app → plan shows -/+; make policy fails with ["delete","create"] is not allowed. Do not apply; revert.
  • Weaken public access: set block_public_policy = false in modules/app-bucket → Trivy (HCL) and conftest (plan) both fail.
  • Lose a tag: remove ManagedBy from default_tags → CKV2_TFLAB_1 fails on the plan.
  • Silent skip: add #checkov:skip=CKV_AWS_18 without a reason → it still passes. Add a review rule for that (ch 13).
  • Cwd trap: run Trivy from modules/ without --ignorefile → ignores vanish.
# nothing billable was created
# in this lab: only files

# remove generated plans (they
# contain resolved values)
find live -name 'tfplan*' -delete

# keep policies/, Makefile and
# the module fixes: Lab 11 runs
# make policy in CI

# scratch tools can stay;
# uninstall if you like:
# pipx uninstall checkov
# brew uninstall trivy conftest
Lab 10 · Build (1/4)

Make the dev roots environment-agnostic

Goal: no literal "dev" left in live/dev/*/*.tf; every environment value comes from an auto-loaded terraform.tfvars. The dev plans must stay at "No changes". Needs Labs 4–9.

cd tf-lab
grep -n '"dev"\|/dev/\|-dev-' live/dev/*/*.tf | grep -v backend.tf
# live/dev/app/main.tf:  name = "/tflab/dev/network/vpc_id"
# live/dev/app/main.tf:  env  = "dev"
# live/dev/app/main.tf:  name = "tflab-dev-app"
# (your list may differ: fix every hit except backend keys)

cd live/dev/app
sed -i.bak \
  -e 's#"/tflab/dev/network/#"/tflab/${var.env}/network/#' \
  -e 's#env *= "dev"#env           = var.env#' \
  -e 's#"tflab-dev-app"#"tflab-${var.env}-app"#' \
  main.tf && rm main.tf.bak
terraform fmt
# environment values: auto-loaded by plain `terraform plan`
# (Lab 11's CI runs plan without -var-file)
cat > terraform.tfvars <<'EOF'
env = "dev"
EOF
cat > ../network/terraform.tfvars <<'EOF'
env      = "dev"
vpc_cidr = "10.10.0.0/16"
EOF

terraform plan -detailed-exitcode; echo "exit=$?"
# No changes. Your infrastructure matches the configuration.
# exit=0
cd ../network
terraform plan -detailed-exitcode; echo "exit=$?"
# exit=0

# same values, now explicit: the refactor is invisible to AWS
git add -A && git commit -m "live: env values in terraform.tfvars"

Exit 2 means something changed: read the diff, you replaced a value that was not identical. variables.tf keeps its defaults from Lab 3; the tfvars files make the environment explicit and win over defaults.

Lab 10 · Build (2/4)

Create prod from the same code and module version

Goal: live/prod/network (10.20.0.0/16) and live/prod/app, each with its own state key. Copy only source files — never .terraform/, state or plans.

cd tf-lab
for c in network app; do
  mkdir -p live/prod/$c
  cp live/dev/$c/*.tf live/dev/$c/.terraform.lock.hcl \
     live/prod/$c/                  # same providers
  sed -i.bak "s#dev/$c/terraform.tfstate#prod/$c/terraform.tfstate#" \
    live/prod/$c/backend.tf && rm live/prod/$c/backend.tf.bak
done

cat > live/prod/network/terraform.tfvars <<'EOF'
env      = "prod"
vpc_cidr = "10.20.0.0/16"
EOF
cat > live/prod/app/terraform.tfvars <<'EOF'
env = "prod"
EOF

# any "dev" left in prod is a bug (→ no output)
grep -n 'dev' live/prod/*/*.tf live/prod/*/*.tfvars
grep -h key live/prod/*/backend.tf
#   key = "prod/network/terraform.tfstate"
#   key = "prod/app/terraform.tfstate"
# producer first
cd live/prod/network
terraform init -backend-config="bucket=$TF_STATE_BUCKET"
terraform plan -out=tfplan
# Plan: N to add, 0 to change, 0 to destroy.
#   + cidr_block = "10.20.0.0/16"
#   + name = "/tflab/prod/network/vpc_id"
terraform apply tfplan

# then the consumer
cd ../app
terraform init -backend-config="bucket=$TF_STATE_BUCKET"
terraform plan -out=tfplan
#   + bucket = "tflab-prod-assets-<ACCT>"
#   + name   = "tflab-prod-app"
terraform apply tfplan

aws s3 ls s3://$TF_STATE_BUCKET --recursive | grep tfstate
#   dev/app/terraform.tfstate
#   dev/network/terraform.tfstate
#   prod/app/terraform.tfstate
#   prod/network/terraform.tfstate

Read the prod plan before applying: only + lines, prod names, the 10.20.0.0/16 CIDR. Any - or ~ means prod is pointing at dev state — stop and check the backend key. All resources here are free (VPC, subnets, IGW, SG, S3, SSM Standard).

Lab 10 · Build (3/4)

One command: plan every component in dependency order

Goal: make plan-all ENV=prod plans network then app, stops at the first error, and leaves tfplan + tfplan.json next to each root for review and for make policy.

mkdir -p scripts && cat > scripts/plan-all.sh <<'EOF'
#!/usr/bin/env bash
set -euo pipefail
ENV=${1:?usage: plan-all.sh dev|prod}
ORDER=(network app)   # producers first
changed=0
for c in "${ORDER[@]}"; do
  dir="live/$ENV/$c"
  echo "== $dir"
  terraform -chdir="$dir" init -input=false \
    -backend-config="bucket=$TF_STATE_BUCKET" >/dev/null
  rc=0
  terraform -chdir="$dir" plan -input=false -no-color \
    -out=tfplan -detailed-exitcode > "$dir/plan.txt" || rc=$?
  case $rc in
    0) echo "   no changes" ;;  2) echo "   CHANGES"; changed=1 ;;
    *) cat "$dir/plan.txt"; echo "   FAILED"; exit 1 ;;
  esac
  terraform -chdir="$dir" show -json tfplan > "$dir/tfplan.json"
done
exit $(( changed ? 2 : 0 ))   # like -detailed-exitcode
EOF
chmod +x scripts/plan-all.sh
printf 'plan.txt\n' >> .gitignore
cat >> Makefile <<'EOF'
ENV ?= dev

.PHONY: plan-all
plan-all:
	scripts/plan-all.sh $(ENV)
EOF
make plan-all ENV=prod
# == live/prod/network
#    no changes
# == live/prod/app
#    no changes

# policies over both environments (Lab 9 target)
make policy PLAN_DIRS="live/dev/network live/dev/app \
  live/prod/network live/prod/app"

git add -A && git commit -m "live/prod + plan-all"

Exit 2 (changes pending) makes make plan-all fail: usable as a drift check.

Lab 10 · Build (4/4)

Optional: the same layout in Terragrunt (plan only)

Goal: feel the trade-off of section 39 without touching real state. The comparison uses its own key prefix tg-compare/, so nothing it plans can affect tf-lab. Requires brew install terragrunt (1.x).

mkdir -p tg-compare/live/prod/network
cd tg-compare/live && cat > root.hcl <<'EOF'
remote_state {
  backend = "s3"
  generate = { path = "backend.tf", if_exists = "overwrite_terragrunt" }
  config = {
    bucket       = "tflab-state-${get_aws_account_id()}"
    key          = "tg-compare/${path_relative_to_include()}/terraform.tfstate"
    region       = "ap-northeast-1"
    encrypt      = true
    use_lockfile = true
  }
}

generate "provider" {
  path      = "provider.tf"
  if_exists = "overwrite_terragrunt"
  contents  = "provider \"aws\" { region = \"ap-northeast-1\" }"
}
EOF
cat > prod/network/terragrunt.hcl <<'EOF'
include "root" {
  path = find_in_parent_folders("root.hcl")
}

terraform {
  source = "../../../../modules//network"
}

inputs = { env = "prod", cidr = "10.20.0.0/16" }
EOF

cd prod
terragrunt run --all -- plan
# Plan: N to add, 0 to change, 0 to destroy.
# (empty key tg-compare/prod/network/… → all "create")

wc -l network/terragrunt.hcl ../../../live/prod/network/*.tf
# do NOT apply: it would be a second prod VPC
cd ../../.. && rm -rf tg-compare

Terragrunt generates backend.tf and provider.tf in .terragrunt-cache/. Check remote_state keys (e.g. use_lockfile) against your Terragrunt release.

Lab 10 · Verify, break it, clean up

Two environments, four states, one code path

Prod is a parameterized copy of dev, not a fork. Prove it, then break the layout in the ways that happen in real teams.

Verify

  • make plan-all ENV=dev and ENV=prod both print "no changes" (exit 0).
  • aws ssm get-parameters-by-path --path /tflab/prod/network returns the prod VPC; its CIDR is 10.20.0.0/16 (aws ec2 describe-vpcs).
  • diff -r live/dev live/prod -x .terraform -x "tfplan*" -x plan.txt shows only backend.tf keys and terraform.tfvars.
  • Both app roots use the same module source (?ref=app-bucket-v…) — promotion is a visible diff when you bump it.
  • make policy passes for all four tfplan.json files.

Break it

  • Copied .terraform: cp -r live/dev/network /tmp/x, edit its key to prod/…, run terraform plan without init → "Backend configuration changed". The dangerous answer is init -migrate-state: it would copy dev state into the prod key. Answer: init -reconfigure.
  • Forgotten key: in a scratch copy keep dev/network/… in the prod backend → the plan wants to change the dev VPC CIDR (forces replacement). Never apply; this is why the grep step exists.
  • Wrong order (think it through): in a brand-new environment, planning app before network is applied fails with ParameterNotFound for /tflab/<env>/network/vpc_id. ORDER=(network app) in the script is what prevents it.
# keep dev AND prod for Lab 11:
# CI plans and applies both

# at the end of the course,
# destroy consumers first:
for e in prod dev; do
  for c in app network; do
    terraform -chdir=live/$e/$c \
      destroy
  done
done

# remove generated files
find live -name 'tfplan*' \
  -o -name plan.txt | xargs rm -f
Lab 11 · Build (1/5)

CI roles: a reusable modules/ci-role

Goal: one small module that creates an IAM role trusted by GitHub OIDC for an exact list of token subjects.

# modules/ci-role/variables.tf
variable "name" { type = string }
variable "oidc_provider_arn" { type = string }
variable "subjects" {
  description = "Exact GitHub OIDC sub claims"
  type        = list(string)
  validation {
    condition     = alltrue([for s in var.subjects : !strcontains(s, "*")])
    error_message = "No wildcards in subjects."
  }
}
variable "managed_policy_arns" {
  type    = list(string)
  default = []
}
variable "inline_policy_json" {
  type    = string
  default = null
}

# modules/ci-role/outputs.tf
output "arn" { value = aws_iam_role.this.arn }
# modules/ci-role/main.tf (1/2)
data "aws_iam_policy_document" "trust" {
  statement {
    actions = ["sts:AssumeRoleWithWebIdentity"]
    principals {
      type        = "Federated"
      identifiers = [var.oidc_provider_arn]
    }
    condition {
      test     = "StringEquals"
      variable = "token.actions.githubusercontent.com:aud"
      values   = ["sts.amazonaws.com"]
    }
    condition {
      test     = "StringEquals"
      variable = "token.actions.githubusercontent.com:sub"
      values   = var.subjects
    }
  }
}

StringEquals with an explicit list: no StringLike, no wildcards, enforced by the validation block (ch 2) so nobody can loosen it by accident.

Lab 11 · Build (2/5)

CI roles: the role resources and the GitHub repo

Goal: finish the module, then create the public repo and read the numeric IDs that the immutable sub claim contains.

# modules/ci-role/main.tf (2/2)
resource "aws_iam_role" "this" {
  name                 = var.name
  assume_role_policy   = data.aws_iam_policy_document.trust.json
  max_session_duration = 3600
}
resource "aws_iam_role_policy_attachment" "managed" {
  for_each   = toset(var.managed_policy_arns)
  role       = aws_iam_role.this.name
  policy_arn = each.value
}
resource "aws_iam_role_policy" "inline" {
  count  = var.inline_policy_json == null ? 0 : 1
  role   = aws_iam_role.this.id
  policy = var.inline_policy_json
}
# create the repo (public: free runner minutes)
cd tf-lab
git rev-parse --git-dir >/dev/null 2>&1 || git init -b main
git add -A && git commit -qm "tf-lab"
gh repo create tf-lab --public --source . --push
gh api repos/<you>/tf-lab --jq '.owner.id, .id'
# 1234567   ← owner_id
# 8901234   ← repo_id

# repos created, renamed or transferred after
# 15 Jul 2026 get sub claims of the form
#   repo:<you>@1234567/tf-lab@8901234:<context>
# older repos: repo:<you>/tf-lab:<context>
# (print a real token's claims: CI/CD deck, Lab 4)
Lab 11 · Build (3/5)

CI roles: live/shared/ci, applied by you

Goal: the OIDC provider and both roles, in a root module that the pipeline never applies.

# live/shared/ci/main.tf (backend key shared/ci/...)
locals {
  repo = "repo:<you>@1234567/tf-lab@8901234"
  st   = "arn:aws:s3:::tflab-state-${data.aws_caller_identity.me.account_id}"
  apply_policies = ["AmazonVPCFullAccess", "AmazonS3FullAccess",
  "AmazonSSMFullAccess", "CloudWatchLogsFullAccess"]
}
data "aws_caller_identity" "me" {}

resource "aws_iam_openid_connect_provider" "gh" {
  url            = "https://token.actions.githubusercontent.com"
  client_id_list = ["sts.amazonaws.com"]
}
module "apply" {
  source            = "../../../modules/ci-role"
  name              = "gha-tf-apply"
  oidc_provider_arn = aws_iam_openid_connect_provider.gh.arn
  subjects = [
    "${local.repo}:environment:dev",
    "${local.repo}:environment:prod",
  ]
  managed_policy_arns = [for p in local.apply_policies :
  "arn:aws:iam::aws:policy/${p}"]
}
module "plan" {
  source            = "../../../modules/ci-role"
  name              = "gha-tf-plan"
  oidc_provider_arn = aws_iam_openid_connect_provider.gh.arn
  subjects = [
    "${local.repo}:pull_request",
    "${local.repo}:ref:refs/heads/main",
  ]
  managed_policy_arns = ["arn:aws:iam::aws:policy/ReadOnlyAccess"]
  inline_policy_json = jsonencode({
    Version = "2012-10-17"
    Statement = [{
      Effect   = "Allow"
      Action   = ["s3:PutObject", "s3:DeleteObject"]
      Resource = "${local.st}/*.tflock"
    }]
  })
}

# terraform init -backend-config="bucket=$TF_STATE_BUCKET" \
#   -backend-config="key=shared/ci/terraform.tfstate"
# terraform apply

ReadOnlyAccess already allows s3:GetObject and s3:ListBucket, so the inline policy only adds lock-file writes. The managed write policies keep the lab short; in real use, scope apply to what each component manages and never give CI IAM write access.

Lab 11 · Build (4/5)

The workflow and the GitHub environments

Goal: environments dev and prod (prod with you as required reviewer), the changed-directories script, and the workflow from ch 11.

R=<you>/tf-lab
ME=$(gh api user --jq .id)
gh api -X PUT repos/$R/environments/dev
gh api -X PUT repos/$R/environments/prod --input - <<EOF
{"reviewers": [{"type": "User", "id": $ME}],
 "deployment_branch_policy":
   {"protected_branches": false,
    "custom_branch_policies": true}}
EOF
gh api -X POST -f name=main \
  repos/$R/environments/prod/deployment-branch-policies
mkdir -p scripts && cat > scripts/changed-dirs.sh <<'EOF'
#!/usr/bin/env bash
set -euo pipefail
git diff --name-only "$BASE" HEAD > /tmp/changed.txt
if grep -q '^modules/' /tmp/changed.txt; then
  ls -d live/dev/* live/prod/*
else
  grep -oE '^live/(dev|prod)/[^/]+' /tmp/changed.txt || true
fi | sort -u | jq -R -s -c 'split("\n")
  | map(select(length > 0))
  | map({dir: ., env: (split("/")[1]),
         name: (split("/")[1:] | join("-"))})'
EOF
chmod +x scripts/changed-dirs.sh
# .github/workflows/terraform.yml
# = the four jobs from ch 11, section 41:
#   detect → checks (PR) → plan (matrix) → apply (push)
# the detect step passes the diff base:
      - id: d
        env:
          BASE: ${{ github.event.pull_request.base.sha
                    || github.event.before }}
        run: echo "dirs=$(scripts/changed-dirs.sh)" \
               >> "$GITHUB_OUTPUT"

# checks job: Lab 9's `make policy` needs its tools.
# Install pinned versions + verify checksums:
      - run: pipx install checkov==<version>
      # trivy + conftest: download the release tarball
      # and its checksums file, then sha256sum -c
# every `terraform init` there needs
#   -backend-config="bucket=tflab-state-<acct>"
# (partial backend config, Lab 0)
# .github/workflows/drift.yml = ch 11, section 43
# replace every <sha> with the full commit SHA of the
# action's current release, then:
git add -A && git commit -m "ci: terraform pipeline"
git push

In this single-person lab, leave "Prevent self-review" off so you can approve your own prod deployment. In a team, turn it on.

Lab 11 · Build (5/5)

One change through the pipeline

Goal: watch a tag change go PR → plan comment → merge → dev apply → prod approval → prod apply, then catch a console edit with the drift job.

git switch -c owner-tag
# add Owner = "<you>" to default_tags in
#   live/dev/app and live/prod/app
git commit -am "tags: add Owner" && git push -u origin HEAD
gh pr create --fill
gh pr checks --watch
# detect → checks → plan (dev-app, prod-app)
gh pr view --comments | head -40
# ### live/dev/app
#   ~ update in-place ... tags_all ... + "Owner" = "<you>"
# Plan: 0 to add, N to change, 0 to destroy.

gh pr merge --squash --delete-branch
gh run watch          # push run on main
# apply (dev-app)  ✓ applied automatically
# apply (prod-app) ⏳ waiting for review
# approve: Actions → the run → Review deployments
#          → prod → Approve (read the plan job log first)
gh run watch          # apply (prod-app) ✓

# drift: change a tag outside Terraform
B=$(cd live/dev/app && terraform output -raw bucket_name)
aws s3api put-bucket-tagging --bucket "$B" --tagging \
  'TagSet=[{Key=Env,Value=hacked}]'
gh workflow run drift.yml && gh run watch
gh issue list
# #7  Drift: live/dev/app

# revert: re-run the whole push run, so it plans
# again and dev applies main (a re-run of only the
# apply job would hit "Saved plan is stale")
gh run rerun <push-run-id>
# ...or codify the change in a PR

The output name bucket_name is whatever live/dev/app exposes for its bucket; adjust it to your Lab 6 outputs. put-bucket-tagging replaces the whole tag set, so the plan will want to restore every tag.

Lab 11 · Verify, break it, clean up

Reviewed plans, gated applies, visible drift

The pipeline should make the safe path the only easy path.

Verify

  • PR shows one plan comment per changed component, and no comment for untouched ones.
  • The apply (prod-app) job waits in "Waiting" until you approve; dev-app does not.
  • CloudTrail: AssumeRoleWithWebIdentity for gha-tf-plan on the PR, gha-tf-apply only in environment jobs.
  • The drift job opens one issue and comments on it on later nights instead of opening duplicates.
  • gh secret list shows no AWS keys.

Break it

  • Plan role writes: in a PR, add terraform apply to the plan job → AWS denies the writes (ReadOnlyAccess). Revert.
  • **Wrong sub:** remove environment: ${{ matrix.c.env }} from the apply job → "Not authorized to perform sts:AssumeRoleWithWebIdentity".
  • Stale plan: while prod waits for approval, apply prod locally, then approve → "Saved plan is stale".
  • Lock file: delete .terraform.lock.hcl in a PR → init -lockfile=readonly fails.
# keep the repo and roles for Lab 12;
# at the very end:
cd live/shared/ci
terraform destroy
gh repo delete <you>/tf-lab --yes

# disable the nightly drift job
# if you keep the repo:
gh workflow disable drift.yml
Lab 12 · Build (1/4)

Secrets: watch a password land in state

Goal: see with your own eyes that sensitive hides a value on screen but stores it in state, plan files and old S3 versions.

mkdir -p live/dev/secrets-demo && cd live/dev/secrets-demo
cat > main.tf <<'EOF'
terraform {
  required_version = ">= 1.14"
  required_providers {
    aws    = { source = "hashicorp/aws", version = "~> 6.0" }
    random = { source = "hashicorp/random", version = "~> 3.7" }
  }
  backend "s3" {
    key          = "dev/secrets-demo/terraform.tfstate"
    region       = "ap-northeast-1"
    use_lockfile = true
  }
}
provider "aws" { region = "ap-northeast-1" }
resource "random_password" "app" {
  length = 32
}
resource "aws_ssm_parameter" "app_secret" {
  name  = "/tflab/dev/secrets-demo/app-secret"
  type  = "SecureString"
  value = random_password.app.result
}
EOF
terraform init -backend-config="bucket=$TF_STATE_BUCKET"
terraform apply   # value = (sensitive value) on screen

# ...but in the state, in clear text:
terraform state pull | jq -r '.resources[]
  | select(.type == "random_password")
  | .instances[0].attributes.result'
# Xk2…   (32 characters)
terraform state pull | jq -r '.resources[]
  | select(.type == "aws_ssm_parameter")
  | .instances[0].attributes.value'
# Xk2…   (the same value, a second time)

# and in every saved plan:
terraform plan -out=tfplan -replace=random_password.app
terraform show -json tfplan | jq -r \
  '.prior_state.values.root_module.resources[]
   | select(.type=="random_password") | .values.result'
# Xk2…

State lives in tflab-state-<acct> with versioning on: each apply keeps a full copy of the old state, including this value. Delete tfplan now (rm tfplan).

Lab 12 · Build (2/4)

Secrets: ephemeral password + write-only argument

Goal: the same SSM parameter, but the password exists only during the run. Terraform 1.11+ and hashicorp/random with the ephemeral random_password.

# replace the two resources in main.tf with:
ephemeral "random_password" "app" {
  length = 32
}

resource "aws_ssm_parameter" "app_secret" {
  name             = "/tflab/dev/secrets-demo/app-secret"
  type             = "SecureString"
  value_wo         = ephemeral.random_password.app.result
  value_wo_version = 1
}

terraform plan
# - random_password.app will be destroyed
#   (only leaves state: it never existed in AWS)
# ~ aws_ssm_parameter.app_secret will be updated in-place
#   + value_wo         = (write-only attribute)
#   + value_wo_version = 1
#   (exact lines vary by provider version)
terraform apply
terraform state pull | jq '.resources[]
  | select(.type == "aws_ssm_parameter")
  | .instances[0].attributes
  | {value, value_wo, value_wo_version}'
# { "value": null, "value_wo": null, "value_wo_version": 1 }
#  (an empty string for value is also "not stored")

# the secret still exists, in SSM only:
aws ssm get-parameter --with-decryption \
  --name /tflab/dev/secrets-demo/app-secret \
  --query Parameter.Value --output text
# 9fQ…   (new value: the ephemeral password of the apply)

terraform plan     # No changes. (a new ephemeral password is
                   # generated, but value_wo_version is still 1)
# rotate: set value_wo_version = 2, then
terraform apply    # ~ update in-place; SSM gets a new value

The exact rendering of value in state depends on the provider version; what matters is that neither the old nor the new password appears in it. Old state versions in S3 still contain the first password: rotating (version 2) makes that copy useless.

Lab 12 · Build (3/4)

Lock down the state bucket

Goal: TLS only, and object access only for you and the two CI roles. Applied from bootstrap/, never by CI.

# bootstrap/state-policy.tf
# A bucket has ONE policy: if Lab 0 already created an
# aws_s3_bucket_policy, merge these statements into it.
variable "admin_role_arn" {
  description = "IAM role ARN (not the STS session ARN)"
  type        = string
}
data "aws_caller_identity" "pol" {}
locals {
  b    = "tflab-state-${data.aws_caller_identity.pol.account_id}"
  acct = data.aws_caller_identity.pol.account_id
}
resource "aws_s3_bucket_policy" "state" {
  bucket = local.b
  policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      { Sid       = "DenyInsecureTransport"
        Effect    = "Deny"
        Principal = "*"
        Action    = "s3:*"
        Resource  = ["arn:aws:s3:::${local.b}",
                     "arn:aws:s3:::${local.b}/*"]
        Condition = { Bool = { "aws:SecureTransport" = "false" } } },
      { Sid       = "OnlyAdminAndCI"
        Effect    = "Deny"
        Principal = "*"
        Action    = ["s3:GetObject", "s3:PutObject",
                     "s3:DeleteObject"]
        Resource  = "arn:aws:s3:::${local.b}/*"
        Condition = { ArnNotLike = { "aws:PrincipalArn" = [
          var.admin_role_arn,
          "arn:aws:iam::${local.acct}:role/gha-tf-plan",
          "arn:aws:iam::${local.acct}:role/gha-tf-apply",
        ] } } },
    ]
  })
}

# your role ARN: take the role name from the session ARN
aws sts get-caller-identity --query Arn --output text
# arn:aws:sts::<acct>:assumed-role/AWSReservedSSO_Admin…_abc/you
aws iam list-roles --query "Roles[?RoleName=='<that name>'].Arn"
# arn:aws:iam::<acct>:role/aws-reserved/sso.amazonaws.com/
#   ap-northeast-1/AWSReservedSSO_Admin…_abc

cd bootstrap && terraform apply \
  -var admin_role_arn='<the role ARN>'

Run terraform plan in live/dev/app right after applying: if it fails with AccessDenied, your ARN is wrong. Fix it from bootstrap/ (local state, so you are not locked out of it). Worst case, the account root user can delete the bucket policy.

Lab 12 · Build (4/4)

Provider hashes for every platform

Goal: a lock file that verifies providers on your Mac and on Linux runners, and a CI init that refuses to change it.

cd live/dev/app
grep -c 'h1:' .terraform.lock.hcl     # per provider:
# 1 h1: (your platform) + many zh: (registry checksums)

terraform providers lock \
  -platform=darwin_arm64 \
  -platform=linux_amd64 \
  -platform=linux_arm64
git diff --stat .terraform.lock.hcl
# 1 file changed, 4 insertions(+)    ← new h1: lines

# do the same in every root module, then commit
for d in ../../*/*/; do
  (cd "$d" && terraform providers lock \
     -platform=darwin_arm64 -platform=linux_amd64 \
     -platform=linux_arm64)
done
git add -A && git commit -m "lock: hashes for 3 platforms"
# what CI runs (Lab 11):
terraform init -input=false -lockfile=readonly \
  -backend-config="bucket=$TF_STATE_BUCKET"

# readonly mode refuses changes to the lock file:
# use a provider that is not locked yet ...
echo 'resource "null_resource" "x" {}' > extra.tf
terraform init -lockfile=readonly \
  -backend-config="bucket=$TF_STATE_BUCKET"
# - Installing hashicorp/null v3.x...
# Error: Provider dependency changes detected
# Changes to the required provider dependencies were
# detected, but the lock file is read-only. ...

rm extra.tf    # undo

Module versions are not in the lock file. Pin registry modules with an exact version and Git modules with ?ref=<full commit SHA> (ch 12).

Lab 12 · Verify, break it, clean up

Secrets out of state, state out of reach

Prove each control with a command, then break it on purpose.

Verify

  • terraform state pull | grep -c <first 6 chars of the SSM value> → 0.
  • aws ssm get-parameter --with-decryption returns the value: AWS has it, Terraform does not.
  • A plan from gha-tf-plan (re-run a PR check) still works with the new bucket policy.
  • curl http://<bucket>.s3.ap-northeast-1.amazonaws.com/ style access (no TLS) is denied.
  • Each root module's lock file lists h1: hashes for three platforms.

Break it

  • No version bump: change length = 40 only → No changes. Bump value_wo_version → update.
  • Ephemeral in a normal argument: set value = ephemeral.random_password.app.result → error: ephemeral values are not valid in that argument.
  • Third role: create a role without the ARN in the policy and try aws s3 cp s3://<bucket>/dev/app/terraform.tfstate - → AccessDenied.
  • Readonly lock: the null provider test above.
cd live/dev/secrets-demo
terraform destroy
# deletes the SSM parameter

# optional: purge old state
# versions of this key only
aws s3api list-object-versions \
  --bucket $TF_STATE_BUCKET \
  --prefix dev/secrets-demo/ \
  --query 'Versions[].VersionId'
# then delete-object --version-id
# for each (keep the bucket policy
#  until the very end of the deck)
Lab 13 · Build (1/4)

Upgrade drill: a provider major in a scratch root

Goal: feel a major provider upgrade without risking real infrastructure. The scratch root has only data sources and outputs, so applying it creates nothing in AWS.

mkdir -p scratch/upgrade && cd scratch/upgrade
cat > main.tf <<'EOF'
terraform {
  required_version = ">= 1.14"
  required_providers {
    aws = { source = "hashicorp/aws", version = "~> 5.0" }
  }
}
provider "aws" { region = "ap-northeast-1" }
data "aws_region" "current" {}
data "aws_caller_identity" "me" {}
output "where" {
  value = format("%s in %s",
    data.aws_caller_identity.me.account_id,
    data.aws_region.current.name)
}
EOF
terraform init     # local state: scratch only
grep -A1 'registry.terraform.io/hashicorp/aws' \
  .terraform.lock.hcl   # version = "5.x.y"
terraform apply -auto-approve
# where = "123456789012 in ap-northeast-1"
# bump the major: "~> 5.0" → "~> 6.0"
sed -i.bak 's/~> 5.0/~> 6.0/' main.tf && rm main.tf.bak

terraform plan
# Error: Inconsistent dependency lock file
#  … locked version selection 5.x.y doesn't match
#  the updated version constraints
#  → run "terraform init -upgrade"

terraform init -upgrade
grep -A1 'registry.terraform.io/hashicorp/aws' \
  .terraform.lock.hcl   # version = "6.x.y"

terraform validate   # Success! (removed args fail here)
terraform plan
# Warning: Deprecated attribute
#   The attribute "name" is deprecated. …

# fix: .name → .region  (new attribute in v6)
sed -i.bak 's/current\.name/current.region/' main.tf
rm main.tf.bak
terraform plan       # No changes. and no warning

Exact error and warning wording varies by Terraform and provider version; what matters is the sequence: constraint → init -upgrade → lock file diff → validate → a no-op plan without warnings. Deprecated attributes show up in plan, not always in validate.

Lab 13 · Build (2/4)

Minor upgrades for real, then refresh modes and drift

Goal: upgrade a live root inside its constraint, then see which plan modes notice a console change.

cd live/dev/network
terraform plan                # No changes. (baseline)

# newest 6.x allowed by "~> 6.0"
terraform init -upgrade
git diff --stat .terraform.lock.hcl
terraform plan                # still No changes.

# hashes for every platform that runs this root
terraform providers lock \
  -platform=linux_amd64 -platform=darwin_arm64
git add .terraform.lock.hcl
git commit -m "dev/network: aws provider 6.x.y"

# timings (run each twice; the first warms caches)
time terraform plan -refresh=false
time terraform plan
time terraform plan -minimal-refresh   # 1.17+
# ClickOps: add a tag in the "console"
VPC=$(aws ec2 describe-vpcs --filters \
  Name=tag:Project,Values=tf-lab Name=tag:Env,Values=dev \
  --query 'Vpcs[0].VpcId' --output text)
aws ec2 create-tags --resources $VPC \
  --tags Key=Owner,Value=clickops

terraform plan -refresh=false     # No changes. (blind)
terraform plan -minimal-refresh   # No changes. (VPC not
                                  #  part of any change)
terraform plan                    # ~ update in-place
#   ~ tags_all = {
#       - "Owner" = "clickops" -> null …
terraform plan -refresh-only      # shows the drift as a
                                  # state update instead

# decide: we do not want it → revert through Terraform
terraform apply                   # review, then yes
terraform plan                    # No changes.

Your timings depend on the root size: dev/network has only a handful of resources, so the differences are small. On a state with hundreds of resources, the refresh is most of the plan time.

Lab 13 · Build (3/4)

OpenTofu on a copy of the state

Goal: run tofu init and tofu plan against the same code without touching the real backend, the real lock file or Terraform's .terraform/ directory.

brew install opentofu && tofu version
cd live/dev/network

# 1) a private copy of the state (contains real IDs:
#    keep it out of Git, delete it afterwards)
terraform state pull > /tmp/dev-network.tfstate

# 2) point THIS directory at the copy, for tofu only:
#    *_override.tf files replace the backend block
cat > backend_override.tf <<'EOF'
terraform {
  backend "local" {
    path = "/tmp/dev-network.tfstate"
  }
}
EOF

# 3) separate data dir: Terraform's .terraform/ untouched
export TF_DATA_DIR=.terraform-tofu
tofu init
tofu plan
# expected: No changes.
# or an error naming a feature / state version that
# OpenTofu does not support yet. Both are results:
# write them down for the ADR (ch 13).

# what changed in the lock file?
git diff .terraform.lock.hcl
# + provider "registry.opentofu.org/hashicorp/aws" {
# +   version = "6.x.y"
# +   hashes  = [ … ]

# 4) undo everything
unset TF_DATA_DIR
rm -rf .terraform-tofu backend_override.tf
git checkout -- .terraform.lock.hcl
rm /tmp/dev-network.tfstate
git status                # clean
terraform plan            # No changes. (real backend)

Never run tofu apply here: the copy is local, so an apply would only diverge the copy from reality. A real migration follows the OpenTofu guide for your exact Terraform version, one root at a time.

Lab 13 · Build (4/4)

Module docs, then the full teardown in the right order

Goal: generated module READMEs, then destroy everything — dependents before dependencies, and the state bucket last because it holds every other state.

brew install terraform-docs
# add the markers once to each module README:
#   <!-- BEGIN_TF_DOCS -->
#   <!-- END_TF_DOCS -->
for m in network app-bucket ci-role; do
  terraform-docs markdown table \
    --output-file README.md --output-mode inject \
    modules/$m
done
git diff --stat modules/     # Inputs/Outputs tables

# ---- teardown: app before network, prod then dev
for env in prod dev; do
  for c in app network; do
    (cd live/$env/$c && terraform destroy)
  done
done
# BucketNotEmpty? empty its versions (right), retry
# Lab 12 demo and the CI roles from Lab 11
(cd live/dev/secrets-demo && terraform destroy)
(cd live/shared/ci && terraform destroy)
gh repo delete <you>/tf-lab   # optional
# ---- bootstrap LAST: its own state is in the bucket
cd bootstrap      # ADMIN_ARN: same role ARN as Lab 12
rm backend.tf                  # 1) state back to local
terraform init -migrate-state  #    answer: yes
# 2) delete the prevent_destroy line in main.tf, then
terraform apply -var admin_role_arn=$ADMIN_ARN  # (lifecycle)

# 3) empty every object version and delete marker
B=$TF_STATE_BUCKET
for kind in Versions DeleteMarkers; do
  aws s3api delete-objects --bucket $B --delete "$(
    aws s3api list-object-versions --bucket $B \
      --query "{Objects: $kind[].{Key:Key,VersionId:VersionId}}" \
      --output json)"
done
# (an empty list makes delete-objects fail: harmless)

terraform destroy -var admin_role_arn=$ADMIN_ARN # 4)

# 5) last: the budget
aws budgets delete-budget --account-id $ACCT \
  --budget-name <name from Lab 0>

delete-objects handles up to 1,000 keys per call — enough for this lab. Migrating bootstrap's state back to local before emptying the bucket matters: emptying first would delete the only copy of that state.

Lab 13 · Verify, break it, clean up

Upgraded, compared, documented — and gone

Each step has a check you can run. The teardown is complete only when the account has nothing tagged Project=tf-lab.

Verify

  • Scratch root: v6 in the lock file, plan "No changes" with no deprecation warning.
  • dev/network lock file commit lists h1: hashes for both linux_amd64 and darwin_arm64.
  • You can say which plan modes saw the Owner=clickops tag and why.
  • tofu plan result recorded; git status clean afterwards.
  • After teardown: aws resourcegroupstaggingapi get-resources --tag-filters Key=Project,Values=tf-lab returns an empty list (it may lag a few minutes).

Break it

  • **Skip init -upgrade:** bump the constraint and run plan → "Inconsistent dependency lock file". Why is that error a feature?
  • One platform: keep only the darwin_arm64 h1: line (delete every zh: line) and push → the Lab 11 CI init fails with a checksum mismatch. With zh: hashes present, a registry install on Linux still verifies.
  • Wrong order: try terraform destroy in live/dev/network before app → read the error (dependent resources or SSM lookups) instead of forcing it.
  • Bootstrap first: reason through what would happen to every live/* state if the bucket went first. (Do not run it.)
# verify nothing is left
aws resourcegroupstaggingapi \
  get-resources --tag-filters \
  Key=Project,Values=tf-lab \
  --query \
  'ResourceTagMappingList[].ResourceARN'
# []

aws s3api head-bucket \
  --bucket $TF_STATE_BUCKET
# ... (404) Not Found

aws iam get-role \
  --role-name gha-tf-apply
# NoSuchEntity

# local clean-up
rm -rf scratch/
# keep the tf-lab repo: it is your
# reference implementation
Lab safety

Cost and cleanup

The labs use only free or near-free AWS resources. The real risks are leftovers, a forgotten resource type, and credentials — not the bill for a VPC.

Can cost money

  • S3: storage and requests for state and app buckets (cents), more if you upload files.
  • KMS customer managed keys, if you choose KMS over SSE-S3: a monthly fee per key, and keys wait 7–30 days before deletion.
  • CloudWatch log groups if something writes to them; set retention.
  • Anything outside the lab rules: NAT gateways, EC2, RDS, EKS bill per hour — the labs never create them.
  • HCP Terraform beyond the Free tier (500 managed resources); GitHub Actions minutes on private repos.

Habits

  • Budget alarm before Lab 1 (Lab 0).
  • Provider default_tags with Project=tf-lab: one tag query finds every leftover.
  • terraform plan before every destroy, and read what will go.
  • OIDC roles for CI, no access keys anywhere; delete them in the teardown.
  • Never put real secrets in lab variables: state and plan files store them in clear text.

Clean up (Lab 13)

  • Destroy live/prod then live/dev, each app before network.
  • Destroy live/dev/secrets-demo (Lab 12) and live/shared/ci (Lab 11); optionally delete <you>/tf-lab.
  • Bootstrap last: migrate its state to local, remove prevent_destroy, empty all object versions, destroy.
  • Check: aws resourcegroupstaggingapi get-resources --tag-filters Key=Project,Values=tf-lab is empty; then delete the budget.
Glossary

Language, state and workflow terms

Short anchors; explain each in your own words.

TermShort meaning
Root / child moduleDirectory you run terraform in (own backend, state) / module called with a module block
ProviderPlugin that maps resources to an API (e.g. hashicorp/aws)
Resource / data sourceObject Terraform manages / object it only reads
Resource addressmodule.network.aws_subnet.public["a"] — identity in state
StateMap from addresses to real IDs and attributes; holds secrets in clear text
BackendWhere state is stored and locked (s3, HCP Terraform, …)
State lock vs lock fileRun-time lock on a state (use_lockfile) vs .terraform.lock.hcl (provider versions, hashes)
DriftReal infrastructure differs from state/config (often ClickOps)
TermShort meaning
Plan / refreshProposed changes (-out file applied as reviewed) / reading real objects before diffing
Replacement (ForceNew)Destroy + create because an argument cannot change in place
moved / removed blockRename an address without touching the resource / stop managing it (destroy = false keeps it)
import blockBrings an existing resource under management through a plan
List block / terraform querySearches existing resources for bulk import (1.14+)
Ephemeral valueValue used during a run but never written to state or plan
Write-only argument*_wo argument sent to the provider, never stored in state
Workspace (CLI)Several states for one configuration; not an environment boundary
Glossary

Ecosystem, policy and delivery terms

Short anchors; explain each in your own words.

TermShort meaning
IaCInfrastructure defined in reviewed, versioned code
OpenTofuMPL-licensed fork of Terraform 1.5 under the Linux Foundation
BUSL 1.1Terraform's source-available license since 1.6
HCP TerraformHashiCorp's hosted runs, state, registry and policies
StacksHCP Terraform feature to deploy components across many deployments
Terragrunt / TerramateOrchestration tools for many roots (DRY config, ordering)
Blast radiusHow much one bad apply or one state can break
action blockProvider operation outside CRUD, run with apply -invoke (1.14+)
Resource identityProvider-defined identifying attributes, used by import and query
TermShort meaning
Policy as codeGuardrails as versioned, tested rules run in CI
CheckovScanner with built-in policies for HCL and plan JSON
Trivy configIaC misconfiguration scanner (absorbed tfsec)
OPA / Rego / conftestPolicy engine, its language, and a CLI to test files with it
Terraform PolicyHCL-based policy as code, GA in Terraform 1.17
SentinelHashiCorp's policy language for HCP Terraform / Enterprise
OIDC federationCI gets short-lived cloud credentials without stored keys
Drift detectionScheduled plan -detailed-exitcode; exit 2 = changes
Break-glassAudited emergency access outside the normal pipeline
Further reading

Primary references

Terraform and the AWS provider release often; when a slide and the docs disagree, the docs win.

Terraform and OpenTofu

  • Terraform language and CLI docs — developer.hashicorp.com/terraform/language, …/terraform/cli
  • Releases and changelogs — github.com/hashicorp/terraform/releases
  • S3 backend (use_lockfile) — developer.hashicorp.com/terraform/language/backend/s3
  • Refactoring with moved — developer.hashicorp.com/terraform/language/modules/develop/refactoring
  • Import blocks and bulk import with terraform query — developer.hashicorp.com/terraform/language/import, …/language/import/bulk
  • Tests — developer.hashicorp.com/terraform/language/tests
  • OpenTofu docs and migration guides — opentofu.org/docs, opentofu.org/docs/intro/migration
  • HCP Terraform legacy Free plan end of life — support.hashicorp.com/hc/en-us/articles/47520801288083

AWS provider, policy and tooling

  • AWS provider docs and v6 upgrade guide — registry.terraform.io/providers/hashicorp/aws/latest/docs/guides/version-6-upgrade
  • AWS provider changelog — github.com/hashicorp/terraform-provider-aws/blob/main/CHANGELOG.md
  • Checkov — www.checkov.io; Trivy misconfiguration scanning — trivy.dev
  • Trivy supply chain incident (Mar 2026) — www.aquasec.com/blog/trivy-supply-chain-attack/
  • OPA and Rego — www.openpolicyagent.org/docs; conftest — www.conftest.dev
  • TFLint — github.com/terraform-linters/tflint; terraform-docs — terraform-docs.io
  • Terragrunt — terragrunt.gruntwork.io; Infracost — www.infracost.io/docs
  • GitHub OIDC to AWS — docs.github.com/actions/security-for-github-actions/security-hardening-your-deployments/configuring-openid-connect-in-amazon-web-services
1 / 1