Senior Platform Engineer · Step 11

SRE Practices

Measure what users experience. Spend reliability deliberately. Respond, learn, and prove recovery.

15 chapters · 60 sections · 15 labs · Use ← → / Space · I for index · O for overview

SRE Practices

Contents

Select a chapter · press I to return here

Chapter 00 · SRE Practices

Orientation and service context

← Contents
Chapter 0 · Section 00

Reliability as a user outcome

Define the decision, the measure, and its limits.

Key point 1

Reliability means users can complete important tasks within acceptable time and correctness bounds. A healthy pod or green dashboard is supporting evidence, not the outcome itself.

Key point 2

Map the practice loop: user journeys → SLIs/SLOs → budget and alerts → on-call/incident response → learning → capacity/resilience/DR.

Chapter 0 · Section 01

The shared service and failure model

Define the decision, the measure, and its limits.

Key point 1

Trace POST /orders through orders and payments; identify failure domains: application, dependency, Kubernetes node, and network. Discuss database and region loss as hypothetical production scenarios, not features of this app.

Key point 2

Name the customer impact of errors, slowness, duplicate charges, and in-memory order loss separately.

Chapter 0 · Section 02

Lab setup and safety boundaries

Define the decision, the measure, and its limits.

Key point 1

Choose local obs kind cluster or the specified CSV/JSON fixtures. Review safe fault knobs, synthetic traffic, and baseline dashboards.

Key point 2

Define the lab stop condition: stop load or faults if the local service does not recover within the planned window; record cleanup and verify normal operation.

Chapter 0 · Section 03

Evidence and study method

Define the decision, the measure, and its limits.

Key point 1

Keep a reliability notebook with assumptions, SLO specification, alert evaluations, incident timeline, action owners, capacity worksheet, and drill results.

Key point 2

Each lab asks for an artifact and a short explanation of decisions, not just a screenshot.

Chapter 0 · Worked example

One journey, several failure modes

Use the example to make a concrete reliability decision.

POST /orders → orders → payments → response
502 from payments: availability failure
201 after 2.1 s: availability pass, latency failure
201 with wrong item: correctness failure in the offline fixture

Interpretation

Start with what a user received. A pod's Ready status does not answer whether the journey succeeded.

Chapter 0 · Common mistakes

What goes wrong

Check these before accepting an SLO, runbook, experiment, or report.

Avoid

Equating healthy Pods with a successful order; mixing live measurements with hypothetical fixture data.

Better check

Find the source data, state assumptions, test the decision, and record who owns the next action.

Chapter 0 · Self-check

Explain it without notes

Answer aloud, then reveal the suggested answer.

What is the service outcome?

A user can create the intended order correctly and within the agreed time.

Lab 0

Map the journey

Complete the lab and keep its artifact for the final review.

1 · Prepare and do

Draw POST /orders → orders → payments and mark app, node, and network failures.

2 · Verify and record

Capture healthy POST status and latency; reset fault knobs; record cleanup.

Evidence: ** draw the shop user journey and dependency map, record a healthy baseline, and verify the fault controls are confined to the local lab.
Chapter 01 · SRE Practices

SRE foundations and user journeys

← Contents
Chapter 1 · Section 04

What SRE changes in operations

Define the decision, the measure, and its limits.

Key point 1

Service level indicators, objectives, agreements, and error budgets; distinguish a customer-facing commitment from an internal engineering target.

Key point 2

Reliability is one feature among others; higher targets have engineering and opportunity costs.

Chapter 1 · Section 05

Choose critical journeys

Define the decision, the measure, and its limits.

Key point 1

Use create order as the measured journey. Discuss read order only as a design exercise: it is unreliable with two independent in-memory replicas and has no read traffic from loadgen.

Key point 2

Define service boundaries and dependent teams; a journey may cross multiple services and still needs one user-facing measure.

Chapter 1 · Section 06

Failure modes and risk

Define the decision, the measure, and its limits.

Key point 1

Build a small failure-mode inventory: symptoms, affected users, detectability, recovery path, and likely blast radius.

Key point 2

Distinguish availability, latency, correctness, durability, and freshness; do not collapse all user harm into HTTP 5xx.

Chapter 1 · Section 07

Ownership and decision rights

Define the decision, the measure, and its limits.

Key point 1

Define service owner, SLO owner, on-call owner, and follow-up action owner; defer incident command roles to Section 32.

Key point 2

Record who can declare an incident, pause a release, approve risk, and end an incident.

Chapter 1 · Worked example

Choose the user boundary

Use the example to make a concrete reliability decision.

Journey: create an order
Eligible: valid POST /orders attempts from users
Not a measured journey: GET /orders/{id}
Why: two replicas have independent maps and duplicate ID sequences

Interpretation

The current read endpoint cannot support a reliable user-facing SLI. A real system would need shared durable state and read traffic.

Chapter 1 · Common mistakes

What goes wrong

Check these before accepting an SLO, runbook, experiment, or report.

Avoid

Defining checkout or read-order reliability for an app that cannot measure them consistently.

Better check

Find the source data, state assumptions, test the decision, and record who owns the next action.

Chapter 1 · Self-check

Explain it without notes

Answer aloud, then reveal the suggested answer.

Why exclude read order?

Two independent in-memory replicas can return 404 or the wrong order; loadgen creates no read traffic.

Lab 1

Write a service charter

Complete the lab and keep its artifact for the final review.

1 · Prepare and do

Name the create-order owner, dependency, impact, and failure modes.

2 · Verify and record

Explain why read order is excluded; define the decision owner.

Evidence: ** write a one-page service charter for create order, with owners, dependency failure modes, user impact, and a note explaining why read order is excluded from the measured SLO.
Chapter 02 · SRE Practices

SLI design and measurement

← Contents
Chapter 2 · Section 08

Event-based SLIs

Define the decision, the measure, and its limits.

Key point 1

Specify eligible events, good events, measurement point, labels, and exclusions. Use good / eligible; show the equivalent bad-event ratio.

Key point 2

Prefer edge or user-visible measurement when possible; compare server-side and synthetic or client-side views.

Chapter 2 · Section 09

Availability and correctness

Define the decision, the measure, and its limits.

Key point 1

Define how validation 4xx, server 5xx, timeouts, failed dependency calls, and duplicate outcomes count for POST /orders.

Key point 2

Handle retries and idempotency: a successful retry may still represent a bad user experience; define the event boundary before counting.

Key point 3

Define fixture-based correctness as successful checked POSTs with actual_order_id = expected_order_id and actual_item = expected_item / successful checked POSTs. Explain that this offline check is not an instrumented production SLI for the current app.

Chapter 2 · Section 10

Latency and low traffic

Define the decision, the measure, and its limits.

Key point 1

Define the latency SLI as eligible POST /orders requests completed within 0.3 seconds divided by eligible requests, matching the Observability lab's 0.3-second histogram bucket; explain why a percentile alone is awkward for budget accounting.

Key point 2

Show effects of sparse traffic, missing samples, scrape gaps, and short windows. Include minimum-volume or no-data handling in alert design.

Chapter 2 · Section 11

SLI implementation and validation

Define the decision, the measure, and its limits.

Key point 1

Translate the definition into metrics and a query; aggregate before division where appropriate, preserve counter reset handling, and avoid averaging per-instance ratios.

Key point 2

Compare a sampled request log with the metric result; document known bias and blind spots.

Chapter 2 · Worked example

Three explicit ratios

Use the example to make a concrete reliability decision.

Availability = good POSTs / eligible POSTs
Latency = POSTs ≤ 0.3 s / eligible POSTs
Fixture correctness = matched ID + item / successful checked POSTs
Fixture: 9/11 available; 7/11 fast; 8/9 correct

Interpretation

A denominator is part of the contract. Decide how invalid requests, timeouts, and retries count before writing a query.

Chapter 2 · Common mistakes

What goes wrong

Check these before accepting an SLO, runbook, experiment, or report.

Avoid

Changing denominators between dashboards; treating a percentile as the proportion under 0.3 seconds.

Better check

Find the source data, state assumptions, test the decision, and record who owns the next action.

Chapter 2 · Self-check

Explain it without notes

Answer aloud, then reveal the suggested answer.

What is the availability denominator?

Eligible user POST /orders events, under the written inclusion and exclusion policy.

Lab 2

Define the ratios

Complete the lab and keep its artifact for the final review.

1 · Prepare and do

Use sre-lab/fixtures/orders-requests.csv to count availability, ≤0.3 s latency, and offline correctness.

2 · Verify and record

Show numerator, denominator, exclusions, and one known measurement gap.

Evidence: ** write availability and latency SLI specifications, calculate ratios from fixtures, and check one result against the dashboard or request log.
Chapter 03 · SRE Practices

SLO design and review

← Contents
Chapter 3 · Section 12

Target and window

Define the decision, the measure, and its limits.

Key point 1

Choose an initial target from user need and historical performance, not from an arbitrary number of nines.

Key point 2

Compare rolling versus calendar windows and short versus long windows; explain why the window changes allowed bad events.

Chapter 3 · Section 13

Worked 30-day SLO

Define the decision, the measure, and its limits.

Key point 1

For the hypothetical 3,000,000 eligible order requests in 30 days at 99.9% availability, calculate 3,000 allowed bad requests; this averages about 1.16 requests/s versus the live loadgen's approximately 5 requests/s. Show the remaining budget after a sample incident.

Key point 2

For a time-based 99.9% SLO, calculate 43.2 minutes of allowed downtime in a 30-day window. Keep the request-based and time-based budgets distinct.

Key point 3

Define separate latency and availability targets only when each leads to a distinct operational decision.

Chapter 3 · Section 14

SLO document

Define the decision, the measure, and its limits.

Key point 1

Record journey, SLI formula, data source, target, window, exclusions, owner, review date, reporting location, and limitations.

Key point 2

Describe how to correct a broken measure or change a target without rewriting history.

Chapter 3 · Section 15

Review cadence and stakeholder agreement

Define the decision, the measure, and its limits.

Key point 1

Review trends, customer feedback, incident history, and support tickets; compare the target with actual user experience.

Key point 2

Set a cadence for revisiting the SLO and record the decision in an auditable change log.

Chapter 3 · Worked example

Count the budget

Use the example to make a concrete reliability decision.

30 days × 24 h × 60 min = 43,200 min
Time-based 99.9% budget = 43.2 min
3,000,000 hypothetical requests × 0.001 = 3,000 bad requests
3,000,000 / 2,592,000 s ≈ 1.16 requests/s

Interpretation

These are two different budget models. The lab load generator runs about 5 requests/s; do not present its data as the 3-million-request example.

Chapter 3 · Common mistakes

What goes wrong

Check these before accepting an SLO, runbook, experiment, or report.

Avoid

Using the request budget and time budget interchangeably; ignoring traffic volume.

Better check

Find the source data, state assumptions, test the decision, and record who owns the next action.

Chapter 3 · Self-check

Explain it without notes

Answer aloud, then reveal the suggested answer.

How many bad requests are allowed?

3,000 of a hypothetical 3,000,000 requests at a 99.9% target.

Lab 3

Propose an SLO

Complete the lab and keep its artifact for the final review.

1 · Prepare and do

Calculate 3,000 allowed bad events from hypothetical 3 million requests and 43.2 minutes from 30 days.

2 · Verify and record

Write target, window, owner, source, exclusions, rationale, and review date.

Evidence: ** publish a one-page orders SLO proposal with a 30-day calculation, rationale, exclusions, and a review decision.
Chapter 04 · SRE Practices

Error budgets and policy

← Contents
Chapter 4 · Section 16

Budget arithmetic

Define the decision, the measure, and its limits.

Key point 1

Derive allowed bad events or bad time from the target; compute consumed and remaining budget over the same window.

Key point 2

Explain why a request-based budget and a time-based budget can answer different questions, and why traffic changes matter.

Chapter 4 · Section 17

Budget as a release signal

Define the decision, the measure, and its limits.

Key point 1

Present Google's example error-budget policy first: when the service exceeds its budget over the preceding four-week window, halt changes and releases except P0 issues or security fixes until it is back within SLO.

Key point 2

Then compare a locally approved softer policy such as staged releases or explicit risk review. State its decision owner, exceptions, and evidence; do not present it as Google's rule.

Chapter 4 · Section 18

Budget policy and conflicts

Define the decision, the measure, and its limits.

Key point 1

State who owns the policy, how deployment risk is assessed, which incidents count, how emergency fixes proceed, and when the policy is reviewed.

Key point 2

Include Google's example rule: a single incident consuming more than 20% of the four-week error budget requires a postmortem (with a P0 action item in that example policy). Explain the difference between that four-week policy and this deck's 30-day worked SLO.

Key point 3

Walk through product versus reliability trade-offs using a concrete release scenario.

Chapter 4 · Section 19

Budget reporting

Define the decision, the measure, and its limits.

Key point 1

Show current SLO, budget spent, burn trend, incident contributions, and data quality in a report a product owner can read.

Key point 2

Avoid misleading single-day percentages and double counting across separate SLIs.

Chapter 4 · Worked example

Apply the published policy

Use the example to make a concrete reliability decision.

30-day fixture: 3,000,000 eligible; 3,200 bad → 200 over budget
Last 28 days: 2,800,000 eligible; 3,060 bad → 260 over budget
Single incident: 800 / 2,800 = 28.6% of four-week budget
Google example: halt releases except P0/security fixes; write postmortem

Interpretation

The 30-day SLO example and Google's four-week sample policy use different windows. Recompute from the matching window before deciding.

Chapter 4 · Common mistakes

What goes wrong

Check these before accepting an SLO, runbook, experiment, or report.

Avoid

Calling a local exception Google's policy; pausing work without a documented decision owner.

Better check

Find the source data, state assumptions, test the decision, and record who owns the next action.

Chapter 4 · Self-check

Explain it without notes

Answer aloud, then reveal the suggested answer.

When does Google's sample freeze releases?

After exceeding the four-week error budget, except P0 and security fixes.

Lab 4

Apply a budget policy

Complete the lab and keep its artifact for the final review.

1 · Prepare and do

Read sre-lab/fixtures/orders-daily.csv and incident-ledger.csv; compute consumed and remaining budget.

2 · Verify and record

Give a release decision under Google's example policy after aligning its four-week window, then state any local exception separately.

Evidence: ** use an incident ledger to calculate budget consumption and write a release recommendation under a documented policy.
Chapter 05 · SRE Practices

Burn-rate alerting

← Contents
Chapter 5 · Section 20

Burn-rate reasoning

Define the decision, the measure, and its limits.

Key point 1

Define burn rate as observed bad-event rate divided by the allowed bad-event rate. At 1×, a 30-day budget lasts about 30 days if the rate persists.

Key point 2

Work through high, medium, and slow burns; show why a high error percentage without enough traffic can be noisy.

Chapter 5 · Section 21

Multiwindow alert design

Define the decision, the measure, and its limits.

Key point 1

Use the SRE Workbook starting parameters for a 30-day 99.9% SLO: 2% budget in 1h/5m → 14.4× → page; 5% in 6h/30m → 6× → page; 10% in 3d/6h → 1× → ticket. Both long and short windows must exceed the same threshold. These are starting points, not universal thresholds.

Key point 2

Explain the short-window guideline of about 1/12 of the long window, plus evaluation interval, missing data, seasonality, and traffic volume.

Chapter 5 · Section 22

Alert quality and routing

Define the decision, the measure, and its limits.

Key point 1

Page on actionable user impact; route slower burns to a ticket or review. Include owner, SLO, current burn, estimated budget impact, dashboard, and runbook.

Key point 2

Suppress or inhibit lower-priority overlapping notifications so one incident does not produce all three alerts. Test recovery, flapping, duplicate alerts, silence expiration, and notification delivery to a test receiver.

Chapter 5 · Section 23

Worked alert exercise

Define the decision, the measure, and its limits.

Key point 1

Calculate burn from fixture windows, evaluate whether a page or ticket should fire, and explain a false-positive scenario.

Chapter 5 · Worked example

Three alert tiers

Use the example to make a concrete reliability decision.

Page: 14.4× over both 1 h and 5 min → 2% budget
Page: 6× over both 6 h and 30 min → 5% budget
Ticket: 1× over both 3 d and 6 h → 10% budget
For 99.9% SLO, 14.4× means bad-event ratio > 0.0144

Interpretation

Short windows confirm the burn is active. Inhibit overlapping lower-priority notifications from the same incident.

Chapter 5 · Common mistakes

What goes wrong

Check these before accepting an SLO, runbook, experiment, or report.

Avoid

Paging on a single short spike; sending three notifications for one incident; ignoring no-data behavior.

Better check

Find the source data, state assumptions, test the decision, and record who owns the next action.

Chapter 5 · Self-check

Explain it without notes

Answer aloud, then reveal the suggested answer.

Why require a short window too?

It confirms that the budget is still burning and lets the alert clear sooner.

Lab 5

Test burn alerts

Complete the lab and keep its artifact for the final review.

1 · Prepare and do

Inspect sre-lab/rules/slo-alerts.yaml: 14.4× 1h/5m, 6× 6h/30m, and 1× 3d/6h.

2 · Verify and record

Run promtool test rules on the packaged test file; use alertmanager.test.yaml with a local receiver to check routing and inhibition.

Evidence: ** define both two-window page rules and the slower ticket rule; use promtool test rules with synthetic series, then use amtool and a test receiver to verify suppression and recovery behavior.
Chapter 06 · SRE Practices

On-call and operational readiness

← Contents
Chapter 6 · Section 24

A sustainable rotation

Define the decision, the measure, and its limits.

Key point 1

Define primary/secondary coverage, handoff, escalation, ownership hours, and a backup when the primary is unavailable.

Key point 2

Track page load, sleep disruption, repeat pages, and time spent on toil; use the data to improve the system.

Chapter 6 · Section 25

Runbooks and access

Define the decision, the measure, and its limits.

Key point 1

A runbook starts with user symptoms, quick checks, safe mitigations, rollback, escalation, and verification.

Key point 2

Rehearse access, permissions, dashboards, and emergency contacts before the shift; avoid embedding secrets in runbooks.

Chapter 6 · Section 26

Toil and automation

Define the decision, the measure, and its limits.

Key point 1

Identify repetitive, manual, reactive work that scales with service growth; prioritize automation by impact and safety.

Key point 2

Automate diagnosis or a reversible recovery step only after documenting its failure modes and fallback.

Chapter 6 · Section 27

Handoffs and readiness reviews

Define the decision, the measure, and its limits.

Key point 1

Review current risks, open incidents, recent deploys, budget state, known noisy alerts, and scheduled changes at handoff.

Chapter 6 · Worked example

The handoff must survive an absence

Use the example to make a concrete reliability decision.

Primary: Alex; backup: Sam
Open risk: payments errors after recent deploy
Current burn: 0.7×; no active page
First action: confirm POST /orders error ratio; rollback only with evidence

Interpretation

A good handoff points to the dashboard, recent changes, escalation path, and a safe first action.

Chapter 6 · Common mistakes

What goes wrong

Check these before accepting an SLO, runbook, experiment, or report.

Avoid

An on-call shift with no backup, no access rehearsal, or a runbook that contains only commands.

Better check

Find the source data, state assumptions, test the decision, and record who owns the next action.

Chapter 6 · Self-check

Explain it without notes

Answer aloud, then reveal the suggested answer.

What must a handoff include?

Risks, changes, current budget, dashboard, runbook, primary and backup contacts.

Lab 6

Rehearse on-call

Complete the lab and keep its artifact for the final review.

1 · Prepare and do

Write a handoff with primary, backup, latest deploy, budget state, and risk.

2 · Verify and record

Have a second person follow the payments runbook without verbal help.

Evidence: ** write a one-page on-call handoff and runbook for a payments failure, then perform a dry run from the runbook alone.
Chapter 07 · SRE Practices

Detection, triage, and declaration

← Contents
Chapter 7 · Section 28

Recognize impact

Define the decision, the measure, and its limits.

Key point 1

Start from user symptoms and SLO signals; correlate changes in traffic, errors, latency, dependencies, and recent deployments.

Key point 2

Separate evidence from hypotheses; mark uncertain timestamps and missing telemetry.

Chapter 7 · Section 29

Severity and declaration

Define the decision, the measure, and its limits.

Key point 1

Define example severity criteria using user impact, scope, duration, and data risk. Declare early when coordination is needed, then revise severity as evidence changes.

Key point 2

Record incident start time, initial scope, known unknowns, and a shared incident channel or log.

Chapter 7 · Section 30

Fast triage

Define the decision, the measure, and its limits.

Key point 1

Check whether the issue is ongoing, widespread, and linked to a recent change; decide the next safe information-gathering step.

Key point 2

Avoid speculative fixes that expand blast radius; use a rollback or traffic shift when it offers a faster, safer mitigation.

Chapter 7 · Section 31

Initial response

Define the decision, the measure, and its limits.

Key point 1

Escalate to the on-call backup or service owner when the first responder cannot contain impact; state the trigger and deadline for escalation. Section 32 defines command roles after declaration.

Chapter 7 · Worked example

Five-minute triage

Use the example to make a concrete reliability decision.

09:00 page: create-order 14.4× burn
09:02 verify user impact and affected region
09:03 correlate with payments rollout
09:05 declare incident and escalate to backup
Unknown: whether rollback will clear errors

Interpretation

Keep observed facts separate from the rollback hypothesis. Early declaration creates a place to coordinate.

Chapter 7 · Common mistakes

What goes wrong

Check these before accepting an SLO, runbook, experiment, or report.

Avoid

Waiting for root cause before declaring; writing a hypothesis as a confirmed fact.

Better check

Find the source data, state assumptions, test the decision, and record who owns the next action.

Chapter 7 · Self-check

Explain it without notes

Answer aloud, then reveal the suggested answer.

When should an incident be declared?

When user impact needs coordination; severity can be revised as facts arrive.

Lab 7

Triage a failure

Complete the lab and keep its artifact for the final review.

1 · Prepare and do

Inject local payments errors; capture SLI impact and recent changes.

2 · Verify and record

Declare severity, separate facts from hypotheses, and state escalation deadline.

Evidence: ** use an injected payments failure to declare severity, produce a five-minute triage note, and name the next mitigation decision.
Chapter 08 · SRE Practices

Incident command and recovery

← Contents
Chapter 8 · Section 32

Command structure

Define the decision, the measure, and its limits.

Key point 1

Incident commander sets priorities and coordinates; operations lead investigates and mitigates; communications lead sends consistent updates.

Key point 2

Use a shared timeline and explicit task assignments; hand off command when the incident outlasts a shift.

Chapter 8 · Section 33

Mitigation before root cause

Define the decision, the measure, and its limits.

Key point 1

Compare rollback, failover, feature disablement, rate limiting, and dependency isolation by speed, reversibility, and risk.

Key point 2

Make one change at a time where possible; record time, actor, expected effect, and observed result.

Chapter 8 · Section 34

Communication

Define the decision, the measure, and its limits.

Key point 1

Write internal and external updates with impact, current state, action, next update time, and uncertainty. Avoid unverified root-cause claims.

Key point 2

Communicate to support and stakeholders at a predictable cadence; preserve a single source of truth.

Chapter 8 · Section 35

Recovery and closure

Define the decision, the measure, and its limits.

Key point 1

Verify user journey and SLI recovery, backlog draining, data integrity, and absence of a hidden second failure before closing.

Key point 2

Record incident duration, customer impact, mitigations, and follow-up owner.

Chapter 8 · Worked example

Mitigate, then explain

Use the example to make a concrete reliability decision.

IC: set priorities and decision cadence
Operations: rollback payments, check user journey
Communications: state impact and next update
Closure: SLI recovers, backlog drains, no data anomaly found

Interpretation

Write the time and observed result of each change. A plausible root cause is not required before mitigation.

Chapter 8 · Common mistakes

What goes wrong

Check these before accepting an SLO, runbook, experiment, or report.

Avoid

Making many changes at once; ending an incident because the Pod is Ready while users still fail.

Better check

Find the source data, state assumptions, test the decision, and record who owns the next action.

Chapter 8 · Self-check

Explain it without notes

Answer aloud, then reveal the suggested answer.

Who decides priorities during an incident?

The incident commander; operations and communications have distinct owners.

Lab 8

Run incident command

Complete the lab and keep its artifact for the final review.

1 · Prepare and do

Assign IC, operations, and communications; keep a timestamped change log.

2 · Verify and record

Send two status updates; mitigate, verify user recovery, and close explicitly.

Evidence: ** run a timed incident simulation with assigned roles, two status updates, a mitigation log, and explicit recovery checks.
Chapter 09 · SRE Practices

Postmortems and learning

← Contents
Chapter 9 · Section 36

Blameless analysis

Define the decision, the measure, and its limits.

Key point 1

Focus on system conditions, decisions made with information available at the time, and gaps in defenses. Avoid hindsight and person-focused blame.

Key point 2

Distinguish trigger, contributing factors, impact, and root-cause hypothesis; support claims with timeline evidence.

Chapter 9 · Section 37

Build the timeline

Define the decision, the measure, and its limits.

Key point 1

Reconcile alerts, deploys, logs, commands, and communications into one chronology; label inferred or approximate times.

Key point 2

Include detection delay, response delay, mitigation, recovery, and customer-visible effects.

Chapter 9 · Section 38

Corrective actions

Define the decision, the measure, and its limits.

Key point 1

Prefer specific changes that reduce recurrence or impact: alert quality, tests, rollout controls, capacity, runbooks, and architecture.

Key point 2

Assign owner, due date, verification method, and priority; track whether the action worked after completion.

Chapter 9 · Section 39

Share and verify learning

Define the decision, the measure, and its limits.

Key point 1

Review the postmortem with affected teams, publish appropriate lessons, and revisit repeat incidents and overdue actions.

Chapter 9 · Worked example

Turn a timeline into action

Use the example to make a concrete reliability decision.

09:00 alert fired; 09:05 declared; 09:18 rollback began
09:24 good-event ratio recovered; 09:40 backlog cleared
Action: add canary error-budget gate; owner + due date
Verification: replay the failure fixture in CI

Interpretation

Describe contributing conditions and evidence. Assign actions with a way to prove they reduce risk.

Chapter 9 · Common mistakes

What goes wrong

Check these before accepting an SLO, runbook, experiment, or report.

Avoid

Blaming a person; actions without owners or verification; silently editing the timeline after review.

Better check

Find the source data, state assumptions, test the decision, and record who owns the next action.

Chapter 9 · Self-check

Explain it without notes

Answer aloud, then reveal the suggested answer.

What makes an action useful?

A specific owner, due date, priority, and verification method.

Lab 9

Write a postmortem

Complete the lab and keep its artifact for the final review.

1 · Prepare and do

Build a timeline from the incident record and identify contributing conditions.

2 · Verify and record

Assign three actions with owner, due date, and verification method.

Evidence: ** write a blameless postmortem from the Lab 8 evidence with an incident timeline and three verifiable actions.
Chapter 10 · SRE Practices

Capacity and overload

← Contents
Chapter 10 · Section 40

Demand and saturation

Define the decision, the measure, and its limits.

Key point 1

Map request rate, concurrency, latency, queue depth, CPU, memory, connections, disk I/O, database limits, and provider quotas.

Key point 2

Explain why averages hide peak demand and why a single bottleneck can dominate end-to-end latency.

Chapter 10 · Section 41

Forecast and headroom

Define the decision, the measure, and its limits.

Key point 1

Estimate peak demand from recent history plus growth and event scenarios; state uncertainty rather than a false precise date.

Key point 2

Set headroom for scaling delay, zonal loss, maintenance, and recovery; review cost and resilience trade-offs.

Chapter 10 · Section 42

Load tests and limits

Define the decision, the measure, and its limits.

Key point 1

Define test goal, traffic shape, success criteria, isolation, stop condition, and safe maximum. Compare observed throughput, latency, errors, and saturation.

Key point 2

Distinguish steady load, spike, stress, and soak tests; use local or disposable environments for the deck.

Chapter 10 · Section 43

Overload controls

Define the decision, the measure, and its limits.

Key point 1

Apply timeouts, bounded queues, concurrency limits, backpressure, load shedding, retry budgets, and circuit breaking; explain when retries amplify overload.

Chapter 10 · Worked example

Plan for a peak and a failure

Use the example to make a concrete reliability decision.

Current peak: 120 rps; projected growth: 25% → 150 rps
One-zone-loss requirement: remaining capacity must handle 150 rps
If each remaining worker safely handles 60 rps, need at least 3
Then check database connections and quotas

Interpretation

State assumptions and uncertainty. CPU alone does not prove that the service can absorb the peak.

Chapter 10 · Common mistakes

What goes wrong

Check these before accepting an SLO, runbook, experiment, or report.

Avoid

Extrapolating from average CPU; running unbounded load; overlooking database or provider quotas.

Better check

Find the source data, state assumptions, test the decision, and record who owns the next action.

Chapter 10 · Self-check

Explain it without notes

Answer aloud, then reveal the suggested answer.

What is headroom for?

Peaks, scaling delay, maintenance, and loss of a failure domain.

Lab 10

Plan capacity

Complete the lab and keep its artifact for the final review.

1 · Prepare and do

Use sre-lab/fixtures/capacity.csv to estimate peak, growth, headroom, and bottleneck.

2 · Verify and record

Run a bounded hey POST test in the local cluster; record stop condition and results.

Evidence: ** calculate a capacity forecast from fixture data and run a bounded local load test; record bottleneck, headroom, and one safe mitigation.
Chapter 11 · SRE Practices

Chaos and resilience experiments

← Contents
Chapter 11 · Section 44

Experiment design

Define the decision, the measure, and its limits.

Key point 1

Define steady-state user outcome, hypothesis, fault, blast radius, observation window, stop condition, and rollback.

Key point 2

Choose a failure mode tied to the dependency map; begin in the local lab and expand scope only after evidence supports it.

Chapter 11 · Section 45

Failure injection

Define the decision, the measure, and its limits.

Key point 1

Test payments errors or latency and a dependency timeout as separate experiments. A controlled pod restart is optional if its hypothesis concerns replica recovery; observe SLI impact, alert timing, and recovery behavior.

Key point 2

A resource deletion or restart is useful only when the experiment defines a hypothesis, measurement, stop condition, and rollback.

Chapter 11 · Section 46

Safety and interpretation

Define the decision, the measure, and its limits.

Key point 1

Establish preflight checks, owner, abort trigger, expected side effects, and cleanup. Do not run during another incident.

Key point 2

Compare observed behavior with the hypothesis; record surprises and limits of the lab environment.

Chapter 11 · Section 47

Turn findings into safeguards

Define the decision, the measure, and its limits.

Key point 1

Propose one change to timeout, retry, fallback, alert, or runbook, then define a repeat experiment that would verify it.

Chapter 11 · Worked example

A bounded experiment

Use the example to make a concrete reliability decision.

Steady state: ≥99.9% good POSTs in baseline sample
Hypothesis: +500 ms payments latency stays below 2 s timeout
Fault: set EXTRA_LATENCY_MS=500 in local shop
Abort: errors persist or recovery misses the planned window
Rollback: set EXTRA_LATENCY_MS=0 and verify

Interpretation

The experiment tests a claim about user behavior. Restarting a pod has value only with the same hypothesis and measurement discipline.

Chapter 11 · Common mistakes

What goes wrong

Check these before accepting an SLO, runbook, experiment, or report.

Avoid

Calling a random deletion a chaos experiment; omitting abort and rollback criteria.

Better check

Find the source data, state assumptions, test the decision, and record who owns the next action.

Chapter 11 · Self-check

Explain it without notes

Answer aloud, then reveal the suggested answer.

What makes a fault injection an experiment?

A steady-state measure, hypothesis, bounded fault, observation, abort, and rollback.

Lab 11

Run a chaos experiment

Complete the lab and keep its artifact for the final review.

1 · Prepare and do

Write steady state, latency hypothesis, observation window, and abort trigger.

2 · Verify and record

Set payments EXTRA_LATENCY_MS=500 locally; measure, reset to 0, and verify recovery.

Evidence: ** run a local payments latency experiment with a written hypothesis, baseline, stop condition, measurements, rollback, and follow-up action.
Chapter 12 · SRE Practices

Disaster recovery design

← Contents
Chapter 12 · Section 48

Recovery objectives and scenarios

Define the decision, the measure, and its limits.

Key point 1

Define RTO as the maximum acceptable recovery duration and RPO as the maximum acceptable data-loss interval for a specific scenario and service.

Key point 2

Compare a pod failure, zone failure, region failure, credential compromise, and data corruption as design scenarios. The live shop app has no persistent store, so its in-memory order state cannot be backed up or restored by the labs.

Chapter 12 · Section 49

Backups and restore paths

Define the decision, the measure, and its limits.

Key point 1

Inventory the state and dependencies a production version would need: durable order store, secrets, DNS, images, infrastructure definitions, and access. Keep this distinct from the existing app.

Key point 2

Design backup frequency, retention, encryption, integrity checks, and an isolated restore test. Labs use dr-orders-events.csv and dr-backup.csv only; a completed backup file alone does not prove recovery.

Chapter 12 · Section 50

DR strategy and trade-offs

Define the decision, the measure, and its limits.

Key point 1

Compare backup-and-restore, pilot light, warm standby, and active-active approaches by expected RTO/RPO, operational complexity, and cost.

Key point 2

Explain failover and failback, split-brain risk, stale data, and what must be checked before returning traffic.

Chapter 12 · Section 51

Plan ownership

Define the decision, the measure, and its limits.

Key point 1

Document trigger, decision authority, contact tree, runbook, dependency order, communication, and measurable success criteria.

Chapter 12 · Worked example

Choose a DR strategy

Use the example to make a concrete reliability decision.

Scenario: hypothetical durable order-store loss
Objective: RTO ≤ 60 min; RPO ≤ 15 min
Backup every 10 min may support RPO, if backups succeed
Restore order: state → service → integrity check → traffic

Interpretation

The present app has only in-memory maps. This strategy is a design exercise backed by CSV fixtures, not a live restore of shop.

Chapter 12 · Common mistakes

What goes wrong

Check these before accepting an SLO, runbook, experiment, or report.

Avoid

Claiming an in-memory app has a backup; choosing an RTO/RPO without a recoverable state design.

Better check

Find the source data, state assumptions, test the decision, and record who owns the next action.

Chapter 12 · Self-check

Explain it without notes

Answer aloud, then reveal the suggested answer.

Can this app meet a data RPO?

No. Its order data is in memory; the CSV plan is hypothetical.

Lab 12

Design DR

Complete the lab and keep its artifact for the final review.

1 · Prepare and do

Use the CSV event and backup fixture for a hypothetical durable order store.

2 · Verify and record

Set RTO/RPO, backup schedule, restore order, integrity checks, and owner.

Evidence: ** design a hypothetical durable shop recovery plan using the documented DR fixtures, including RTO/RPO, backup schedule, restore order, and data-integrity checks. Do not imply the in-memory app can be restored.
Chapter 13 · SRE Practices

Disaster recovery drills

← Contents
Chapter 13 · Section 52

Tabletop first

Define the decision, the measure, and its limits.

Key point 1

Walk a scenario through declaration, decision points, access, communications, restoration, and return to service; record unclear ownership and missing prerequisites.

Key point 2

Include one inject such as a failed backup, unavailable operator, or stale DNS record.

Chapter 13 · Section 53

Local restore exercise

Define the decision, the measure, and its limits.

Key point 1

Restore dr-backup.csv into an isolated output CSV, run ID and item integrity checks against dr-orders-events.csv, and explain which later events are missing. Do not replay a live order journey against the unchanged app.

Key point 2

Measure fixture restore plus integrity-check elapsed time as exercise RTO; derive simulated RPO from the source and backup event timestamps. Compare these exercise values with the stated objectives without claiming production recovery performance.

Chapter 13 · Section 54

Failover and failback decisions

Define the decision, the measure, and its limits.

Key point 1

Tabletop only: a single obs kind cluster has no second destination for live failover. Decide when to move traffic, how to verify destination health and data consistency, and when to abort.

Key point 2

Walk the reverse path and reconciliation of hypothetical writes during the failover window; do not claim a live failback test.

Chapter 13 · Section 55

Drill report and improvements

Define the decision, the measure, and its limits.

Key point 1

Record timestamps, measured objectives, unmet dependencies, data validation results, decision delays, and prioritized actions.

Chapter 13 · Worked example

Measure the fixture drill

Use the example to make a concrete reliability decision.

Failure at 12:00; newest recoverable event at 11:52
Simulated RPO = 8 min
Declare at 12:00; fixture restored and checked at 12:28
Exercise RTO = 28 min
Failover and failback: tabletop only

Interpretation

A measured CSV exercise does not validate a production RTO or RPO. Record missing events and every integrity check.

Chapter 13 · Common mistakes

What goes wrong

Check these before accepting an SLO, runbook, experiment, or report.

Avoid

Calling a fixture restore production proof; describing tabletop failover as a live test.

Better check

Find the source data, state assumptions, test the decision, and record who owns the next action.

Chapter 13 · Self-check

Explain it without notes

Answer aloud, then reveal the suggested answer.

What did the fixture drill prove?

Only its simulated data gap and measured fixture restore time, not live failover.

Lab 13

Drill recovery

Complete the lab and keep its artifact for the final review.

1 · Prepare and do

Tabletop the failure and copy sre-lab/fixtures/dr-backup.csv to a separate output file.

2 · Verify and record

Compare missing IDs/items; calculate simulated RPO and exercise RTO; record gaps.

Evidence: ** run a tabletop plus fixture restore drill, report exercise RTO and simulated RPO, and update the plan from observed gaps.
Chapter 14 · SRE Practices

Capstone reliability review

← Contents
Chapter 14 · Section 56

Assemble the service review

Define the decision, the measure, and its limits.

Key point 1

Combine user journey, SLI/SLO document, error-budget ledger, alert test, on-call runbook, incident record, capacity worksheet, chaos result, and DR report.

Key point 2

Identify inconsistencies: a journey missing a measure, an alert without a response, a recovery objective without a tested restore path.

Chapter 14 · Section 57

Make a risk decision

Define the decision, the measure, and its limits.

Key point 1

Present current budget, recent incidents, projected demand, and open actions. Recommend ship, ship with guardrails, delay, or invest in reliability; state who decides.

Key point 2

Separate facts, assumptions, and recommendations.

Chapter 14 · Section 58

Review with stakeholders

Define the decision, the measure, and its limits.

Key point 1

Explain reliability status in customer terms, including impact and uncertainty; defend target and alert choices with evidence.

Key point 2

Turn feedback into a prioritized plan with owners and verification criteria.

Chapter 14 · Section 59

Final self-check

Define the decision, the measure, and its limits.

Key point 1

Given a new incident and traffic dataset, calculate SLI and burn, choose an alert response, assign incident roles, state recovery checks, and propose the next experiment.

Chapter 14 · Worked example

Make a decision from evidence

Use the example to make a concrete reliability decision.

SLO: 99.9%; remaining budget: 14%
Open P1 action: retry storm mitigation
Peak forecast: 25% growth; one-zone headroom unverified
Decision: delay risky rollout until capacity test and action close

Interpretation

State which facts are measured, which are assumptions, and who owns the release decision.

Chapter 14 · Common mistakes

What goes wrong

Check these before accepting an SLO, runbook, experiment, or report.

Avoid

Giving a ship recommendation without tying it to budget, incidents, capacity, and open actions.

Better check

Find the source data, state assumptions, test the decision, and record who owns the next action.

Chapter 14 · Self-check

Explain it without notes

Answer aloud, then reveal the suggested answer.

What must the final recommendation show?

The measured facts, assumptions, risk decision, owner, and next verification step.

Lab 14

Present a reliability review

Complete the lab and keep its artifact for the final review.

1 · Prepare and do

Collect SLO, budget, alert, incident, capacity, chaos, and DR artifacts.

2 · Verify and record

Give a 10-minute recommendation with facts, assumptions, owner, and next checks.

Evidence: ** deliver a concise reliability review packet and a 10-minute tabletop presentation; pass only if every major claim is backed by an artifact.
Back matter

Lab environment

The deck uses the Observability shop app and synthetic fixtures.

Local tools

Docker, kind, kubectl, helm, jq, hey, promtool, amtool. Cluster obs; namespaces shop and monitoring; services orders and payments.

Data boundary

POST /orders is the measured journey. Read order is unreliable across two independent in-memory replicas. DR data is synthetic CSV, not app state.

Back matter

Fixture contract

Use these columns to make every calculation reproducible.

Request and aggregate data

orders-requests.csv: timestamp_utc,request_id,route,status_code,duration_seconds,expected_order_id,actual_order_id,expected_item,actual_item. orders-daily.csv: date_utc,eligible_requests,bad_availability,slow_requests,checked_correctness,bad_correctness.

DR data

dr-orders-events.csv and dr-backup.csv: event_id,created_at_utc,order_id,item. dr-manifest.json: backup_completed_at_utc,failure_at_utc. Compare missing IDs and items after fixture restore.

Back matter

Download lab inputs

All files sit beside this HTML deck and open locally.

Back matter

Calculation sheet

Check the units and denominator before using a number.

availability = good / eligible
latency = events ≤ 0.3 s / eligible
correctness = matching ID and item / successful checked POSTs
allowed bad events = eligible × (1 − target)
burn rate = observed bad ratio / (1 − target)
time budget = window × (1 − target)
exercise RTO = restore verified − declared failure
simulated RPO = failure − latest recoverable event

Worked value

At 99.9% over 30 days, 3,000,000 eligible events permit 3,000 bad events. A time-based target permits 43.2 minutes; it is a different budget model.

Back matter

Fixture answer key

Check these results only after doing the calculations yourself.

Requests and budget

orders-requests.csv: 11 eligible, 9 availability-good (81.8%), 7 within 0.3 s (63.6%). Among 9 successful orders, 8 match ID and item (88.9%). orders-daily.csv: 3,000,000 eligible, 3,200 bad versus a 3,000-event budget.

Policy and recovery

Last 28 fixture days: 2,800,000 eligible and 3,060 bad versus a 2,800-event budget. INC-001 uses 800/2,800 = 28.6% of that budget. DR backup has 4 of 6 events; failure at 12:00 and latest recoverable event at 11:52 give simulated RPO 8 minutes.

Back matter

Command reference

Run only against the local obs cluster and reset fault knobs afterward.

kubectl config current-context
kubectl -n shop get deploy,svc
kubectl -n shop set env deploy/payments FAIL_RATE=0.3
kubectl -n shop set env deploy/payments FAIL_RATE=0 EXTRA_LATENCY_MS=0
# In another terminal: kubectl -n shop port-forward svc/orders 8080:8080
hey -z 30s -c 5 -m POST -d '{"item":"book","amount":42}' http://localhost:8080/orders
promtool check rules sre-lab/rules/slo-alerts.yaml
promtool test rules sre-lab/rules/slo-alerts.test.yaml

Safety

Confirm the current context is the local kind obs cluster. Use a short, bounded load test. The packaged alert rules consume synthetic precomputed ratio series; add real recording rules before any deployment.

Back matter

Sources

Start with primary references and verify version-sensitive commands before use.

1 / 1