Measure what users experience. Spend reliability deliberately. Respond, learn, and prove recovery.
15 chapters · 60 sections · 15 labs · Use ← → / Space · I for index · O for overview
Select a chapter · press I to return here
Define the decision, the measure, and its limits.
Reliability means users can complete important tasks within acceptable time and correctness bounds. A healthy pod or green dashboard is supporting evidence, not the outcome itself.
Map the practice loop: user journeys → SLIs/SLOs → budget and alerts → on-call/incident response → learning → capacity/resilience/DR.
Define the decision, the measure, and its limits.
Trace POST /orders through orders and payments; identify failure domains: application, dependency, Kubernetes node, and network. Discuss database and region loss as hypothetical production scenarios, not features of this app.
Name the customer impact of errors, slowness, duplicate charges, and in-memory order loss separately.
Define the decision, the measure, and its limits.
Choose local obs kind cluster or the specified CSV/JSON fixtures. Review safe fault knobs, synthetic traffic, and baseline dashboards.
Define the lab stop condition: stop load or faults if the local service does not recover within the planned window; record cleanup and verify normal operation.
Define the decision, the measure, and its limits.
Keep a reliability notebook with assumptions, SLO specification, alert evaluations, incident timeline, action owners, capacity worksheet, and drill results.
Each lab asks for an artifact and a short explanation of decisions, not just a screenshot.
Use the example to make a concrete reliability decision.
POST /orders → orders → payments → response
502 from payments: availability failure
201 after 2.1 s: availability pass, latency failure
201 with wrong item: correctness failure in the offline fixtureStart with what a user received. A pod's Ready status does not answer whether the journey succeeded.
Check these before accepting an SLO, runbook, experiment, or report.
Equating healthy Pods with a successful order; mixing live measurements with hypothetical fixture data.
Find the source data, state assumptions, test the decision, and record who owns the next action.
Answer aloud, then reveal the suggested answer.
A user can create the intended order correctly and within the agreed time.
Complete the lab and keep its artifact for the final review.
Draw POST /orders → orders → payments and mark app, node, and network failures.
Capture healthy POST status and latency; reset fault knobs; record cleanup.
shop user journey and dependency map, record a healthy baseline, and verify the fault controls are confined to the local lab.Define the decision, the measure, and its limits.
Service level indicators, objectives, agreements, and error budgets; distinguish a customer-facing commitment from an internal engineering target.
Reliability is one feature among others; higher targets have engineering and opportunity costs.
Define the decision, the measure, and its limits.
Use create order as the measured journey. Discuss read order only as a design exercise: it is unreliable with two independent in-memory replicas and has no read traffic from loadgen.
Define service boundaries and dependent teams; a journey may cross multiple services and still needs one user-facing measure.
Define the decision, the measure, and its limits.
Build a small failure-mode inventory: symptoms, affected users, detectability, recovery path, and likely blast radius.
Distinguish availability, latency, correctness, durability, and freshness; do not collapse all user harm into HTTP 5xx.
Define the decision, the measure, and its limits.
Define service owner, SLO owner, on-call owner, and follow-up action owner; defer incident command roles to Section 32.
Record who can declare an incident, pause a release, approve risk, and end an incident.
Use the example to make a concrete reliability decision.
Journey: create an order
Eligible: valid POST /orders attempts from users
Not a measured journey: GET /orders/{id}
Why: two replicas have independent maps and duplicate ID sequencesThe current read endpoint cannot support a reliable user-facing SLI. A real system would need shared durable state and read traffic.
Check these before accepting an SLO, runbook, experiment, or report.
Defining checkout or read-order reliability for an app that cannot measure them consistently.
Find the source data, state assumptions, test the decision, and record who owns the next action.
Answer aloud, then reveal the suggested answer.
Two independent in-memory replicas can return 404 or the wrong order; loadgen creates no read traffic.
Complete the lab and keep its artifact for the final review.
Name the create-order owner, dependency, impact, and failure modes.
Explain why read order is excluded; define the decision owner.
Define the decision, the measure, and its limits.
Specify eligible events, good events, measurement point, labels, and exclusions. Use good / eligible; show the equivalent bad-event ratio.
Prefer edge or user-visible measurement when possible; compare server-side and synthetic or client-side views.
Define the decision, the measure, and its limits.
Define how validation 4xx, server 5xx, timeouts, failed dependency calls, and duplicate outcomes count for POST /orders.
Handle retries and idempotency: a successful retry may still represent a bad user experience; define the event boundary before counting.
Define fixture-based correctness as successful checked POSTs with actual_order_id = expected_order_id and actual_item = expected_item / successful checked POSTs. Explain that this offline check is not an instrumented production SLI for the current app.
Define the decision, the measure, and its limits.
Define the latency SLI as eligible POST /orders requests completed within 0.3 seconds divided by eligible requests, matching the Observability lab's 0.3-second histogram bucket; explain why a percentile alone is awkward for budget accounting.
Show effects of sparse traffic, missing samples, scrape gaps, and short windows. Include minimum-volume or no-data handling in alert design.
Define the decision, the measure, and its limits.
Translate the definition into metrics and a query; aggregate before division where appropriate, preserve counter reset handling, and avoid averaging per-instance ratios.
Compare a sampled request log with the metric result; document known bias and blind spots.
Use the example to make a concrete reliability decision.
Availability = good POSTs / eligible POSTs
Latency = POSTs ≤ 0.3 s / eligible POSTs
Fixture correctness = matched ID + item / successful checked POSTs
Fixture: 9/11 available; 7/11 fast; 8/9 correctA denominator is part of the contract. Decide how invalid requests, timeouts, and retries count before writing a query.
Check these before accepting an SLO, runbook, experiment, or report.
Changing denominators between dashboards; treating a percentile as the proportion under 0.3 seconds.
Find the source data, state assumptions, test the decision, and record who owns the next action.
Answer aloud, then reveal the suggested answer.
Eligible user POST /orders events, under the written inclusion and exclusion policy.
Complete the lab and keep its artifact for the final review.
Use sre-lab/fixtures/orders-requests.csv to count availability, ≤0.3 s latency, and offline correctness.
Show numerator, denominator, exclusions, and one known measurement gap.
Define the decision, the measure, and its limits.
Choose an initial target from user need and historical performance, not from an arbitrary number of nines.
Compare rolling versus calendar windows and short versus long windows; explain why the window changes allowed bad events.
Define the decision, the measure, and its limits.
For the hypothetical 3,000,000 eligible order requests in 30 days at 99.9% availability, calculate 3,000 allowed bad requests; this averages about 1.16 requests/s versus the live loadgen's approximately 5 requests/s. Show the remaining budget after a sample incident.
For a time-based 99.9% SLO, calculate 43.2 minutes of allowed downtime in a 30-day window. Keep the request-based and time-based budgets distinct.
Define separate latency and availability targets only when each leads to a distinct operational decision.
Define the decision, the measure, and its limits.
Record journey, SLI formula, data source, target, window, exclusions, owner, review date, reporting location, and limitations.
Describe how to correct a broken measure or change a target without rewriting history.
Define the decision, the measure, and its limits.
Review trends, customer feedback, incident history, and support tickets; compare the target with actual user experience.
Set a cadence for revisiting the SLO and record the decision in an auditable change log.
Use the example to make a concrete reliability decision.
30 days × 24 h × 60 min = 43,200 min
Time-based 99.9% budget = 43.2 min
3,000,000 hypothetical requests × 0.001 = 3,000 bad requests
3,000,000 / 2,592,000 s ≈ 1.16 requests/sThese are two different budget models. The lab load generator runs about 5 requests/s; do not present its data as the 3-million-request example.
Check these before accepting an SLO, runbook, experiment, or report.
Using the request budget and time budget interchangeably; ignoring traffic volume.
Find the source data, state assumptions, test the decision, and record who owns the next action.
Answer aloud, then reveal the suggested answer.
3,000 of a hypothetical 3,000,000 requests at a 99.9% target.
Complete the lab and keep its artifact for the final review.
Calculate 3,000 allowed bad events from hypothetical 3 million requests and 43.2 minutes from 30 days.
Write target, window, owner, source, exclusions, rationale, and review date.
orders SLO proposal with a 30-day calculation, rationale, exclusions, and a review decision.Define the decision, the measure, and its limits.
Derive allowed bad events or bad time from the target; compute consumed and remaining budget over the same window.
Explain why a request-based budget and a time-based budget can answer different questions, and why traffic changes matter.
Define the decision, the measure, and its limits.
Present Google's example error-budget policy first: when the service exceeds its budget over the preceding four-week window, halt changes and releases except P0 issues or security fixes until it is back within SLO.
Then compare a locally approved softer policy such as staged releases or explicit risk review. State its decision owner, exceptions, and evidence; do not present it as Google's rule.
Define the decision, the measure, and its limits.
State who owns the policy, how deployment risk is assessed, which incidents count, how emergency fixes proceed, and when the policy is reviewed.
Include Google's example rule: a single incident consuming more than 20% of the four-week error budget requires a postmortem (with a P0 action item in that example policy). Explain the difference between that four-week policy and this deck's 30-day worked SLO.
Walk through product versus reliability trade-offs using a concrete release scenario.
Define the decision, the measure, and its limits.
Show current SLO, budget spent, burn trend, incident contributions, and data quality in a report a product owner can read.
Avoid misleading single-day percentages and double counting across separate SLIs.
Use the example to make a concrete reliability decision.
30-day fixture: 3,000,000 eligible; 3,200 bad → 200 over budget
Last 28 days: 2,800,000 eligible; 3,060 bad → 260 over budget
Single incident: 800 / 2,800 = 28.6% of four-week budget
Google example: halt releases except P0/security fixes; write postmortemThe 30-day SLO example and Google's four-week sample policy use different windows. Recompute from the matching window before deciding.
Check these before accepting an SLO, runbook, experiment, or report.
Calling a local exception Google's policy; pausing work without a documented decision owner.
Find the source data, state assumptions, test the decision, and record who owns the next action.
Answer aloud, then reveal the suggested answer.
After exceeding the four-week error budget, except P0 and security fixes.
Complete the lab and keep its artifact for the final review.
Read sre-lab/fixtures/orders-daily.csv and incident-ledger.csv; compute consumed and remaining budget.
Give a release decision under Google's example policy after aligning its four-week window, then state any local exception separately.
Define the decision, the measure, and its limits.
Define burn rate as observed bad-event rate divided by the allowed bad-event rate. At 1×, a 30-day budget lasts about 30 days if the rate persists.
Work through high, medium, and slow burns; show why a high error percentage without enough traffic can be noisy.
Define the decision, the measure, and its limits.
Use the SRE Workbook starting parameters for a 30-day 99.9% SLO: 2% budget in 1h/5m → 14.4× → page; 5% in 6h/30m → 6× → page; 10% in 3d/6h → 1× → ticket. Both long and short windows must exceed the same threshold. These are starting points, not universal thresholds.
Explain the short-window guideline of about 1/12 of the long window, plus evaluation interval, missing data, seasonality, and traffic volume.
Define the decision, the measure, and its limits.
Page on actionable user impact; route slower burns to a ticket or review. Include owner, SLO, current burn, estimated budget impact, dashboard, and runbook.
Suppress or inhibit lower-priority overlapping notifications so one incident does not produce all three alerts. Test recovery, flapping, duplicate alerts, silence expiration, and notification delivery to a test receiver.
Define the decision, the measure, and its limits.
Calculate burn from fixture windows, evaluate whether a page or ticket should fire, and explain a false-positive scenario.
Use the example to make a concrete reliability decision.
Page: 14.4× over both 1 h and 5 min → 2% budget
Page: 6× over both 6 h and 30 min → 5% budget
Ticket: 1× over both 3 d and 6 h → 10% budget
For 99.9% SLO, 14.4× means bad-event ratio > 0.0144Short windows confirm the burn is active. Inhibit overlapping lower-priority notifications from the same incident.
Check these before accepting an SLO, runbook, experiment, or report.
Paging on a single short spike; sending three notifications for one incident; ignoring no-data behavior.
Find the source data, state assumptions, test the decision, and record who owns the next action.
Answer aloud, then reveal the suggested answer.
It confirms that the budget is still burning and lets the alert clear sooner.
Complete the lab and keep its artifact for the final review.
Inspect sre-lab/rules/slo-alerts.yaml: 14.4× 1h/5m, 6× 6h/30m, and 1× 3d/6h.
Run promtool test rules on the packaged test file; use alertmanager.test.yaml with a local receiver to check routing and inhibition.
promtool test rules with synthetic series, then use amtool and a test receiver to verify suppression and recovery behavior.Define the decision, the measure, and its limits.
Define primary/secondary coverage, handoff, escalation, ownership hours, and a backup when the primary is unavailable.
Track page load, sleep disruption, repeat pages, and time spent on toil; use the data to improve the system.
Define the decision, the measure, and its limits.
A runbook starts with user symptoms, quick checks, safe mitigations, rollback, escalation, and verification.
Rehearse access, permissions, dashboards, and emergency contacts before the shift; avoid embedding secrets in runbooks.
Define the decision, the measure, and its limits.
Identify repetitive, manual, reactive work that scales with service growth; prioritize automation by impact and safety.
Automate diagnosis or a reversible recovery step only after documenting its failure modes and fallback.
Define the decision, the measure, and its limits.
Review current risks, open incidents, recent deploys, budget state, known noisy alerts, and scheduled changes at handoff.
Use the example to make a concrete reliability decision.
Primary: Alex; backup: Sam
Open risk: payments errors after recent deploy
Current burn: 0.7×; no active page
First action: confirm POST /orders error ratio; rollback only with evidenceA good handoff points to the dashboard, recent changes, escalation path, and a safe first action.
Check these before accepting an SLO, runbook, experiment, or report.
An on-call shift with no backup, no access rehearsal, or a runbook that contains only commands.
Find the source data, state assumptions, test the decision, and record who owns the next action.
Answer aloud, then reveal the suggested answer.
Risks, changes, current budget, dashboard, runbook, primary and backup contacts.
Complete the lab and keep its artifact for the final review.
Write a handoff with primary, backup, latest deploy, budget state, and risk.
Have a second person follow the payments runbook without verbal help.
payments failure, then perform a dry run from the runbook alone.Define the decision, the measure, and its limits.
Start from user symptoms and SLO signals; correlate changes in traffic, errors, latency, dependencies, and recent deployments.
Separate evidence from hypotheses; mark uncertain timestamps and missing telemetry.
Define the decision, the measure, and its limits.
Define example severity criteria using user impact, scope, duration, and data risk. Declare early when coordination is needed, then revise severity as evidence changes.
Record incident start time, initial scope, known unknowns, and a shared incident channel or log.
Define the decision, the measure, and its limits.
Check whether the issue is ongoing, widespread, and linked to a recent change; decide the next safe information-gathering step.
Avoid speculative fixes that expand blast radius; use a rollback or traffic shift when it offers a faster, safer mitigation.
Define the decision, the measure, and its limits.
Escalate to the on-call backup or service owner when the first responder cannot contain impact; state the trigger and deadline for escalation. Section 32 defines command roles after declaration.
Use the example to make a concrete reliability decision.
09:00 page: create-order 14.4× burn
09:02 verify user impact and affected region
09:03 correlate with payments rollout
09:05 declare incident and escalate to backup
Unknown: whether rollback will clear errorsKeep observed facts separate from the rollback hypothesis. Early declaration creates a place to coordinate.
Check these before accepting an SLO, runbook, experiment, or report.
Waiting for root cause before declaring; writing a hypothesis as a confirmed fact.
Find the source data, state assumptions, test the decision, and record who owns the next action.
Answer aloud, then reveal the suggested answer.
When user impact needs coordination; severity can be revised as facts arrive.
Complete the lab and keep its artifact for the final review.
Inject local payments errors; capture SLI impact and recent changes.
Declare severity, separate facts from hypotheses, and state escalation deadline.
payments failure to declare severity, produce a five-minute triage note, and name the next mitigation decision.Define the decision, the measure, and its limits.
Incident commander sets priorities and coordinates; operations lead investigates and mitigates; communications lead sends consistent updates.
Use a shared timeline and explicit task assignments; hand off command when the incident outlasts a shift.
Define the decision, the measure, and its limits.
Compare rollback, failover, feature disablement, rate limiting, and dependency isolation by speed, reversibility, and risk.
Make one change at a time where possible; record time, actor, expected effect, and observed result.
Define the decision, the measure, and its limits.
Write internal and external updates with impact, current state, action, next update time, and uncertainty. Avoid unverified root-cause claims.
Communicate to support and stakeholders at a predictable cadence; preserve a single source of truth.
Define the decision, the measure, and its limits.
Verify user journey and SLI recovery, backlog draining, data integrity, and absence of a hidden second failure before closing.
Record incident duration, customer impact, mitigations, and follow-up owner.
Use the example to make a concrete reliability decision.
IC: set priorities and decision cadence
Operations: rollback payments, check user journey
Communications: state impact and next update
Closure: SLI recovers, backlog drains, no data anomaly foundWrite the time and observed result of each change. A plausible root cause is not required before mitigation.
Check these before accepting an SLO, runbook, experiment, or report.
Making many changes at once; ending an incident because the Pod is Ready while users still fail.
Find the source data, state assumptions, test the decision, and record who owns the next action.
Answer aloud, then reveal the suggested answer.
The incident commander; operations and communications have distinct owners.
Complete the lab and keep its artifact for the final review.
Assign IC, operations, and communications; keep a timestamped change log.
Send two status updates; mitigate, verify user recovery, and close explicitly.
Define the decision, the measure, and its limits.
Focus on system conditions, decisions made with information available at the time, and gaps in defenses. Avoid hindsight and person-focused blame.
Distinguish trigger, contributing factors, impact, and root-cause hypothesis; support claims with timeline evidence.
Define the decision, the measure, and its limits.
Reconcile alerts, deploys, logs, commands, and communications into one chronology; label inferred or approximate times.
Include detection delay, response delay, mitigation, recovery, and customer-visible effects.
Define the decision, the measure, and its limits.
Prefer specific changes that reduce recurrence or impact: alert quality, tests, rollout controls, capacity, runbooks, and architecture.
Assign owner, due date, verification method, and priority; track whether the action worked after completion.
Define the decision, the measure, and its limits.
Review the postmortem with affected teams, publish appropriate lessons, and revisit repeat incidents and overdue actions.
Use the example to make a concrete reliability decision.
09:00 alert fired; 09:05 declared; 09:18 rollback began
09:24 good-event ratio recovered; 09:40 backlog cleared
Action: add canary error-budget gate; owner + due date
Verification: replay the failure fixture in CIDescribe contributing conditions and evidence. Assign actions with a way to prove they reduce risk.
Check these before accepting an SLO, runbook, experiment, or report.
Blaming a person; actions without owners or verification; silently editing the timeline after review.
Find the source data, state assumptions, test the decision, and record who owns the next action.
Answer aloud, then reveal the suggested answer.
A specific owner, due date, priority, and verification method.
Complete the lab and keep its artifact for the final review.
Build a timeline from the incident record and identify contributing conditions.
Assign three actions with owner, due date, and verification method.
Define the decision, the measure, and its limits.
Map request rate, concurrency, latency, queue depth, CPU, memory, connections, disk I/O, database limits, and provider quotas.
Explain why averages hide peak demand and why a single bottleneck can dominate end-to-end latency.
Define the decision, the measure, and its limits.
Estimate peak demand from recent history plus growth and event scenarios; state uncertainty rather than a false precise date.
Set headroom for scaling delay, zonal loss, maintenance, and recovery; review cost and resilience trade-offs.
Define the decision, the measure, and its limits.
Define test goal, traffic shape, success criteria, isolation, stop condition, and safe maximum. Compare observed throughput, latency, errors, and saturation.
Distinguish steady load, spike, stress, and soak tests; use local or disposable environments for the deck.
Define the decision, the measure, and its limits.
Apply timeouts, bounded queues, concurrency limits, backpressure, load shedding, retry budgets, and circuit breaking; explain when retries amplify overload.
Use the example to make a concrete reliability decision.
Current peak: 120 rps; projected growth: 25% → 150 rps
One-zone-loss requirement: remaining capacity must handle 150 rps
If each remaining worker safely handles 60 rps, need at least 3
Then check database connections and quotasState assumptions and uncertainty. CPU alone does not prove that the service can absorb the peak.
Check these before accepting an SLO, runbook, experiment, or report.
Extrapolating from average CPU; running unbounded load; overlooking database or provider quotas.
Find the source data, state assumptions, test the decision, and record who owns the next action.
Answer aloud, then reveal the suggested answer.
Peaks, scaling delay, maintenance, and loss of a failure domain.
Complete the lab and keep its artifact for the final review.
Use sre-lab/fixtures/capacity.csv to estimate peak, growth, headroom, and bottleneck.
Run a bounded hey POST test in the local cluster; record stop condition and results.
Define the decision, the measure, and its limits.
Define steady-state user outcome, hypothesis, fault, blast radius, observation window, stop condition, and rollback.
Choose a failure mode tied to the dependency map; begin in the local lab and expand scope only after evidence supports it.
Define the decision, the measure, and its limits.
Test payments errors or latency and a dependency timeout as separate experiments. A controlled pod restart is optional if its hypothesis concerns replica recovery; observe SLI impact, alert timing, and recovery behavior.
A resource deletion or restart is useful only when the experiment defines a hypothesis, measurement, stop condition, and rollback.
Define the decision, the measure, and its limits.
Establish preflight checks, owner, abort trigger, expected side effects, and cleanup. Do not run during another incident.
Compare observed behavior with the hypothesis; record surprises and limits of the lab environment.
Define the decision, the measure, and its limits.
Propose one change to timeout, retry, fallback, alert, or runbook, then define a repeat experiment that would verify it.
Use the example to make a concrete reliability decision.
Steady state: ≥99.9% good POSTs in baseline sample
Hypothesis: +500 ms payments latency stays below 2 s timeout
Fault: set EXTRA_LATENCY_MS=500 in local shop
Abort: errors persist or recovery misses the planned window
Rollback: set EXTRA_LATENCY_MS=0 and verifyThe experiment tests a claim about user behavior. Restarting a pod has value only with the same hypothesis and measurement discipline.
Check these before accepting an SLO, runbook, experiment, or report.
Calling a random deletion a chaos experiment; omitting abort and rollback criteria.
Find the source data, state assumptions, test the decision, and record who owns the next action.
Answer aloud, then reveal the suggested answer.
A steady-state measure, hypothesis, bounded fault, observation, abort, and rollback.
Complete the lab and keep its artifact for the final review.
Write steady state, latency hypothesis, observation window, and abort trigger.
Set payments EXTRA_LATENCY_MS=500 locally; measure, reset to 0, and verify recovery.
payments latency experiment with a written hypothesis, baseline, stop condition, measurements, rollback, and follow-up action.Define the decision, the measure, and its limits.
Define RTO as the maximum acceptable recovery duration and RPO as the maximum acceptable data-loss interval for a specific scenario and service.
Compare a pod failure, zone failure, region failure, credential compromise, and data corruption as design scenarios. The live shop app has no persistent store, so its in-memory order state cannot be backed up or restored by the labs.
Define the decision, the measure, and its limits.
Inventory the state and dependencies a production version would need: durable order store, secrets, DNS, images, infrastructure definitions, and access. Keep this distinct from the existing app.
Design backup frequency, retention, encryption, integrity checks, and an isolated restore test. Labs use dr-orders-events.csv and dr-backup.csv only; a completed backup file alone does not prove recovery.
Define the decision, the measure, and its limits.
Compare backup-and-restore, pilot light, warm standby, and active-active approaches by expected RTO/RPO, operational complexity, and cost.
Explain failover and failback, split-brain risk, stale data, and what must be checked before returning traffic.
Define the decision, the measure, and its limits.
Document trigger, decision authority, contact tree, runbook, dependency order, communication, and measurable success criteria.
Use the example to make a concrete reliability decision.
Scenario: hypothetical durable order-store loss
Objective: RTO ≤ 60 min; RPO ≤ 15 min
Backup every 10 min may support RPO, if backups succeed
Restore order: state → service → integrity check → trafficThe present app has only in-memory maps. This strategy is a design exercise backed by CSV fixtures, not a live restore of shop.
Check these before accepting an SLO, runbook, experiment, or report.
Claiming an in-memory app has a backup; choosing an RTO/RPO without a recoverable state design.
Find the source data, state assumptions, test the decision, and record who owns the next action.
Answer aloud, then reveal the suggested answer.
No. Its order data is in memory; the CSV plan is hypothetical.
Complete the lab and keep its artifact for the final review.
Use the CSV event and backup fixture for a hypothetical durable order store.
Set RTO/RPO, backup schedule, restore order, integrity checks, and owner.
shop recovery plan using the documented DR fixtures, including RTO/RPO, backup schedule, restore order, and data-integrity checks. Do not imply the in-memory app can be restored.Define the decision, the measure, and its limits.
Walk a scenario through declaration, decision points, access, communications, restoration, and return to service; record unclear ownership and missing prerequisites.
Include one inject such as a failed backup, unavailable operator, or stale DNS record.
Define the decision, the measure, and its limits.
Restore dr-backup.csv into an isolated output CSV, run ID and item integrity checks against dr-orders-events.csv, and explain which later events are missing. Do not replay a live order journey against the unchanged app.
Measure fixture restore plus integrity-check elapsed time as exercise RTO; derive simulated RPO from the source and backup event timestamps. Compare these exercise values with the stated objectives without claiming production recovery performance.
Define the decision, the measure, and its limits.
Tabletop only: a single obs kind cluster has no second destination for live failover. Decide when to move traffic, how to verify destination health and data consistency, and when to abort.
Walk the reverse path and reconciliation of hypothetical writes during the failover window; do not claim a live failback test.
Define the decision, the measure, and its limits.
Record timestamps, measured objectives, unmet dependencies, data validation results, decision delays, and prioritized actions.
Use the example to make a concrete reliability decision.
Failure at 12:00; newest recoverable event at 11:52
Simulated RPO = 8 min
Declare at 12:00; fixture restored and checked at 12:28
Exercise RTO = 28 min
Failover and failback: tabletop onlyA measured CSV exercise does not validate a production RTO or RPO. Record missing events and every integrity check.
Check these before accepting an SLO, runbook, experiment, or report.
Calling a fixture restore production proof; describing tabletop failover as a live test.
Find the source data, state assumptions, test the decision, and record who owns the next action.
Answer aloud, then reveal the suggested answer.
Only its simulated data gap and measured fixture restore time, not live failover.
Complete the lab and keep its artifact for the final review.
Tabletop the failure and copy sre-lab/fixtures/dr-backup.csv to a separate output file.
Compare missing IDs/items; calculate simulated RPO and exercise RTO; record gaps.
Define the decision, the measure, and its limits.
Combine user journey, SLI/SLO document, error-budget ledger, alert test, on-call runbook, incident record, capacity worksheet, chaos result, and DR report.
Identify inconsistencies: a journey missing a measure, an alert without a response, a recovery objective without a tested restore path.
Define the decision, the measure, and its limits.
Present current budget, recent incidents, projected demand, and open actions. Recommend ship, ship with guardrails, delay, or invest in reliability; state who decides.
Separate facts, assumptions, and recommendations.
Define the decision, the measure, and its limits.
Explain reliability status in customer terms, including impact and uncertainty; defend target and alert choices with evidence.
Turn feedback into a prioritized plan with owners and verification criteria.
Define the decision, the measure, and its limits.
Given a new incident and traffic dataset, calculate SLI and burn, choose an alert response, assign incident roles, state recovery checks, and propose the next experiment.
Use the example to make a concrete reliability decision.
SLO: 99.9%; remaining budget: 14%
Open P1 action: retry storm mitigation
Peak forecast: 25% growth; one-zone headroom unverified
Decision: delay risky rollout until capacity test and action closeState which facts are measured, which are assumptions, and who owns the release decision.
Check these before accepting an SLO, runbook, experiment, or report.
Giving a ship recommendation without tying it to budget, incidents, capacity, and open actions.
Find the source data, state assumptions, test the decision, and record who owns the next action.
Answer aloud, then reveal the suggested answer.
The measured facts, assumptions, risk decision, owner, and next verification step.
Complete the lab and keep its artifact for the final review.
Collect SLO, budget, alert, incident, capacity, chaos, and DR artifacts.
Give a 10-minute recommendation with facts, assumptions, owner, and next checks.
The deck uses the Observability shop app and synthetic fixtures.
Docker, kind, kubectl, helm, jq, hey, promtool, amtool. Cluster obs; namespaces shop and monitoring; services orders and payments.
POST /orders is the measured journey. Read order is unreliable across two independent in-memory replicas. DR data is synthetic CSV, not app state.
Use these columns to make every calculation reproducible.
orders-requests.csv: timestamp_utc,request_id,route,status_code,duration_seconds,expected_order_id,actual_order_id,expected_item,actual_item. orders-daily.csv: date_utc,eligible_requests,bad_availability,slow_requests,checked_correctness,bad_correctness.
dr-orders-events.csv and dr-backup.csv: event_id,created_at_utc,order_id,item. dr-manifest.json: backup_completed_at_utc,failure_at_utc. Compare missing IDs and items after fixture restore.
All files sit beside this HTML deck and open locally.
Check the units and denominator before using a number.
availability = good / eligible
latency = events ≤ 0.3 s / eligible
correctness = matching ID and item / successful checked POSTs
allowed bad events = eligible × (1 − target)
burn rate = observed bad ratio / (1 − target)
time budget = window × (1 − target)
exercise RTO = restore verified − declared failure
simulated RPO = failure − latest recoverable eventAt 99.9% over 30 days, 3,000,000 eligible events permit 3,000 bad events. A time-based target permits 43.2 minutes; it is a different budget model.
Check these results only after doing the calculations yourself.
orders-requests.csv: 11 eligible, 9 availability-good (81.8%), 7 within 0.3 s (63.6%). Among 9 successful orders, 8 match ID and item (88.9%). orders-daily.csv: 3,000,000 eligible, 3,200 bad versus a 3,000-event budget.
Last 28 fixture days: 2,800,000 eligible and 3,060 bad versus a 2,800-event budget. INC-001 uses 800/2,800 = 28.6% of that budget. DR backup has 4 of 6 events; failure at 12:00 and latest recoverable event at 11:52 give simulated RPO 8 minutes.
Run only against the local obs cluster and reset fault knobs afterward.
kubectl config current-context
kubectl -n shop get deploy,svc
kubectl -n shop set env deploy/payments FAIL_RATE=0.3
kubectl -n shop set env deploy/payments FAIL_RATE=0 EXTRA_LATENCY_MS=0
# In another terminal: kubectl -n shop port-forward svc/orders 8080:8080
hey -z 30s -c 5 -m POST -d '{"item":"book","amount":42}' http://localhost:8080/orders
promtool check rules sre-lab/rules/slo-alerts.yaml
promtool test rules sre-lab/rules/slo-alerts.test.yamlConfirm the current context is the local kind obs cluster. Use a short, bounded load test. The packaged alert rules consume synthetic precomputed ratio series; add real recording rules before any deployment.
Start with primary references and verify version-sensitive commands before use.