Self-study deck · Prometheus 3 · Loki · OpenTelemetry · Tempo · Grafana 13 · CloudWatch

Observability

Metrics, logs and traces you can actually debug with: Prometheus and PromQL, alerting with Alertmanager, structured logs in Loki, OpenTelemetry traces through the Collector into Tempo, RED and USE dashboards in Grafana, and the same ideas on Amazon CloudWatch — with a lab for every chapter.

Use ← → / Space to navigate · I contents · O overview · T theme · F fullscreen

Observability

Contents

Click any line to jump there · press I to return to this page

Chapter 0

Orientation

What observability is for, the stack this deck builds, the versions it covers, and how to study it.

Mental model

Observability answers three questions during an incident

Telemetry is only worth its cost if it shortens the path from "something is wrong" to "this change fixes it".

1 · Is it broken?

  • Detect from the user's point of view: error rate, latency, availability.
  • Signals: metrics and alerts on symptoms (ch 2–4).
  • Cheap to store, cheap to query, aggregated by design.
  • Output: a page or a ticket, with a runbook link.

2 · Where is it broken?

  • Narrow down: which service, route, dependency, node, zone, version.
  • Signals: RED/USE dashboards (ch 10) and traces (ch 8–9).
  • A trace shows which hop of a request spent the time or failed.
  • Output: one component to look at.

3 · Why is it broken?

  • Explain: the exact error, input, config, deploy or resource limit.
  • Signals: structured logs (ch 6–7), events, profiles.
  • High detail, high volume: queried by trace_id, not read end to end.
  • Output: a hypothesis you can test and a fix.
Mental model

Monitoring vs observability

Monitoring checks the failure modes you predicted. Observability lets you ask new questions without shipping new code.

Monitoring

  • Known unknowns: "alert when disk > 90 %".
  • Fixed dashboards and thresholds.
  • Answers "is it up?" well, "why is checkout slow for one tenant?" poorly.
  • Still essential: alerts start every investigation.

Observability

  • Unknown unknowns: slice by any dimension after the fact.
  • Needs rich context: high-cardinality fields in logs and traces, linked by IDs.
  • A property of the system you instrument, not a tool you buy.
  • Measured by how fast an engineer new to the service can explain an incident.

This deck's stance

  • Metrics for detection and trends (cheap, low cardinality).
  • Traces for location (sampled, request-scoped).
  • Logs for detail (structured, linked by trace_id).
  • One pipeline (OpenTelemetry) and one UI (Grafana), plus CloudWatch on AWS.
Deck map

The telemetry pipeline, chapter by chapter

Read left to right: instrument, collect, store, query, act. The same shape applies to the open-source stack and to CloudWatch.

Instrumentshop app (Go)client_golang (ch 2)slog JSON logsch 6OTel SDK spansch 8CollectPrometheus scrapeSD + relabel (ch 2, 5)Grafana AlloyDaemonSet (ch 7)OTel Collectorotel-gateway (ch 9)StorePrometheus TSDBremote write (ch 5)Lokilabels + chunks (ch 7)Tempotraces (ch 9)Query and actPromQLch 3LogQLch 7TraceQLch 9GrafanaRED / USE (ch 10)Alertmanagerroutes (ch 4)On AWS the same jobs are done by EMF, the CloudWatch agent / ADOT, CloudWatch Metrics, Logs, X-Ray and alarms (ch 11–12).

Chapter 13 closes the loop: a debugging workflow from alert to root cause, alert quality, cost and cardinality governance, and monitoring the monitoring.

Versions covered

What this deck assumes (checked Nov 2026)

These projects release monthly. When a slide and the official docs disagree, the docs win.

Prometheus and Grafana

  • Prometheus 3.15 (3.5 is the LTS line). UTF-8 metric and label names, OTLP receiver, Remote Write 2.0, new UI.
  • Native histograms stable since 3.8, but opt-in (scrape_native_histograms); classic histograms are not converted.
  • Grafana 13 (Apr 2026): dynamic dashboards on by default, dashboard schema v2, Git Sync GA.

Logs and traces

  • Loki 3.7. Promtail is end of life since 2 Mar 2026 → Grafana Alloy (1.21).
  • Loki and Tempo Helm charts moved to grafana-community/helm-charts (Loki: 16 Mar 2026).
  • Tempo 3: Kafka-based write path in microservices mode, no ingesters; monolithic mode needs no Kafka; TraceQL metrics GA.
  • A re-architected Loki (Kafka ingestion, new query engine) was announced in Apr 2026; treat it as rolling out.

OpenTelemetry and AWS

  • OTel traces, metrics and logs are stable; Profiles is public Alpha (Mar 2026).
  • Span events are being deprecated in favor of events emitted through the Logs API.
  • CloudWatch native OTel metrics with PromQL (preview Apr 2026, GA mid-2026).
  • X-Ray SDKs and daemon: maintenance since 25 Feb 2026, end of support 25 Feb 2027 → use OpenTelemetry / ADOT.

Chart and image versions in the labs are not pinned on the slides: run helm search repo <chart> and pin what you install in your own values files.

The demo app

shop: two Go services that every lab observes

Small enough to read in one sitting, realistic enough to fail in interesting ways: a synchronous dependency with tunable errors and latency.

kind cluster obs · namespace shoploadgencurl loop ~5 rpsordersPOST /orderspaymentsPOST /chargeFAIL_RATE=0..1EXTRA_LATENCY_MSGET /orders/{id}/metrics /healthzTelemetry added lab by lab: metrics (2), logs (6), traces (8).

What the labs add

  • Lab 0: services, images, kind, kube-prometheus-stack.
  • Lab 2: http_requests_total, http_request_duration_seconds, ServiceMonitors.
  • Lab 4: recording and alerting rules, Alertmanager routes.
  • Lab 6–7: JSON logs with trace_id, Loki and Alloy.
  • Lab 8–9: OpenTelemetry SDK, Collector, Tempo, exemplars.
  • Lab 10: RED and USE dashboards as code.
  • Lab 11–12: the same ideas on CloudWatch.
  • Lab 13: a game day from alert to root cause.
Lab environment

What you need before Lab 0

Labs 0–10 and 13 run on a laptop with kind and cost nothing. Labs 11–12 use a small AWS sandbox with a budget alarm.

Machine

  • Docker Desktop, OrbStack or colima with 6 GB RAM and 4 CPUs for the full stack (Prometheus, Grafana, Loki, Tempo, Collector, app).
  • macOS or Linux shell; ~10 GB free disk.
  • AWS sandbox account, AWS CLI v2, export AWS_REGION=ap-northeast-1 (Labs 11–12).

Tools

  • kind, kubectl, helm, Go 1.25+ (the images build with golang:1.27).
  • promtool and amtool (ship with the Prometheus and Alertmanager releases).
  • logcli (Loki release), jq, hey (local load tests).
  • Optional: k9s to watch the cluster.
obs-lab/
├── go.mod      # module obs-lab
├── Dockerfile  # ARG APP
├── apps/
│   ├── orders/main.go
│   └── payments/main.go
├── deploy/
│   ├── kind.yaml
│   ├── kps-values.yaml
│   └── shop.yaml
├── rules/      # Lab 4
└── dashboards/ # Lab 10

namespaces:
  monitoring  kps (Lab 0)
  shop        orders, payments, loadgen
  logging     Loki, Alloy   (Lab 7)
  otel        otel-gateway  (Lab 8)
  tracing     Tempo         (Lab 9)
How to use this deck

A study loop that sticks

Read a part, run its lab, then answer the self-check out loud without notes.

Per part

  • Read the concept slides; redraw the diagrams from memory.
  • Type the queries yourself in Prometheus, Grafana Explore or Logs Insights; predict the result first.
  • Study the "Common mistakes" slide: each one is a real outage or a surprise bill.

Per chapter

  • Do the matching lab in the Hands-on labs part.
  • Do every "Break it" step and explain the symptom before you fix it.
  • Answer the self-check questions; revisit what you hesitate on.

Navigation

  • ← → or Space to move, I for Contents, O for overview.
  • T toggles the theme, F fullscreen.
  • Every Contents line is a link.
  • "(ch N)" points inside this deck; SLOs and burn-rate alerts live in the SRE deck.

Order matters: Prometheus (ch 2) before PromQL (ch 3); OpenTelemetry basics (ch 8) before the Collector and Tempo (ch 9); every signal before dashboards (ch 10) and the debugging workflow (ch 13).

Chapter 1

Observability foundations

What each signal answers, what it costs, how cardinality decides the bill, and the RED / USE methods that tell you what to measure.

Chapter 1 · Part 1.1

Signals and the mental model

Metrics, logs, traces, profiles and events: what each one is good at, and how they fit into one pipeline.

Mental model

Monitoring vs observability

Monitoring checks the failures you predicted. Observability lets you ask new questions about failures you did not predict — without shipping new code.

Monitoring

  • Known unknowns: "Is CPU above 90%?", "Is the error rate above 1%?"
  • Predefined checks, dashboards and alerts.
  • Answers whether something is wrong.
  • Still essential: alerts are monitoring.

Observability

  • Unknown unknowns: "Why are only EU card payments slow since 14:05?"
  • High-context telemetry you can slice by any dimension after the fact.
  • Answers where and why.
  • A property of the system + its telemetry, not a product you buy.

The three questions of an incident

  • Is it broken? — alerts on symptoms (metrics).
  • Where is it broken? — dashboards and traces narrow to a service, route, dependency.
  • Why is it broken? — logs, trace attributes, profiles and change events show the cause.

Rule of thumb: metrics tell you something is wrong, traces tell you where, logs (and profiles) tell you why. Good tooling lets you jump between them with one click (ch 9).

Signals

The five signals at a glance

Each signal trades detail for cost differently. You need all of them, in different volumes.

SignalWhat it isBest at answeringCost shapeIn this deck
MetricsNumeric samples aggregated over time, identified by name + labelsIs it broken? Trends, alerting, capacityPer series (label combinations); cheap per eventPrometheus (ch 2–5), CloudWatch (ch 11)
LogsTimestamped records of discrete events, ideally structuredWhy? Details of one event, errors, auditPer byte ingested and storedslog, Loki (ch 6–7), CloudWatch Logs (ch 12)
TracesTree of spans: one request across servicesWhere? Which hop is slow or failingPer span; usually sampledOpenTelemetry, Tempo (ch 8–9), X-Ray (ch 12)
ProfilesStack samples of CPU, memory, locks over timeWhich function burns CPU or memoryPer sample; continuous profiling is low overheadMentioned only; OTel Profiles is Alpha (Mar 2026)
EventsChanges: deploys, config flips, scaling, incidentsWhat changed just before it broke?Low volume, high valueGrafana annotations (ch 10), ch 13

"Three pillars" (metrics, logs, traces) is a useful starting point, not a law: what matters is that the signals share identity (service, trace ID) so you can correlate them.

Pipeline

Every signal flows through the same six stages

Tools differ per stage; the stages do not. Most design decisions (sampling, dropping, labels) happen before storage.

InstrumentSDKs, exporters, agentsCollectscrape / receiveProcessfilter, enrich, sampleStoreTSDB, object storeQuery + visualizePromQL, LogQL, TraceQLActalert, page, auto-healMetricsclient_golang, node_exporter → Prometheus scrape → relabel → TSDB → PromQL → AlertmanagerLogsslog JSON to stdout → Alloy DaemonSet → parse, labels → Loki chunks in object storage → LogQLTracesOTel SDK → OTLP → Collector (batch, tail sampling) → Tempo → TraceQLAWSEMF / agent / ADOT → CloudWatch Metrics + Logs + X-Ray → Logs Insights, PromQL → alarms → SNSCost leverThe earlier you drop, aggregate or sample, the cheaper everything after it — but you can never query what you dropped.
Correlation

One request, seen three ways

A POST /orders that failed because payments timed out. The shared fields are what make the signals one story.

# METRIC — one sample in one series (aggregated, no request detail)
http_requests_total{job="orders",method="POST",route="/orders",code="502"} 17

# LOG — one structured line from orders (detail of this one event)
{"time":"2026-11-08T14:05:12.481Z","level":"ERROR","msg":"payment failed",
 "service":"orders","route":"/orders","status":502,"duration_ms":2004,
 "order_id":"o-81231","trace_id":"4bf92f3577b34da6a3ce929d0e0e4736",
 "span_id":"00f067aa0ba902b7","error":"payments: context deadline exceeded"}

# TRACE — the request as a tree of spans (where the time went)
trace 4bf92f3577b34da6a3ce929d0e0e4736
└─ orders  POST /orders            2004 ms  status=ERROR
   ├─ orders  validate order          3 ms
   └─ payments POST /charge        2000 ms  status=ERROR (timeout)

The joins

  • service / job / service.name: the same service name everywhere.
  • route: the template (/orders/{id}), never the raw path.
  • trace_id in logs → open the trace (ch 9).
  • Exemplars attach a trace_id to a metric sample → jump from a latency spike to one slow trace (ch 2, ch 9).
  • Deploy annotations line up with the moment the graph changed (ch 10).
Collection model

Pull vs push

Prometheus pulls; OpenTelemetry, Loki and CloudWatch mostly receive pushes. Both work; know what each gives you.

Pull (scrape)

  • The collector fetches /metrics from each target on an interval.
  • Free health check: a failed scrape sets up == 0.
  • The server controls load and timing.
  • Needs service discovery and network reachability to every target.
  • Awkward for short-lived jobs (they exit before the next scrape).

Push

  • The app or an agent sends data to an endpoint (OTLP, Loki push API, PutMetricData).
  • Works for short-lived jobs, serverless, and across NAT or firewalls.
  • Silence is ambiguous: dead, or just idle? You need explicit heartbeats.
  • Receivers need rate limits to survive a misbehaving client.

In practice

  • Kubernetes metrics: pull (Prometheus or an agent that scrapes).
  • Logs: an agent tails files and pushes (Alloy → Loki).
  • Traces: always push (SDK → Collector → backend).
  • Lambda / batch: push (EMF logs, OTLP, Pushgateway as last resort).
  • Prometheus 3 can also receive OTLP pushes (ch 2).
Landscape

The tools this deck uses, by stage

Open source on Kubernetes first, then the AWS managed equivalents. The concepts transfer; only the query language changes.

StageMetricsLogsTracesAWS managed
Instrumentclient_golang, OTel SDK, exporterslog/slog (Go), OTel log bridgeOTel SDK, otelhttpADOT, CloudWatch agent, EMF
Collect / processPrometheus scrape, Alloy, OTel CollectorGrafana Alloy (Promtail is EOL)OTel CollectorCloudWatch agent, ADOT, AMP managed scrapers
StorePrometheus TSDB; Thanos / Mimir long termLokiTempoCloudWatch, AMP, X-Ray
QueryPromQLLogQLTraceQLMetrics Insights, PromQL, Logs Insights
Visualize / actGrafana, AlertmanagerGrafana, Loki rulerGrafanaCloudWatch dashboards and alarms, Managed Grafana

Versions (Nov 2026): Prometheus 3.15, Loki 3.7, Alloy 1.21, Tempo 3.x, Grafana 13. See the versions slide in Orientation.

Chapter 1 · Part 1.2

Instrumentation and cardinality economics

Where to put telemetry, and why one careless label can cost more than the whole cluster.

Instrumentation

White-box vs black-box

Black-box sees what users see; white-box sees why. Use both, for different jobs.

Black-box

  • Probes from outside: HTTP checks, synthetic transactions, ping, TLS expiry.
  • Tools: blackbox_exporter (ch 2), CloudWatch Synthetics (ch 11).
  • Catches DNS, load balancer, certificate and network failures the app never sees.
  • Says nothing about why.

White-box

  • Telemetry emitted by the code and runtime: request counters, histograms, logs, spans.
  • Tools: client libraries, OTel SDKs, exporters for databases and queues.
  • Shows internal queues, retries, dependency latency.
  • Blind when the process is down or unreachable.

Use them together

  • Page on user-visible symptoms; black-box is the purest symptom.
  • Debug with white-box.
  • If white-box says healthy and black-box says down → look between them: DNS, LB, network, TLS.
  • Probe from where users are (another region, outside the VPC).
Instrumentation

Where to instrument the shop app

Instrument at boundaries: everything that enters, leaves, waits or retries. Internal functions only when profiling says so.

loadgenclientsnamespace shopordersPOST /orders, GET /orders/{id}paymentsPOST /chargeexternal PSPnot in this labinboundoutbound call① inbound: RED per route② outbound: client latency + errors③ business: orders_created_total① inbound: RED per route④ amounts: payments_charge_amount⑤ runtime: goroutines, GC, memoryAlso: queues (depth, age of oldest message), pools (in use / max), retries and timeouts, caches (hit ratio), and every dependency call.

The server side of each hop (①) gives RED for that service; the client side (②) measures the dependency as the caller experiences it, including network and retries. They differ, and the difference is information.

Cardinality

Cardinality: the number of distinct series

A metric name is not one series. Every unique combination of label values is a separate series with its own memory and storage.

http_requests_total{method, route, code}

method: GET, POST                      → 2
route:  /orders, /orders/{id}, /healthz → 3
code:   200, 201, 400, 404, 500, 502   → 6

worst case: 2 × 3 × 6 = 36 series per pod
× 3 pods (pod label from the scrape)   = 108 series

# now add the raw path instead of the route:
path: /orders/o-1, /orders/o-2, ...    → unbounded
→ 1 new series per order ID, forever (until retention)

# and a histogram multiplies again:
http_request_duration_seconds_bucket
  (11 default buckets + "+Inf") = 12 series per combination
  + _sum + _count → 14 series per label combination

Why it hurts

  • Prometheus keeps every active series in memory (head block): roughly a few KiB per series including index — plan with measurements, not folklore (ch 5).
  • Query cost grows with series touched, not with result size.
  • Managed backends bill per active series or per sample (AMP, Grafana Cloud) — or per metric per dimension combination (CloudWatch custom metrics).
  • Churn (series that appear and vanish, e.g. pod names on every deploy) inflates the index even when the active count looks fine.
Cardinality

Labels that explode, and what to do instead

A label is a dimension you will aggregate or filter by. If its values are unbounded, it belongs in logs or traces, not metrics.

Tempting labelWhy it explodesDo this instead
user_id, order_id, emailOne series per user or orderPut it in logs and span attributes; metrics stay aggregate
Raw URL path /orders/o-81231IDs in the pathLabel with the route template /orders/{id}
Error message textFree text, often with IDs insideA small enum reason="timeout"; full message in logs
Client IP, user agentUnboundedLogs; or bucket (country, browser family)
Timestamps, request IDs, trace IDsUnique per requestExemplars (one trace ID per sample, not a label)
Query string, SQL textUnboundedA normalized operation name (db.operation.name)

Test before you ship: count the realistic number of values for every label and multiply. If the product is above a few thousand per service, stop and redesign.

Data reduction

Aggregate, sample, or keep: three ways to reduce volume

Each technique loses something different. Choose per signal and per use case.

Aggregate

  • Turn events into counts and distributions (metrics, recording rules, span metrics).
  • Keeps: exact totals, rates, latency distributions.
  • Loses: the individual event.
  • Do it before sampling, so totals stay correct (ch 9).

Sample

  • Keep a fraction of traces or debug logs.
  • Head sampling: decide at the start (cheap, blind to outcome).
  • Tail sampling: decide after the trace completes (keep errors and slow ones).
  • Loses: guaranteed coverage of every request.

Tier retention

  • Hot: 7–15 days raw, fast queries (Prometheus, Loki, Tempo).
  • Warm: downsampled metrics for months (Thanos / Mimir, CloudWatch roll-ups).
  • Cold: compressed logs in S3 for audit.
  • Different signals, different retention: traces days, metrics months, audit logs years.
Cost

What drives the bill for each signal

Know the unit you pay for. That unit is what you control.

SignalUnit that drives costBiggest leverTypical trap
Metrics (self-hosted)Active series × retention; query loadDrop unused metrics and labels at scrape (metric_relabel_configs)Histograms with many buckets × many labels
Metrics (CloudWatch)Custom metrics (each name + dimension combination), API calls; OTel metrics per GB ingested (2026)Fewer dimensions; EMF instead of PutMetricData loopsPer-instance or per-pod dimensions
LogsGB ingested + GB stored (+ GB scanned by queries)Levels, sampling, dropping noisy lines at the agentDebug logs left on; logging full request bodies
TracesSpans ingested and storedSampling (head or tail)100% tracing of health checks
QueriesData scanned (Logs Insights, Loki), series touchedNarrow time range and selectors firstDashboards refreshing every 5 s over 30 days

Chapter 13 turns this into governance: budgets per team, limits per tenant, and a monthly review of top series and top log streams.

Chapter 1 · Part 1.3

Methods: golden signals, RED, USE

Three checklists that decide what to measure first, so dashboards are designed instead of accumulated.

Golden signals

The four golden signals

From Google's SRE book: if you can measure only four things about a user-facing system, measure these.

Latency

  • Time to serve a request.
  • Separate successful from failed requests: a fast 500 is not good latency.
  • Use percentiles or distributions, not averages.

Traffic

  • Demand on the system: requests per second, sessions, messages per second.
  • Context for every other signal: errors at 2 req/s vs 2,000 req/s.

Errors

  • Rate of failed requests: explicit (5xx), implicit (200 with wrong content), policy (slower than 1 s counts as failed).
  • Usually expressed as a ratio of traffic.

Saturation

  • How full the most constrained resource is: CPU, memory, queue, connection pool.
  • Leading indicator: latency rises before errors appear.
  • Often a prediction: "disk full in 4 hours".

RED and USE below are two specializations: RED for services (requests), USE for resources (things requests consume). Together they cover the golden signals.

RED

RED: for every request-driven service

Tom Wilkie's method. Same three panels for every service, so any engineer can read any dashboard.

Rate

  • Requests per second, per service and route.
  • sum by (route) (rate(http_requests_total{job="orders"}[5m]))

Errors

  • Failed requests per second, or as a ratio of rate.
  • sum(rate(http_requests_total{job="orders",code=~"5.."}[5m])) / sum(rate(http_requests_total{job="orders"}[5m]))

Duration

  • Latency distribution: p50, p90, p99, plus a heatmap.
  • histogram_quantile(0.99, sum by (le) (rate(http_request_duration_seconds_bucket{job="orders"}[5m])))

These queries are explained in ch 3 and turned into dashboards in ch 10. RED does not cover saturation: pair it with USE for the resources each service runs on.

USE

USE: for every resource

Brendan Gregg's method. For each resource, check utilization, saturation and errors — in that order, and quickly.

ResourceUtilizationSaturationErrors
CPU (node)1 - avg(rate(node_cpu_seconds_total{mode="idle"}[5m]))Load average / cores; run queueMachine check errors (rare)
CPU (container)Usage / limit (or / request)CFS throttling: container_cpu_cfs_throttled_periods_total—
MemoryWorking set / limitSwapping, page faults, reclaimOOM kills (kube_pod_container_status_last_terminated_reason)
DiskBusy time, space used %I/O queue length, wait timeDevice errors, filesystem read-only
NetworkBytes/s vs link capacityDrops, retransmits, conntrack table fullInterface errors
Software: pools, queuesConnections in use / maxWaiters, queue depth, age of oldest messageRejected or timed-out acquisitions

Kubernetes queries for each row are in ch 5 and ch 10. The Linux A to Z deck covers the underlying tools (vmstat, iostat, sar).

Symptoms vs causes

Page on symptoms, investigate causes

Users feel symptoms. Causes are many, change over time, and often heal themselves.

Symptom alerts (page)

  • Error ratio for POST /orders above budget.
  • p99 latency of checkout above 1 s for 10 minutes.
  • Black-box probe failing from outside.
  • Few, stable, tied to user impact.

Cause signals (dashboards, tickets)

  • CPU at 95% on one node.
  • A pod restarted.
  • Disk 80% full (ticket, not page — unless it fills in hours).
  • Useful for diagnosis; noisy as pages.

Why this matters

  • High CPU with happy users is efficiency, not an incident.
  • Cause-based paging creates alert fatigue and gaps: you cannot list every cause.
  • Exception: causes that predict a certain, imminent outage (certificate expiring, disk full in 2 h).

Alert design is ch 4; alert quality and on-call hygiene is ch 13; SLO-based paging is in the SRE deck.

SLIs

Where SLIs and SLOs fit

An SLI is a carefully chosen RED measurement expressed as "good events / valid events". This deck builds the metrics; the SRE deck builds the objectives.

SLI (indicator)

  • Availability: non-5xx requests / all requests.
  • Latency: requests faster than 300 ms / all requests — needs a histogram bucket at 0.3 s.
  • Measured as close to the user as possible (load balancer or edge > app).

SLO (objective)

  • Target over a window: 99.9% of requests succeed over 30 days.
  • Error budget: the 0.1% you may spend on failures and releases.
  • Burn-rate alerts page when the budget burns too fast (preview in ch 4).

What this deck gives the SRE deck

  • Histogram buckets placed at SLO thresholds (ch 2).
  • Recording rules for ratios over several windows (ch 4).
  • Dashboards with the SLI front and centre (ch 10).
  • CloudWatch Application Signals SLOs (ch 11).
Worked example

A signal plan for the shop app

Write this table before writing instrumentation code. Lab 1 asks you to fill it in for the shop app.

QuestionSignalName / query (built later)Alert?
Are orders failing?Metric (R, E)http_requests_total{job="orders",route="/orders",code=~"5.."} ratioPage (SLO burn, SRE deck)
Are orders slow?Metric (D)http_request_duration_seconds p99 per routePage if above SLO
Is payments the cause?Metric + tracepayments RED; orders → payments span durationNo — dashboard
Which orders failed and why?Logs{namespace="shop", app="orders"} | json | status >= 500No
Is revenue flowing?Business metricrate(orders_created_total[5m]) vs last weekTicket on large drop
Are pods healthy and sized?USE (Kubernetes)CPU throttling, memory vs limit, restartsTicket; page only on crash loop
Is the site reachable?Black-boxprobe_success for /healthz via blackbox_exporterPage
Common mistakes

Foundation mistakes that cost months later

Each of these looks harmless on day one.

Design

  • Collecting everything "just in case" without a signal plan.
  • Averages instead of percentiles or histograms for latency.
  • Different service names in metrics, logs and traces → no correlation.
  • Paging on causes (CPU, restarts) instead of symptoms.

Cardinality

  • User IDs, raw paths or error strings as metric labels.
  • Histograms with 30 buckets × 6 labels "for precision".
  • Ignoring churn: pod and container labels on every deploy.
  • No limits: one bad deploy can take Prometheus down (OOM).

Habits that prevent them

  • A signal plan per service (question → signal → query → alert?).
  • A label review in code review: "what are all the values?"
  • Shared resource attributes: service.name, deployment.environment.name.
  • Budgets and limits from day one (ch 5, ch 13).
Self-check

Foundations: can you answer these?

Answer out loud, with an example from the shop app.

Signals

  • Which signal answers "is it broken", "where", and "why"?
  • What makes logs, metrics and traces correlatable?
  • Give one strength and one weakness of pull-based collection.
  • Why is OTel Profiles not yet something to depend on in production?

Cardinality and cost

  • How many series does a histogram with 11 buckets and labels method (2) × route (3) × code (6) produce per pod?
  • Why must order_id never be a metric label? Where should it go?
  • What does each signal cost per unit (series, byte, span)?
  • Why aggregate before sampling?

Methods

  • List the four golden signals and the RED and USE letters.
  • Why are averages bad for latency?
  • Give two symptom alerts and two cause signals for the shop app.
  • When is paging on a cause acceptable?

Practice: Lab 1 in Hands-on labs — observe the shop app with only black-box checks, logs and Kubernetes metrics, write its signal plan, and see why RED needs app metrics.

Chapter 2

Prometheus fundamentals

How Prometheus stores time series, the four metric types and how to instrument with them, scraping and relabeling, and the exporters around it.

Chapter 2 · Part 2.1

Architecture and data model

A pull-based server with a local time-series database, a rule engine and an HTTP API — and what a "series" really is.

Architecture

The Prometheus ecosystem in one picture

One binary does discovery, scraping, storage, rule evaluation and querying. Everything else plugs in around it.

Prometheus serverApps with /metricsclient librariesExportersnode, blackbox, postgresPushgatewaybatch jobs onlyKubernetes APIobjects to discoverScrape managerpull + relabelService discoverykubernetes, file, EC2TSDBhead, WAL, blocksRule enginerecord, alertPromQL engineHTTP API + UI:9090Remote write · OTLP receiver (optional, ch 5)Alertmanagergroup, route, notify (ch 4)Slack, email, PagerDutyGrafanadashboards (ch 10)pull

Prometheus is deliberately single-node: each server is independent and reliable on its own. Scale and long-term storage come from sharding and remote write (ch 5).

Data model

A time series is a name plus a set of labels

Change any label value and you have a different series. Each series is a stream of (timestamp, value) samples.

# exposition format (what /metrics returns)
# HELP http_requests_total HTTP requests handled.
# TYPE http_requests_total counter
http_requests_total{method="POST",route="/orders",code="201"} 1027
http_requests_total{method="POST",route="/orders",code="502"} 17
http_requests_total{method="GET",route="/orders/{id}",code="200"} 4410

# after the scrape, Prometheus adds target labels:
http_requests_total{job="orders", instance="10.244.1.7:8080",
  namespace="shop", pod="orders-7d9c...", method="POST",
  route="/orders", code="502"}  @1762610712.481  17

# the metric name is itself a label:
{__name__="http_requests_total", job="orders", ...}

Terms

  • Sample: one float64 value (or one native histogram) + a millisecond timestamp.
  • Series: all samples with the same label set.
  • Metric name: __name__; matches [a-zA-Z_:][a-zA-Z0-9_:]* by convention — Prometheus 3 also accepts any UTF-8 (quoted).
  • Target labels: job, instance and whatever relabeling adds.
  • Label names starting with __ are reserved for internal use.
  • Values are always float64: booleans become 0/1, enums become labels or separate series.
TSDB

How the local storage works

Recent data lives in memory with a write-ahead log; older data is in immutable, compacted blocks on disk.

Head block (memory)

  • All active series and their most recent samples (roughly the last 2–3 hours).
  • Every sample is first appended to the WAL on disk, so a crash loses nothing that was acknowledged.
  • Full chunks are memory-mapped to disk to reduce RAM.
  • Memory use ≈ active series × per-series overhead — the reason cardinality matters.

Persistent blocks (disk)

  • Every 2 hours the head is cut into a block: chunks + index + metadata.
  • Blocks are immutable; deletes are tombstones until compaction.
  • Compaction merges blocks into larger ones (up to 10% of retention, max 31 days).
  • Gorilla-style compression: ~1–2 bytes per sample on disk.

Retention flags

  • --storage.tsdb.retention.time (default 15d).
  • --storage.tsdb.retention.size (e.g. 50GB; oldest blocks dropped first).
  • --storage.tsdb.path (default data/).
  • Local disk only: use a reliable block volume, not NFS.
  • Long retention and global views → Thanos, Mimir or AMP (ch 5).
Staleness

Lookback delta and staleness markers

A query at time t takes, for each series, the newest sample at or before t — if it is not too old and not marked stale.

Lookback delta

  • Default 5 minutes (--query.lookback-delta).
  • An instant query at t returns a series only if it has a sample in (t − 5m, t].
  • So a series keeps "existing" for up to 5 minutes after its last sample — unless a staleness marker ends it.

Staleness markers

  • When a target disappears, or a series is missing from a scrape, Prometheus writes a special stale sample.
  • Queries stop returning that series right away instead of after 5 minutes.
  • Works only for scraped data; samples pushed via remote write or OTLP rely on the lookback delta.
  • Pods replaced by a deploy → old series go stale, new ones start (churn).

Consequences

  • Scrape interval must be well below 5 minutes (common: 15–60 s).
  • Graphs show gaps when scrapes fail, not flat lines.
  • A metric you stop exporting disappears; alerts on it silently stop — use absent() (ch 3).
  • Timestamps in exposition output are rarely a good idea: they disable staleness handling.
Prometheus 3

What changed in Prometheus 3 (2024 → Nov 2026)

Prometheus 3.0 shipped in Nov 2024; 3.15 is current and 3.5 is the long-term support line.

ChangeSinceWhat it means for you
New web UI (Mantine-based)3.0Query, targets, rules, TSDB status; the old UI is gone
UTF-8 metric and label names3.0{"http.server.request.duration", "service.name"="orders"} — OTel names can be stored unescaped
OTLP receiver3.0--web.enable-otlp-receiver → push OTLP metrics to /api/v1/otlp/v1/metrics
Remote Write 2.03.0 (experimental)Smaller payloads, metadata, exemplars and created timestamps in one protocol
scrape_classic_histograms renamed always_scrape_classic_histograms3.0Update scrape configs when upgrading from 2.x
Range selectors are left-open3.0[5m] excludes a sample exactly at t − 5m; edge cases in rate and *_over_time changed slightly
Native histograms stable3.8Opt-in per scrape job with scrape_native_histograms: true; the old feature flag is a no-op since 3.9

Check the release notes before relying on anything newer than this table; experimental features need --enable-feature=... and may change.

Running it

A minimal configuration and how to reload it

One YAML file for scraping and rules; flags for storage and features. Validate every change with promtool.

# prometheus.yml
global:
  scrape_interval: 15s       # default 1m
  scrape_timeout: 10s        # default 10s, must be <= interval
  evaluation_interval: 15s   # rules; default 1m
  external_labels:           # added when data leaves this server
    cluster: obs

rule_files:
  - /etc/prometheus/rules/*.yml

alerting:
  alertmanagers:
    - static_configs: [{targets: ['alertmanager:9093']}]

scrape_configs:
  - job_name: prometheus
    static_configs: [{targets: ['localhost:9090']}]
# validate before deploying
promtool check config prometheus.yml
# Checking prometheus.yml
#  SUCCESS: prometheus.yml is valid prometheus config file syntax

# run (container)
docker run -p 9090:9090 -v $PWD:/etc/prometheus \
  prom/prometheus:v3.15.1 \
  --config.file=/etc/prometheus/prometheus.yml \
  --storage.tsdb.retention.time=15d \
  --web.enable-lifecycle

# reload without restart
kill -HUP <pid>
curl -X POST localhost:9090/-/reload   # needs --web.enable-lifecycle

# health and readiness
curl localhost:9090/-/healthy
curl localhost:9090/-/ready

With the Prometheus Operator (ch 5) you never edit this file: the operator generates it from ServiceMonitor, PodMonitor and PrometheusRule objects and reloads Prometheus for you.

Chapter 2 · Part 2.2

Metric types and instrumentation

Counter, gauge, histogram, summary — and native histograms — with naming rules and Go code for the shop app.

Metric types

Four types, one question each

The type is a contract with the query language: it decides which functions make sense.

TypeSemanticsExamplesQuery with
CounterOnly goes up; resets to 0 on process restarthttp_requests_total, orders_created_total, node_cpu_seconds_totalrate(), increase() — never the raw value
GaugeGoes up and down; a current valuego_goroutines, node_memory_MemAvailable_bytes, queue depth, temperatureRaw value, avg_over_time, max_over_time, deriv, predict_linear
HistogramCounts observations into buckets + sum + counthttp_request_duration_seconds, payments_charge_amounthistogram_quantile() on rate() of buckets; ratios per bucket
SummaryClient-side quantiles + sum + countLegacy Go GC pause summaries, some librariesRead quantiles as-is; cannot aggregate quantiles across instances

Info-style metrics are gauges with value 1 that carry metadata as labels, e.g. go_info{version="go1.27.1"}, kube_pod_info{node=...}; you join them with other series (ch 3).

Counters and gauges

Counters reset, gauges lie between scrapes

Two subtle facts that explain most beginner query bugs.

Counters

  • A restart resets the counter to 0. rate() and increase() detect the drop and compensate.
  • So never compute max - min of a counter yourself: a reset breaks it.
  • Count events, not states: orders_created_total, not orders_in_progress.
  • Initialize label combinations you know (e.g. code="500" at 0) so a first error is a visible increase, not a new series.

Gauges

  • You only see the value at scrape time; a spike between two scrapes is invisible.
  • For "how much happened" use a counter even if a gauge seems natural (bytes sent, jobs processed).
  • For in-flight work, a gauge is right: http_requests_in_flight.
  • _over_time functions summarize gauges over a window (ch 3).

Choose the type

  • Can it only increase? → counter (_total).
  • Is it a current level? → gauge.
  • Do you need a distribution (latency, size)? → histogram.
  • Need exact quantiles on one instance and never aggregate? → summary (rare).
Histograms

Classic histograms: cumulative buckets

Each bucket counts observations less than or equal to its upper bound le. Buckets are counters, so you rate() them.

# TYPE http_request_duration_seconds histogram
http_request_duration_seconds_bucket{route="/orders",le="0.05"}  812
http_request_duration_seconds_bucket{route="/orders",le="0.1"}   950
http_request_duration_seconds_bucket{route="/orders",le="0.25"}  1003
http_request_duration_seconds_bucket{route="/orders",le="0.3"}   1010
http_request_duration_seconds_bucket{route="/orders",le="0.5"}   1018
http_request_duration_seconds_bucket{route="/orders",le="1"}     1022
http_request_duration_seconds_bucket{route="/orders",le="2.5"}   1044
http_request_duration_seconds_bucket{route="/orders",le="+Inf"}  1044
http_request_duration_seconds_sum{route="/orders"}               71.3
http_request_duration_seconds_count{route="/orders"}             1044

# 1010 of 1044 requests took <= 300 ms  → 96.7%
# +Inf bucket == _count; _sum/_count = mean (68 ms)

Design the buckets

  • Put a bucket boundary exactly at each SLO threshold (here 0.3 s): then "% under 300 ms" is exact, not estimated.
  • Cover the realistic range: below the fastest and above the timeout.
  • Go default prometheus.DefBuckets: .005 … 10 s (11 buckets) — fine for web latency.
  • Each bucket is one series per label combination: 10 buckets × 30 label combinations × 3 pods = 900 series.
  • Quantiles are interpolated inside a bucket: wide buckets → large error (ch 3).
  • Changing buckets later breaks queries over the change boundary.
Histograms vs summaries

Why histograms usually win

Summaries compute quantiles inside each process. That is precise for one process and useless for the fleet.

PropertyHistogram (classic)Summary
Aggregate across podsYes: sum bucket rates, then compute the quantileNo: averaging p99s is mathematically meaningless
Choose quantile after the factYes, any quantile at query timeNo, fixed in code (e.g. 0.5, 0.9, 0.99)
AccuracyLimited by bucket widthConfigurable error, exact per process
Client costCheap: increment a counterHigher: streaming quantile estimation
Server costOne series per bucketOne series per quantile
SLO "fraction under X"Exact if X is a bucket boundaryNot possible

Default to histograms. Native histograms (next slide) remove most of the bucket-design pain.

Native histograms

Native histograms: high resolution in one series

Stable since Prometheus 3.8, still opt-in. Buckets are exponential and sparse, and the whole histogram is a single series.

How they work

  • Bucket boundaries grow by a fixed factor (schema): e.g. factor ≈ 1.09 gives ~9% relative error.
  • Only buckets that received observations are stored (sparse).
  • One series per label combination instead of one per bucket + _sum + _count.
  • No bucket design up front; quantiles are far more accurate.

Turning them on

  • Instrument: set NativeHistogramBucketFactor (Go) — the client then exposes both classic and native.
  • Scrape: scrape_native_histograms: true in the scrape config; native histograms need the protobuf scrape format, which is then negotiated first.
  • Keep classic buckets too during migration: always_scrape_classic_histograms: true.
  • Query: histogram_quantile(0.99, sum(rate(http_request_duration_seconds[5m]))) — no _bucket, no le (more functions in ch 3).

Caveats (Nov 2026)

  • Remote-write receivers and long-term stores must support them (Grafana Mimir does; check your backend's docs).
  • Exact SLO thresholds: histogram_fraction estimates within the bucket; classic buckets at the threshold are exact.
  • The classic text exposition format does not carry native histograms: the target must serve protobuf.

In this deck the shop app exposes both forms; Lab 2 scrapes native histograms with a local Prometheus so you can compare queries side by side.

Naming

Naming and label rules

Names are an API: dashboards, alerts and other teams depend on them. Follow the conventions and they explain themselves.

Metric names

  • <namespace>_<what>_<unit>[_total]: http_request_duration_seconds, orders_created_total.
  • Base units: seconds, bytes, meters, ratios 0–1 (not ms, MB or %).
  • Counters end in _total; _count, _sum, _bucket are reserved for histograms and summaries.
  • The unit is in the name, never in a label.
  • snake_case; no type or label names inside the name (requests_by_route_total is wrong).

Labels

  • Bounded values only (see ch 1: cardinality).
  • A label should be something you would sum by or filter on.
  • Do not duplicate target labels (job, instance, pod) — Prometheus adds them.
  • Sum over any label should still be meaningful: do not mix type="total" with its parts.
  • Keep the label set stable over time; empty label = absent label.

Good vs bad

  • ✓ http_requests_total{route="/orders/{id}"}
  • ✗ http_requests_total{path="/orders/o-81231"}
  • ✓ process_resident_memory_bytes
  • ✗ memory_usage_mb
  • ✓ cache_requests_total{result="hit"}
  • ✗ cache_hits_total + cache_requests_total that double count
Instrumentation

Instrumenting the shop app with client_golang

One middleware gives RED for every route; r.Pattern (Go 1.23+) keeps route bounded. promauto registers on the default registry, which already has go_* and process_* collectors.

// apps/internal/metrics/metrics.go (module obs-lab)
var requests = promauto.NewCounterVec(prometheus.CounterOpts{
    Name: "http_requests_total",
    Help: "HTTP requests by method, route template and status code.",
}, []string{"method", "route", "code"})

var duration = promauto.NewHistogramVec(prometheus.HistogramOpts{
    Name:    "http_request_duration_seconds",
    Help:    "HTTP request latency.",
    Buckets: []float64{.01, .025, .05, .1, .25, .3, .5, 1, 2.5, 5},
    NativeHistogramBucketFactor:     1.1, // also expose native
    NativeHistogramMaxBucketNumber:  100,
    NativeHistogramMinResetDuration: time.Hour,
}, []string{"method", "route"})

type recorder struct {
    http.ResponseWriter
    code int
}

func (r *recorder) WriteHeader(c int) {
    r.code = c; r.ResponseWriter.WriteHeader(c)
}
// Middleware wraps the whole mux; the mux sets r.Pattern,
// e.g. "GET /orders/{id}", before calling the handler.
func Middleware(next http.Handler) http.Handler {
    return http.HandlerFunc(func(w http.ResponseWriter,
        r *http.Request) {
        start := time.Now()
        rec := &recorder{ResponseWriter: w, code: 200}
        next.ServeHTTP(rec, r)
        route := "unmatched"                      // 404/405
        if _, p, ok := strings.Cut(r.Pattern, " "); ok {
            route = p                             // "/orders/{id}"
        }
        requests.WithLabelValues(r.Method, route,
            strconv.Itoa(rec.code)).Inc()
        duration.WithLabelValues(r.Method, route).
            Observe(time.Since(start).Seconds())
    })
}

// apps/orders/main.go
mux.Handle("GET /metrics", promhttp.HandlerFor(prometheus.
    DefaultGatherer, promhttp.HandlerOpts{EnableOpenMetrics: true}))
srv := &http.Server{Addr: ":8080",
    Handler: metrics.Middleware(mux)}
Business metrics

Business metrics and a second histogram

RED says the service works; business metrics say the business works. They are often the first to show a silent failure.

// apps/orders: count successful orders
var ordersCreated = promauto.NewCounter(prometheus.CounterOpts{
    Name: "orders_created_total",
    Help: "Orders created successfully.",
})
// in the POST /orders handler, after payments succeeded:
ordersCreated.Inc()

// apps/payments: distribution of charged amounts
var chargeAmount = promauto.NewHistogram(prometheus.HistogramOpts{
    Name:    "payments_charge_amount",
    Help:    "Charged amount per successful charge (currency units).",
    Buckets: prometheus.ExponentialBuckets(1, 2, 12), // 1 … 2048
})
// in the POST /charge handler, after the charge:
chargeAmount.Observe(amount)

Notes

  • ExponentialBuckets(start, factor, count) and LinearBuckets(start, width, count) generate bucket lists.
  • Amounts are not seconds or bytes, so the name has no unit suffix; document the currency in Help.
  • Do not add customer_id or currency with many values as labels.
  • A drop in rate(orders_created_total[5m]) while RED looks fine = logic bug, not an outage — worth a ticket alert (ch 4).
  • Business metrics in metrics; per-order detail (order_id) in logs (ch 6).
Exemplars

Exemplars: a trace ID attached to a sample

An exemplar is one example observation (value + labels such as trace_id) stored next to a series. It is the bridge from a latency graph to one real trace.

// observe with an exemplar when the request is traced (ch 8)
sc := trace.SpanContextFromContext(r.Context())
obs := duration.WithLabelValues(r.Method, route)
if sc.IsSampled() {
    obs.(prometheus.ExemplarObserver).ObserveWithExemplar(
        secs, prometheus.Labels{"trace_id": sc.TraceID().String()})
} else {
    obs.Observe(secs)
}

# OpenMetrics exposition (needs EnableOpenMetrics: true)
http_request_duration_seconds_bucket{route="/orders",le="2.5"} 1044 \
  # {trace_id="4bf92f3577b34da6a3ce929d0e0e4736"} 2.004 1762610712.48

Requirements

  • Exposition in OpenMetrics (or protobuf) format: plain text format has no exemplars.
  • Prometheus stores them only with --enable-feature=exemplar-storage (in-memory ring buffer, size set by storage.exemplars.max_exemplars).
  • Query: /api/v1/query_exemplars; Grafana shows them as dots on time series panels.
  • Exemplar labels are limited to 128 UTF-8 characters in total.
  • Exemplars do not create series: this is how a trace ID gets near metrics without exploding cardinality.
  • Wired end to end in ch 9.
Chapter 2 · Part 2.3

Scraping, service discovery and relabeling

How Prometheus finds targets, decides what to scrape, and rewrites labels before and after the scrape.

Scrape config

Anatomy of a scrape job

A scrape job is: where to find targets, how to talk to them, how to rewrite labels, and limits.

scrape_configs:
  - job_name: shop-pods               # becomes job="shop-pods" by default
    scrape_interval: 15s               # overrides global
    scrape_timeout: 10s
    metrics_path: /metrics             # default
    scheme: http                       # default
    kubernetes_sd_configs:             # WHERE: discovery
      - role: pod
        namespaces: {names: [shop]}
    relabel_configs:                   # BEFORE scrape: targets
      - source_labels: [__meta_kubernetes_pod_container_port_name]
        regex: http
        action: keep
      - source_labels: [__meta_kubernetes_pod_label_app]
        target_label: job
    metric_relabel_configs:            # AFTER scrape: samples
      - source_labels: [__name__]
        regex: go_gc_.*
        action: drop
    sample_limit: 5000                 # LIMITS: fail scrape above
    label_limit: 30

Defaults worth knowing

  • scrape_interval: 1m; scrape_timeout: 10s; evaluation_interval: 1m.
  • instance defaults to __address__ (host:port).
  • Labels beginning with __ are dropped after target relabeling (except they can be copied first).
  • __param_<name> sets URL query parameters; __scheme__ and __metrics_path__ override scheme and path.
  • Use one job per kind of target: limits and intervals are per job.
  • With the Operator, a ServiceMonitor produces exactly this kind of job (ch 5).
Target health

up and the synthetic scrape metrics

Every scrape writes a few series about itself. They are your first stop when a graph is empty.

SeriesMeaningTypical use
up1 if the scrape succeeded, 0 if it failedup == 0 alert per job; avg by (job) (up)
scrape_duration_secondsHow long the scrape tookClose to scrape_timeout → slow exporter, will start failing
scrape_samples_scrapedSamples exposed by the targetSpot a target that suddenly exposes 10× more
scrape_samples_post_metric_relabelingSamples kept after metric_relabel_configsCheck that your drop rules work
scrape_series_addedNew series created by this scrapeChurn detector: should be ~0 in steady state

The Targets page (/targets) shows the last error per target ("context deadline exceeded", "connection refused", "sample limit exceeded"). Service discovery (/service-discovery) shows discovered labels before and after relabeling — including dropped targets.

Kubernetes SD

Kubernetes service discovery roles

Prometheus watches the API server and turns objects into targets with __meta_kubernetes_* labels you can relabel.

RoleOne target perUseful meta labelsUse for
nodeNode (kubelet address)__meta_kubernetes_node_name, _node_label_<name>kubelet, cAdvisor
podDeclared container port (or pod IP if none)_namespace, _pod_name, _pod_label_<name>, _pod_container_port_name, _pod_annotation_<name>Apps without a Service; DaemonSets
serviceService port_service_name, _service_port_nameBlack-box probing of Services
endpointsEndpoint address + portservice + pod labelsApps behind a Service (classic)
endpointsliceEndpointSlice endpoint + portservice + pod labels, _endpointslice_*Same as endpoints; scales better
ingressIngress path_ingress_name, _ingress_pathBlack-box probing of public URLs

Meta label names are abbreviated above: each starts with __meta_kubernetes_ (e.g. __meta_kubernetes_pod_label_app). Label names are sanitized: app.kubernetes.io/name becomes ..._label_app_kubernetes_io_name.

Relabeling

Where relabeling happens

Two different hooks with the same syntax. Mixing them up is the most common relabeling bug.

Service discovery__meta_* labelsrelabel_configsper targetScrapeGET /metricsmetric_relabel_configsper sampleTSDBstored seriesDecide WHICH targets to scrapeand set job, namespace, pod labels.Sees __meta_*, __address__.Runs once per target per SD update.Decide WHICH samples to keepand drop or rewrite labels.Sees __name__ and final labels.Runs on every sample of every scrape: keep it cheap.Also: write_relabel_configs (per remote_write endpoint) and alert_relabel_configs (before alerts go to Alertmanager).

Dropping a metric in metric_relabel_configs saves storage and memory, but the target still exposes and Prometheus still parses it. Removing it at the source is cheaper when you own the code.

Relabeling

Relabel actions with examples

Each rule: join source_labels with separator (default ;), match against regex (fully anchored), then act.

Actions

ActionEffect
replace (default)Set target_label to replacement (with $1 groups)
keep / dropKeep or drop the target / sample if regex matches
keepequal / dropequalKeep / drop if source equals target_label value
labelmapCopy labels whose names match regex
labeldrop / labelkeepRemove / retain labels by name
hashmodSet target_label to hash(source) mod modulus (sharding)
lowercase / uppercaseChange case of the joined source value
# copy all pod labels as-is
- action: labelmap
  regex: __meta_kubernetes_pod_label_(.+)

# namespace and pod labels
- source_labels: [__meta_kubernetes_namespace]
  target_label: namespace
- source_labels: [__meta_kubernetes_pod_name]
  target_label: pod

# opt-in by annotation (prometheus.io/scrape=true)
- source_labels:
    [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
  regex: "true"
  action: keep

# drop a noisy label from samples (metric_relabel_configs)
- regex: pod_template_hash|controller_revision_hash
  action: labeldrop
Relabeling

honor_labels, limits and other safety settings

A target can expose labels that collide with the labels Prometheus attaches. And a broken target can flood you. Both are configurable.

Label conflicts

  • Target exposes job="x" and Prometheus wants job="shop".
  • Default honor_labels: false: the scraped label is renamed exported_job.
  • honor_labels: true: the scraped value wins — only for Pushgateway and federation, where the source labels are the truth.
  • honor_timestamps: true (default) uses timestamps in the exposition if present.

Limits per scrape

  • sample_limit: scrape fails (all samples rejected, up = 0) above N samples after relabeling.
  • label_limit, label_name_length_limit, label_value_length_limit.
  • target_limit: maximum targets per job after relabeling.
  • body_size_limit: uncompressed response size.
  • A clear failure beats a silent cardinality explosion.

Recommended baseline

  • Limits on every job that scrapes application code.
  • Alert on up == 0 and on scrapes failing because of limits (the target error says why).
  • scrape_timeout < scrape_interval.
  • Track scrape_series_added and prometheus_tsdb_head_series (ch 5).
Chapter 2 · Part 2.4

Exporters and special cases

Metrics for software you do not own, black-box probes, batch jobs, OTLP pushes and federation.

Exporters

The exporter pattern and node_exporter

An exporter translates a system's own stats into Prometheus metrics on scrape. It runs next to the system, not inside it.

node_exporter (Linux hosts)

  • Port 9100; in kube-prometheus-stack it runs as a DaemonSet.
  • node_cpu_seconds_total{mode} — CPU time per mode (counter).
  • node_memory_MemAvailable_bytes, node_memory_MemTotal_bytes.
  • node_filesystem_avail_bytes{mountpoint}, node_disk_io_time_seconds_total.
  • node_network_receive_bytes_total{device}, node_load1.
  • Textfile collector: --collector.textfile.directory reads *.prom files written by cron jobs.

Common exporters

  • blackbox_exporter: probes HTTP, TCP, ICMP, DNS, gRPC (next slide).
  • postgres_exporter, mysqld_exporter, redis_exporter: database stats.
  • kube-state-metrics: state of Kubernetes objects (ch 5).
  • cloudwatch_exporter / YACE: CloudWatch metrics into Prometheus (watch API cost).
  • Many apps expose /metrics natively: CoreDNS, etcd, Argo CD, NGINX Ingress, Envoy.

Rules for exporters

  • One exporter instance per monitored instance; avoid "one exporter for all databases" (the up metric loses meaning).
  • Run it as a sidecar or next to the process, scraped by Prometheus.
  • Check its cardinality before rollout: some expose a series per table or per query.
  • Exporters for managed services (RDS) see only what the service API exposes.
Black-box probing

blackbox_exporter: probes as metrics

Prometheus asks the exporter to probe a target; relabeling moves the real target into a URL parameter.

# blackbox.yml (exporter config)
modules:
  http_2xx:
    prober: http
    timeout: 5s
    http:
      valid_status_codes: []   # empty = 2xx
      preferred_ip_protocol: ip4

# prometheus.yml
- job_name: blackbox-http
  metrics_path: /probe
  params: {module: [http_2xx]}
  static_configs:
    - targets: ['http://orders.shop.svc:8080/healthz']
  relabel_configs:
    - source_labels: [__address__]
      target_label: __param_target   # ?target=...
    - source_labels: [__param_target]
      target_label: instance         # keep the URL as instance
    - target_label: __address__
      replacement: blackbox:9115     # actually scrape the exporter

Key probe metrics

  • probe_success — 1 or 0: the symptom to alert on.
  • probe_duration_seconds — total probe time.
  • probe_http_status_code, probe_http_duration_seconds{phase} (resolve, connect, tls, processing, transfer).
  • probe_ssl_earliest_cert_expiry — Unix time; alert at < 14 days: probe_ssl_earliest_cert_expiry - time() < 14*86400.
  • up here means "the exporter answered", not "the target is up".
  • With the Operator, a Probe CRD generates this job (ch 5).
Custom collector

Writing a collector for state you read on demand

For values you already have elsewhere (a queue length, a pool size), compute them at scrape time instead of updating gauges constantly.

type queueCollector struct {
    depth *prometheus.Desc
    q     *Queue
}

func newQueueCollector(q *Queue) *queueCollector {
    return &queueCollector{q: q, depth: prometheus.NewDesc(
        "orders_queue_depth",                    // name
        "Orders waiting for payment.",           // help
        []string{"priority"}, nil)}              // variable, const labels
}

func (c *queueCollector) Describe(ch chan<- *prometheus.Desc) {
    ch <- c.depth
}

func (c *queueCollector) Collect(ch chan<- prometheus.Metric) {
    for prio, n := range c.q.DepthByPriority() {  // called on each scrape
        ch <- prometheus.MustNewConstMetric(
            c.depth, prometheus.GaugeValue, float64(n), prio)
    }
}

// main: prometheus.MustRegister(newQueueCollector(q))

When to use what

  • Direct instrumentation (Counter.Inc(), Histogram.Observe()): events in your own code.
  • Collector: state read from somewhere at scrape time — a queue, a pool, /proc, an API.
  • Collect must be fast and safe to call concurrently; slow collectors make scrapes time out.
  • Use GaugeValue or CounterValue correctly: a counter from an external source must never go down except on reset.
  • Test with testutil.CollectAndCompare from client_golang/prometheus/testutil.
Pushgateway

Pushgateway: for batch jobs, and nothing else

A service-level batch job (a nightly backup) finishes before any scrape. It pushes its result; Prometheus scrapes the Pushgateway.

# end of a nightly job: push the result, grouped by job name
cat <<EOF | curl --data-binary @- \
  http://pushgateway:9091/metrics/job/orders_backup
# TYPE backup_last_success_timestamp_seconds gauge
backup_last_success_timestamp_seconds $(date +%s)
# TYPE backup_duration_seconds gauge
backup_duration_seconds 312
EOF

# Prometheus scrape job for the gateway
- job_name: pushgateway
  honor_labels: true      # keep job="orders_backup" from the push
  static_configs: [{targets: ['pushgateway:9091']}]

# alert: backup older than 26 hours
time() - backup_last_success_timestamp_seconds > 26*3600

Pitfalls

  • Pushed series never expire: a job that stopped running still looks present. Delete groups (DELETE /metrics/job/<name>) or alert on the timestamp.
  • up reflects the Pushgateway, not the jobs.
  • Single point of failure and bottleneck if every pod pushes.
  • Never use it to get around network limits for long-running services: use an agent that scrapes locally and remote-writes, or OTLP.
  • For many short jobs (Lambda), prefer push protocols designed for it: OTLP or CloudWatch EMF (ch 11).
OTLP and federation

Pushing OTLP into Prometheus 3, and federation

Two ways to get metrics into a Prometheus without scraping the source directly.

# Prometheus 3: accept OTLP metrics pushes
prometheus --web.enable-otlp-receiver ...
# endpoint: http://prometheus:9090/api/v1/otlp/v1/metrics

# prometheus.yml
otlp:
  promote_resource_attributes:   # resource attrs → labels
    - service.name
    - deployment.environment.name
    - k8s.namespace.name
  # translation_strategy: UnderscoreEscapingWithSuffixes (default)
  #   http.server.request.duration → http_server_request_duration_seconds
storage:
  tsdb:
    out_of_order_time_window: 30m  # recommended for pushed OTLP

# OTel SDK env: OTEL_EXPORTER_OTLP_METRICS_ENDPOINT=
#   http://prometheus:9090/api/v1/otlp/v1/metrics

Notes

  • service.namespace/service.name → job, service.instance.id → instance; other resource attributes go to target_info unless promoted.
  • Use cumulative temporality (the default for most SDKs); delta support is experimental.
  • Pushed data has no up and no staleness markers.
  • Full name translation rules are in ch 8.
  • Federation: /federate?match[]={job="orders"} exposes current series for another Prometheus to scrape. Fine for a few aggregated series (recording rules) in a global view; not for copying everything — large responses are slow and time out. Prefer remote write to Thanos, Mimir or AMP (ch 5).
Common mistakes

Prometheus mistakes seen in real clusters

Each one has caused an outage of either the app or the monitoring.

Instrumentation

  • Raw URL path as a label → millions of series.
  • Milliseconds or megabytes in names (latency_ms) instead of base units.
  • Summaries everywhere, then trying to average p99 across pods.
  • No histogram bucket at the SLO threshold.
  • Gauge used to count events (lost between scrapes).

Scraping

  • Filtering targets in metric_relabel_configs instead of relabel_configs.
  • honor_labels: true on normal app jobs → apps overwrite job/instance.
  • No sample_limit → one bad deploy OOMs Prometheus.
  • Pushgateway for long-running services; stale pushed series alerting forever.
  • scrape_timeout equal to a 1 s interval.

Habits that prevent them

  • promtool check config and promtool check metrics in CI.
  • Look at /metrics of every new service in review.
  • Alert on up == 0 and on scrape_series_added spikes.
  • One middleware per language that sets route labels from the router.
  • Read the Targets and Service discovery pages before debugging queries.
Self-check

Prometheus fundamentals: can you answer these?

Explain with the exact config field, function or metric.

Architecture and storage

  • What is in the head block, and what does the WAL protect?
  • Default retention, and the two flags that change it?
  • What is the lookback delta and how do staleness markers shorten it?
  • Name three things that changed in Prometheus 3.
  • Why does Prometheus not cluster itself?

Types and instrumentation

  • Why must you never read a raw counter value in a graph?
  • Why can't summaries be aggregated across pods?
  • What three series families does a classic histogram expose?
  • What two settings turn on native histograms end to end?
  • How does the shop middleware keep route bounded?

Scraping and exporters

  • relabel_configs vs metric_relabel_configs: when does each run?
  • How does the blackbox job move the target URL into ?target=?
  • What happens when a scrape exceeds sample_limit?
  • When is honor_labels: true correct?
  • Two Pushgateway pitfalls?

Practice: Lab 2 in Hands-on labs — instrument orders and payments with client_golang, scrape them with ServiceMonitors, then compare classic and native histograms in a local Prometheus with a blackbox probe.

Chapter 3

PromQL

The query language behind every Prometheus graph, alert and recording rule: selectors, rates, aggregation, joins, histograms and the traps that produce confident wrong answers.

Chapter 3 · Part 3.1

Selectors and data types

What a PromQL expression returns, how series are selected, and how time enters a query.

Data types

Four types, and which functions accept which

Most PromQL errors ("expected type range vector") are type errors. Know what each expression returns before you nest it.

Expression types

TypeWhat it isExample
Instant vectorOne sample per series, all at the same evaluation timehttp_requests_total{job="orders"}
Range vectorA window of samples per series, ending at the evaluation timehttp_requests_total{job="orders"}[5m]
ScalarA single floating-point number, no labels0.05, scalar(...), time()
StringA string literal; rarely used outside label_replace"orders"

Rules of thumb

  • Graphs and alerts need an instant vector (or scalar) as the final result.
  • Functions like rate, increase, avg_over_time take a range vector and return an instant vector.
  • You cannot graph a range vector directly; wrap it in a function.
  • Aggregations (sum, max, …) take and return instant vectors.
  • With native histograms a sample can hold a whole histogram instead of a float (section 11).
Data types

Instant vector vs range vector on the time axis

Both are evaluated "at" time t. An instant selector takes the latest sample within the lookback window; a range selector takes every sample inside the range.

time →15s scrape intervalevaluation time trange selector [5m]: every sample in (t−5m, t]lookback 5m: newest sample onlyInstant selector returns the sample at x=960 (newest within 5 minutes of t). If a series has no sample in the last 5 minutes, it is absent from the result.

Prometheus 3 made range selectors and the lookback window left-open: a sample exactly at t−5m is excluded. Lookback is set by --query.lookback-delta (default 5m).

Selectors

Label matchers

A selector is a metric name plus label matchers. Regular expressions are RE2 and fully anchored.

# exact match and negative match
http_requests_total{job="orders", code!="200"}

# regex (anchored: "5.." matches 500..599, not "1500")
http_requests_total{job=~"orders|payments", code=~"5.."}

# negative regex
http_requests_total{route!~"/healthz|/metrics"}

# name as a label: every metric starting with http_
{__name__=~"http_.*", job="orders"}

# Prometheus 3: UTF-8 names must be quoted
{"http.server.request.duration", "http.route"="/orders"}

# INVALID: matches every series in the TSDB
{job=~".*"}      # error: needs a matcher that can't be empty

Matcher semantics

  • = equal, != not equal, =~ regex match, !~ regex no-match.
  • A missing label equals the empty string: {team=""} selects series without team.
  • Every selector needs at least one matcher that does not match the empty string.
  • Regex =~"orders" is the same as ="orders" (anchored), but slower.
  • The metric name is the label __name__; foo{} ≡ {__name__="foo"}.
  • Prometheus 3 accepts UTF-8 metric and label names (for OTel names with dots) in quoted form.
Time modifiers

offset and @: move the evaluation time

offset shifts relative to the evaluation time; @ pins an absolute time. Both apply to a single selector, not the whole expression.

# requests per second now vs exactly one week ago
sum(rate(http_requests_total{job="orders"}[5m]))
/
sum(rate(http_requests_total{job="orders"}[5m] offset 1w))

# negative offset (look "ahead" inside a past range query)
rate(http_requests_total[5m] offset -10m)

# pin to a Unix timestamp
http_requests_total{job="orders"} @ 1762560000

# the value at the end of the graph's range, for every step
topk(5, rate(http_requests_total[5m] @ end()))

When to use them

  • Week-over-week ratios for traffic dashboards (offset 1w).
  • "Top 5 at the end of the window" without the series changing at every step: @ end().
  • start() and end() refer to the query range; in an instant query both equal the evaluation time.
  • Modifier order is [5m] offset 1w or @ … offset … attached to the selector.
  • Comparing to the past is noisy at low traffic: compare rates over longer windows.
Queries over time

Instant queries, range queries and the HTTP API

A graph is many instant evaluations: one per step between start and end. Alerts and recording rules are instant evaluations.

# instant query: evaluate once at "time" (default: now)
curl -sG localhost:9090/api/v1/query \
  --data-urlencode 'query=sum by (job) (rate(http_requests_total[5m]))' \
  | jq -r '.data.result[] | "\(.metric.job)  \(.value[1])"'
# orders    5.03
# payments  5.03

# range query: evaluate at start, start+step, … end
curl -sG localhost:9090/api/v1/query_range \
  --data-urlencode 'query=sum(rate(http_requests_total[5m]))' \
  --data-urlencode "start=$(date -u -v-1H +%s)" \
  --data-urlencode "end=$(date -u +%s)" \
  --data-urlencode 'step=60' | jq '.data.result[0].values | length'
# 61

# from the CLI
promtool query instant http://localhost:9090 'up{namespace="shop"}'

What to remember

  • resultType is vector, matrix, scalar or string.
  • In a range query, [5m] is re-evaluated at every step: a step larger than the range skips data.
  • Grafana chooses step from the panel width ($__interval); a 30-day panel may step every few minutes.
  • Values come back as strings ("5.03") to keep float precision; NaN and +Inf are possible.
  • date -v-1H is BSD/macOS; on Linux use date -u -d "-1 hour" +%s.
Tools

Where to write and debug PromQL

Write queries where you can see intermediate results, then move them into rules and dashboards.

Prometheus UI (3.x)

  • Rebuilt UI with autocomplete and inline function docs.
  • Table tab: the instant result with all labels — start here.
  • Graph tab: a range query over the chosen window.
  • Tree view explains each sub-expression and its result.
  • Status → Targets, TSDB Status, Rules, Alerts.

Grafana Explore

  • Same queries with the data source of your dashboards.
  • Builder and Code modes; "Explain" describes each step.
  • Shows $__interval and $__rate_interval resolution.
  • Split view: metrics next to Loki logs or Tempo traces (ch 9).

Habits that save time

  • First run the bare selector in Table view: count series, read labels.
  • Add one function at a time and re-check labels.
  • Check count(...) before sum(...) to know how many series you aggregate.
  • Copy final queries into a rules file and test with promtool (ch 4).
Chapter 3 · Part 3.2

Counters: rate, irate, increase

Counters only go up — except when they reset. Every useful counter query turns them into per-second rates.

Counters

A raw counter is almost never what you want

http_requests_total is a running total since process start. Its absolute value depends on uptime, and it drops to zero on restart.

# raw value: grows forever, drops on restart
http_requests_total{job="orders", code="200"}

# per-second rate over 5m, reset-safe
rate(http_requests_total{job="orders", code="200"}[5m])

# total increase over 1h (= rate × 3600)
increase(orders_created_total[1h])

# how many times did the counter reset in 1h?
resets(http_requests_total{job="orders"}[1h])

Properties and questions

  • Monotonically increasing while the process lives; resets to 0 on restart (new pod, crash, deploy).
  • Per-series: every pod has its own counter. Name ends in _total (ch 2).
  • How many requests per second now? → rate.
  • How many orders in the last hour? → increase.
  • Is the error share growing? → ratio of two rates.
  • None of these questions use the raw value.
rate()

How rate() handles resets and window edges

rate looks at all samples in the range, adds back any drop as a reset, and extrapolates to the window boundaries.

counter valuerange [5m]restart: value drops to near 0 → treated as a resetreset-corrected increase= 140 + (280→175 rise)then ÷ seconds in windowextrapolated to thewindow edges

Because of extrapolation, increase() returns non-integers (e.g. 59.8 requests) even for integer counters. It is an estimate, not a count. Use rate/increase only on counters; on a gauge a decrease is misread as a reset.

Counters

rate vs irate vs increase

Same input, different questions. rate is the default for dashboards and the only sane choice for alerts.

FunctionUsesOutputUse forAvoid for
rate(c[5m])All samples in the windowAverage per-second rate over the windowDashboards, alerts, recording rules—
irate(c[5m])Only the last two samples in the windowInstantaneous per-second rateZoomed-in, high-resolution graphs of fast-moving countersAlerts (too spiky); long-range graphs (steps skip most data)
increase(c[1h])All samples in the windowrate × window secondsHuman-readable totals: "orders in the last hour"Summing many short windows to get a total (edge errors add up)
delta(g[1h])First and last sampleChange of a gaugeGauges such as queue lengthCounters (no reset handling)
deriv(g[10m])Linear regression of all samplesPer-second slope of a gaugeTrend of memory or disk usageCounters

rate needs at least two samples in the window, otherwise the series disappears from the result.

Choosing the window

How long should the [range] be?

Too short and series flicker in and out; too long and spikes vanish. The scrape interval sets the floor.

The floor: 4 × scrape interval

  • With scrape_interval: 15s, use at least [1m]; [5m] is a common default.
  • One failed scrape must still leave ≥ 2 samples in the window.
  • Windows shorter than 2 scrapes return nothing most of the time.

Grafana $__rate_interval

  • max($__interval + scrape interval, 4 × scrape interval).
  • Scrape interval comes from the data source setting (default 15s): set it to your real scrape interval.
  • Grows with the zoom level, so no step skips data.
  • Use it in every rate/increase in dashboards (ch 10).

Alerts and rules

  • Fixed windows ([5m], [1h]) so the meaning never changes.
  • Short windows react fast but are noisy; long windows are stable but slow.
  • Recording rules name the window: …:rate5m (ch 4).
  • Burn-rate alerts pair a long and a short window (SRE deck).
rate(http_requests_total{job="orders"}[$__rate_interval])   # Grafana panel
rate(http_requests_total{job="orders"}[5m])                  # alert / recording rule
Rate then aggregate

Always rate first, then sum

Each pod has its own counter that resets independently. Summing raw counters first creates a fake series whose "resets" are restarts of one pod.

# RIGHT: per-series rate, then aggregate
sum by (job) (rate(http_requests_total[5m]))

# WRONG (needs a subquery to even parse): sum, then rate
rate(sum by (job) (http_requests_total)[5m:30s])
# one pod restarts → the sum drops by that pod's total
# → rate sees a "reset" → huge false spike

# RIGHT: ratios are ratios of sums of rates
sum by (job) (rate(http_requests_total{code=~"5.."}[5m]))
/
sum by (job) (rate(http_requests_total[5m]))

Why

  • rate detects resets per series; once series are summed, a drop in one pod is indistinguishable from a reset of the total.
  • Same reason recording rules store rate results, never sums of raw counters (ch 4).
  • Averaging per-pod ratios (avg(error_ratio)) gives a quiet pod the same weight as a busy one: divide sums instead.
  • Lab 3 restarts orders and shows the false spike side by side.
Gauges

Functions for gauges: trends, predictions, windows

Gauges go up and down: queue length, memory in use, free disk. Do not rate them; summarize them over time.

# memory trend (bytes per second) of shop pods
deriv(container_memory_working_set_bytes{namespace="shop",container!=""}[15m])

# will the root filesystem fill within 4 hours?
predict_linear(node_filesystem_avail_bytes{mountpoint="/"}[6h], 4 * 3600) < 0

# peak and average over a window
max_over_time(container_memory_working_set_bytes{namespace="shop"}[1h])
avg_over_time(kube_deployment_status_replicas_available{namespace="shop"}[1h])

# how often did a value change? (config reloads, flapping state)
changes(kube_pod_container_status_restarts_total{namespace="shop"}[1h])

The *_over_time family

  • avg_, min_, max_, sum_, count_, last_, stddev_, quantile_over_time.
  • present_over_time and absent_over_time test presence (section 11).
  • All take a range vector and aggregate per series over time; sum by aggregates across series at one time.
  • predict_linear fits a line: good for disks, poor for bursty memory.
Chapter 3 · Part 3.3

Aggregation and vector matching

Collapse series with by/without, then combine vectors with arithmetic, comparisons and joins.

Aggregation

Aggregation operators and by / without

Aggregations work across series at a single time. by keeps only the listed labels; without drops the listed labels and keeps the rest.

Operators

OperatorResult
sum, avg, min, maxArithmetic across series
countNumber of series in each group
groupValue 1 per group (only the label set matters)
stddev, stdvarSpread across series
topk(k, v), bottomk(k, v)k largest / smallest series, original labels kept
quantile(φ, v)φ-quantile across series (not latency!)
count_values("lbl", v)Count series per distinct value
limitk, limit_ratioExperimental in 3.x (feature flag)
# RPS per job and route (all other labels dropped)
sum by (job, route) (rate(http_requests_total[5m]))

# same, but keep everything except pod and instance
sum without (pod, instance) (rate(http_requests_total[5m]))

# how many shop pods are being scraped?
count by (job) (up{namespace="shop"})

# how many deployments run 1, 2, 3 … replicas?
count_values("replicas", kube_deployment_spec_replicas{namespace="shop"})

# containers per image in the shop namespace
count by (image) (kube_pod_container_info{namespace="shop"})

without is safer in shared rules: labels added later (e.g. cluster) survive. by is clearer on dashboards.

Aggregation

topk and friends in graphs: a classic surprise

topk is evaluated at every step of a range query, so a graph of "top 5" can show many more than 5 lines.

What happens

  • Step 1: routes A–E are top 5. Step 2: F replaces E.
  • The graph now has 6 series, each with gaps.
  • Same for bottomk and for sort (which has no effect in range queries).

Fixes

  • Pin the ranking at one time: topk(5, … @ end()) then filter with and (below).
  • Use a table or bar gauge panel with an instant query.
  • Rank by a longer window: topk(5, sum by (route) (rate(x[1h]))).
# top 5 routes by traffic, chosen once at the end
sum by (route) (rate(http_requests_total[5m]))
  and on (route)
topk(5, sum by (route) (rate(http_requests_total[1h] @ end())))
Binary operators

Arithmetic, comparison and precedence

Operators work between scalars and vectors. Comparisons filter by default; add bool to get 0/1 instead.

# vector ∘ scalar: applied to every sample
container_memory_working_set_bytes{namespace="shop"} / 1024 / 1024

# comparison filters: only series above 0.5 s remain
job_route:http_request_duration_seconds:p99_5m > 0.5

# bool: keeps all series, value 1 (true) or 0 (false)
job_route:http_request_duration_seconds:p99_5m > bool 0.5

# precedence, highest first
#   ^
#   *  /  %  atan2
#   +  -
#   ==  !=  <=  <  >=  >
#   and  unless
#   or
a + b * c        # = a + (b * c)
2 ^ 3 ^ 2        # ^ is right-associative: 2 ^ 9 = 512

Gotchas

  • Arithmetic between vectors drops the metric name from the result.
  • A filter comparison keeps the left-hand value, not 1.
  • bool is required when comparing two scalars.
  • Division by zero gives +Inf (x/0) or NaN (0/0); every comparison with NaN is false, so a NaN ratio never fires an alert.
  • Use parentheses generously: readers do not remember precedence.
Vector matching

One-to-one matching with on and ignoring

For a / b Prometheus pairs series whose label sets are identical. When they differ, tell it which labels to match on.

# error ratio per job: both sides have exactly {job}
sum by (job) (rate(http_requests_total{code=~"5.."}[5m]))
/
sum by (job) (rate(http_requests_total[5m]))

# without aggregation the left side has code="500",
# the right side has code="200" etc. → no pairs
rate(http_requests_total{code="500"}[5m])
/ ignoring (code)
sum without (code) (rate(http_requests_total[5m]))

# match only on job and route
sum by (job, route, code) (rate(http_requests_total{code=~"5.."}[5m]))
/ on (job, route) group_left
sum by (job, route) (rate(http_requests_total[5m]))

Matching rules

  • Default: labels must match exactly (metric name ignored).
  • on (l1, l2): match only on these labels.
  • ignoring (l1): match on all labels except these.
  • One-to-one: each side may have at most one series per match group, otherwise "many-to-many matching not allowed".
  • The last example keeps code from the left side, so it is many-to-one: it needs group_left (next slide).
  • Series without a partner are silently dropped — check counts.
Vector matching

Many-to-one joins with group_left

The classic use: copy a label from an "info" metric (value always 1) onto a usage metric, e.g. which node a pod runs on.

CPU per pod (many)namespace, pod → valuekube_pod_info (one per pod)namespace, pod, node … = 1Result per podnamespace, pod, node → CPU* on (namespace, pod) group_left (node)multiplying by 1 keeps the value; node is copied from the right side
# CPU cores used per pod, labelled with its node
sum by (namespace, pod) (
  rate(container_cpu_usage_seconds_total{
         namespace="shop", container!=""}[5m])
)
* on (namespace, pod) group_left (node)
  kube_pod_info

# then roll up per node
sum by (node) (
  sum by (namespace, pod) (
    rate(container_cpu_usage_seconds_total{
           container!=""}[5m]))
  * on (namespace, pod) group_left (node)
    kube_pod_info
)

group_left = the left side has many series per match; group_right the reverse. Labels listed in the parentheses are copied from the "one" side. If the "one" side has duplicates (e.g. two kube-state-metrics replicas), the join fails with "found duplicate series" — aggregate it first with max by (namespace, pod, node).

Set operators

and, or, unless: filter by presence, not value

Set operators match label sets like on(); the value always comes from the left side (or whichever side exists, for or).

# alert on error ratio only where there is real traffic
job:http_request_errors:ratio_rate5m > 0.05
  and on (job)
sum by (job) (rate(http_requests_total[5m])) > 1

# series that exist on the left but not on the right:
# deployments with fewer available replicas than desired
kube_deployment_spec_replicas{namespace="shop"}
  unless on (namespace, deployment)
(kube_deployment_spec_replicas == kube_deployment_status_replicas_available)

# fill a missing series with 0 (careful: no labels!)
sum(rate(http_requests_total{code=~"5.."}[5m])) or vector(0)

Semantics

  • a and b: series of a that have a match in b.
  • a unless b: series of a with no match in b.
  • a or b: all of a, plus series of b not matched in a.
  • vector(0) has no labels: or vector(0) only works when the left side is aggregated to no labels too.
  • Use on (…) to restrict which labels decide the match.
Cookbook

RED queries for the shop app

Rate, Errors, Duration per service and route. These become recording rules in ch 4 and dashboard panels in ch 10.

# Rate: requests per second per service and route
sum by (job, route) (rate(http_requests_total{namespace="shop"}[5m]))

# Errors: share of 5xx responses per service
sum by (job) (rate(http_requests_total{namespace="shop", code=~"5.."}[5m]))
/
sum by (job) (rate(http_requests_total{namespace="shop"}[5m]))

# Duration: p99 latency per service and route (classic histogram)
histogram_quantile(0.99,
  sum by (job, route, le) (rate(http_request_duration_seconds_bucket{namespace="shop"}[5m])))

# Duration: mean latency (sum / count)
sum by (job) (rate(http_request_duration_seconds_sum{namespace="shop"}[5m]))
/
sum by (job) (rate(http_request_duration_seconds_count{namespace="shop"}[5m]))

# Business: orders per minute, and charged amount per minute
sum(rate(orders_created_total[5m])) * 60
sum(rate(payments_charge_amount_sum[5m])) * 60

# Saturation of the service's pods: CPU used / CPU requested
sum by (pod) (rate(container_cpu_usage_seconds_total{namespace="shop", container!=""}[5m]))
/
sum by (pod) (kube_pod_container_resource_requests{namespace="shop", resource="cpu"})
Chapter 3 · Part 3.4

Histograms, subqueries and pitfalls

Latency percentiles done right, nested time windows, missing-series checks and query cost.

Classic histograms

How histogram_quantile estimates a percentile

A classic histogram is a set of cumulative counters, one per bucket upper bound le. The quantile is interpolated inside one bucket.

le=0.050.10.250.51+Infcumulative count per bucket (per-second rates)p99p99 falls in bucket (0.25, 0.5]: result assumes a uniform spread inside it
# p99 per route over the last 5 minutes
histogram_quantile(0.99,
  sum by (job, route, le) (
    rate(http_request_duration_seconds_bucket{job="orders"}[5m])
  )
)

# keep le in the by() clause: without it the
# series have no le label and the result is empty

Rules: rate the _bucket series, sum by (…, le), then histogram_quantile. If the quantile lands in the +Inf bucket, the result is the largest finite bound. Averaging percentiles across pods is meaningless; aggregate buckets instead.

Classic histograms

Bucket design and what the error means

The answer can only be as precise as the bucket containing the quantile. Choose buckets around your SLO threshold.

Error and design

  • A p99 between buckets 0.25 and 0.5 can be off by up to 0.25 s.
  • Put a bucket exactly at the SLO threshold (e.g. 0.3 s): then "fraction faster than 0.3 s" is exact.
  • Typical HTTP set: 5 ms … 10 s, roughly ×2–2.5 per step, 10–15 buckets.
  • Every bucket is a series per label combination: buckets × routes × codes × pods adds up (ch 5).
  • Check the boundaries the app really exposes: count by (le) (http_request_duration_seconds_bucket{job="orders"}).
# share of requests faster than 300 ms
# (exact only if a bucket le="0.3" exists)
sum by (job) (rate(
  http_request_duration_seconds_bucket{le="0.3"}[5m]))
/
sum by (job) (rate(
  http_request_duration_seconds_count[5m]))

# Apdex-style score: satisfied ≤ 0.3 s,
# tolerating ≤ 1 s
(
  sum(rate(http_request_duration_seconds_bucket{
        job="orders", le="0.3"}[5m]))
+ sum(rate(http_request_duration_seconds_bucket{
        job="orders", le="1"}[5m]))
) / 2
/ sum(rate(http_request_duration_seconds_count{
        job="orders"}[5m]))
# le="1" includes the satisfied requests,
# so (S + (S + T)) / 2 = S + T / 2
Native histograms

Native histograms in PromQL

Stable since Prometheus 3.8 but opt-in (scrape_native_histograms). One series holds the whole distribution with exponential buckets: no _bucket, no le.

# p99: same function, no _bucket suffix, no le label
histogram_quantile(0.99,
  sum by (job) (rate(http_request_duration_seconds{job="orders"}[5m])))

# request rate and mean latency from the same series
histogram_count(sum by (job) (rate(http_request_duration_seconds[5m])))
histogram_avg(sum by (job) (rate(http_request_duration_seconds[5m])))

# share of requests between 0 and 0.3 s
histogram_fraction(0, 0.3,
  sum by (job) (rate(http_request_duration_seconds[5m])))

# spread
histogram_stddev(sum by (job) (rate(http_request_duration_seconds[5m])))

Why they matter

  • High resolution everywhere, without choosing buckets in advance.
  • Far fewer series than classic histograms with many buckets.
  • histogram_fraction for any threshold, not only existing bucket bounds.
  • The client must expose them (client_golang: NativeHistogramBucketFactor) and Prometheus must scrape them (ch 2).
  • A metric can be exposed in both forms during a migration; dashboards must pick one.
Subqueries

Subqueries: a range over the result of an expression

expr[range:resolution] evaluates expr at each resolution step inside range, producing a range vector you can feed to *_over_time.

# highest 5m request rate seen in the last hour
max_over_time(
  sum by (job) (rate(http_requests_total{namespace="shop"}[5m]))[1h:1m]
)

# was the error ratio above 5% for at least 10 of the last 30 minutes?
count_over_time(
  (job:http_request_errors:ratio_rate5m > 0.05)[30m:1m]
) >= 10

# resolution omitted → global evaluation_interval
avg_over_time(job:http_request_errors:ratio_rate5m[1h:])

Use with care

  • Each subquery runs the inner expression many times: [1h:1m] = 60 evaluations per step of the outer query.
  • Prefer a recording rule for the inner expression, then max_over_time(rule[1h]) reads stored samples.
  • The resolution should be a multiple of the rule's evaluation interval; finer just repeats samples.
  • Do not use a subquery to rate a sum of counters (section 09).
Missing data

Detecting absence, and editing labels

An expression over nothing returns nothing — not zero. Alerts on "error rate > 5%" stay silent when the target vanishes.

# 1 with labels {job="orders"} when no such series exists
absent(up{job="orders"})

# no sample at all in the last 10 minutes
absent_over_time(http_requests_total{job="orders"}[10m])

# target exists but scrape fails
up{namespace="shop"} == 0

# derive a label: instance "10.244.1.7:8080" → ip "10.244.1.7"
label_replace(up{namespace="shop"}, "ip", "$1", "instance", "(.*):.*")

# join two labels into one
label_join(up{namespace="shop"}, "where", "/", "namespace", "pod")

Notes

  • absent() only copies labels from equality matchers in the selector; regex matchers give no labels.
  • Use absent per critical job; up == 0 tells you which instance is down.
  • label_replace(v, dst, replacement, src, regex): anchored regex; if it does not match, the series is returned unchanged.
  • Time helpers: time(), timestamp(v), hour(), day_of_week() (UTC).
  • Example: data staleness in seconds = time() - timestamp(up{job="orders"}).
Query cost

Why a query is slow, and how to make it cheap

Cost ≈ series selected × samples per series × steps. Dashboards multiply it by panels and viewers.

Expensive patterns

  • Selectors without a narrow label: rate(http_requests_total[5m]) over the whole cluster.
  • Long ranges on high-cardinality metrics: [7d] on per-pod buckets.
  • Subqueries with fine resolution.
  • Unanchored-looking regexes like =~".*orders.*" on many series.
  • Dozens of panels each recomputing the same p99.

Cheaper patterns

  • Narrow first: job, namespace, route.
  • Recording rules for anything used in more than one place (ch 4).
  • Aggregate away pod and instance early.
  • Use $__rate_interval and sensible panel resolutions.
  • Check series counts before shipping a dashboard.

Where to look

  • Prometheus flags --query.timeout (default 2m) and --query.max-samples (default 50,000,000).
  • Error "query processing would load too many samples into memory" = max-samples hit.
  • Query log: query_log_file in the global config.
  • Metrics: prometheus_engine_query_duration_seconds.
  • Cardinality analysis is covered in ch 5.
Common mistakes

PromQL mistakes that produce confident wrong answers

Each of these returns a number. That is what makes them dangerous.

Counters and rates

  • sum then rate (false spikes on every restart).
  • rate on a gauge (decreases read as resets).
  • [30s] window with a 30s scrape interval → empty or flickering results.
  • irate in alerts.
  • Expecting integers from increase.

Aggregation and joins

  • Averaging ratios or percentiles across pods.
  • Dropping le before histogram_quantile → empty result.
  • group_left against a metric that has duplicates.
  • Joins that silently drop series without a partner.
  • topk in a range graph showing more than k lines.

Missing data

  • Alerting on error_ratio > 0.05 with no traffic: 0/0 = NaN, the alert never fires and never tells you why.
  • No absent() / up == 0 alert: a dead target looks healthy.
  • or vector(0) on a labelled expression.
  • Dashboards that show "No data" and are read as "no errors".
Self-check

PromQL: can you answer these?

Write the query first, then explain why it is correct.

Selectors and counters

  • What does {job=~"orders"} match, and why is {job=~".*"} rejected?
  • What changed about range boundaries in Prometheus 3?
  • Why does increase() return 59.8 instead of 60?
  • How long should the rate window be with a 30s scrape interval?
  • Why must rate come before sum?

Aggregation and matching

  • When do you prefer without over by?
  • Write the 5xx ratio per job and route.
  • Copy the node label onto per-pod memory usage.
  • What does a > bool 0.5 return, compared to a > 0.5?
  • Why can a "top 5" graph show 8 lines?

Histograms and pitfalls

  • Write p99 latency per route from a classic histogram.
  • Where should bucket boundaries be, and why?
  • How do the queries change for native histograms?
  • What does a subquery cost, and what replaces it?
  • Which query tells you a job has disappeared?

Practice: Lab 3 in Hands-on labs — RED queries on the shop app via the HTTP API, a node join, injected errors and latency, and the sum-then-rate trap during a restart.

Chapter 4

Recording rules, alerting and Alertmanager

Precompute what dashboards and alerts need, write alerts that wake people only for real symptoms, route them with Alertmanager, and test all of it before it ships.

Chapter 4 · Part 4.1

Recording rules

Store the result of an expensive or shared expression as a new series.

Recording rules

Why precompute

A recording rule evaluates a PromQL expression on a schedule and writes the result back to the TSDB as a new metric.

Speed

  • A dashboard p99 over 30 days reads one series per route instead of every bucket of every pod.
  • Alert evaluation becomes cheap and predictable.
  • Subqueries over rules read stored samples instead of recomputing.

Consistency

  • One definition of "error ratio" for dashboards, alerts and SLO reports.
  • Changing the definition happens in one place, under review.
  • Names document the aggregation level and window.

Building blocks

  • Layer rules: per-route rate → per-job ratio → SLO windows.
  • Federation or remote write can ship only the aggregated rules (ch 5).
  • Cost: each rule creates series; aggregate before recording, never record per-pod data you do not need.
Rule files

Anatomy of a rules file

Rules live in groups. Groups run in parallel; rules inside a group run in order, so later rules can use earlier results from the same evaluation.

# rules/shop.rules.yml
groups:
  - name: shop-red
    interval: 30s     # default: global evaluation_interval
    rules:
      - record: job_route:http_requests:rate5m
        expr: |
          sum by (job, route) (
            rate(http_requests_total{namespace="shop"}[5m]))
      - record: job:http_requests:rate5m
        expr: sum by (job) (job_route:http_requests:rate5m)
      - record: job:http_request_errors:rate5m
        expr: |
          sum by (job) (rate(
            http_requests_total{namespace="shop", code=~"5.."}[5m]))
      - record: job:http_request_errors:ratio_rate5m
        expr: job:http_request_errors:rate5m / job:http_requests:rate5m
      - record: job_route:http_request_duration_seconds:p99_5m
        expr: |
          histogram_quantile(0.99, sum by (job, route, le) (rate(
            http_request_duration_seconds_bucket{namespace="shop"}[5m])))
        labels:
          quantile: "0.99"   # static labels added to the result

Fields

  • groups[].name must be unique per file.
  • interval: evaluation interval for the group.
  • limit: max series a rule may produce (0 = no limit); exceeding it fails the rule.
  • record: the new metric name (must be a valid metric name).
  • expr: any PromQL expression returning an instant vector.
  • labels: added or overwritten on every result series.
  • Loaded via rule_files: in prometheus.yml, or from PrometheusRule objects with the Operator (section 14).
Naming

The level:metric:operations convention

From the Prometheus docs: the name tells you which labels remain, which metric it came from, and what was done to it.

PartMeaningExample
levelLabels kept by the aggregation, joined by _job_route, job, namespace
metricSource metric name; drop _total after ratehttp_requests
operationsOperations applied, newest firstrate5m, ratio_rate5m, p99_5m

Examples

  • job_route:http_requests:rate5m — sum by (job, route) (rate(…[5m])).
  • job:http_request_errors:ratio_rate5m — 5xx share per job over 5m.
  • namespace_pod:container_cpu_usage:rate5m — CPU per pod.
  • job:http_request_duration_seconds:p99_5m — p99 per job.
  • Keep the unit in the name (_seconds, _bytes).

Colons are reserved for recording rules: never use them in metric names you instrument.

Layering

Aggregate correctly when you build on rules

Rules are reused far from where they were written. The two common errors are averaging ratios and re-aggregating percentiles.

# RIGHT: per-job ratio from per-job sums
job:http_request_errors:ratio_rate5m =
  job:http_request_errors:rate5m / job:http_requests:rate5m

# WRONG: average of per-route ratios
avg by (job) (job_route:http_request_errors:ratio_rate5m)
# a route with 1 request/hour weighs as much as /orders

# RIGHT: per-job p99 from buckets
histogram_quantile(0.99,
  sum by (job, le) (rate(http_request_duration_seconds_bucket[5m])))

# WRONG: average or max of per-route p99s
avg by (job) (job_route:http_request_duration_seconds:p99_5m)

# Longer windows: build from rates, not from ratios
job:http_request_errors:ratio_rate1h =
  sum by (job) (rate(http_requests_total{code=~"5.."}[1h]))
  / sum by (job) (rate(http_requests_total[1h]))

Guidelines

  • Record numerators and denominators separately; derive ratios from them.
  • Keep le in recorded bucket rates if you need percentiles at a higher level.
  • Put dependent rules in the same group, in order, to avoid one-cycle lag.
  • Rules that depend on another group see its result from the previous evaluation.
  • SLO windows (5m, 30m, 1h, 6h, 1d, 3d) belong to the SRE deck; the same patterns apply.
Chapter 4 · Part 4.2

Alerting rules and alert design

An alert is a PromQL condition plus a duration, labels for routing and annotations for the human.

Alerting rules

Anatomy of an alerting rule

Every series returned by expr is one alert instance. No series → no alert.

groups:
  - name: shop-alerts
    rules:
      - alert: ShopHighErrorRate
        expr: |
          job:http_request_errors:ratio_rate5m{job=~"orders|payments"}
            > 0.05
          and on (job) job:http_requests:rate5m > 0.5
        for: 5m
        keep_firing_for: 5m
        labels:
          severity: critical
          team: shop
        annotations:
          summary: >-
            {{ $labels.job }}: {{ $value | humanizePercentage }}
            of requests fail
          description: >-
            5xx ratio above 5% for 5 minutes on {{ $labels.job }}.
            Check payments FAIL_RATE and recent deploys.
          runbook_url: >-
            https://runbooks.example.internal/shop/high-error-rate

Fields

  • alert: name, becomes label alertname.
  • expr: condition; the value is available as $value.
  • for: how long the condition must hold before firing (pending state).
  • keep_firing_for: keep firing this long after the condition clears (avoids flapping resolves; Prometheus ≥ 2.42).
  • labels: used by Alertmanager for routing, grouping and inhibition — keep them few and stable.
  • annotations: text for humans; templated; never used for routing.
  • The and on (job) … > 0.5 guard stops alerts on 1 failed request out of 2.
Lifecycle

Inactive → pending → firing → resolved

Prometheus evaluates the rule every interval and sends active alerts to Alertmanager repeatedly; Alertmanager decides whom to notify and when.

inactiveexpr returns nothingpendingexpr true, < forfiringexpr true ≥ forkeep firingexpr false < keep_firing_forresolvedsent oncetrueheld forfalseelapsedfalse before for elapsed → resettrue again → still firingWhat you can queryALERTS{alertname="ShopHighErrorRate", alertstate="pending" | "firing", …labels} = 1 while activeALERTS_FOR_STATE: when the alert became active (restores the for-timer after a Prometheus restart)

Effective delay to the first notification ≈ rule interval + for + Alertmanager group_wait. With 30s + 5m + 30s, expect ~6 minutes. That is a design decision, not an accident.

Templating

Annotations a tired on-call engineer can act on

Annotations use Go templates. Give the number, the scope, the likely cause and a link.

annotations:
  summary: >-
    p99 {{ $value | humanizeDuration }}
    on {{ $labels.job }} {{ $labels.route }}
  description: |
    p99 latency above 500 ms for 10 minutes.
    Cluster: {{ $externalLabels.cluster }}
    Dashboard: https://grafana.example.internal/d/shop-red
      ?var-job={{ $labels.job }}
  runbook_url: >-
    https://runbooks.example.internal/shop/{{ $labels.alertname }}

# rendered:
#   summary: p99 742ms on payments /charge

Available in templates

  • $labels (or .Labels): labels of the alert instance.
  • $value (or .Value): value of expr for this instance.
  • $externalLabels, $externalURL.
  • Functions: humanize, humanize1024, humanizeDuration, humanizePercentage, humanizeTimestamp, printf, query.
  • $value is the value at the current evaluation; it changes over time — do not put it in labels (it would create a new alert every evaluation).
Alert design

Page on symptoms, ticket on causes

Users feel symptoms (errors, latency, unavailability). Causes (CPU, disk, a pod restart) matter only when they lead to symptoms or will soon.

AlertKindSeverity / action
5xx ratio > 5% for 5m on ordersSymptomcritical → page now
p99 latency > 500 ms for 10mSymptomcritical or warning, depending on the SLO
up == 0 for all orders podsSymptom (total outage of a dependency)critical → page
Disk predicted full within 4 hImminent causewarning → page in working hours or ticket
CPU > 90% on one nodeCauseDashboard, not an alert (unless it breaks something)
One pod restarted onceCauseNothing; repeated CrashLoopBackOff → warning ticket
Prometheus cannot reach AlertmanagerMeta-monitoring (ch 13)critical via an independent path

Every page must be urgent, actionable and need a human. If the response is "wait and see", it is a ticket. Paging policy and SLO-based paging are covered in the SRE deck.

Alert design

Against flapping and noise

A noisy alert trains people to ignore the pager. These knobs remove most noise without hiding real problems.

Duration and windows

  • for: filters short spikes; typical 2–15 m.
  • Use a rate window ≥ for / 2, so the condition is smooth.
  • keep_firing_for: prevents resolve → fire → resolve cycles.
  • Hysteresis: fire at 5%, but evaluate the clear condition separately only if needed (a second rule with unless).

Guards

  • Minimum traffic: and on (job) job:http_requests:rate5m > 0.5.
  • Exclude expected states: unless on (namespace) kube_namespace_labels{label_maintenance="true"} (needs that label exported).
  • Alert on aggregates (job), not on every pod, unless the pod matters on its own.
  • Alertmanager grouping and inhibition (section 14) for one notification per incident.

Coverage you need anyway

  • up == 0 per job, and absent(up{job="orders"}) when the target disappears entirely.
  • Scrape errors: up plus scrape_samples_scraped == 0.
  • Rule evaluation failures: prometheus_rule_evaluation_failures_total (ch 13).
  • Watchdog (always firing) to prove the pipeline works end to end.
Shop alerts

A complete alert set for the shop app

Three symptom alerts and one presence alert. Lab 4 loads a trimmed version of this set (no ShopTargetDown) as a PrometheusRule.

- alert: ShopHighErrorRate
  expr: job:http_request_errors:ratio_rate5m{job=~"orders|payments"} > 0.05 and on (job) job:http_requests:rate5m > 0.5
  for: 5m
  labels: {severity: critical, team: shop}
  annotations: {summary: "{{ $labels.job }}: {{ $value | humanizePercentage }} 5xx"}

- alert: ShopHighLatencyP99
  expr: |
    histogram_quantile(0.99, sum by (job, le) (rate(http_request_duration_seconds_bucket{job=~"orders|payments"}[5m]))) > 0.5
  for: 10m
  labels: {severity: warning, team: shop}
  annotations: {summary: "{{ $labels.job }} p99 {{ $value | humanizeDuration }}"}

- alert: ShopTargetDown
  expr: up{job=~"orders|payments"} == 0
  for: 2m
  labels: {severity: warning, team: shop}
  annotations: {summary: "{{ $labels.job }} pod {{ $labels.pod }} not scraped"}

- alert: ShopJobAbsent
  expr: absent(up{job="orders"})
  for: 5m
  labels: {severity: critical, team: shop, namespace: shop}  # absent() has no namespace
  annotations: {summary: "No orders targets at all: deployment or ServiceMonitor gone"}
Preview

Where this goes next: burn-rate alerts

Threshold alerts ("5% errors") ignore how much failure your SLO can afford. Burn-rate alerts page on how fast the error budget is being spent.

The idea

  • SLO 99.9% → error budget 0.1% of requests over 30 days.
  • Burn rate = observed error ratio ÷ budget ratio. Burn rate 1 spends the budget in exactly 30 days.
  • Burn rate 14.4 for 1 hour spends 2% of a 30-day budget (14.4 × 1 h ÷ 720 h).
  • Multi-window: require both a long (1h) and a short (5m) window to be burning, so the alert also clears quickly.
# page: 2% of the 30-day budget in 1 hour (SLO 99.9%)
(
  job:http_request_errors:ratio_rate1h{job="orders"} > (14.4 * 0.001)
and
  job:http_request_errors:ratio_rate5m{job="orders"} > (14.4 * 0.001)
)

SLIs, SLOs, error budgets and the full multiwindow, multi-burn-rate alert table are covered in the SRE deck. The recording rules from section 12 are its inputs.

Chapter 4 · Part 4.3

Alertmanager

Deduplicate, group, route, inhibit and silence alerts, then deliver notifications.

Alertmanager

What happens between "firing" and a message in Slack

Prometheus only says "these alerts are active". Alertmanager turns that stream into a few notifications for the right people.

Prometheus Arule evaluationPrometheus BHA replicaAlertmanager (cluster of 2–3, gossip on :9094)Dedupsame labels = 1Inhibitmute by ruleSilencemute by humanRouterouting treeGroup + waitgroup_by, timersSlackPagerDutyWebhookEach Prometheus sends to every Alertmanager instance directly (no load balancer); the cluster dedups notifications.

Prometheus re-sends active alerts periodically and sets an end time; if Alertmanager stops hearing about an alert, it resolves it. That is why a dead Prometheus looks like "all clear" — hence Watchdog (ch 13).

Grouping

Grouping and the three timers

Alerts with equal values for the group_by labels go into one notification. Timers decide how quickly and how often that group is (re)sent.

route:
  receiver: default
  group_by: [alertname, namespace, job]
  group_wait: 30s       # wait for more alerts before the first message
  group_interval: 5m    # min time before sending an updated group
  repeat_interval: 4h   # re-send an unchanged, still-firing group

# group_by: ['...'] disables grouping (one message per alert)

Timers

TimerDefaultEffect
group_wait30sFirst notification of a new group is delayed to collect siblings
group_interval5mNew or resolved alerts in an existing group wait at most this long
repeat_interval4hReminder for a group with no changes; set ≥ group_interval

Example: 3 orders pods go down within 20 s → one message listing 3 alerts after ~30 s. A 4th pod 2 minutes later → an updated message at the next group_interval tick, not immediately.

Routing

The routing tree

Every alert enters at the root and walks down. The first matching child wins unless it sets continue: true. Children inherit settings they do not override.

route:
  receiver: default-slack     # fallback for everything
  group_by: [alertname, namespace]
  routes:
    - matchers: [alertname = "Watchdog"]
      receiver: deadmans-switch
      repeat_interval: 1m
    - matchers: [team = "shop", severity = "critical"]
      receiver: shop-pager
      group_wait: 10s
      continue: true          # also try the next routes
    - matchers: [team = "shop"]
      receiver: shop-slack
      routes:
        - matchers: [severity = "info"]
          receiver: "null"    # drop
    - matchers: [namespace =~ "kube-system|monitoring"]
      receiver: platform-slack

receivers:
  - name: "null"   # no configs = drop
  # … other receivers on the next slide

How matching works

  • matchers: list of label op "value" with =, !=, =~, !~; all must match (AND).
  • The older match / match_re fields are deprecated; use matchers.
  • Depth-first: an alert goes to the deepest matching node.
  • No child matches → the parent node handles it.
  • Test routing before deploying: amtool config routes test (section 15).
  • The shop critical alert above goes to shop-pager and shop-slack.
Receivers

Receivers: where notifications go

A receiver is a named list of integrations. One receiver can notify several systems.

receivers:
  - name: shop-slack
    slack_configs:
      - api_url_file: /etc/alertmanager/secrets/slack-url
        channel: '#shop-alerts'
        send_resolved: true
        title: '{{ template "slack.shop.title" . }}'
        text: '{{ template "slack.shop.text" . }}'

  - name: shop-pager
    pagerduty_configs:
      - routing_key_file: /etc/alertmanager/secrets/pd-key   # Events API v2

  - name: lab-webhook
    webhook_configs:
      - url: http://alert-sink.monitoring.svc:8080/
        send_resolved: true

  - name: oncall-mail
    email_configs:
      - to: shop-oncall@example.com   # needs global smtp_* settings

Notes

  • Prefer *_file fields (api_url_file, routing_key_file) to keep secrets out of the config.
  • send_resolved defaults differ: true for webhook and PagerDuty, false for Slack and email.
  • Webhook receives JSON: status, alerts[] (labels, annotations, startsAt, endsAt), groupLabels, commonLabels, externalURL.
  • Other integrations include Opsgenie, Microsoft Teams, Telegram, SNS, Webex, Discord.
  • Lab 4 uses the webhook receiver with a tiny echo service.
Inhibition and silences

Muting: by rule (inhibition) and by hand (silences)

Inhibition suppresses alerts that are consequences of another firing alert. Silences mute matching alerts for a time window, with an author and a comment.

inhibit_rules:
  # if the whole job is gone, don't also send latency/error alerts
  - source_matchers: [alertname = "ShopJobAbsent"]
    target_matchers: [alertname != "ShopJobAbsent"]
    equal: [team]

  # critical hides warning of the same alert on the same job
  - source_matchers: [severity = "critical"]
    target_matchers: [severity = "warning"]
    equal: [alertname, job]
export AM=http://localhost:9093

# silence during planned work (default duration 1h)
amtool silence add --alertmanager.url=$AM \
  --author="$USER" --duration=2h \
  --comment="payments DB migration CHG-1234" \
  job=payments

amtool silence query --alertmanager.url=$AM
amtool silence expire --alertmanager.url=$AM <silence-id>

# what is firing right now?
amtool alert query --alertmanager.url=$AM team=shop

equal matters: without it, any critical alert anywhere would inhibit every warning in the company. Silences are stored in Alertmanager (replicated in the cluster) and expire on their own — always set a short duration and a comment.

Time intervals

Mute or activate routes on a schedule

Named time intervals (top-level time_intervals) can mute a route (mute_time_intervals) or make it active only then (active_time_intervals).

time_intervals:
  - name: offhours
    time_intervals:
      - weekdays: ['saturday', 'sunday']
        location: Asia/Tokyo
      - times:
          - start_time: '19:00'
            end_time: '24:00'
          - start_time: '00:00'
            end_time: '09:00'
        location: Asia/Tokyo

route:
  routes:
    - matchers: [team = "shop", severity = "warning"]
      receiver: shop-slack
      mute_time_intervals: [offhours]    # warnings wait for office hours
    - matchers: [team = "shop", severity = "critical"]
      receiver: shop-pager               # always active

Notes

  • Fields: times, weekdays, days_of_month, months, years, location (IANA time zone; default UTC).
  • Multiple entries in one interval are OR-ed; fields inside one entry are AND-ed.
  • Muted alerts are not lost: they are still active and are sent when the mute ends if still firing.
  • The older top-level key mute_time_intervals still works but time_intervals is preferred.
  • Silences are for one-off events; time intervals for recurring schedules.
Templates

Notification templates

Default messages are generic. A short template with counts, summaries and links makes alerts readable on a phone.

{{/* /etc/alertmanager/config/shop.tmpl */}}
{{ define "slack.shop.title" -}}
[{{ .Status | toUpper }}
{{- if eq .Status "firing" }}:{{ .Alerts.Firing | len }}{{ end }}]
{{ .CommonLabels.alertname }} · {{ .CommonLabels.job }}
{{- end }}

{{ define "slack.shop.text" -}}
{{ range .Alerts -}}
• {{ .Annotations.summary }}
{{- with .Annotations.runbook_url }} <{{ . }}|runbook>{{ end }}
{{ end -}}
{{- end }}

# alertmanager.yml
templates:
  - /etc/alertmanager/config/*.tmpl

Template data

  • .Status: firing or resolved for the group.
  • .Alerts (with .Firing and .Resolved), each with .Labels, .Annotations, .StartsAt, .GeneratorURL.
  • .GroupLabels, .CommonLabels, .CommonAnnotations, .ExternalURL.
  • Functions include toUpper, join, title, safeHtml.
  • Test rendering with amtool template render before shipping.
High availability

Running Alertmanager highly available

Run 2–3 replicas as a cluster. They gossip notification logs and silences, so a group is notified once even though every replica receives every alert.

How it works

  • Peers connect with --cluster.peer=<host>:9094 (TCP/UDP gossip).
  • Each Prometheus sends every alert to all replicas.
  • Replicas wait a position-based delay before notifying; the first one records it in the shared log, others skip.
  • Silences are replicated to all peers.

Failure modes

  • Load balancer in front of Alertmanager → each replica sees a subset of alerts: broken grouping.
  • Network partition → possible duplicate notifications (Alertmanager prefers duplicates over losing them).
  • One replica only → a restart drops in-flight groups and timers.

In kube-prometheus-stack

  • alertmanager.alertmanagerSpec.replicas: 3 creates the cluster; the Operator wires peers.
  • Prometheus discovers all Alertmanager pods through the Operator.
  • Storage for silences: a PVC (storage:) so silences survive restarts.
On Kubernetes

Rules and Alertmanager config with the Prometheus Operator

In kube-prometheus-stack, rules are PrometheusRule objects and Alertmanager config comes from Helm values or AlertmanagerConfig objects (ch 5).

apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: shop-rules
  namespace: shop
  labels:
    release: kps      # matched by the default ruleSelector
spec:
  groups:
    - name: shop-red
      rules:
        - record: job:http_requests:rate5m
          expr: |
            sum by (job) (
              rate(http_requests_total{namespace="shop"}[5m]))
    - name: shop-alerts
      rules:
        - alert: ShopTargetDown
          expr: up{job=~"orders|payments"} == 0
          for: 2m
          labels: {severity: warning, team: shop}

What the chart gives you

  • A large default alert set (kubernetes-*, node-*, prometheus-*, alertmanager-*) using severity critical/warning/info.
  • Watchdog: always firing, route it to a dead man's switch.
  • InfoInhibitor: suppresses info alerts unless a warning/critical fires in the same namespace.
  • Alertmanager config: alertmanager.config in values (a full alertmanager.yml).
  • AlertmanagerConfig CRs (namespaced): the Operator adds a namespace matcher so teams only route their own alerts; selected by alertmanagerConfigSelector.
  • Without release: kps, a PrometheusRule is silently ignored.
Chapter 4 · Part 4.4

Testing rules and routing

Rules are code: lint them, unit-test them, and test routing before anyone gets paged by a typo.

Validation

Static checks with promtool and amtool

Fast checks that catch syntax errors, bad PromQL, duplicate rules and broken Alertmanager config.

# rules: YAML structure + PromQL parse + duplicate rules
promtool check rules rules/shop.rules.yml
# Checking rules/shop.rules.yml
#   SUCCESS: 9 rules found

# lint options: duplicate-rules (default), all, none
promtool check rules --lint=all rules/shop.rules.yml

# full Prometheus config including referenced rule files
promtool check config prometheus.yml

# Alertmanager config + templates
amtool check-config alertmanager.yml
# Checking 'alertmanager.yml'  SUCCESS
# Found:
#  - global config
#  - route
#  - 1 inhibit rules
#  - 4 receivers
#  - 1 templates

What they do not catch

  • A query that parses but selects nothing (wrong label value).
  • A threshold that never fires, or always fires.
  • An alert that routes to the wrong receiver.
  • A rule that is valid but never loaded (missing release: kps label).
  • → unit tests (next slides) and checking Status → Rules in the UI.
Unit tests

promtool test rules: feed synthetic series, assert results

Tests run the real rule engine against series you define, at times you choose. No Prometheus server needed.

# rules/shop.test.yml
rule_files:
  - shop.rules.yml
evaluation_interval: 1m

tests:
  - interval: 1m
    input_series:
      # 6 errors/min and 54 ok/min → 10% errors, 1 req/s
      - series: >-
          http_requests_total{namespace="shop",job="orders",code="500"}
        values: '0+6x20'
      - series: >-
          http_requests_total{namespace="shop",job="orders",code="200"}
        values: '0+54x20'
    alert_rule_test:
      - eval_time: 3m          # still pending (for: 5m)
        alertname: ShopHighErrorRate
        exp_alerts: []
      - eval_time: 10m
        alertname: ShopHighErrorRate
        exp_alerts:
          - exp_labels:
              {job: orders, severity: critical, team: shop}
            # a rule with annotations also needs exp_annotations

How to read it

  • interval: spacing of the input samples.
  • values uses expanding notation (next slide).
  • promql_expr_test (not shown; used in Lab 4) checks any expression at an eval_time against exp_samples.
  • alert_rule_test checks which alerts are firing (pending alerts are not listed).
  • exp_labels must list all labels of the alert, including those from expr and labels:; alertname is implied.
  • Annotations are compared too: without exp_annotations the test expects none, so list the rendered ones (exp_annotations: {summary: "…"}).
  • Run: promtool test rules rules/shop.test.yml → SUCCESS.
Unit tests

Series notation and test patterns

A compact language to describe counters, gaps and staleness.

values notation

NotationExpands to
0+6x40 6 12 18 24 (start, +step, 4 more times)
100-10x3100 90 80 70
1x31 1 1 1 (shorthand for 1+0x3)
1 2 _ 4_ = no sample at that step
_x3three missing samples
1 2 stalestaleness marker: series ends
0+6x10 0+6x10counter reset in the middle

What to test for every alert

  • Fires when it should (after for).
  • Does not fire while pending, and for short spikes.
  • Does not fire with too little traffic (guard clause).
  • Absence: drop the series with _x10 and expect ShopJobAbsent.
  • Counter reset in the middle does not cause a false fire.
  • Labels match what routing expects (team, severity).
Routing tests

Test the routing tree and run everything in CI

Ask Alertmanager which receiver a label set would reach, then automate all checks on every pull request.

# show the tree
amtool config routes --config.file=alertmanager.yml

# which receivers get a critical shop alert?
amtool config routes test --config.file=alertmanager.yml \
  team=shop severity=critical alertname=ShopHighErrorRate
# shop-pager,shop-slack

# fail if the result differs from what you expect
amtool config routes test --config.file=alertmanager.yml \
  --verify.receivers=shop-pager,shop-slack \
  team=shop severity=critical
# .github/workflows/monitoring.yml (excerpt)
jobs:
  rules:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@<sha>
      - name: install promtool and amtool
        run: |
          # pin exact versions; download from the GitHub releases
          ./ci/install-prom-tools.sh
      - run: promtool check rules rules/*.rules.yml
      - run: promtool test rules rules/*.test.yml
      - run: amtool check-config alertmanager/alertmanager.yml
      - run: ./ci/test-routes.sh   # amtool routes test cases

With the Operator, keep rules in plain rule files in Git, test them, and generate the PrometheusRule wrapper (Lab 4 does this with a heredoc). Tests on the CR YAML itself need the spec extracted first.

Common mistakes

Alerting mistakes that cost sleep or hide outages

Most bad alerting is not missing alerts: it is alerts nobody trusts.

Rules

  • $value or a timestamp in alert labels → a new alert every evaluation.
  • No for: → pages on one bad scrape.
  • Ratio alerts without a traffic guard (or with NaN when traffic is zero) → either noise or silence.
  • No absent() / up == 0 alerts for the jobs you care about.
  • Averaging ratios or percentiles in recording rules.

Alertmanager

  • Load balancer in front of the Alertmanager cluster.
  • group_by too fine (a message per pod) or too coarse (everything in one).
  • Inhibit rules without equal:.
  • Silences without expiry discipline; "temporary" 30-day silences.
  • No Watchdog route: a dead pipeline looks like a quiet week.

Process

  • Rules changed in the UI or by hand, not in Git.
  • No unit tests; first test is a real incident.
  • PrometheusRule without release: kps → never loaded, nobody notices.
  • Alerts without runbook links or owners (team label).
  • Paging on causes (CPU, restarts) instead of symptoms.
Self-check

Rules and alerting: can you answer these?

Answer with the exact field, label or command.

Recording rules

  • What does job_route:http_requests:rate5m tell you about its expression?
  • Why record numerator and denominator separately?
  • In which order are rules in one group evaluated, and why does it matter?
  • Why is avg of per-route p99s wrong?

Alerting rules

  • What are the alert states, and what moves an alert between them?
  • What does keep_firing_for prevent?
  • Why must $value never be in labels?
  • Estimate the delay from first error to first notification.
  • Which alert fires when the orders Deployment is deleted?

Alertmanager and tests

  • What do group_wait, group_interval, repeat_interval control?
  • When does an alert reach two receivers?
  • Why must equal: be set on inhibit rules?
  • What does 0+6x20 expand to?
  • Which command proves a critical shop alert reaches shop-pager?

Practice: Lab 4 in Hands-on labs — recording rules and alerts for the shop app tested with promtool, loaded as a PrometheusRule, routed by Alertmanager to a webhook sink, then broken with a missing label, inhibition and a silence.

Chapter 5

Prometheus on Kubernetes and at scale

kube-prometheus-stack and the Operator CRDs, the metrics Kubernetes gives you for free, keeping cardinality under control, and running Prometheus highly available with long-term storage.

Chapter 5 · Part 5.1

kube-prometheus-stack and the Operator

One Helm release that installs a full monitoring stack, and CRDs that turn scrape config into Kubernetes objects.

kube-prometheus-stack

What one Helm release installs

The chart prometheus-community/kube-prometheus-stack bundles the Prometheus Operator, its CRDs and a ready-to-use stack. In the labs the release is kps in namespace monitoring.

Workloads

  • Prometheus Operator: watches CRDs and generates config.
  • Prometheus (StatefulSet managed by the Operator).
  • Alertmanager (StatefulSet managed by the Operator).
  • Grafana (subchart) with Prometheus as default data source.
  • kube-state-metrics (Deployment) and node-exporter (DaemonSet).

Content

  • ServiceMonitors for the API server, kubelet/cAdvisor, CoreDNS, kube-state-metrics, node-exporter, the Operator, Prometheus and Alertmanager.
  • PrometheusRules: recording and alerting rules from the kubernetes-mixin and node-mixin.
  • Grafana dashboards as ConfigMaps (cluster, namespace, pod, node views).
  • Watchdog alert: always firing, used as a dead man's switch (ch 13).

Names with release kps

  • Prometheus service: kps-kube-prometheus-stack-prometheus.
  • Alertmanager service: kps-kube-prometheus-stack-alertmanager.
  • Grafana service: kps-grafana.
  • Pod: prometheus-kps-kube-prometheus-stack-prometheus-0.
  • Check with kubectl -n monitoring get svc,prometheus,alertmanager.

Chart names are derived from the release name and truncated; if you pick another release name, list the services instead of guessing them.

Operator

How the Operator turns objects into a running Prometheus

You never edit prometheus.yml. You create Kubernetes objects; the Operator renders the config, stores it in a Secret and the config-reloader sidecar reloads Prometheus.

You (Git, Helm, kubectl)ServiceMonitor / PodMonitorwhat to scrapePrometheusRulerecording + alerting rulesPrometheus CRreplicas, retention, selectorsAlertmanagerConfig / Alertmanager CRrouting, receiversPrometheus Operatorwatch + reconcileNamespace monitoringSecretprometheus.yaml.gzConfigMapsrule filesPrometheus podprometheus + config-reloaderStatefulSet prometheus-kps-…created and updated by the Operator

The reloader calls Prometheus's /-/reload endpoint after the generated config changes, so most changes apply without a restart. Invalid rules are rejected by the Operator's admission webhook (if enabled) or reported in the Operator logs.

CRDs

The CRDs and what each one replaces

All are in the API group monitoring.coreos.com. Most are v1; PrometheusAgent and ScrapeConfig are v1alpha1, AlertmanagerConfig is v1alpha1/v1beta1 — check kubectl api-resources.

CRDReplaces in plain PrometheusTypical owner
PrometheusThe server Deployment/StatefulSet, flags, global configPlatform team
PrometheusAgentA Prometheus in agent mode (scrape + remote write only)Platform team
AlertmanagerThe Alertmanager clusterPlatform team
ServiceMonitorA scrape_config that discovers targets through a Service's EndpointsApp team
PodMonitorA scrape_config that discovers Pods directly (no Service needed)App team
ProbeA blackbox_exporter scrape of URLs or IngressesApp or platform team
ScrapeConfigAny raw scrape config: static targets, file, HTTP, EC2, Consul SD…Platform team
PrometheusRuleA rule file (recording + alerting rules)App team
AlertmanagerConfigA namespaced sub-tree of the Alertmanager routing configApp team
ThanosRulerThanos Ruler instances (rules over Thanos Query)Platform team
Selectors

Why your ServiceMonitor is ignored: the release label

The Prometheus object selects monitors and rules by label. The chart's default makes it select only objects labeled release: <release name>.

# values.yaml (kube-prometheus-stack) — defaults
prometheus:
  prometheusSpec:
    serviceMonitorSelectorNilUsesHelmValues: true
    podMonitorSelectorNilUsesHelmValues: true
    ruleSelectorNilUsesHelmValues: true
    probeSelectorNilUsesHelmValues: true
    scrapeConfigSelectorNilUsesHelmValues: true
# → rendered Prometheus CR:
#   serviceMonitorSelector:
#     matchLabels: {release: kps}
#   serviceMonitorNamespaceSelector: {}   # all namespaces

# inspect what YOUR Prometheus selects
kubectl -n monitoring get prometheus \
  kps-kube-prometheus-stack-prometheus \
  -o jsonpath='{.spec.serviceMonitorSelector}{"\n"}'
# {"matchLabels":{"release":"kps"}}

Two ways to make it work

  • Label every object: metadata.labels.release: kps on each ServiceMonitor, PodMonitor and PrometheusRule (what the labs do).
  • Select everything: set the …NilUsesHelmValues values to false → empty selectors → every monitor in every namespace is picked up.
  • Multi-tenant clusters: use a team label instead (serviceMonitorSelector: {matchLabels: {prometheus: shared}}) and restrict namespaces with serviceMonitorNamespaceSelector.
  • Debug order: label matches? → namespace selected? → Service selector matches pods? → port name matches? → target in Status → Targets.
ServiceMonitor

ServiceMonitor for orders: every field that matters

A ServiceMonitor selects Services; Prometheus then scrapes each ready endpoint (pod IP) behind them. port refers to the Service port name, not a number.

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: orders
  namespace: shop
  labels:
    release: kps                 # picked up by kps
spec:
  selector:
    matchLabels: {app: orders}   # Service labels
  namespaceSelector:
    matchNames: [shop]           # default: own namespace
  endpoints:
    - port: http                 # Service port NAME
      path: /metrics
      interval: 30s
      scrapeTimeout: 10s
      metricRelabelings:         # after scrape
        - sourceLabels: [__name__]
          regex: go_gc_.*
          action: drop
  sampleLimit: 5000              # fail scrape above this
  jobLabel: app                  # job = Service label "app"

What Prometheus gets

  • Job name serviceMonitor/shop/orders/0 internally; the job label is the Service name unless jobLabel is set.
  • Target labels added: namespace, service, pod, container, endpoint, instance (pod IP:port).
  • relabelings run before the scrape (on discovered targets); metricRelabelings run after (on scraped samples).
  • sampleLimit makes the whole scrape fail (up == 0) when exceeded — a hard cardinality guard.
  • Lab conventions expect job="orders" and job="payments": the Services are named orders and payments.
PodMonitor and PrometheusRule

Scraping pods without a Service, and shipping rules with the app

Use a PodMonitor for pods that have no Service (workers, batch, sidecars). Rules live next to the app as a PrometheusRule.

apiVersion: monitoring.coreos.com/v1
kind: PodMonitor          # example: a queue worker
metadata:                 # (not part of the shop app)
  name: worker
  namespace: shop
  labels: {release: kps}
spec:
  selector:
    matchLabels: {app: worker}
  podMetricsEndpoints:
    - port: metrics            # container port NAME
      interval: 30s
      scrapeTimeout: 10s
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: orders
  namespace: shop
  labels: {release: kps}
spec:
  groups:
    - name: orders.rules
      rules:
        - record: job_route:http_requests:rate5m
          expr: sum by (job, route) (
                  rate(http_requests_total[5m]))
        - alert: OrdersHighErrorRatio
          expr: |
            sum(rate(http_requests_total{job="orders",code=~"5.."}[5m]))
              / sum(rate(http_requests_total{job="orders"}[5m])) > 0.05
          for: 10m
          labels: {severity: page}
          annotations:
            summary: "orders 5xx ratio {{ $value | humanizePercentage }}"

Rule groups from all selected PrometheusRules are merged into the Prometheus config. Group names must be unique per PrometheusRule; validate with promtool check rules in CI before the Operator sees them (ch 4).

Helm values

Values you will change on day one

Defaults are sized for a demo. These are the knobs that decide cost, durability and who can add monitors.

# deploy/kps-values.yaml
prometheus:
  prometheusSpec:
    retention: 10d             # chart default 10d
    retentionSize: 40GiB       # delete oldest blocks first
    replicas: 2                # HA pair (section 19)
    externalLabels:
      cluster: obs
    scrapeInterval: 30s
    evaluationInterval: 30s
    resources:
      requests: {cpu: 500m, memory: 2Gi}
      limits: {memory: 4Gi}
    storageSpec:
      volumeClaimTemplate:
        spec:
          accessModes: [ReadWriteOnce]
          resources: {requests: {storage: 50Gi}}
    enforcedSampleLimit: 20000 # cap for every monitor
alertmanager:
  alertmanagerSpec: {replicas: 3}
grafana:
  adminPassword: change-me     # use an existing Secret

Why these

  • Storage: without storageSpec Prometheus uses emptyDir — every pod restart loses all data.
  • Retention by time and size; whichever triggers first deletes the oldest blocks.
  • Memory limit, no CPU limit is a common choice: CPU throttling slows queries and rule evaluation; an OOM is visible and alertable.
  • externalLabels identify this Prometheus to remote storage, Thanos and Alertmanager; cluster is the minimum.
  • enforcedSampleLimit protects the server from a single team's cardinality bug.
  • Upgrades: CRDs are not upgraded by helm upgrade; follow the chart's upgrade notes for each major version.
Chapter 5 · Part 5.2

Kubernetes metrics

Where each Kubernetes metric comes from, and the queries that answer "is my pod starved, throttled or killed?".

Sources

Four sources, four kinds of truth

Kubernetes metrics come from different components. Knowing the source tells you the label set and the meaning.

SourceScraped fromAnswersExample metrics
kubelet / cAdvisorEach node's kubelet /metrics/cadvisorWhat containers actually usecontainer_cpu_usage_seconds_total, container_memory_working_set_bytes, container_cpu_cfs_throttled_periods_total
kubeletKubelet /metrics, /metrics/probesVolumes, pod lifecycle, probe resultskubelet_volume_stats_available_bytes, prober_probe_total
kube-state-metricsOne Deployment that watches the APIWhat the API declares (desired vs actual state)kube_pod_status_phase, kube_deployment_status_replicas_available, kube_pod_container_resource_requests
node-exporterDaemonSet, host namespacesThe node: CPU, memory, disk, networknode_cpu_seconds_total, node_memory_MemAvailable_bytes, node_filesystem_avail_bytes
Control planeAPI server, etcd, scheduler, controller-managerIs the cluster itself healthy?apiserver_request_total, apiserver_request_duration_seconds, etcd_server_has_leader

On managed control planes (EKS, GKE) etcd, scheduler and controller-manager are often not scrapeable; disable those ServiceMonitors in the chart values to avoid permanently-down targets.

kube-state-metrics

Desired vs actual: kube-state-metrics queries

kube-state-metrics turns object state into series. It does not measure usage; it reports what the API server knows.

# Deployments not at desired replicas
kube_deployment_spec_replicas
  != kube_deployment_status_replicas_available

# pods in Pending/Unknown/Failed (alert with `for: 15m`)
sum by (namespace, pod) (
  kube_pod_status_phase{phase=~"Pending|Unknown|Failed"}
) > 0

# containers restarting in the last hour
increase(kube_pod_container_status_restarts_total[1h]) > 3

# last termination reason was OOMKilled
kube_pod_container_status_last_terminated_reason{reason="OOMKilled"} == 1

# containers in CrashLoopBackOff
kube_pod_container_status_waiting_reason{reason="CrashLoopBackOff"} == 1

# nodes not Ready
kube_node_status_condition{condition="Ready",status="true"} == 0

Things to know

  • One series per object (per container, per condition, per phase): state is a 1/0 value on the matching label.
  • Labels and annotations of objects are not exported by default: allow them with --metric-labels-allowlist (e.g. pods=[team]) → kube_pod_labels.
  • Join usage with ownership via kube_pod_owner or kube_pod_info and group_left (ch 3).
  • Restart counters survive pod restarts (the API keeps the count); they reset when the pod is recreated.
  • kube-state-metrics is a single Deployment: on large clusters shard it (--shard, --total-shards) or watch its own scrape duration.
CPU

CPU: usage, requests and throttling

CPU is a compressible resource: a container over its limit is throttled, not killed. Throttling shows up as latency, not as errors.

# CPU cores used per pod (cAdvisor)
sum by (namespace, pod) (
  rate(container_cpu_usage_seconds_total{
         container!="", namespace="shop"}[5m]))

# usage vs request (1.0 = using exactly the request)
sum by (namespace, pod) (
  rate(container_cpu_usage_seconds_total{container!=""}[5m]))
/
sum by (namespace, pod) (
  kube_pod_container_resource_requests{resource="cpu"})

# share of CFS periods that were throttled
sum by (namespace, pod, container) (
  rate(container_cpu_cfs_throttled_periods_total[5m]))
/
sum by (namespace, pod, container) (
  rate(container_cpu_cfs_periods_total[5m]))

Reading the numbers

  • container!="" drops the pod-level cgroup series that cAdvisor also exports (otherwise you double count).
  • A throttling ratio of 25%+ on a latency-sensitive service is a common reason for p99 spikes even at low average CPU.
  • Throttling happens per 100 ms CFS period: a bursty Go or JVM service can be throttled while its 5-minute average is far below the limit.
  • Fixes: raise or remove the CPU limit, set GOMAXPROCS to match the limit (Go 1.25+ does this automatically from the cgroup limit), reduce parallelism.
  • Requests drive scheduling; usage far above requests means noisy neighbours, far below means wasted capacity.
Memory

Memory: working set, limits and OOMKills

Memory is not compressible. A container above its memory limit is killed by the kernel OOM killer; the pod shows OOMKilled and restarts.

# working set per container (what the OOM decision is based on)
sum by (namespace, pod, container) (
  container_memory_working_set_bytes{container!="", namespace="shop"})

# working set as a share of the limit
sum by (namespace, pod, container) (
  container_memory_working_set_bytes{container!=""})
/
sum by (namespace, pod, container) (
  kube_pod_container_resource_limits{resource="memory"}) > 0.9

# OOM kill events (cAdvisor counter, recent kubelets)
increase(container_oom_events_total{namespace="shop"}[1h]) > 0

# node memory actually available
node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes

Pitfalls

  • Use working set, not container_memory_usage_bytes: usage includes page cache that the kernel can reclaim.
  • Working set still includes active file cache; a log-heavy app can look "close to the limit" without being in danger.
  • Go: set GOMEMLIMIT to ~90% of the container limit so the GC works harder before the kernel kills the process.
  • An OOMKill only appears as a restart and a last_terminated_reason; alert on restarts + reason, not on the event alone.
  • Pods with no memory limit can be killed when the node runs out of memory (eviction or system OOM): watch node-level metrics too.
Node and control plane

Node and cluster health queries

The kubernetes-mixin rules shipped by the chart already alert on most of these; read them before writing your own.

# node CPU utilization (0–1), excluding idle and iowait
1 - avg by (instance) (
  rate(node_cpu_seconds_total{mode=~"idle|iowait"}[5m]))

# filesystem will be full within 4 hours
predict_linear(
  node_filesystem_avail_bytes{fstype!~"tmpfs|overlay"}[6h],
  4*3600) < 0


# API server error ratio by verb
sum by (verb) (rate(apiserver_request_total{code=~"5.."}[5m]))
  / sum by (verb) (rate(apiserver_request_total[5m]))

# API server p99 latency for mutating calls
histogram_quantile(0.99, sum by (le, verb) (rate(
  apiserver_request_duration_seconds_bucket{
    verb=~"POST|PUT|PATCH|DELETE"}[5m])))

Where to look first

  • Built-in alerts: KubePodCrashLooping, KubePodNotReady, KubeDeploymentReplicasMismatch, KubeContainerWaiting, NodeFilesystemSpaceFillingUp, KubeCPUOvercommit.
  • Built-in dashboards: "Kubernetes / Compute Resources / Namespace (Pods)" and "Node Exporter / Nodes".
  • USE method for nodes and pods in ch 10 builds on exactly these queries.
  • On EKS, the API server is managed: watch apiserver_request_* and throttling (429s), not etcd.
Chapter 5 · Part 5.3

Cardinality and performance

Active series drive memory, CPU and cost. Find the expensive ones and stop them at the door.

Cardinality

Where series come from

Series count = product of distinct values of every label on a metric, summed over targets. One unbounded label multiplies everything else.

A bounded metric

  • http_request_duration_seconds_bucket{method, route, code, le}
  • 3 methods × 8 routes × 5 codes × 12 buckets = 1,440 series per pod.
  • × 6 pods = 8,640 series. Fine.

The same metric, one bad label

  • Add user_id with 50,000 active users …
  • 1,440 × 50,000 = 72 million series per pod (in theory).
  • In practice: OOMKilled Prometheus, failing scrapes, a TSDB that takes hours to replay its WAL.

Usual suspects

  • IDs: user, order, session, request, trace.
  • Raw URL paths (/orders/8f2c…) instead of route templates (/orders/{id}).
  • Error messages, SQL text, IPs, user agents, pod names of short-lived jobs.
  • Histograms with many buckets × many labels.
  • Autoscaling churn: each new pod name creates new series.

Put unbounded values in logs and traces (ch 6, ch 9), where they are cheap per event. Metrics are for aggregates over bounded label sets.

Finding them

Find the top metrics and labels

Start with the TSDB status page (Status → TSDB Status in the UI) — it is computed from the head block and costs almost nothing.

# total active series in the head block
prometheus_tsdb_head_series

# TSDB stats API: top metrics, labels, label pairs
curl -s localhost:9090/api/v1/status/tsdb | jq '.data |
  {series: .headStats.numSeries,
   topMetrics: .seriesCountByMetricName[:5],
   topLabels:  .labelValueCountByLabelName[:5]}'

# which jobs add the most series per scrape
topk(10, sum by (job) (scrape_samples_post_metric_relabeling))

# churn: new series per scrape, by job
topk(10, sum by (job) (rate(scrape_series_added[10m])))

# series per metric name (EXPENSIVE: touches every series)
topk(10, count by (__name__) ({__name__=~".+"}))

How to read them

  • seriesCountByMetricName: the metric to fix first.
  • labelValueCountByLabelName: the label with the most distinct values — usually the culprit.
  • scrape_samples_scraped vs …_post_metric_relabeling: how much your drop rules already remove.
  • scrape_series_added high and steady = churn (new pods, rotating label values) even if the total looks flat.
  • Run the count by (__name__) query only on small servers or with a narrow selector; on a large server it can take seconds and lots of memory.
  • Grafana Cloud and Mimir have cardinality UIs; open-source Prometheus has the TSDB page and these queries.
Controls

Stop cardinality at the door

Fix in the code first (route templates, no IDs). Until the fix ships, drop or rewrite at scrape time.

# ServiceMonitor endpoint: metricRelabelings (after scrape)
metricRelabelings:
  # drop a whole metric
  - sourceLabels: [__name__]
    regex: http_client_request_duration_seconds_bucket
    action: drop
  # remove a label from every series
  - regex: user_id
    action: labeldrop
  # collapse raw paths into one value
  - sourceLabels: [path]
    regex: /orders/.+
    targetLabel: path
    replacement: /orders/{id}

# hard limits per scrape (ServiceMonitor spec)
sampleLimit: 5000
labelLimit: 30
labelValueLengthLimit: 200

Know the trade-offs

  • labeldrop can create duplicate series (two series that only differed by that label) → scrape errors "duplicate sample for timestamp". Aggregate in the app instead, or drop the metric.
  • Drop rules still pay the scrape and parse cost; only storage and memory are saved.
  • Limits fail the whole scrape: up == 0 for that target. Alert on prometheus_target_scrapes_exceeded_sample_limit_total increasing.
  • enforcedSampleLimit on the Prometheus CR caps every monitor, whatever teams set.
  • OTel pipelines can do the same earlier: filter and transform processors in the Collector (ch 9).
Sizing

Memory, disk and query cost

Prometheus memory scales mainly with active head series and churn; disk scales with samples ingested per second × retention.

Memory

  • Measure, don't guess: process_resident_memory_bytes / prometheus_tsdb_head_series gives your bytes per series (often a few KiB including overhead).
  • Head block holds the last ~2–3 h of series in memory; churn makes it hold many more series than are "active now".
  • Queries need memory too: a dashboard over 30 days × 100k series can use more than ingestion.
  • Keep 30–50% headroom over observed peak (WAL replay after a restart is a peak).

Disk

  • Compressed samples are roughly 1–2 bytes each on disk.
  • Estimate: rate(prometheus_tsdb_head_samples_appended_total[1h]) × retention seconds × ~2 bytes.
  • Example: 50k samples/s × 10 days ≈ 43 billion samples ≈ 60–90 GB.
  • Use a local SSD/EBS gp3 volume; NFS is not supported for the TSDB.

Query guards (flags)

  • --query.timeout (default 2m).
  • --query.max-samples (default 50,000,000 samples loaded per query).
  • --query.max-concurrency (default 20).
  • Recording rules for every dashboard query that aggregates many series (ch 4).
  • In the Operator these map to spec.query fields on the Prometheus CR.

Bytes-per-sample and bytes-per-series figures are rules of thumb; they change with Prometheus versions and label sizes. Always compute them from your own server's metrics.

Slow queries

Finding slow queries and slow rules

Most "Prometheus is slow" tickets are one expensive dashboard or one heavy recording rule.

# prometheus.yml (global) — log every query to a file
global:
  query_log_file: /prometheus/query.log
# Operator: spec.queryLogFile on the Prometheus CR

# rule groups that take longer than their interval
prometheus_rule_group_last_duration_seconds
  > prometheus_rule_group_interval_seconds

# missed rule evaluations (group overran its interval)
increase(prometheus_rule_group_iterations_missed_total[1h]) > 0

# query latency by stage
prometheus_engine_query_duration_seconds{quantile="0.9"}

# scrape durations close to the timeout
max by (job) (scrape_duration_seconds) > 8

Usual fixes

  • Replace dashboard queries over raw high-cardinality metrics with recording rules.
  • Narrow selectors: always filter by job or namespace before regexes on other labels.
  • Avoid long ranges with short steps ([30d] with a 15 s step).
  • Split a slow rule group: groups are evaluated in parallel, rules inside a group sequentially.
  • Slow scrapes: the exporter is slow (e.g. a database exporter running queries); raise scrapeTimeout only after fixing the exporter.
Chapter 5 · Part 5.4

HA, remote write and long-term storage

One Prometheus is a single point of failure with local disk. Replicas, remote write and object storage fix both.

HA

High availability: two identical replicas

Prometheus has no clustering. HA means running two (or more) identical servers that scrape the same targets and send the same alerts.

How it works

  • Each replica scrapes all targets and evaluates all rules independently.
  • Both send alerts to all Alertmanager instances; Alertmanager deduplicates identical alerts (same labels) and its own cluster gossips notification state.
  • Each replica has a distinct external label (the Operator adds prometheus_replica by default) so remote storage can deduplicate.
  • Data differs slightly between replicas (scrape timing); queries against one replica only see its gaps.

Operator settings

  • replicas: 2 on the Prometheus CR (StatefulSet with two pods).
  • replicaExternalLabelName (default prometheus_replica).
  • Pod anti-affinity so both replicas never share a node.
  • A PodDisruptionBudget (the chart can create one).
  • Alertmanager: replicas: 3 forms a gossip cluster automatically.

What HA does not solve

  • Retention is still local disk on each replica.
  • No global view across clusters.
  • Queries hit one replica: gaps from its restart are visible unless something merges both.
  • → Remote write or Thanos sidecar to a long-term store that deduplicates.
Remote write

Remote write: stream samples to another system

Prometheus appends every sample to its WAL; remote write tails the WAL and sends batches over HTTP to any compatible receiver.

# Prometheus CR (or kps values under prometheusSpec)
remoteWrite:
  - url: https://mimir.example.com/api/v1/push
    name: longterm
    queueConfig:
      capacity: 10000
      maxShards: 50
      maxSamplesPerSend: 2000
    writeRelabelConfigs:          # only what you need off-cluster
      - sourceLabels: [__name__]
        regex: "up|job_.*:.*|http_request_duration_seconds_bucket"
        action: keep
# AMP: add sigv4: {region: ap-northeast-1} and
# url .../workspaces/<id>/api/v1/remote_write

# health of the queue
rate(prometheus_remote_storage_samples_failed_total[5m])
prometheus_remote_storage_shards
# lag in seconds
prometheus_remote_storage_highest_timestamp_in_seconds
  - ignoring(remote_name, url) group_right
    prometheus_remote_storage_queue_highest_sent_timestamp_seconds

Concepts

  • Shards: parallel senders; scale up automatically to maxShards when the queue falls behind.
  • Retries with backoff on 5xx and 429; 4xx (other than 429) drops the batch.
  • If the receiver is down longer than the WAL keeps data (~2 h), samples are lost.
  • Remote Write 2.0 (spec in Prometheus 3, experimental): string interning, native histograms, metadata and exemplars in one message; select it with protobuf_message: io.prometheus.write.v2.Request in remote_write when the receiver supports it.
  • Receivers: Prometheus itself (--web.enable-remote-write-receiver), Mimir, Thanos Receive, AMP, Cortex, VictoriaMetrics, many vendors.
  • writeRelabelConfigs is the cheapest cost control for paid backends.
Agent mode

Agent mode: scrape and forward, nothing else

A Prometheus started with --agent has no local querying, rules or alerting. It scrapes, writes a WAL and remote-writes. Ideal for edge clusters that report to a central store.

Agent mode

  • Flag: --agent (Prometheus 3); in the Operator, the PrometheusAgent CRD.
  • Uses the same scrape config, service discovery and ServiceMonitors.
  • Much less memory than a full server: no head block for queries, WAL truncated as soon as data is sent.
  • Recent Operator versions can run it as a DaemonSet (mode: DaemonSet, behind a feature gate) to scrape only node-local pods.

When to use

  • Many small clusters → one central Mimir/Thanos/AMP.
  • You already have a central query layer and want no local Prometheus to operate.
  • Alternatives with the same role: Grafana Alloy (prometheus.scrape + prometheus.remote_write), the OTel Collector prometheus receiver, AMP managed collectors.

What you lose

  • No local queries: if the link to the central store is down, you are blind (data waits in the WAL).
  • No local alerting: rules run centrally (Mimir ruler, Thanos Ruler, AMP rules).
  • Debug with the agent's own metrics (prometheus_remote_storage_*) and the /agent UI page.
Long-term storage

Three architectures for long-term, global queries

All three keep Prometheus-compatible querying (PromQL) and put blocks in object storage. They differ in how data arrives and who operates it.

Thanos (sidecar)Prometheus2h blocksThanos sidecarupload + StoreAPIObject storage (S3)blocks, compacted + downsampledStore gatewayreads S3Queriermerge + dedupMimir / Thanos Receive (push)Prometheusor agent / AlloyDistributorHA dedupIngestersrecent dataObject storage (S3)blocksQueriers + store gatewaysremote writeAmazon Managed Service for PrometheusPrometheusagent / CollectorManaged collectoragentless scraper (EKS)AMP workspaceremote write (SigV4), PromQL APIRules + alert managermanaged; Grafana / AMG on top
Long-term storage

Choosing between them

The question is less "which is best" than "who operates it, and how does data get there".

Thanos sidecarMimir / Thanos ReceiveAMP
Data pathUpload completed blocks (pull for recent data)Remote write pushRemote write push or managed scraping
You operateSidecar, store, querier, compactorA distributed system (+ Kafka in newer Mimir designs)Nothing server-side; pay per sample and query
Recent dataQueried live from each Prometheus (sidecar StoreAPI)Held by ingesters until flushed to blocksManaged
AlertingLocal Prometheus (or Thanos Ruler globally)Mimir ruler + Alertmanager, or local PrometheusAMP rules + alert manager → SNS
Long rangesDownsampled blocks (5m, 1h)Recording rules; query sharding and cachingRecording rules; managed query engine
Cost shapeYour compute + S3Your compute + S3 (+ Kafka if used)Per sample ingested, stored and queried
Good forFew clusters, keep local PrometheusMany clusters, multi-tenant platformAWS shops, no ops team for the backend

Mixed setups are common: local Prometheus for alerting close to the workload, plus remote write of a curated subset to a central store for global dashboards and long retention.

Thanos and Mimir

Deduplication, downsampling and multi-tenancy

The details you need to know before you operate either system.

Deduplication

  • Thanos Querier: --query.replica-label=prometheus_replica merges the two HA replicas at query time.
  • Mimir distributor: HA tracker accepts samples from one replica per cluster at a time (default labels cluster and __replica__); fails over when the elected replica stops sending.
  • Both require consistent external labels: same cluster, different replica label.

Downsampling and retention

  • Thanos Compactor creates 5m and 1h downsampled blocks (raw data must exist >40 h and >10 days respectively before those levels are produced).
  • Downsampling speeds up long-range queries; it does not save space by itself (raw is kept unless you set shorter raw retention).
  • Mimir: retention per tenant; no downsampling — rely on recording rules for long ranges.

Multi-tenancy and sharding

  • Tenant per request: Mimir reads X-Scope-OrgID, Thanos Receive reads THANOS-TENANT (configurable); limits per tenant (series, ingestion rate).
  • Prometheus functional sharding: one Prometheus per team/namespace.
  • Hash sharding: shards: N on the Prometheus CR splits targets with hashmod; each shard sees only part of the data, so cross-shard queries need a global layer.
Common mistakes

Prometheus on Kubernetes: mistakes that cost an outage or a bill

Each of these shows up in real clusters within the first months.

Operator

  • ServiceMonitor without release: kps → silently ignored.
  • port: 8080 instead of the port name → no targets.
  • Service selector matches no pods (labels differ) → endpoints empty.
  • No storageSpec → all data lost on every restart.
  • Editing the generated Secret by hand → overwritten by the Operator.

Kubernetes metrics

  • Summing cAdvisor CPU without container!="" → double counting.
  • Using container_memory_usage_bytes for OOM risk → false alarms from page cache.
  • CPU limits on latency-sensitive services without watching throttling.
  • Keeping etcd/scheduler monitors on EKS → targets down forever.

Scale

  • IDs or raw paths as labels; no sampleLimit anywhere.
  • Remote writing everything to a paid backend instead of a curated keep list.
  • Two HA replicas without distinct replica labels → the backend rejects or double counts.
  • Expecting Prometheus memory to stay flat while pods churn every few minutes.
Self-check

Prometheus on Kubernetes: can you answer these?

Answer with the exact field, flag, label or query.

Operator

  • Which CRD would you use for a worker without a Service?
  • Why does a new ServiceMonitor produce no target, and what are the four things you check?
  • Where does the Operator put the generated Prometheus config, and how does it get reloaded?
  • What is the difference between relabelings and metricRelabelings?
  • What does sampleLimit do when exceeded?

Kubernetes metrics

  • Which component exports kube_pod_status_phase, and which exports container_cpu_usage_seconds_total?
  • Write the CPU throttling ratio query for one container.
  • Why is working set a better OOM signal than usage?
  • How do you find containers whose last restart was an OOMKill?
  • Why might a pod be throttled with 20% average CPU?

Scale

  • Where do you find the metric with the most series without running an expensive query?
  • What can go wrong with labeldrop?
  • How does Prometheus behave when the remote write receiver is down for 10 minutes? For 5 hours?
  • What does agent mode remove, and when is that acceptable?
  • How do Thanos and Mimir deduplicate an HA pair?

Practice: Lab 5 in Hands-on labs — inspect the Operator's selectors, find a cardinality bomb with the TSDB status API and stop it with metricRelabelings and sampleLimit, trigger CPU throttling and an OOMKill, then remote-write a curated subset to a second Prometheus.

Chapter 6

Structured logging

Logs that machines can query and humans can still read: a field schema, slog in Go, trace correlation, safety and cost, and how logs leave a Kubernetes pod.

Chapter 6 · Part 6.1

Why structured logs

A log line is an event with fields. Treat its shape as an API that dashboards, alerts and on-call engineers depend on.

Structure

The same event, three ways

Only the structured forms can be filtered, aggregated and joined without fragile regular expressions.

Unstructured text

  • 2026-11-08 10:15:02 ERROR payment failed for order o-123 after 2013ms (502)
  • Readable, but every query is a regex.
  • A wording change breaks every dashboard and alert that parsed it.
  • Numbers are strings: no avg(duration) without parsing.

logfmt

  • time=2026-11-08T10:15:02Z level=ERROR msg="payment failed" order_id=o-123 duration_ms=2013 status=502
  • Key=value pairs, still easy to read in a terminal.
  • Flat only: no nested objects or arrays.
  • Loki parses it with | logfmt (ch 7).

JSON

  • {"time":"2026-11-08T10:15:02Z","level":"ERROR","msg":"payment failed","service":"orders","order_id":"o-123","route":"POST /orders","status":502,"duration_ms":2013}
  • Typed values (numbers, booleans), nesting possible.
  • Native for CloudWatch Logs Insights, Loki | json, OpenSearch.
  • The deck's choice for the shop app.

The message (msg) stays a short, constant string ("payment failed"). Everything that varies goes into fields. Constant messages are what you count and alert on.

Schema

The shop field schema

A small, fixed set of keys used by every service. Agree on names once; dashboards and queries in ch 7, 9 and 12 rely on them.

KeyTypeExampleMeaning
timeRFC 3339 string2026-11-08T10:15:02.123ZWhen the event happened (UTC, sub-second)
levelstringINFO, WARN, ERROR, DEBUGSeverity (next slide)
msgstringpayment failedConstant event name, no variables inside
servicestringordersSame value as OTel service.name (ch 8)
trace_id, span_idhex string4bf92f3577b34da6a3ce929d0e0e4736Link to the trace (ch 9); absent when no span
order_idstringo-123Business key: lets support find "my order" in seconds
routestringPOST /ordersRoute template, never the raw path
statusinteger502HTTP status returned
duration_msinteger2013Request duration in milliseconds

Align names with OpenTelemetry where a convention exists (service.name, http.route, http.response.status_code) if you export logs through OTel; the shop app keeps short JSON keys and maps them in the pipeline.

Levels

Levels: what each one promises

A level is a routing decision: who should look, and how soon. Use few levels, consistently.

LevelUse it forNot forTypical production volume
DEBUGDetails for developers while diagnosingAnything someone needs in an incidentOff by default; switch on per service, temporarily
INFONormal business events: request served, order created, job finishedEvery loop iteration or cache hitThe bulk of volume; budget it
WARNUnexpected but handled: retry succeeded, fallback used, 4xx from a clientErrors you will never act onLow; trend it
ERRORThe operation failed and the caller saw it: 5xx, failed job, lost messageExpected outcomes (validation failures, 404)Rare; each one should be explainable

Do not page on log levels alone: alert on rates of a constant message or on metrics (ch 4, ch 7). Go's log/slog has exactly DEBUG (-4), INFO (0), WARN (4) and ERROR (8); levels in between are allowed but rarely worth it.

Logs vs events vs metrics

When a log is the right signal

Logs are the most flexible and the most expensive signal per unit of information. Use each signal for what it is cheap at.

Use a log when

  • You need the details of one occurrence: which order, which input, which error.
  • Fields are high-cardinality (IDs, emails, error text).
  • Volume is bounded: per request, per job, per state change.
  • An auditor or support engineer needs the record.

Use a metric when

  • You need counts, rates, percentiles over time.
  • You alert on it: metrics are cheaper and faster to evaluate.
  • The question is "how many / how slow", not "which one".
  • Labels are bounded (ch 5, section 18).

Use a trace span when

  • You need timing across services for one request.
  • You want the log context of a request without logging every step: span events and attributes.
  • Logs still carry trace_id so you can jump between them (ch 9).

A "wide event" approach (one rich structured log or span per request with many fields) replaces dozens of scattered log lines and is easier to query. The request log in section 21 is exactly that.

Log as an API

Treat the log format as a contract

Other people build on your fields: dashboards, alerts, SIEM rules, support runbooks. Breaking them silently breaks those tools.

Rules

  • Keys are snake_case, stable, documented in the repo (docs/logging.md).
  • Never reuse a key with a different type (status is always an integer).
  • Add fields freely; rename or remove only with a deprecation window.
  • One JSON object per line, UTF-8, no pretty printing, no trailing text.
  • Timestamps in UTC (slog writes +00:00, equal to Z).

Enforce it

  • A shared logging package (the shop app uses apps/internal/logx).
  • Unit tests that decode a log line and assert required keys exist.
  • Lint against fmt.Println / log.Printf in services.
  • A pipeline stage that drops or flags non-JSON lines (ch 7).
  • Review new fields for PII (section 22).
Chapter 6 · Part 6.2

Logging in Go with slog

The standard library's structured logger: handlers, attributes, context, and how to put trace IDs on every line.

slog basics

log/slog: logger, handler, record

Since Go 1.21 the standard library has structured logging. A Logger builds Records; a Handler decides how to format and where to write them.

package main

import (
	"log/slog"
	"os"
)

func main() {
	h := slog.NewJSONHandler(os.Stdout, &slog.HandlerOptions{
		Level: slog.LevelInfo,   // drop DEBUG
	})
	log := slog.New(h).With("service", "orders")
	slog.SetDefault(log)         // also captures log.Printf output

	log.Info("order created", "order_id", "o-123", "amount", 42.5)
	log.Debug("cache miss", "key", "o-123")   // not written
	log.Error("payment failed",
		slog.String("order_id", "o-124"),
		slog.Int("status", 502))
}
{"time":"2026-11-08T10:15:02.123456+00:00","level":"INFO",
 "msg":"order created","service":"orders",
 "order_id":"o-123","amount":42.5}
{"time":"2026-11-08T10:15:02.130211+00:00","level":"ERROR",
 "msg":"payment failed","service":"orders",
 "order_id":"o-124","status":502}

# (each object is one line in reality; wrapped here)

With returns a child logger whose attributes are pre-formatted once, so per-call cost stays low. The typed forms (slog.String, slog.Int, LogAttrs) avoid allocations and catch key/value mismatches at compile time; go vet also reports odd key/value pairs.

slog features

Groups, LogValuer, dynamic level and source

Four features that matter in services.

// groups → nested JSON
log.Info("charge", slog.Group("card",
	slog.String("brand", "visa"),
	slog.String("last4", "4242")))
// {"msg":"charge","card":{"brand":"visa","last4":"4242"}}

// LogValuer: a type decides how it is logged
type Token string
func (Token) LogValue() slog.Value {
	return slog.StringValue("[REDACTED]")
}
log.Info("login", "token", Token("s3cr3t"))  // "token":"[REDACTED]"

// change the level at runtime (e.g. from an admin endpoint)
var lvl slog.LevelVar          // zero value = INFO
_ = lvl.UnmarshalText(
	[]byte(os.Getenv("LOG_LEVEL")))
h := slog.NewJSONHandler(os.Stdout, &slog.HandlerOptions{
	Level:     &lvl,
	AddSource: false,  // true: "source":{file,line,…}
})
lvl.Set(slog.LevelDebug)       // takes effect immediately

Why each matters

  • Groups keep related fields together and avoid key collisions between libraries.
  • LogValuer is the safest redaction: the secret type can never be logged in clear, wherever it ends up.
  • LevelVar lets you enable DEBUG on one pod for ten minutes without a redeploy.
  • UnmarshalText accepts DEBUG, INFO, WARN, ERROR (case-insensitive, optional offset like INFO+2); on error the level stays INFO.
  • AddSource costs a runtime.Callers per record; useful for ERROR, expensive for hot INFO paths.
  • ReplaceAttr in HandlerOptions rewrites or removes any attribute (renaming keys, redaction by key) — section 22.
Request logging

One request log per request: the middleware

A single "wide" INFO line per request with route, status and duration replaces scattered "start/end" lines.

// recorder wraps http.ResponseWriter and keeps the status code
func Middleware(log *slog.Logger, next http.Handler) http.Handler {
	return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
		start := time.Now()
		rec := &recorder{ResponseWriter: w, status: http.StatusOK}
		next.ServeHTTP(rec, r)
		route := r.Pattern // set by ServeMux (Go 1.23+)
		if route == "" {
			route = "unmatched"
		}
		lvl := slog.LevelInfo
		switch {
		case rec.status >= 500:
			lvl = slog.LevelError
		case rec.status >= 400:
			lvl = slog.LevelWarn
		}
		log.LogAttrs(r.Context(), lvl, "request",
			slog.String("route", route),
			slog.Int("status", rec.status),
			slog.Int64("duration_ms",
				time.Since(start).Milliseconds()))
	})
}

Design notes

  • Wrap the ServeMux directly: it sets Pattern on the request it receives; a wrapper in between that clones the request (r.WithContext, http.TimeoutHandler) hides it.
  • route is the template (GET /orders/{id}), so it is safe to index or aggregate.
  • Level from the status: 5xx → ERROR, 4xx → WARN.
  • LogAttrs(r.Context(), …) passes the context, so a handler can add trace_id (next slide).
  • Health checks and /metrics scrapes are noise: skip them or log them at DEBUG.
  • Business events (order created with order_id) are logged inside the handler, not here.
Trace correlation

Put trace_id and span_id on every line

A small handler wrapper reads the active span from the context. Every …Context log call then carries the IDs, and logs and traces link both ways (ch 9).

import "go.opentelemetry.io/otel/trace"

type traceHandler struct{ slog.Handler }

func (h traceHandler) Handle(ctx context.Context, r slog.Record) error {
	if sc := trace.SpanContextFromContext(ctx); sc.IsValid() {
		r.AddAttrs(
			slog.String("trace_id", sc.TraceID().String()),
			slog.String("span_id", sc.SpanID().String()))
	}
	return h.Handler.Handle(ctx, r)
}

// keep the wrapper when attributes or groups are added
func (h traceHandler) WithAttrs(a []slog.Attr) slog.Handler {
	return traceHandler{h.Handler.WithAttrs(a)}
}
func (h traceHandler) WithGroup(n string) slog.Handler {
	return traceHandler{h.Handler.WithGroup(n)}
}

log := slog.New(traceHandler{slog.NewJSONHandler(os.Stdout, nil)})
log.ErrorContext(ctx, "payment failed", "order_id", id)

Rules

  • Use the …Context methods (InfoContext, ErrorContext, LogAttrs(ctx, …)); log.Info has no context and no IDs.
  • Without an OpenTelemetry SDK the span context is invalid and the fields are simply omitted — safe to ship before ch 8.
  • Trace IDs are 32 hex characters, span IDs 16 (W3C format, ch 8).
  • Never make trace_id a Loki label or a metric label: it is unique per request (ch 7).
  • Enabled is inherited from the embedded handler, so level filtering still works.
Errors

Logging errors well

An error log is useful only if it says what failed, for which entity, and why — once.

// wrap with context as the error travels up
if err != nil {
	return fmt.Errorf("charge order %s: %w", id, err)
}

// log once, at the boundary that handles it
if err := svc.Create(ctx, o); err != nil {
	log.ErrorContext(ctx, "create order failed",
		"order_id", o.ID,
		"err", err)            // err.Error() as a string
	http.Error(w, "payment failed", http.StatusBadGateway)
	return
}
// {"level":"ERROR","msg":"create order failed","order_id":"o-124",
//  "err":"charge order o-124: Post \"http://payments:8080/charge\":
//         context deadline exceeded", ...}

// errors.Is / As still work on the wrapped error
if errors.Is(err, context.DeadlineExceeded) { /* timeout */ }

Habits

  • Log or return, not both: logging at every layer produces five lines for one failure.
  • Wrap with %w and the entity ID; the final message reads like a stack of causes.
  • Key err for the error string; add err_type or a stable error code if you want to aggregate by kind.
  • Stack traces: only for panics and truly unexpected errors; they are large and multi-line (section 23).
  • Recover panics in middleware and log them as ONE JSON line with the stack in a field.
  • Record the same error on the span (span.RecordError, SetStatus) so the trace shows it (ch 9).
OTel log bridge

Sending slog records through OpenTelemetry

Instead of (or in addition to) stdout, a bridge turns slog records into OTel log records and exports them over OTLP with the resource and trace context attached.

import (
	"go.opentelemetry.io/contrib/bridges/otelslog"
	"go.opentelemetry.io/otel/log/global"
)

// after the LoggerProvider is set up (ch 8):
// global.SetLoggerProvider(lp)
logger := otelslog.NewLogger("orders")

logger.InfoContext(ctx, "order created", "order_id", id)
// → OTLP log record:
//   body: "order created"
//   attributes: order_id=o-123
//   trace_id/span_id from ctx, resource: service.name=orders

stdout or OTLP?

  • stdout + node agent (Alloy, Fluent Bit, CloudWatch agent): simplest, survives collector outages, works for every container including third-party ones. The deck's default.
  • OTLP via the bridge: structured end to end, trace context and resource attributes attached without parsing, one pipeline for all signals.
  • Many teams do both during migration; avoid double ingestion by sending each stream to only one backend.
  • The OTel Logs API is meant for bridges and for emitting events; application code keeps using its normal logging library.
Chapter 6 · Part 6.3

Safety and cost

Logs are copied everywhere and kept for months. What you put in them is a security and budget decision.

PII and secrets

What must never reach a log

Logs are replicated to agents, backends, exports, laptops and tickets. Assume everyone with log access can read every field.

Never log

  • Passwords, tokens, API keys, session cookies, Authorization headers.
  • Full card numbers (PAN), CVV; bank details.
  • Private keys, connection strings with passwords.
  • Full request or response bodies of user-facing APIs.

Log with care (minimize, mask, or hash)

  • Emails, phone numbers, names, addresses, IP addresses (personal data under GDPR and similar laws).
  • User IDs: often needed; prefer an internal opaque ID over email.
  • Free-text fields typed by users (they paste secrets into them).
  • Error messages from libraries that echo input (SQL errors with values).

Controls

  • Allowlist what you log (named fields), never "log the whole struct".
  • Secret types implement LogValuer (previous section).
  • ReplaceAttr redacts by key as a safety net.
  • Pipeline masking: CloudWatch Logs data protection policies (ch 12), Alloy/Collector processors.
  • Short retention and restricted access for logs that may contain personal data.

Logging regulations and retention limits depend on your jurisdiction and contracts; involve your security or privacy team when defining the schema.

Redaction

A safety net with ReplaceAttr

ReplaceAttr is called for every attribute before it is written. Use it to rename keys and to redact known-sensitive keys anywhere, including inside groups.

var sensitive = map[string]bool{
	"password": true, "token": true, "authorization": true,
	"card_number": true, "cvv": true, "email": true,
}

func replace(groups []string, a slog.Attr) slog.Attr {
	if sensitive[strings.ToLower(a.Key)] {
		return slog.String(a.Key, "[REDACTED]")
	}
	return a
}

h := slog.NewJSONHandler(os.Stdout, &slog.HandlerOptions{
	ReplaceAttr: replace,
})
slog.New(h).Info("signup", "email", "a@example.com", "plan", "pro")
// {"time":"…","level":"INFO","msg":"signup",
//  "email":"[REDACTED]","plan":"pro"}

Limits of key-based redaction

  • It only sees keys. A secret inside msg or inside an error string passes through.
  • A value that is a struct logged with %+v-style formatting is one string: redact by type (LogValuer) instead.
  • Built-in keys (time, level, msg, source) also pass through ReplaceAttr (with empty groups); do not drop them by accident.
  • Test it: a unit test that logs a known secret and asserts it is absent from the output.
  • Pipeline masking (regex for card numbers, emails) is the second net, not the first.
Volume and cost

Where log cost comes from

Most backends charge by ingested GB, and some by GB stored or scanned. Volume is driven by a few chatty lines, not by the average.

Cost drivers

  • Ingestion (GB/month): CloudWatch Logs, Grafana Cloud, Datadog, most SaaS.
  • Storage/retention: object storage for Loki is cheap; indexed stores are not.
  • Query scan: CloudWatch Logs Insights charges per GB scanned (ch 12).
  • Index/labels: high-cardinality labels in Loki multiply streams (ch 7).

Find the top talkers

  • Count lines per msg and service: usually 3–5 messages are most of the volume.
  • Look for per-loop or per-item logs inside request handlers.
  • Health checks and readiness probes logged at INFO every few seconds × every pod.
  • Stack traces repeated on every retry.
  • DEBUG accidentally left on in one deployment.

Reduce it

  • Delete or downgrade lines nobody queries (check query history).
  • Aggregate into a metric (count of "cache miss") instead of a line per event.
  • Sample success logs; keep 100% of WARN and ERROR.
  • Shorter retention for DEBUG/INFO, longer for audit.
  • Set a log budget per service and alert on it (ch 13).

Quick estimate: 200 bytes/line × 50 req/s × 86,400 s ≈ 864 MB/day ≈ 26 GB/month for one request log line per request. Check your backend's current price per GB before deciding.

Sampling

Sampling and rate limiting logs

Logs are not sampled by default. Sample only what you can afford to lose, and keep everything that explains a failure.

// keep all non-2xx; keep 1 in 10 successful request logs
type sampler struct{ n atomic.Uint64 }

func (s *sampler) keep(status int) bool {
	if status >= 400 {
		return true
	}
	return s.n.Add(1)%10 == 0
}

// in the middleware
if s.keep(rec.status) {
	log.LogAttrs(ctx, lvl, "request", attrs...)
}

// better: decide per trace, so a sampled request has
// ALL its lines (and its trace) — use the trace flag
if trace.SpanContextFromContext(ctx).IsSampled() { … }

Choices

  • Per line (1 in N): cheap, but a single request may lose half its lines.
  • Per trace: keep logs of requests whose trace is sampled — logs and traces stay consistent (ch 9).
  • Rate limiting: at most N lines per second per message; protects the pipeline from error storms (one failing dependency → millions of identical lines).
  • At the collector: Alloy (loki.process with a sampling stage) or the OTel Collector filter/probabilistic sampler can sample without code changes.
  • Record the sample rate in a field (sample_rate: 10) so counts from logs can be scaled back.
  • Never sample audit logs or security events.
Audit vs application logs

Not all logs are the same

Different consumers, retention and integrity requirements. Keep them in separate streams.

Application logsAudit logsAccess logs
PurposeDebugging, operationsWho did what, when — accountabilityEvery request at the edge (LB, ingress, API gateway)
Examplesshop request logs, errorsAdmin changed a price; Kubernetes API audit; CloudTrailALB access logs, NGINX ingress logs
SamplingAllowed for success pathsNeverSometimes (very high volume)
RetentionDays to weeksMonths to years (policy-driven)Weeks to months
IntegrityBest effortTamper-evident, restricted write and deleteUsually stored in S3
AccessEngineersSecurity, compliance; few engineersEngineers, security

Do not build audit trails on the application log pipeline: a log level change or a sampling rule must never be able to delete an audit record.

Chapter 6 · Part 6.4

Logs in Kubernetes

How a line written to stdout becomes a file on the node, and how an agent ships it with Kubernetes metadata.

Path of a log line

From fmt.Fprintln(os.Stdout) to a backend

Containers write to stdout/stderr. The container runtime writes them to files on the node; the kubelet rotates them; a node agent tails them.

Node obs-workerorders containerstdout / stderrcontainerdCRI log format/var/log/pods/shop_orders-…/orders/0.log (+ rotated files)/var/log/containers/*.logsymlinks with pod, ns, containerkubeletrotation, kubectl logsNode agent (DaemonSet)Alloy · Fluent Bit · OTel Collector · CloudWatch agenttailAPI server (k8s metadata)pod labels, namespace, nodeBackendsLokich 7CloudWatch Logsch 12OpenSearch / SaaS

kubectl logs reads the same files through the kubelet. If the file is rotated away or the pod is deleted, kubectl logs cannot show it — only your backend can.

CRI format

What is actually in the file

The runtime prefixes every line. Agents must parse this CRI format before your JSON.

# /var/log/pods/shop_orders-…_<uid>/orders/0.log
2026-11-08T10:15:02.123456789Z stdout F {"level":"INFO",…}
2026-11-08T10:15:02.200000000Z stderr F panic: runtime error: …
2026-11-08T10:15:03.000000000Z stdout P {"msg":"huge line, part 1…
2026-11-08T10:15:03.000000000Z stdout F …part 2"}
#           timestamp             stream tag  content
# tag F = full line, P = partial (line was split by the runtime)

# useful commands
kubectl -n shop logs deploy/orders --tail=20
kubectl -n shop logs <pod> --previous   # crashed container
kubectl -n shop logs -l app=orders --since=10m --prefix
kubectl -n shop logs <pod> -c orders --timestamps

Details that bite

  • Very long lines are split by the runtime into partial (P) chunks (containerd splits at 16 KiB by default); the agent must reassemble them (cri stage in Alloy does).
  • stdout and stderr are interleaved in one file; the stream is a field, not a separate file.
  • Rotation is done by the kubelet: containerLogMaxSize (default 10Mi) and containerLogMaxFiles (default 5) per container.
  • A busy container can rotate faster than a slow agent reads → lost lines. Watch agent lag metrics.
  • --previous shows the last terminated container: the first place to look after a crash.
  • Docker (dockershim) wrote a JSON-wrapped format instead; it is gone since Kubernetes 1.24.
Collection patterns

Node agent, sidecar, or direct export

The node-level DaemonSet is the default. The other two patterns solve specific problems.

PatternHowProsCons
Node agent (DaemonSet)One agent per node tails /var/log/podsZero app changes; covers every container; one configOnly sees stdout/stderr; needs hostPath access; per-node resource use
Sidecar streamingApp writes to a file; a sidecar tails it to its own stdoutWorks for legacy apps that only log to filesDuplicate storage on disk; more containers per pod
Sidecar agentAn agent container in the pod ships directlyPer-app config and credentials; works on Fargate-style nodes without DaemonSetsOne agent per pod: cost and config sprawl
Direct export (OTLP)App sends logs over OTLP to a CollectorFully structured, trace context includedLogs lost if the app crashes before flushing; library needed per language

EKS on Fargate has no DaemonSets: use its built-in Fluent Bit log router (configured with a ConfigMap) or a sidecar. CloudWatch side: ch 12.

Multiline and enrichment

Multiline stack traces and Kubernetes metadata

Two jobs every node agent must do well.

Multiline

  • A Java/Python stack trace printed as text becomes one line per frame — dozens of separate "events".
  • Best fix: the app logs the stack trace inside one JSON field (escaped newlines).
  • Otherwise: a multiline stage in the agent that joins lines until the next line starting with a timestamp (loki.process → stage.multiline with firstline regex in Alloy).
  • Go panics print to stderr in plain text; recover them in middleware and log JSON.

Enrichment

  • The agent adds namespace, pod, container, node, and selected pod labels (app).
  • Those become Loki labels (low cardinality) or stored fields — choose carefully (ch 7).
  • Never turn pod into a primary label on huge clusters without thinking: each pod creates new streams.
  • The agent queries the API server: on big clusters use node-local filtering (watch only pods on its own node).

Agent checklist

  • Resource limits and a memory buffer limit.
  • Position file on a hostPath so restarts do not re-send or skip.
  • Backpressure: what happens when the backend is down (buffer to disk, then drop?).
  • Self-metrics scraped by Prometheus (lines read, sent, dropped, lag).
  • Promtail is end of life (2 Mar 2026): new installs use Grafana Alloy (ch 7).
Common mistakes

Structured logging mistakes

Each of these turns logs into either noise, a liability or a large bill.

Shape

  • Variables inside msg ("payment failed for o-123") → impossible to count by message.
  • Same key with different types across services (status "502" vs 502).
  • Raw URL paths in route.
  • Pretty-printed or multi-line JSON → one event split into many lines.
  • Mixing fmt.Println lines with JSON lines.

Content

  • Logging whole request bodies or structs → secrets and PII in logs.
  • Logging the same error at every layer.
  • ERROR for expected outcomes (404, validation) → nobody trusts ERROR anymore.
  • No trace_id: logs and traces cannot be joined.
  • Using log.Info instead of log.InfoContext → IDs missing.

Operations

  • DEBUG left on in production for weeks.
  • Health checks logged at INFO on every probe.
  • Relying on kubectl logs for incidents: gone after rotation or pod deletion.
  • Audit records on the same sampled pipeline as app logs.
  • No volume budget, discovered on the monthly invoice.
Self-check

Structured logging: can you answer these?

Answer precisely; write the code for the Go questions.

Design

  • Why must msg be a constant string?
  • Which ten keys make up the shop log schema, and what type is status?
  • When is a log the right signal instead of a metric or a span?
  • What does each of the four levels promise to the reader?
  • Why keep audit logs on a separate pipeline?

Go and slog

  • What is the difference between log.Info and log.InfoContext for trace correlation?
  • Write a handler wrapper that adds trace_id and span_id.
  • Why must the request middleware wrap the ServeMux directly to see r.Pattern?
  • Two ways to keep a secret out of logs with slog, and the limits of each.
  • How do you switch one service to DEBUG without a redeploy?

Kubernetes

  • Where does a container's stdout end up on the node, and in what format?
  • What do the F and P tags mean?
  • Which kubelet settings control log rotation, and what are their defaults?
  • Name three collection patterns and when each is needed.
  • How do you handle a multi-line stack trace?

Practice: Lab 6 in Hands-on labs — add the shared logx package to orders and payments (schema, request middleware, trace-ready handler, redaction), then break it with DEBUG volume, a leaked secret and a non-JSON line.

Chapter 7

Loki and LogQL

A log store that indexes labels instead of text: how Loki stores streams, how Alloy ships them, how LogQL filters, parses and turns logs into metrics, and how to keep it healthy.

Chapter 7 · Part 7.1

Loki architecture

Streams, chunks and a small index: why Loki is cheap to run and why label design decides everything.

Core idea

Index the labels, not the text

Loki stores log lines compressed in object storage and indexes only a small set of labels per stream. Queries select streams by label, then scan their lines in parallel.

Stream = one unique label set

  • {namespace="shop", app="orders", container="orders", level="INFO"} is one stream.
  • Every line pushed with exactly those labels is appended to it, in time order.
  • Change one label value and you have a new stream.
  • Same model as a Prometheus series, but values are log lines, not numbers.

What gets indexed

  • Only the label set → which chunks hold which stream for which time range.
  • Line content is not indexed: |= "timeout" is a brute-force scan.
  • Result: index is tiny, ingest is cheap, storage is object storage.
  • Scans are fast because they are split by time and stream and run in parallel.

Consequences for you

  • Few labels, low cardinality: namespace, app, container, level.
  • High-cardinality values (trace_id, order_id, user_id) stay in the line or in structured metadata.
  • A narrow stream selector + short time range is the main performance lever.
  • Too many streams hurt ingesters and queries far more than large volume does.
Write and read paths

From a container line to a Grafana panel

Writes are replicated to ingesters and flushed as chunks; reads merge recent data from ingesters with flushed chunks from storage.

Write pathAlloytails pod logsDistributorvalidate, rate limitIngesters (×RF)WAL + in-memory chunksObject storagechunks + TSDB indexCompactorindex compaction, retentionpushhash by streamflushRead pathGrafana / logcliLogQLQuery frontendsplit by time, cacheQueriersscan + mergeIndex gatewaywhich chunks?Ruleralerts, recordingsub-queriesrecent data from ingesterschunks

In monolithic mode all of these run in one process (-target=all); the boxes are still there, just in-process. RF = replication factor (3 by default, 1 in the lab).

Components

Who does what

Know these names: they appear in metrics, logs, Helm values and error messages.

ComponentJobNotes
DistributorReceives pushes, validates labels and limits, hashes each stream to ingestersStateless; first place 429 rate-limit errors come from
IngesterBuilds chunks in memory, writes a WAL, flushes chunks to storage, serves recent dataStateful; replication factor 3 by default
Query frontendSplits queries by time, shards them, queues them, caches resultsStateless; put it in front of queriers
Query schedulerHolds the query queue between frontend and queriersOptional; lets frontends scale independently
QuerierExecutes LogQL over ingesters + storage, deduplicates replicasScale out for read load
Index gatewayServes TSDB index lookupsKeeps index downloads off queriers
CompactorCompacts index files, applies retention and deletesExactly one per cluster
RulerEvaluates LogQL alerting and recording rulesSends alerts to Alertmanager, recorded series via remote write
Pattern ingesterDetects recurring line patternsFeeds Grafana Logs Drilldown pattern views
Streams and chunks

How a stream becomes chunks and index entries

Ingesters keep one open chunk per stream, then cut and flush it. Many tiny streams mean many tiny chunks — the classic Loki performance problem.

Chunk lifecycle (Loki 3.x defaults)

  • Lines for a stream are appended to an open, compressed chunk in memory (and to the WAL).
  • Cut when it reaches chunk_target_size (1.5 MB compressed)…
  • …or after chunk_idle_period (30m) without new lines…
  • …or at max_chunk_age (2h).
  • Flushed chunks go to object storage; the TSDB index maps label set + time → chunk.
  • Low-volume streams flush small chunks: more objects, more index, slower queries.
# schema_config: decides index type and layout
schema_config:
  configs:
    - from: "2024-04-01"   # applies to data from this day
      store: tsdb           # index type (current)
      object_store: s3      # filesystem in the lab
      schema: v13           # needed for structured metadata
      index:
        prefix: index_
        period: 24h         # one index table per day

# never edit an existing entry: add a new one with
# a future "from" date to change schema or store

boltdb-shipper and schemas before v13 are legacy. New installs use tsdb + v13; structured metadata and OTLP ingestion depend on it.

Deployment modes

Monolithic, simple scalable, distributed

Same binary, different -target. Pick by volume and by how much you want to operate.

ModeHow it runsRough size (chart guidance)Status
Monolithic (single binary)One process, -target=all; filesystem or object storageUp to a few tens of GB/dayLab choice; simplest to run
Simple scalableread, write, backend targets; object storage requiredUp to about 1 TB/dayDeprecated; the chart notes it is removed in Loki 4
Distributed (microservices)Each component as its own Deployment/StatefulSet; object storage requiredAbove ~1 TB/dayRecommended for large clusters

Helm chart (Nov 2026)

  • OSS chart lives in grafana-community/helm-charts since 16 Mar 2026.
  • Community chart 12.0.0 renamed SingleBinary to Monolithic (old name still accepted).
  • Built-in MinIO subchart is deprecated: bring your own S3/GCS/Azure bucket.

Where Loki is heading

  • GrafanaCON (Apr 2026) announced a re-architecture: Kafka-backed ingestion, a new query engine and planner, and a columnar "DataObjects" format.
  • Treat it as rolling out, not the default: check the release notes of your version before planning around it.
Where a field belongs

Label, structured metadata, or line content

Structured metadata (Loki 3, schema v13) attaches key/value pairs to each line without creating streams: the right home for IDs you want to filter on.

Decide per field

FieldPut it inWhy
namespace, app, containerLabelLow cardinality, used in every selector
levelLabel (or rely on detected_level)Handful of values
trace_id, span_idStructured metadataUnique per request; filter with | trace_id="…"
order_id, user_idLine (JSON) or structured metadataUnbounded cardinality
route, status, duration_msLine (JSON), parsed at query timeNeeded only by some queries
podLabel is acceptableChurns on every rollout: watch stream counts

OTLP ingestion

  • Loki accepts OTLP logs natively at /otlp/v1/logs.
  • A default set of resource attributes becomes index labels, dots → underscores: e.g. service.name → service_name, k8s.namespace.name → k8s_namespace_name.
  • Other resource and log attributes become structured metadata.
  • Loki 3 also derives service_name for non-OTLP pushes and can add detected_level.
  • Bloom filters to accelerate structured-metadata searches are experimental.
Chapter 7 · Part 7.2

Collecting with Grafana Alloy

Promtail is gone: Alloy discovers pods, tails their logs, processes lines and pushes streams to Loki.

Agent choice

Promtail is end of life; Alloy replaces it

Grafana Alloy is Grafana's OpenTelemetry Collector distribution with native Prometheus and Loki pipelines. It is the supported way to ship logs to Loki.

Timeline

  • Promtail: end of life on 2 Mar 2026 — no further fixes, including security fixes.
  • Grafana Agent (static and flow): end of life on 1 Nov 2025.
  • Alloy 1.21 (Oct 2026) is current at the time of writing.

Migrate a Promtail config

  • alloy convert --source-format=promtail --output=config.alloy promtail.yaml
  • Add --bypass-errors to convert what it can and report the rest.
  • Review the output: relabel rules and stages translate 1:1 into components.
  • Run old and new side by side on a test namespace and compare stream counts.

How Alloy works

  • A config is a graph of components, each with arguments and exports.
  • Exports of one component are wired into arguments of another (forward_to, targets).
  • Syntax: Alloy configuration syntax (HCL-like; formerly "River").
  • Built-in UI on port 12345 shows the live graph and each component's health.
Pipeline

The Kubernetes log pipeline in five components

Discover pods, turn metadata into labels, read the logs, process lines, push to Loki.

1 · discover

  • discovery.kubernetes
  • Lists pods (here: this node only) with __meta_kubernetes_* labels.
→

2 · relabel

  • discovery.relabel
  • Metadata → namespace, pod, container, app.
→

3 · read

  • loki.source.kubernetes
  • Tails container logs via the API; no hostPath.
→

4 · process

  • loki.process
  • Parse JSON, labels, metadata, timestamps, drop noise.
→

5 · write

  • loki.write
  • Batches, pushes to /loki/api/v1/push, retries.

Alternative source: local.file_match + loki.source.file reading /var/log/pods/* from a hostPath (Promtail's classic model). Both work; the API source needs no node mounts but puts load on the API server at scale.

Alloy config

config.alloy for a DaemonSet

Each Alloy pod only discovers pods on its own node (HOSTNAME is set to the node name by the Alloy Helm chart), so no line is shipped twice.

discovery.kubernetes "pods" {
  role = "pod"
  selectors {
    role  = "pod"
    field = "spec.nodeName=" +
      coalesce(sys.env("HOSTNAME"), constants.hostname)
  }
}

discovery.relabel "pods" {
  targets = discovery.kubernetes.pods.targets
  rule {
    source_labels = ["__meta_kubernetes_namespace"]
    target_label  = "namespace"
  }
  rule {
    source_labels = ["__meta_kubernetes_pod_name"]
    target_label  = "pod"
  }
  rule {
    source_labels = ["__meta_kubernetes_pod_container_name"]
    target_label  = "container"
  }
  // app rule: continued on the right
  rule {
    source_labels = ["__meta_kubernetes_pod_label_app"]
    target_label  = "app"
  }
}

loki.source.kubernetes "pods" {
  targets    = discovery.relabel.pods.output
  forward_to = [loki.process.shop.receiver]
}

loki.process "shop" {
  // processing stages: next slide
  forward_to = [loki.write.default.receiver]
}

loki.write "default" {
  endpoint {
    url = "http://loki.logging:3100/loki/api/v1/push"
  }
}

// wiring: <component>.<label>.<export>
//  discovery.relabel.pods.output -> targets
//  loki.process.shop.receiver    -> log receiver
loki.process

Stages: parse once at the edge, label sparingly

Stages run in order on every line. The shop apps log slog JSON (ch 6); other namespaces pass through untouched.

loki.process "shop" {
  stage.match {
    selector = "{namespace=\"shop\"}"
    stage.json {
      expressions = {
        level    = "level",
        trace_id = "trace_id",
        ts       = "time",
      }
    }
    stage.labels {
      values = { level = "" }  // "" = same name
    }
    stage.structured_metadata {
      values = { trace_id = "" }
    }
    stage.timestamp {
      source = "ts"
      format = "RFC3339Nano"
    }
  }
  stage.drop {
    expression          = ".*GET /healthz.*"
    drop_counter_reason = "healthz"
  }
  forward_to = [loki.write.default.receiver]
}

What each stage does

  • stage.match: run nested stages only for lines whose stream matches a LogQL selector.
  • stage.json: extracts fields into a temporary map (nothing is sent as a label yet).
  • stage.labels: promotes level to a stream label (values INFO, WARN, ERROR).
  • stage.structured_metadata: attaches trace_id per line, no new stream.
  • stage.timestamp: uses the app's own time instead of read time.
  • stage.drop: discards noise before it costs anything.
  • Others you will meet: stage.cri/stage.docker (file sources), stage.multiline, stage.logfmt, stage.regex, stage.static_labels, stage.label_drop, stage.output.
Label strategy

Labels that scale and labels that page you at night

Stream count ≈ product of label value counts. Keep it in the thousands per tenant, not the millions.

Good labels

LabelValues
namespacetens
app / service_nametens to hundreds
containera few per app
level3–5
cluster, enva few

Bad labels

LabelWhy
trace_id, request_idOne stream per request
order_id, user_id, emailUnbounded, and PII in the index
status, route, duration_msMultiply streams; parse at query time
ip, url with query stringUnbounded
filename per rotationNew stream on every rotation

Rules of thumb

  • If you would not put it in a Prometheus label, do not put it in a Loki label.
  • Labels describe where a line came from, not what it says.
  • Need to filter by an ID? Structured metadata or a line filter.
  • Check stream counts after every agent change (logcli series --analyze-labels).
Other sources

More ways in: files, CRDs, events and OpenTelemetry

Choose by where logs live and which pipeline you already run.

SourceUse whenNotes
local.file_match + loki.source.fileDaemonSet reading /var/log/pods on the nodeNeeds a hostPath mount (alloy.mounts.varlog: true in the chart) and stage.cri to strip the CRI prefix
loki.source.kubernetesDaemonSet or clustered Deployment without host mountsUses the API server log endpoint; filter by node or enable clustering to avoid duplicates
loki.source.podlogsTeams declare what to collect with a PodLogs custom resourceLike a ServiceMonitor for logs
loki.source.kubernetes_eventsShip Kubernetes Events as log linesRun one instance (or clustering); events otherwise expire after about an hour
otelcol.receiver.* → otelcol.exporter.otlphttp to /otlpApps or Collectors already emit OTLP logsResource attributes become labels/structured metadata (section 24)
OTel Collector filelog receiverYou standardise on the Collector instead of AlloyExport with otlphttp to Loki's OTLP endpoint
Chapter 7 · Part 7.3

LogQL log queries

Select streams, filter lines, parse fields, filter on them and reshape the output.

Anatomy

A log query is a pipeline

A stream selector, then stages joined by |. Each stage sees the output of the one before.

{namespace="shop", app="orders"}         # 1 stream selector  (index lookup: which chunks)
  |= "payment failed"                    # 2 line filter      (cheap substring scan)
  | json                                 # 3 parser           (extracts every JSON key as a label)
  | status >= 500 and duration_ms > 250  # 4 label filters    (typed comparisons)
  | line_format "{{.order_id}} {{.route}} {{.status}} {{.duration_ms}}ms"   # 5 formatting

# sample result line (labels shown in Grafana's details pane):
# o-4182 /orders 502 311ms

Order matters for speed

  • Selector first: fewer streams = fewer chunks downloaded.
  • Line filters before parsers: drop most lines with a cheap substring check.
  • Parse only what survives, then filter on fields.

Two kinds of query

  • Log query: returns lines (this section).
  • Metric query: wraps a log query in a range aggregation like rate(… [5m]) and returns series (section 27).

Where to run it

  • Grafana Explore with the Loki data source (builder or code mode).
  • Grafana Logs Drilldown for query-less browsing.
  • logcli query in a terminal (section 28).
Selectors and line filters

Choosing streams, then matching text

Matchers work like PromQL. Line filters are substring or regex matches on the raw line.

Stream selector matchers

MatcherExample
= equal{app="orders"}
!= not equal{namespace="shop", container!="istio-proxy"}
=~ regex{app=~"orders|payments"}
!~ not regex{namespace!~"kube-.*"}

Line filters

FilterKeeps lines that…
|= "error"contain the string (case-sensitive)
!= "healthz"do not contain it
|~ "(?i)timeout|refused"match the RE2 regex
!~ "GET /metrics"do not match
|> "<_> level=ERROR <_>"match a pattern (<_> = anything)
!> "<_> level=DEBUG <_>"do not match the pattern

A selector needs at least one matcher that cannot match an empty value: {app=~".*"} alone is rejected. Chained filters are ANDed: |= "payment" != "retry".

Parsers

Turning a line into labels at query time

Parsers add extracted labels to each line. Nothing is stored: you pay the parse cost on every query.

# json: every key (nested keys joined with _)
{app="orders"} | json
# {"level":"ERROR","msg":"payment failed",
#  "order_id":"o-41","route":"/orders","status":502}
#  -> labels level_extracted, msg, order_id, route, status

# json with only the fields you need (faster)
{app="orders"} | json status="status", route="route"

# logfmt: key=value lines
{app="legacy"} | logfmt

# pattern: positional, for fixed formats
{app="nginx"}
  | pattern `<ip> - - [<_>] "<method> <uri> <_>" <code> <size>`

# regexp: named RE2 groups
{app="legacy"} | regexp `took (?P<ms>\d+)ms`

# keep only lines that parsed cleanly
{app="orders"} | json | __error__ = ""

Details that trip people up

  • A parsed key that collides with an existing stream label gets the suffix _extracted: with level already a label, | json yields level_extracted.
  • Invalid characters in keys become _ (http.status → http_status).
  • Failed parses set __error__ (e.g. JSONParserErr) instead of dropping the line; filter with | __error__="".
  • Structured metadata needs no parser: {app="orders"} | trace_id="4bf9…".
  • | unpack reverses Alloy's stage.pack.
  • Backticks avoid escaping in patterns and regexes.
Label filters and formatting

Filtering on fields and reshaping output

After parsing, compare fields with types: numbers, durations and byte sizes are understood.

# typed comparisons
{app="orders"} | json | status >= 500
{app="orders"} | json | duration_ms > 250
{app="api"} | logfmt | latency > 250ms     # "312ms", "1.2s"
{app="api"} | logfmt | size > 1MB

# boolean logic
{app="orders"} | json
  | level_extracted = "ERROR" or status >= 500

# reshape the line
{app="orders"} | json
  | line_format "{{.status}} {{.route}} {{.order_id}}"

# rename or create labels
{app="orders"} | json | label_format code=status

# keep or drop labels in the result
{app="orders"} | json | keep route, status
{app="orders"} | json | drop msg, time

# strip ANSI colour codes
{app="legacy"} | decolorize

Good to know

  • String equality route = "/orders" and regex route =~ "/orders.*" work on any label.
  • Comparing a non-number with > sets __error__ for that line.
  • line_format and label_format use Go templates plus functions such as ToUpper, trunc, div.
  • keep/drop cut result size; useful before metric queries to reduce series labels.
  • Filtering on a parsed field is far slower than a line filter on the same text. When you can, write |= "\"status\":502" first, then parse.
Query performance

Fast and slow versions of the same question

"Show me failed payments for order o-4182 in the last hour."

Slow

  • {namespace=~".+"} | json | order_id="o-4182"
  • Every stream in the cluster, every line parsed, 24h default range in some tools.
  • Times out or hits max_query_series / query limits.

Better

  • {namespace="shop", app="orders"} |= "o-4182" | json
  • Two streams-worth of chunks (per level), a substring filter discards almost everything before JSON parsing.
  • Time range: the last hour.

Best

  • If order_id were structured metadata: {app="orders"} | order_id="o-4182".
  • Even better: start from the trace (ch 9) and jump to logs by trace_id.

Watch the query statistics in Grafana (Query inspector → Stats): bytes processed and lines processed tell you whether the selector or the filter needs work.

Chapter 7 · Part 7.4

LogQL metric queries and alerting

Counting lines, aggregating extracted values, and alerting from logs with the ruler.

Range aggregations

Counting log lines over time

Wrap a log query in a range and an aggregation function; the result is a set of time series, just like PromQL.

Log range aggregations

FunctionReturns per stream
count_over_time(q [5m])Number of lines in the window
rate(q [5m])Lines per second
bytes_over_time(q [5m])Bytes of log lines in the window
bytes_rate(q [5m])Bytes per second
absent_over_time(q [5m])1 if there were no lines (for "logs stopped" alerts)
# error lines per second per app
sum by (app) (
  rate({namespace="shop", level="ERROR"}[5m])
)

# top 5 noisiest containers by bytes
topk(5,
  sum by (namespace, container) (
    bytes_rate({namespace=~".+"}[5m])
  )
)

# 5xx responses per route, parsed from the line
sum by (route) (
  count_over_time({app="orders"} | json
    | status >= 500 [5m])
)

Aggregate with sum by (…): without it you get one series per stream, including per pod, which is rarely what a panel needs.

Unwrapped ranges

Aggregating numbers that live inside the line

unwrap turns an extracted label into the sample value, so you can compute sums, averages and quantiles from logs.

# p99 latency per route from duration_ms in the JSON
quantile_over_time(0.99,
  {app="orders"} | json
    | __error__ = ""
    | unwrap duration_ms [5m]
) by (route)

# total charged amount per minute
sum(
  sum_over_time({app="payments"} |= "charged"
    | json | unwrap amount [1m])
)

# values like "312ms" or "1.5s": convert to seconds
avg_over_time({app="api"} | logfmt
  | unwrap duration(latency) [5m]) by (route)

# "10MB" style values
sum_over_time({app="up"} | logfmt
  | unwrap bytes(size) [5m])

Functions on unwrapped values

  • sum_over_time, avg_over_time, min_over_time, max_over_time, stddev_over_time, stdvar_over_time.
  • quantile_over_time(φ, …): quantile over the unwrapped values in the window (sharded execution may approximate).
  • first_over_time, last_over_time; rate / rate_counter on unwrapped values.
  • Grouping goes after the range for these: quantile_over_time(…) by (route).
  • Lines whose value does not parse set __error__: filter them or the query errors.
  • Cost: every matching line is read and parsed at query time. For dashboards you open every day, emit a real metric (ch 2) or use a recording rule.
Logs as RED

Shop service health from logs alone

Useful when a component has no metrics (a vendor binary, a legacy job). For your own services, prefer the Prometheus metrics from ch 2.

# Rate: requests per second (one log line per request)
sum(rate({app="orders"} |= "\"route\"" [5m]))

# Errors: share of requests with status >= 500
sum(count_over_time({app="orders"} | json
      | status >= 500 [5m]))
/
sum(count_over_time({app="orders"} | json
      | status > 0 [5m]))

# Duration: p95 per route
quantile_over_time(0.95,
  {app="orders"} | json | __error__ = ""
    | unwrap duration_ms [5m]) by (route)

Compared with metrics

  • Log-based RED needs one log line per request: sampling logs (ch 6) makes these numbers wrong.
  • Queries scan raw data: seconds per panel at volume, vs milliseconds for PromQL.
  • Retention follows log retention, often shorter than metric retention.
  • Good for: ad-hoc questions, low-traffic or third-party components, alerts on specific messages ("OOM", "certificate expired").
Loki ruler

Alerting rules written in LogQL

Same rule file format as Prometheus (ch 4); only the expression language differs. Alerts go to the same Alertmanager.

# rules for tenant "fake" (auth disabled):
# /etc/loki/rules/fake/shop.yaml
groups:
  - name: shop-logs
    interval: 1m
    rules:
      - alert: OrdersPaymentErrors
        expr: |
          sum(count_over_time(
            {namespace="shop", app="orders", level="ERROR"}
              |= "payment failed" [5m])) > 10
        for: 2m
        labels:
          severity: warning
        annotations:
          summary: "orders: {{ $value }} payment failures in 5m"
          runbook_url: https://runbooks.example/shop/payments

      - alert: OrdersLogsMissing
        expr: absent_over_time({namespace="shop", app="orders"}[10m])
        for: 5m
        labels: {severity: warning}
# Loki config (ruler block)
ruler:
  storage:
    type: local
    local:
      directory: /etc/loki/rules   # <dir>/<tenant>/*.yaml
  rule_path: /var/loki/rules-temp  # scratch space
  alertmanager_url: http://alertmanager-operated.monitoring:9093
  enable_api: true

# check what the ruler loaded / what is firing
curl -s localhost:3100/loki/api/v1/rules
curl -s localhost:3100/prometheus/api/v1/alerts | jq

# with object storage instead of local:
# lokitool rules sync (or the ruler API) uploads
# rule groups per tenant
Recording rules

Precompute log metrics into Prometheus

A recording rule evaluates a LogQL metric query on a schedule and remote-writes the result as a Prometheus series. Dashboards then query cheap PromQL.

# Loki rule file
groups:
  - name: shop-log-metrics
    interval: 1m
    rules:
      - record: app:log_errors:rate5m
        expr: |
          sum by (app) (
            rate({namespace="shop", level="ERROR"}[5m]))

# Loki config: where recorded series go
ruler:
  wal:
    dir: /var/loki/ruler-wal
  remote_write:
    enabled: true
    clients:
      prom:              # any name; url is one line:
        url: http://kps-kube-prometheus-stack-prometheus
             .monitoring:9090/api/v1/write

Making it work

  • Prometheus must accept remote write: in kube-prometheus-stack set prometheus.prometheusSpec.enableRemoteWriteReceiver: true.
  • Recorded series get the rule's labels; add a source="loki" label to tell them apart.
  • Naming follows ch 4: level:metric:operations.
  • Use it for anything shown on a shared dashboard or used in more than one alert.
  • The ruler WAL buffers samples if Prometheus is briefly unreachable.
Chapter 7 · Part 7.5

Operating Loki

Retention, limits, ingestion errors, the command line, and the metrics that tell you Loki is in trouble.

Retention

Deleting data on purpose

Retention is applied by the compactor. Without it, data lives as long as the bucket keeps it.

compactor:
  working_directory: /var/loki/compactor
  retention_enabled: true
  retention_delete_delay: 2h   # grace before deleting
  delete_request_store: s3     # required; lab: filesystem

limits_config:
  retention_period: 744h          # 31 days, global default
  retention_stream:               # per-selector overrides
    - selector: '{namespace="dev"}'
      priority: 1
      period: 72h
    - selector: '{namespace="shop", level="DEBUG"}'
      priority: 2
      period: 24h

Rules

  • Minimum retention period is 24h.
  • Highest priority wins when several retention_stream selectors match.
  • Per-tenant overrides live in the runtime config (overrides:), reloadable without restart.
  • retention_period: 0s = keep forever (the default). Decide explicitly.
  • Object-storage lifecycle rules must be longer than Loki retention, or Loki will find index entries for missing chunks.
  • Targeted deletes (GDPR, leaked secret): log deletion API /loki/api/v1/delete with a selector and time range.
Limits

Per-tenant limits that protect the cluster

Defaults below are Loki 3.x limits_config defaults; Helm charts may override some.

LimitDefaultWhat happens when exceeded
ingestion_rate_mb / ingestion_burst_size_mb4 MB/s / 6 MB per tenant (global strategy: shared by all distributors)Push rejected with HTTP 429 (rate_limited)
max_global_streams_per_user5000New streams rejected (stream_limit); existing ones keep working
max_label_names_per_series15Push rejected (400)
max_line_size256KBLine rejected (line_too_long) unless max_line_size_truncate
reject_old_samples_max_age168h (1 week)Old lines rejected (greater_than_max_sample_age)
max_query_series500Metric query fails: too many series — aggregate more
max_entries_limit_per_query5000Log query returns at most this many lines

Raise limits deliberately per tenant in the runtime config. A team asking for more streams usually has a label problem, not a capacity problem.

Ingestion errors

When lines do not arrive

Clients see HTTP errors; Loki counts every rejected line by reason. Alert on the counter, not on someone noticing a gap.

loki_discarded_samples_total{reason}

reasonMeaningUsual fix
rate_limitedTenant over ingestion rateRaise limit or cut noisy logs
per_stream_rate_limitOne stream too hotAdd a label that splits it (e.g. pod)
stream_limitToo many active streamsRemove high-cardinality labels
line_too_longLine > max_line_sizeTruncate or fix the logger
greater_than_max_sample_ageTimestamp too oldAgent backlog or wrong timestamp parsing
too_far_behindOut-of-order beyond the windowClock skew or bad timestamp stage

Out-of-order writes

  • Accepted by default (unordered_writes: true) as long as a line is not older than the stream's newest line minus half of max_chunk_age (1h with defaults).
  • Typical cause of too_far_behind: stage.timestamp parsing a wrong field, or an agent replaying an old backlog.
  • Alloy shows push errors in its own logs and in loki_write_* metrics.
logcli

Loki from the terminal

Handy for scripts, incident notes and checking label cardinality.

export LOKI_ADDR=http://localhost:3100

logcli labels                         # label names
logcli labels app                     # values of one label
logcli series '{namespace="shop"}'    # streams that match

# cardinality per label for a selector
logcli series --analyze-labels '{namespace="shop"}'
# Total Streams:  12
# Unique Labels:  5
# Label Name  Unique Values  Found In Streams
# pod         6              12
# level       3              12
# ...

logcli query --since=1h --limit=20 \
  '{app="orders"} |= "payment failed" | json'

logcli query --tail '{namespace="shop", level="ERROR"}'

# metric query at one instant
logcli instant-query \
  'sum by (app) (rate({namespace="shop"}[5m]))'

Tips

  • --since, --from/--to (RFC3339) set the range; default is the last hour.
  • --limit caps lines (default 30); --batch pages through large results.
  • -o raw prints just the lines (pipe into jq); -o jsonl includes labels.
  • --org-id sets the tenant when multi-tenancy is on.
  • The same APIs: /loki/api/v1/query_range, /loki/api/v1/labels, /loki/api/v1/series, /loki/api/v1/tail.
Health signals

Watching Loki itself

Loki and Alloy expose Prometheus metrics; scrape them with a ServiceMonitor (release: kps) and alert on a few.

Metrics to watch

MetricTells you
loki_distributor_bytes_received_totalIngest volume per tenant
loki_discarded_samples_totalLines rejected, by reason
loki_ingester_memory_streamsActive streams held in ingesters
loki_request_duration_secondsLatency per route (push, query_range)
loki_ingester_wal_*WAL health and replay
loki_write_* (Alloy)Agent-side send errors and retries

Speed levers

  • Results cache (memcached) in the query frontend: repeated dashboard queries are nearly free.
  • Chunks cache: avoids refetching hot chunks from object storage.
  • Query splitting by time (split_queries_by_interval) and sharding by stream run pieces in parallel.
  • More queriers help only if the selector is narrow; a full-cluster regex scan is slow at any size.
  • In the lab both caches are switched off to save memory.
Common mistakes

Loki mistakes that hurt later

Most Loki outages and bills trace back to labels.

Labels and agents

  • trace_id, order_id or status as labels → millions of streams, stream_limit rejections.
  • Running Promtail in 2026: no security fixes since 2 Mar 2026.
  • Loki source on every DaemonSet pod without a node filter → each line shipped N times.
  • Timestamp stage parsing the wrong field → too_far_behind rejections.

Queries

  • {namespace=~".+"} with a 7-day range from a dashboard.
  • Parsing JSON before a cheap line filter.
  • Forgetting level is already a label, then filtering | json | level="ERROR" (it is level_extracted).
  • Log-based RED on sampled logs.
  • No sum by → one series per stream, max_query_series errors.

Operations

  • Retention never configured: storage grows forever.
  • Bucket lifecycle shorter than Loki retention.
  • Simple scalable mode chosen for a new install (deprecated).
  • No alert on loki_discarded_samples_total.
  • Built-in MinIO used as "real" storage.
Self-check

Loki and LogQL: can you answer these?

Answer with the exact component, label, stage or function.

Architecture and Alloy

  • What does Loki index, and what does it not?
  • When is an ingester chunk cut and flushed?
  • Why is simple scalable mode a poor choice for a new install?
  • Where should trace_id go, and why not a label?
  • How does a DaemonSet of Alloy avoid shipping each line twice?

LogQL

  • Why put |= "o-4182" before | json?
  • Why does | json produce level_extracted in the shop logs?
  • Write: 5xx lines per route per second for orders.
  • Write: p99 of duration_ms per route.
  • What does __error__ contain after a failed parse, and how do you drop those lines?

Operations

  • Which component applies retention, and what else must be set?
  • A team gets stream_limit rejections. What do you look at first?
  • Which metric tells you lines are being discarded, and by which reason?
  • How does a LogQL recording rule reach Prometheus?
  • Which command shows unique values per label for a selector?

Practice: Lab 7 in Hands-on labs — Loki (monolithic) and Alloy in logging, a Grafana data source, LogQL on the shop logs and a ruler alert; then break it with a high-cardinality label.

Chapter 8

OpenTelemetry fundamentals

One vendor-neutral way to produce traces, metrics and logs: the API/SDK split, OTLP, context propagation, resources and semantic conventions, and instrumenting the shop app in Go.

Chapter 8 · Part 8.1

The OpenTelemetry model

What OpenTelemetry is, how its pieces fit, and what is stable today.

What it is

OpenTelemetry is a standard for producing and moving telemetry — not a backend

A CNCF project: one specification, per-language APIs and SDKs, a wire protocol, a Collector, and shared attribute names.

What you get

  • Specification: data model and behaviour for traces, metrics, logs (and profiles).
  • API: what libraries and your code call (tracer.Start, counter.Add).
  • SDK: the implementation that samples, batches and exports.
  • Instrumentation libraries: ready-made spans/metrics for HTTP, gRPC, SQL, Kafka, …
  • OTLP: the protocol between SDKs, Collectors and backends.
  • Collector: receive, process, export (ch 9).
  • Semantic conventions: standard attribute names (section 31).

What it is not

  • Not a storage or UI: traces go to Tempo/Jaeger/X-Ray, metrics to Prometheus/CloudWatch, logs to Loki/CloudWatch Logs.
  • Not a replacement for good logging or metric design: it standardises the plumbing.
  • Not one binary: each language has its own SDK with its own maturity.

Why platform teams adopt it

  • Instrument once, switch or mix backends by changing Collector config.
  • Same context propagation across Go, Java, Python, Node services.
  • Same resource attributes on all signals → correlation (ch 9).
  • AWS moved X-Ray instrumentation to OpenTelemetry (ch 12).
Architecture

From a function call to a backend

Libraries depend only on the API. The application owner installs the SDK once, at startup, and decides where data goes.

Process: orders (Go)Your codetracer.Start, counter.AddInstrumentation libsotelhttp, otelgrpc, otelsqlOpenTelemetry APIno-op unless an SDK is registeredProvidersTracer, Meter, LoggerProcessors / readerssampler, batchExportersOTLP gRPC/HTTPSDK implements the APICollectorotel-gateway (ch 9)OTLP :4317 / :4318Tempo / X-RaytracesPrometheus / CloudWatchmetricsLoki / CloudWatch Logslogs

Sending straight from the SDK to a backend also works; a Collector in between lets you batch, enrich, sample, redact and fan out without redeploying apps.

API vs SDK

Two layers, two audiences

This split is why a library can ship instrumentation without forcing a vendor or even an SDK on its users.

Who uses what

LayerUsed byContains
APILibrary authors, application codeInterfaces: TracerProvider, Tracer, Span, MeterProvider, instruments, propagators; no-op default
SDKApplication owner, once at startupSamplers, span processors, metric readers, views, resource, exporters
InstrumentationApplication owner picksWrappers for frameworks and clients, built on the API
ExporterConfigured in the SDKOTLP (default choice), Prometheus, console/stdout

Signals and their providers

  • Traces: TracerProvider → Tracer → Span.
  • Metrics: MeterProvider → Meter → instruments: Counter, UpDownCounter, Histogram, Gauge, and async Observable* variants.
  • Logs: LoggerProvider with a bridge API — you keep using slog/log4j/logging and a bridge forwards records with trace context.
  • Providers are created once and shut down on exit to flush buffered data.
OTLP

The OpenTelemetry Protocol

Protobuf messages over gRPC or HTTP. Every SDK, the Collector and most backends speak it.

Transports

TransportDefault portPaths / notes
gRPC4317Services for traces, metrics, logs on one port
HTTP/protobuf4318/v1/traces, /v1/metrics, /v1/logs
HTTP/JSON4318Same paths; handy for debugging with curl

Behaviour worth knowing

  • Exporters retry retryable errors (gRPC UNAVAILABLE, HTTP 429/502/503/504) with backoff.
  • Servers can return partial success: some items rejected, with a count and message.
  • Compression: gzip is supported and usually worth enabling across networks.
  • Spec default protocol for OTEL_EXPORTER_OTLP_PROTOCOL is http/protobuf; Go has separate otlptracegrpc and otlptracehttp packages instead.
  • Backends with native OTLP intake: Prometheus 3 (/api/v1/otlp/v1/metrics), Loki (/otlp/v1/logs), Tempo, CloudWatch (ch 11–12).

Endpoint scheme matters for gRPC exporters: http://otel-gateway.otel:4317 means plaintext, https://… means TLS.

Maturity

What is stable in Nov 2026

Stability is per signal and per language. Check the status table for your SDK before relying on a feature.

AreaStatusNotes
Traces (API, SDK, OTLP)StableMature in all major languages
Metrics (API, SDK, OTLP)StableExponential histograms map to Prometheus native histograms
Logs (data model, bridge API, SDK, OTLP)Stable specLanguage SDK maturity varies; check yours
ProfilesPublic Alpha (Mar 2026)Not for production-critical use; protocol not frozen
Span eventsBeing phased outAnnounced early 2026: emit events through the Logs API; existing span events keep working
CollectorCore modules 1.x; contrib 0.xComponent stability is per component (alpha/beta/stable) in its README
Semantic conventionsPer areaHTTP stable (v1.23.0, 2023); database client stable (2025); messaging, GenAI still in development
Metrics model

OTel metrics in Prometheus terms

Same ideas as ch 2 with a few extra concepts: temporality, views and exponential histograms.

Instrument → Prometheus type

OTel instrumentPrometheus result
Counter (monotonic)counter (_total)
UpDownCountergauge
Gauge / ObservableGaugegauge
Histogram (explicit buckets)classic histogram (_bucket, _sum, _count)
Histogram (exponential aggregation)native histogram

Extra concepts

  • Temporality: cumulative (Prometheus style: running total) or delta (per interval). Prometheus needs cumulative; the Collector deltatocumulative processor converts.
  • Views change aggregation, bucket boundaries or drop attributes at the SDK, before export — the place to cut cardinality.
  • Export interval: periodic reader, 60 s by default (OTEL_METRIC_EXPORT_INTERVAL).
  • Exemplars: trace IDs attached to measurements recorded inside a sampled span (ch 9).
  • This deck keeps client_golang for the shop metrics (ch 2) and uses OTel for traces; both are valid choices.
Chapter 8 · Part 8.2

Context propagation and baggage

How a trace survives a network hop, a goroutine and a queue.

Propagation

One trace across two services

The caller injects its span context into request headers; the callee extracts it and starts a child span. Without both halves you get two unrelated traces.

ordersSERVER spanPOST /ordersCLIENT spanPOST (to payments)childpaymentsSERVER spanPOST /chargeINTERNAL spanfraud checktraceparent: 00-<trace-id>-<client-span-id>-01inject: propagator writes headersextract: propagator reads headersAll four spans share one trace ID; each parent ID points to the span that caused it.

In Go, otelhttp.NewTransport injects and otelhttp.NewHandler extracts — but only with a propagator registered via otel.SetTextMapPropagator. Forgetting it is the number one "broken traces" cause.

traceparent

The W3C Trace Context header

The default propagator in every OpenTelemetry SDK. Example from the W3C specification:

traceparent: 00-0af7651916cd43dd8448eb211c80319c-b7ad6b7169203331-01
             │  │                                │                │
             │  trace-id (16 bytes, 32 hex)      parent-id        trace-flags (01 = sampled)
             version (00)                        (8 bytes, 16 hex: the caller's span ID)

tracestate: congo=t61rcWkgMzE,rojo=00f067aa0ba902b7      # optional vendor-specific key=value list

Rules

  • All-zero trace or parent IDs are invalid; a callee must then start a new trace.
  • The sampled flag carries the head-sampling decision downstream: ParentBased samplers honour it (ch 9).
  • tracestate lets vendors carry extra data without breaking others; keep it small.

Other formats you will meet

  • B3 (Zipkin): X-B3-TraceId … or single b3 header.
  • AWS X-Ray: X-Amzn-Trace-Id: Root=1-…;Parent=…;Sampled=1 (ch 12).
  • Jaeger (uber-trace-id), legacy.
  • Configure several at once: OTEL_PROPAGATORS=tracecontext,baggage,xray.
In-process context

Go: the trace lives in context.Context

Span context travels inside ctx. Every function that might create a span or call out must take ctx and pass it on.

// good: the child span and the outgoing request
// both inherit the request's trace
func create(w http.ResponseWriter, r *http.Request) {
	ctx := r.Context()               // set by otelhttp
	ctx, span := tracer.Start(ctx, "reserve stock")
	defer span.End()

	req, _ := http.NewRequestWithContext(ctx,
		"POST", payURL, body)
	resp, err := client.Do(req)      // traceparent injected
	...
}

// background work that outlives the request:
// keep the trace link, drop the cancellation
go func(ctx context.Context) {
	_, span := tracer.Start(ctx, "send receipt")
	defer span.End()
	...
}(context.WithoutCancel(ctx))
// bad: each of these starts a NEW trace
ctx := context.Background()          // context dropped
tracer.Start(context.TODO(), "x")
client.Post(payURL, "application/json", body)
                                     // no ctx at all
http.NewRequest("POST", payURL, body)
                                     // ctx-less request

// bad: span never ends -> never exported
_, span := tracer.Start(ctx, "charge")
if err != nil { return err }         // leaked
span.End()

// rule: start, then immediately
//   defer span.End()

Other languages hide the context in thread-locals or async-locals (Java, Python, Node), so it usually "just works" — until you hand work to a thread pool or callback.

Across queues

Propagating through messages

Queues break the request/response chain. Put the context into message headers on produce and extract it on consume.

prop := otel.GetTextMapPropagator()

// producer: PRODUCER span, inject into headers
ctx, span := tracer.Start(ctx, "publish orders",
	trace.WithSpanKind(trace.SpanKindProducer))
headers := propagation.MapCarrier{}
prop.Inject(ctx, headers)            // traceparent, baggage
msg.Headers = toKafkaHeaders(headers)
span.End()

// consumer: extract, then CONSUMER span
carrier := propagation.MapCarrier(fromKafkaHeaders(msg))
pctx := prop.Extract(context.Background(), carrier)
ctx, span = tracer.Start(pctx, "process orders",
	trace.WithSpanKind(trace.SpanKindConsumer))
defer span.End()

Child or link?

  • One message → one processing: the consumer span can be a child of the producer span.
  • Batch consumer handling 100 messages from 100 traces: start one span and add span links to each message's context (trace.WithLinks).
  • Messaging semantic conventions (messaging.system, messaging.destination.name, messaging.operation.type) are still in development; instrumentation libraries for Kafka, SQS and NATS set them for you.
  • For SQS, carry context in message attributes; AWS SDK instrumentation can do it.
Baggage

Key/value context that travels with the request

Baggage propagates application-defined pairs (baggage header) to every downstream service. It is not added to spans automatically.

// header on the wire
baggage: shop.tenant=acme,
  shop.experiment=checkout-v2

// set at the edge
m, _ := baggage.NewMember(
	"shop.tenant", "acme")
b, _ := baggage.New(m)
ctx = baggage.ContextWithBaggage(ctx, b)

// read anywhere downstream
t := baggage.FromContext(ctx).
	Member("shop.tenant").Value()

// copy onto a span when useful
span.SetAttributes(attribute.String(
	"shop.tenant", t))

Use it for

  • Tenant or customer tier, to filter traces across all services.
  • Experiment or feature-flag variant.
  • A synthetic-test marker so downstream services can tag or drop test traffic.

Risks

  • Sent on every outgoing request, including to third parties: never put secrets, tokens or PII in it.
  • Clients can forge it: do not use it for authorisation decisions.
  • Size adds to every hop; W3C only guarantees 64 members / 8192 bytes.
  • Strip or reset incoming baggage at your public edge.
Trust boundaries

Where traces should start and stop

Propagation is a security and data-quality decision at the edge of your system.

Incoming from the internet

  • A client can send any traceparent, including one with sampled=1 on every request.
  • Options: ignore incoming context at the public edge (new root span) and record the client's value as a link, or honour it only from trusted callers.
  • Sampling decisions should be yours, not the client's.

Outgoing to third parties

  • Headers leak internal trace IDs and baggage to vendors.
  • Use a client without the OTel transport for external calls, or a propagator that injects only traceparent.
  • Some APIs reject unknown headers.

Configuration

  • OTEL_PROPAGATORS (default tracecontext,baggage) selects propagators via env in SDKs that read it.
  • In Go, the composite propagator you register is the one used everywhere.
  • Mixed estates: add b3 or xray so older services keep joining traces during migration.
Chapter 8 · Part 8.3

Resources and semantic conventions

Who emitted the data, and standard names for what it describes.

Resource

The resource: identity shared by every signal

A resource is a set of attributes describing the entity producing telemetry. It is attached once per provider and repeated on every span, metric and log.

AttributeExampleWhy it matters
service.nameordersRequired; default is unknown_service:<executable> — always set it
service.version1.4.2 or git SHACompare before/after a deploy
service.namespaceshopGroups services; with Prometheus forms job="shop/orders"
service.instance.idpod UID or random UUIDDistinguishes replicas; becomes Prometheus instance
deployment.environment.namelab, prodRenamed from deployment.environment in semconv 1.27
k8s.namespace.name, k8s.pod.name, k8s.deployment.name, k8s.node.nameshop, orders-7d9…Usually added by the Collector k8sattributes processor (ch 9)
cloud.provider, cloud.region, cloud.account.idaws, ap-northeast-1Resource detectors (EC2, ECS, EKS) or the Collector
telemetry.sdk.language, telemetry.sdk.versiongo, 1.xAdded by the SDK
Building the resource

Detectors, environment variables and the Collector

Set identity in code or env; let detectors and the Collector add infrastructure attributes.

res, err := resource.New(ctx,
	resource.WithFromEnv(),      // OTEL_SERVICE_NAME,
	                             // OTEL_RESOURCE_ATTRIBUTES
	resource.WithTelemetrySDK(), // telemetry.sdk.*
	resource.WithHost(),         // host.name
	resource.WithProcess(),      // process.* (PID, runtime)
	resource.WithAttributes(
		semconv.ServiceVersion(version),
	),
)

# Kubernetes Deployment: identity from env
env:
  - name: OTEL_SERVICE_NAME
    value: orders
  - name: OTEL_RESOURCE_ATTRIBUTES
    value: deployment.environment.name=lab,service.namespace=shop

Who adds what

  • App: service.*, deployment.environment.name — things only the team knows.
  • SDK detectors: host, process, container ID; contrib detectors for AWS EC2/ECS/EKS and other clouds.
  • Collector k8sattributes: pod, namespace, deployment, node from the API server, matched by pod IP or UID.
  • Collector resourcedetection: env, system, ec2, eks, … on the node.
  • Conflicts: decide one owner per attribute; processors can be set to not overwrite (override: false).
Semantic conventions

HTTP: old names vs stable names

HTTP conventions were declared stable in semconv v1.23.0. Instrumentation that still emits the old names can emit both during migration.

Attribute and metric renames

Old (experimental)Stable
http.methodhttp.request.method
http.status_codehttp.response.status_code
http.targeturl.path + url.query
http.urlurl.full
net.peer.name / net.host.nameserver.address
http.server.duration (ms)http.server.request.duration (s)
http.client.duration (ms)http.client.request.duration (s)

Rules for HTTP spans

  • Server span name: {method} {http.route}, e.g. POST /orders — low cardinality, never the raw path.
  • Client span name: {method}, optionally with a URL template.
  • http.route is the template (/orders/{id}), url.path the actual path.
  • Status: server spans are errors for 5xx only; client spans for 4xx and 5xx.
  • Opt-in env for libraries in transition: OTEL_SEMCONV_STABILITY_OPT_IN=http (new names) or http/dup (both).
  • Duration histogram default buckets (s): 0.005 … 10.
Beyond HTTP

Databases, messaging, errors and your own attributes

Use an existing convention whenever one exists; dashboards and backends understand them.

Common conventions

AreaKey attributesStatus
Database clientdb.system.name, db.namespace, db.operation.name, db.query.text (sanitised)Stable (2025)
Messagingmessaging.system, messaging.destination.name, messaging.operation.typeDevelopment
RPC / gRPCrpc.system, rpc.service, rpc.methodCheck the semconv site
Exceptionsexception.type, exception.message, exception.stacktraceRecorded by span.RecordError
Errorserror.type (e.g. timeout, 500)Stable with HTTP

Your own attributes

  • Namespace them: shop.order.items, shop.customer.tier — never bare type or id.
  • Lowercase, dot-separated, snake_case within a segment.
  • Keep values bounded if they will become metric labels (spans can hold high-cardinality values like an order ID).
  • Never put secrets or full card numbers in attributes; redact in the Collector as a safety net (ch 9).
OTel → Prometheus

How OTel names become Prometheus names

When OTLP metrics land in Prometheus (OTLP receiver, or Collector → remote write), names and resources are translated.

Default translation (UnderscoreEscapingWithSuffixes)

OpenTelemetryPrometheus
http.server.request.duration (s, histogram)http_server_request_duration_seconds + _bucket/_sum/_count
orders.created (Counter, unit {order})orders_created_total ({…} units dropped)
attribute http.response.status_codelabel http_response_status_code
resource service.namespace + service.namejob="shop/orders"
resource service.instance.idinstance
other resource attributestarget_info series (join with group_left, ch 3)
# prometheus.yml (Prometheus 3)
otlp:
  # copy selected resource attributes onto every series
  promote_resource_attributes:
    - service.namespace
    - deployment.environment.name
    - k8s.namespace.name
  # keep UTF-8 names with dots (no _ escaping):
  # translation_strategy: NoUTF8EscapingWithSuffixes

# start flag that enables the endpoint
#   --web.enable-otlp-receiver
# push to: /api/v1/otlp/v1/metrics
Chapter 8 · Part 8.4

Instrumenting the shop app

The Go SDK in practice: setup, HTTP in and out, manual spans, configuration, and zero-code options for other languages.

Go SDK setup

One function wires traces for the whole process

Create the resource, the exporter and the provider; register the provider and the propagator globally; return the shutdown function.

import (
	"context"
	"go.opentelemetry.io/otel"
	"go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc"
	"go.opentelemetry.io/otel/propagation"
	"go.opentelemetry.io/otel/sdk/resource"
	sdktrace "go.opentelemetry.io/otel/sdk/trace"
)

func setupOTel(ctx context.Context) (func(context.Context) error, error) {
	res, err := resource.New(ctx,
		resource.WithFromEnv(), resource.WithTelemetrySDK())
	if err != nil {
		return nil, err
	}
	exp, err := otlptracegrpc.New(ctx) // OTEL_EXPORTER_OTLP_ENDPOINT
	if err != nil {
		return nil, err
	}
	tp := sdktrace.NewTracerProvider(
		sdktrace.WithBatcher(exp), sdktrace.WithResource(res))
	otel.SetTracerProvider(tp)
	otel.SetTextMapPropagator(propagation.NewCompositeTextMapPropagator(
		propagation.TraceContext{}, propagation.Baggage{}))
	return tp.Shutdown, nil
}

Notes

  • WithBatcher uses the batch span processor: queue 2048, batch 512, 5 s delay (spec defaults, tunable with OTEL_BSP_*).
  • Default sampler is ParentBased(AlwaysSample); the Go SDK also reads OTEL_TRACES_SAMPLER / _ARG.
  • Call the returned shutdown on exit (with a timeout) or the last batch is lost.
  • Without SetTracerProvider, every tracer is a no-op: no error, no data.
  • Without SetTextMapPropagator, spans exist but each service starts its own trace.
  • Metrics follow the same pattern: sdkmetric.NewMeterProvider(sdkmetric.WithReader(sdkmetric.NewPeriodicReader(exp))) + otel.SetMeterProvider.
HTTP in and out

otelhttp: server spans, client spans, propagation

Wrap the whole handler chain once, outermost, so logging and metrics middleware (ch 2, ch 6) run inside the span. Wrap the client transport to create CLIENT spans and inject headers.

// server: otelhttp outermost, mux innermost
handler := otelhttp.NewHandler(
	metrics.Middleware(logx.Middleware(log, routeSpan(mux))),
	"orders", // fallback span name
	otelhttp.WithFilter(func(r *http.Request) bool {
		return r.URL.Path != "/healthz" && r.URL.Path != "/metrics"
	}))

// mux sets r.Pattern ("POST /orders"); name the span after it
func routeSpan(next http.Handler) http.Handler {
	return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
		next.ServeHTTP(w, r)
		if _, route, ok := strings.Cut(r.Pattern, " "); ok {
			span := trace.SpanFromContext(r.Context())
			span.SetName(r.Pattern)
			span.SetAttributes(attribute.String("http.route", route))
		}
	})
}
// client: requests made with ctx join the trace
client = &http.Client{Timeout: 2 * time.Second,
	Transport: otelhttp.NewTransport(http.DefaultTransport)}
req, _ := http.NewRequestWithContext(ctx, "POST", payURL, body)
resp, err := client.Do(req)

What you get

  • SERVER span POST /orders with http.request.method, http.route, http.response.status_code, url.path, server.address.
  • Every log line and metric observation in the request sees a valid span context → trace_id in logs, exemplars on histograms (ch 9).
  • Probes and scrapes are not traced (WithFilter returns false).
  • CLIENT span per outgoing request, child of the active span, with traceparent injected.
  • 5xx responses mark the server span as error.
  • Recent otelhttp releases also read r.Pattern themselves (otelhttp.WithRouteTag was deprecated and later removed); routeSpan keeps the result independent of the version.
  • otelhttp records http.server.request.duration through the global MeterProvider (no-op if none is set).
Manual spans

Adding business detail the library cannot know

Library spans show HTTP; your spans show what the request was doing.

var tracer = otel.Tracer("obs-lab/orders")

func create(w http.ResponseWriter, r *http.Request) {
	ctx := r.Context()
	span := trace.SpanFromContext(ctx)  // the otelhttp span
	...
	span.SetAttributes(
		attribute.String("shop.order.id", o.ID),
		attribute.Int("shop.order.items", 1))

	ctx, pay := tracer.Start(ctx, "charge payment")
	defer pay.End()
	if err := charge(ctx, o); err != nil {
		pay.RecordError(err)                 // exception.* event
		pay.SetStatus(codes.Error, "charge failed")
		http.Error(w, "payment failed", 502)
		return
	}
	...
}

Guidelines

  • Name spans after the operation, not the data: charge payment, not charge o-4182.
  • Put IDs in attributes; high-cardinality values are fine on spans.
  • Set status Error only for real failures; leave Unset for success (Ok is for explicit overrides).
  • RecordError stores the error as an exception event; with span events being phased out, check what your SDK version emits.
  • Tracer name = instrumentation scope (module path), not the service name.
  • One span per meaningful step (DB call, remote call, expensive computation) — not per function.
Configuration by environment

The OTEL_* variables

Defined by the specification so the same Deployment settings work across languages. Support varies slightly per SDK.

VariableExampleEffect
OTEL_SERVICE_NAMEordersSets service.name (wins over OTEL_RESOURCE_ATTRIBUTES)
OTEL_RESOURCE_ATTRIBUTESdeployment.environment.name=labExtra resource attributes, comma-separated
OTEL_EXPORTER_OTLP_ENDPOINThttp://otel-gateway.otel:4317Base endpoint for all signals (_TRACES_ENDPOINT etc. per signal)
OTEL_EXPORTER_OTLP_PROTOCOLgrpc, http/protobufTransport, in SDKs that pick it at runtime
OTEL_TRACES_SAMPLER / _ARGparentbased_traceidratio / 0.25Head sampling (ch 9)
OTEL_PROPAGATORStracecontext,baggagePropagators (default shown)
OTEL_TRACES_EXPORTER, OTEL_METRICS_EXPORTER, OTEL_LOGS_EXPORTERotlp, console, noneExporter choice in agents and autoconfigure SDKs
OTEL_METRIC_EXPORT_INTERVAL60000Metric export period in ms
OTEL_SDK_DISABLEDtrueTurn the SDK off without a code change
Zero-code instrumentation

Instrumenting without changing code

Agents and wrappers load instrumentation libraries at startup and configure the SDK from OTEL_* variables.

LanguageHowNotes
Java-javaagent:opentelemetry-javaagent.jar (e.g. via JAVA_TOOL_OPTIONS)Most complete: hundreds of libraries
Pythonopentelemetry-bootstrap -a install then opentelemetry-instrument python app.pyInstalls instrumentations matching your packages
Node.jsnode --require @opentelemetry/auto-instrumentations-node/register app.jsMust load before other modules
.NETOpenTelemetry .NET Automatic Instrumentation (CLR profiler)Also a NuGet-based option
GoeBPF-based: OpenTelemetry Go auto-instrumentation and OBI (OpenTelemetry eBPF Instrumentation)Needs elevated privileges; covers common libraries; compile-time instrumentation is a separate SIG effort

Zero-code gets you HTTP, RPC and DB spans quickly. Business attributes, custom spans and good span names still need a few lines of code.

OpenTelemetry Operator

Injecting instrumentation in Kubernetes

The Operator manages Collectors (OpenTelemetryCollector CR) and injects language agents into pods via an Instrumentation CR and an annotation.

apiVersion: opentelemetry.io/v1alpha1
kind: Instrumentation
metadata:
  name: default
  namespace: shop
spec:
  exporter:
    endpoint: http://otel-gateway.otel:4318
  propagators: [tracecontext, baggage]
  sampler:
    type: parentbased_traceidratio
    argument: "0.25"
  resource:
    resourceAttributes:
      deployment.environment.name: lab
---
# in the Deployment's pod template
metadata:
  annotations:
    instrumentation.opentelemetry.io/inject-java: "true"
    # inject-python, inject-nodejs, inject-dotnet,
    # inject-go (+ otel-go-auto-target-exe: /app)

How it works

  • A mutating webhook adds an init container that copies the agent into a shared volume.
  • It sets JAVA_TOOL_OPTIONS / PYTHONPATH / NODE_OPTIONS and OTEL_* env (service name from the workload, k8s resource attributes).
  • Pods must be (re)created after the CR exists: injection happens at admission.
  • Go injection runs a privileged eBPF sidecar and must be enabled in the Operator.
  • The Operator needs cert-manager (or another way to provide webhook certificates).
  • This deck installs the Collector with its Helm chart instead (Lab 8); the Operator is the next step when many teams want auto-instrumentation.
Common mistakes

OpenTelemetry mistakes that break traces

Most "OTel does not work" tickets are one of these.

Setup

  • No otel.SetTracerProvider: all spans are no-ops, silently.
  • No propagator registered: every service starts its own trace.
  • Shutdown never called: the last seconds of spans vanish on exit or rollout.
  • service.name unset → unknown_service:app in every backend.
  • gRPC exporter pointed at port 4318 (or HTTP at 4317).

Code

  • context.Background() or client.Post in the request path → broken parent chain.
  • Missing defer span.End() on early returns.
  • Span names with IDs (GET /orders/o-41): unbounded names break span metrics.
  • Status Error on 404s from servers.
  • Secrets or PII in attributes or baggage.

Conventions

  • Mixing old (http.status_code) and new names across services without http/dup during migration.
  • Custom attributes without a namespace (id, type).
  • Dashboards built on deployment.environment after instrumentation moved to deployment.environment.name.
  • Forgetting the unit: http.server.request.duration is seconds, not ms.
Self-check

OpenTelemetry fundamentals: can you answer these?

Answer with the exact API, header, attribute or variable.

Model

  • Why do libraries depend on the API and not the SDK?
  • What happens when nothing registers a TracerProvider?
  • Which ports and paths does OTLP use by default?
  • Which signals are stable, and what is the status of profiles?
  • Which OTel instrument becomes a Prometheus gauge?

Propagation

  • Decode 00-0af7…319c-b7ad…3331-01: what are the three fields after the version?
  • Which two otelhttp wrappers inject and extract?
  • Why does context.Background() in a handler break a trace?
  • Child span or span link for a batch consumer? Why?
  • Name two risks of baggage.

Resources and conventions

  • Which attribute is required, and what is its default?
  • What is the stable name for http.status_code? For http.server.duration?
  • What should a server span for GET /orders/o-41 be called?
  • How do service.namespace and service.name appear in Prometheus?
  • What does OTEL_SEMCONV_STABILITY_OPT_IN=http/dup do?

Practice: Lab 8 in Hands-on labs — the otel-gateway Collector with a debug exporter, orders and payments instrumented with the Go SDK and otelhttp, and one trace spanning both; then break propagation and watch the trace split.

Chapter 9

Traces, the Collector and Tempo

Read a trace like a profiler for a distributed request, decide which traces to keep, run the Collector as the telemetry router, query Tempo with TraceQL, and connect traces to metrics and logs.

Chapter 9 · Part 9.1

Spans in depth

Kinds, status, attributes, events and links: what a span records and how a waterfall is read.

Spans

Anatomy of a span

A span is one timed operation. A trace is the tree of spans that share one trace ID.

# one span from the shop app, as the SDK exports it (simplified)
trace_id:        4bf92f3577b34da6a3ce929d0e0e4736
span_id:         00f067aa0ba902b7
parent_span_id:  a3ce929d0e0e4736        # empty on root
name:            "POST /charge"          # route, not URL
kind:            SPAN_KIND_SERVER
start, end:      01.120 s, 01.388 s     # 268 ms
status:          ERROR "card declined"
attributes:
  http.request.method:       POST
  http.route:                /charge
  http.response.status_code: 502
resource:                                # who produced it (ch 8)
  service.name: payments
  deployment.environment.name: lab
events:  [ exception {exception.type, …} ]
links:   []

What each part is for

  • IDs: 16-byte trace ID, 8-byte span ID; the parent ID builds the tree.
  • Name: groups spans for analysis; a URL with IDs in it makes every span unique.
  • Kind: who called whom (next slide).
  • Status: Unset, Ok or Error — the error flag queries filter on.
  • Attributes: what happened in this operation (semantic conventions, ch 8).
  • Resource: the process that emitted it, shared by all its spans.
  • Events / links: timestamped points inside the span, and references to other traces.
Spans

Span kinds tell you which side of a call you are looking at

An HTTP call between two services produces two spans: a CLIENT span in the caller and a SERVER span in the callee.

The five kinds

KindMeaningShop example
SERVERHandles an incoming synchronous requestorders handling POST /orders
CLIENTMakes an outgoing synchronous requestorders calling payments
PRODUCERHands a message to a broker or queuepublish order.created
CONSUMERProcesses a message from a brokerworker handling order.created
INTERNALIn-process work (the default)validateOrder()

Why the pair matters

  • CLIENT duration − SERVER duration ≈ network + queueing + connection setup.
  • A CLIENT span with no SERVER child: the callee is not instrumented, or its spans were dropped.
  • Service graphs are built from CLIENT→SERVER (and PRODUCER→CONSUMER) pairs (section 37).
  • Pick the kind deliberately in manual spans; auto-instrumentation (otelhttp) sets it for you.
Spans

Status: when is a span an error?

Status is not the HTTP status code. The semantic conventions define the mapping so dashboards and TraceQL agree.

Status codes

  • Unset (default): the instrumentation made no judgment.
  • Error: the operation failed; add a description.
  • Ok: an explicit "this succeeded" set by the application; it overrides later Error from instrumentation. Use rarely.

HTTP mapping (semconv)

  • Server span: 5xx → Error; 4xx → left Unset (the client sent a bad request, the server did its job).
  • Client span: 4xx and 5xx → Error.
  • Network failure, timeout or no response → Error on the client span.
  • otelhttp applies this mapping for you.
// Go: record a failure on the current span
import (
    "go.opentelemetry.io/otel/codes"
    "go.opentelemetry.io/otel/trace"
)

span := trace.SpanFromContext(ctx)
if err != nil {
    span.RecordError(err) // event
    span.SetStatus(codes.Error,
        "charge failed")
    return err
}
Spans

Attributes, events and links

Attributes describe the whole operation; events mark moments inside it; links connect spans that are related but not parent and child.

Attributes

  • Key-value pairs: string, bool, int, float, or arrays of them.
  • Prefer semantic convention keys (http.route, db.system.name, messaging.system).
  • Custom keys get a namespace: shop.order.items, shop.customer.tier.
  • High-cardinality values (order ID) are fine on spans — traces are sampled and not aggregated like metrics.
  • Never secrets or raw PII (ch 6).

Events

  • Timestamped name + attributes inside a span: exception, retry, cache miss.
  • span.RecordError(err) adds an exception event.
  • Direction (2026): the Span Event API is being deprecated in favor of emitting events as log records through the Logs API, correlated by trace context. Existing span events keep working; follow your SDK release notes.

Links

  • A span can reference spans in other traces.
  • Batch consumer: one span processes 50 messages → 50 links to the producers' spans.
  • Long jobs split into several traces, linked to the trigger.
  • Links are set at span start in most SDKs: trace.WithLinks(...).
  • Grafana shows links in the span detail; follow them to the related trace.
Reading traces

Reading a waterfall: the critical path

Total latency is decided by the longest chain of sequential work, not by the sum of all spans.

orders POST /orders (SERVER) 412 msorders SELECT customers 38 msorders GET inventory (CLIENT) 60 ms — parallel with chargeorders POST /charge (CLIENT) 300 mspayments POST /charge (SERVER) 285 mspayments fraud-check (INTERNAL) 240 msorders INSERT order 20 ms0 ms412 mscritical path: POST /orders → POST /charge → fraud-check

Speeding up GET inventory (off the critical path) saves nothing; fixing fraud-check (240 of 412 ms) does.

Reading traces

Patterns you will recognise in a waterfall

Most latency problems have a shape. Learn the shapes and a trace tells you the fix.

Shape in the waterfallWhat it usually meansNext step
Staircase of identical short DB/HTTP spansN+1 queries or a call inside a loopBatch the call; check the code path in code.function attributes
One long child span covering most of the parentA slow dependencyOpen the dependency's SERVER span; check its own children
Gap before the first childWork before any call: lock wait, GC, CPU, a missing spanAdd a manual span or check profiles
CLIENT span much longer than its SERVER childNetwork, DNS, connection pool exhaustion, queueingCheck client pool metrics and DNS spans
Repeated CLIENT spans with errors, then successRetriesCount retries; check timeouts and backoff
Wide fan-out, total = slowest branchTail latency amplificationHedge, cache or cut the fan-out
Chapter 9 · Part 9.2

Sampling

Keep the traces that explain problems, pay for fewer of the boring ones, and keep the decision consistent across services.

Sampling

Why sample, and the two places to decide

Tracing every request of a busy service is expensive to export, store and query. The question is which traces you can afford to lose.

Head sampling — in the SDK, at the root spanRoot span startsdecision nowSampled flagtraceparent …-01 / -00Every downstream service follows the flagParentBased samplerTail sampling — in the Collector, after the trace endsAll spansexported (100 %)Buffer by tracedecision_waitPolicieserror? slow?Keepto TempoDrop

Head

  • Cheap: unsampled spans are never exported.
  • Blind: decides before knowing if the request fails or is slow.

Tail

  • Sees the whole trace: keep all errors and slow ones.
  • Costs memory and needs every span of a trace on one Collector.
Sampling

Head sampling in the SDK

Use ParentBased everywhere so a service never breaks a trace that an upstream service decided to keep.

// Go SDK: 10 % of new traces, follow the parent otherwise
import sdktrace "go.opentelemetry.io/otel/sdk/trace"

tp := sdktrace.NewTracerProvider(
    sdktrace.WithSampler(
        sdktrace.ParentBased(sdktrace.TraceIDRatioBased(0.10)),
    ),
    sdktrace.WithBatcher(exporter),
    sdktrace.WithResource(res),
)

# the same with environment variables (no code change)
OTEL_TRACES_SAMPLER=parentbased_traceidratio
OTEL_TRACES_SAMPLER_ARG=0.10

Built-in samplers

SamplerBehaviour
AlwaysOn / always_onRecord and export everything (SDK default is parentbased_always_on)
AlwaysOff / always_offNothing
TraceIDRatioBased(r)Keep a fraction, decided from the trace ID, so every service agrees
ParentBased(root)Follow the parent's sampled flag; use root only for new traces

A service that uses plain TraceIDRatioBased with a different ratio than its callers creates broken traces: some spans kept, their parents dropped. Wrap it in ParentBased.

Sampling

Tail sampling with the Collector

The tail_sampling processor (contrib) waits for a trace's spans, then evaluates policies. A trace is kept if any policy says sample (unless a drop policy matches).

processors:
  tail_sampling:
    decision_wait: 10s          # wait for late spans of a trace
    num_traces: 50000           # traces held in memory
    expected_new_traces_per_sec: 100
    policies:
      - name: keep-errors
        type: status_code
        status_code: {status_codes: [ERROR]}
      - name: keep-slow
        type: latency
        latency: {threshold_ms: 500}
      - name: keep-checkout-route
        type: string_attribute
        string_attribute: {key: http.route, values: [/orders]}
      - name: baseline
        type: probabilistic
        probabilistic: {sampling_percentage: 5}

Policy types you will use

  • status_code, latency, probabilistic, string_attribute, numeric_attribute, boolean_attribute.
  • rate_limiting (spans per second), span_count, trace_state, ottl_condition.
  • and (all sub-policies), composite (rate shares per sub-policy), drop (explicitly drop, e.g. health checks).
  • Sizing: memory ≈ num_traces × spans per trace × span size. Spans arriving after decision_wait are handled by the decision cache (if configured) or treated as a new trace.
  • Watch the processor's own metrics for dropped-too-early and late spans.
Sampling

Tail sampling at scale: route by trace ID

Every span of a trace must reach the same Collector instance. With several gateway replicas, add a load-balancing tier in front.

orders podagentpayments podagentworker podagentLB Collectorsloadbalancing exporterGateway 1tail_samplingGateway 2tail_samplingGateway 3tail_samplinghash(trace_id) → the same gateway for every span
# tier 1: route spans by trace ID
exporters:
  loadbalancing:
    routing_key: traceID
    protocol:
      otlp:
        tls: {insecure: true}
    resolver:
      k8s:
        service: otel-sampler.otel   # headless service
        ports: [4317]

When a gateway pod is added or removed, the hash ring changes and in-flight traces can be split. Keep the sampling tier stable and size it for peak, not average.

Sampling

What sampling does to everything else

Sampled traces are a biased subset. Anything that counts must be computed before the sampler, or from metrics.

Derive RED before sampling

  • Request rate, error rate and latency come from metrics (ch 2) or from span metrics generated before tail_sampling in the pipeline (section 37).
  • Counting spans in Tempo after tail sampling over-counts errors: you kept all of them, but only 5 % of successes.

Consistency

  • TraceIDRatioBased is consistent because every service hashes the same trace ID.
  • OpenTelemetry's consistent probability sampling records the threshold in tracestate (ot=th:…) so later stages can adjust counts.
  • Mixing head and tail sampling: tail only sees what head kept.

Practical defaults

  • Small systems: keep 100 %, no sampling at all.
  • Growing: parentbased_always_on in SDKs + tail sampling (errors, slow, 5–10 % baseline).
  • Very large: head sampling at the edge + tail sampling in gateways.
  • Always keep exemplar-linked traces in mind: an exemplar may point to a trace the tail sampler dropped.
Chapter 9 · Part 9.3

The OpenTelemetry Collector

A vendor-neutral process that receives, transforms and exports telemetry: the router between your apps and every backend.

Collector

Building blocks and pipelines

Components are declared once, then wired into pipelines per signal under service.pipelines. A component that is declared but not used in a pipeline is not started.

Collector processReceiversotlp, prometheus, filelogProcessors (ordered)memory_limiter → … → batchExportersotlp, prometheusremotewriteConnectorsspanmetrics, servicegraphBackendsTempo, Prometheus, Lokitraces pipeline: otlp → memory_limiter, k8sattributes, tail_sampling, batch → otlp/tempo + spanmetricsmetrics pipeline: spanmetrics (connector as receiver) → batch → prometheusremotewriteExtensions (not in pipelines): health_check, pprof, zpages, auth (basicauth, sigv4auth, oauth2client)A connector is an exporter in one pipeline and a receiver in another: traces in, metrics out.
Collector

A complete gateway config

This is the shape of otel-gateway in the labs: OTLP in, Kubernetes metadata added, traces to Tempo, span metrics to Prometheus.

receivers:
  otlp:
    protocols:
      grpc: {endpoint: 0.0.0.0:4317}
      http: {endpoint: 0.0.0.0:4318}
processors:
  memory_limiter:
    check_interval: 1s
    limit_percentage: 80
    spike_limit_percentage: 20
  k8sattributes: {}            # see next slides
  batch: {}
connectors:
  spanmetrics: {}
exporters:
  otlp/tempo:
    endpoint: tempo.tracing.svc:4317
    tls: {insecure: true}
  prometheusremotewrite:
    endpoint: http://prometheus-operated.monitoring.svc:9090/api/v1/write
extensions:
  health_check: {endpoint: 0.0.0.0:13133}
service:
  extensions: [health_check]
  pipelines:
    traces:
      receivers: [otlp]
      processors: [memory_limiter, k8sattributes, batch]
      exporters: [otlp/tempo, spanmetrics]
    metrics/spanmetrics:
      receivers: [spanmetrics]
      processors: [memory_limiter, batch]
      exporters: [prometheusremotewrite]
  telemetry:
    metrics:
      readers:
        - pull:
            exporter:
              prometheus: {host: 0.0.0.0, port: 8888}

type/name (e.g. otlp/tempo) gives several instances of one component type. Validate before deploying: otelcol-contrib validate --config=config.yaml.

Collector

Processors that every production pipeline needs

Order matters: processors run in the order listed in the pipeline.

ProcessorPurposeWhere in the pipeline
memory_limiterRefuses data when memory crosses the limit, so the Collector backs off instead of OOM-crashingFirst
k8sattributesAdds k8s.namespace.name, k8s.pod.name, k8s.deployment.name, … by matching the sender's podEarly, before anything that filters on them
resource / attributesInsert, upsert, delete or hash resource / span attributesMiddle
transformOTTL statements: rename, redact, set values conditionallyMiddle
filterDrop spans, metrics or logs matching OTTL conditions (health checks, noisy debug logs)Before expensive work
tail_samplingKeep or drop whole traces (section 34)After enrichment, before batch
batchGroups data into larger requests; fewer, bigger exportsLast

Newer Collector versions can also batch inside the exporter (sending_queue batching in exporterhelper); the batch processor is still widely used. Check the docs for the version you run.

Collector

Enrich, redact and filter with k8sattributes, transform and filter

Do metadata and hygiene once in the Collector instead of in every service.

processors:
  k8sattributes:
    auth_type: serviceAccount
    # RBAC: get/list/watch pods,
    # namespaces, replicasets
    extract:
      metadata: [k8s.namespace.name,
        k8s.pod.name, k8s.deployment.name,
        k8s.node.name]
    pod_association:
      - sources:
          - from: resource_attribute
            name: k8s.pod.ip
      - sources:
          - from: connection  # sender IP
  resource:
    attributes:
      - key: deployment.environment.name
        value: lab
        action: upsert
  transform:
    error_mode: ignore
    trace_statements:
      - context: span
        statements:
          - delete_key(attributes,
              "http.request.header.authorization")
          - replace_pattern(attributes["url.query"],
              "token=[^&]+", "token=REDACTED")
  filter/health:
    error_mode: ignore
    traces:
      span:
        - 'attributes["url.path"] == "/healthz"'

from: connection only works when the Collector sees the pod IP directly (agent DaemonSet, or a gateway without a proxy in front). Behind a load balancer, send k8s.pod.ip as a resource attribute from the agent.

Collector

Deployment patterns: agent, gateway, or both

Agents sit next to the workload; gateways are a central, scalable tier. Most clusters end up with both.

Agent (DaemonSet or sidecar)

  • One per node: apps send to $(NODE_IP):4317 or a node-local service.
  • Reads node-local data: container log files (filelog), kubelet stats (kubeletstats), host metrics.
  • k8sattributes with from: connection works.
  • Small memory budget; keep processing light.

Gateway (Deployment + Service)

  • A few replicas behind a Service; scales with HPA.
  • Holds credentials to backends (one place to rotate).
  • Heavy work: tail sampling (with trace-ID routing), span metrics, redaction.
  • In the labs: otel-gateway in namespace otel, apps export to it directly.

Choosing

  • Small cluster, traces only: gateway alone is fine (the labs).
  • Logs from files or node metrics: you need agents.
  • Many backends or tenants: agent → gateway.
  • Managed alternatives: Grafana Alloy (embeds Collector components), ADOT on AWS (ch 12).
Collector

Distributions and how to run it

The Collector is a toolkit: a distribution is a binary with a chosen set of components.

Distributions

DistributionContents
otelcol (core)Small set: OTLP, debug, batch, memory_limiter
otelcol-contribAlmost every community component (tail_sampling, k8sattributes, spanmetrics, prometheusremotewrite)
otelcol-k8sComponents curated for Kubernetes
Custom (ocb builder)Only what you list in a manifest: smaller binary, smaller attack surface
ADOTAWS Distro for OpenTelemetry: AWS exporters and SigV4 auth (ch 12)
Grafana AlloyGrafana's collector; Alloy 1.21 adds an OpenTelemetry Engine (public preview)
# Helm (open-telemetry repo), contrib image, gateway mode
helm upgrade --install otel-gateway \
  open-telemetry/opentelemetry-collector -n otel \
  --set mode=deployment \
  --set image.repository=otel/opentelemetry-collector-contrib \
  -f gateway-values.yaml   # 'config:' merged into chart defaults

# or with the OpenTelemetry Operator:
# kind: OpenTelemetryCollector  (spec.mode, spec.config)

# Validate locally before you ship
docker run --rm -v $PWD/config.yaml:/c.yaml \
  otel/opentelemetry-collector-contrib \
  validate --config=/c.yaml
Collector

Monitor the Collector itself

A Collector that silently drops data is worse than none. Scrape its internal metrics (port 8888 above) and alert on refusals and failures.

Key internal metrics (as scraped by Prometheus)

MetricMeaning
otelcol_receiver_accepted_spans_totalSpans received successfully
otelcol_receiver_refused_spans_totalRejected at the receiver (often memory_limiter back-pressure)
otelcol_exporter_sent_spans_totalDelivered to the backend
otelcol_exporter_send_failed_spans_totalExport failures after retries
otelcol_exporter_queue_size / _capacitySending queue fill level
# alert: the Collector is failing to export
sum by (exporter) (
  rate(otelcol_exporter_send_failed_spans_total[5m])
) > 0

# queue more than 80 % full: backend too slow
max by (exporter) (
  otelcol_exporter_queue_size
  / otelcol_exporter_queue_capacity
) > 0.8

Same pattern for metrics and logs: _metric_points_ and _log_records_ variants. Exact names can change between Collector versions; check curl :8888/metrics on yours. The debug exporter (verbosity: detailed) is the quickest way to see what a pipeline really carries.

Chapter 9 · Part 9.4

Tempo and TraceQL

A trace store that only needs object storage, and a query language that selects spans by what they did.

Tempo

Tempo in one slide

Tempo stores traces in object storage as Parquet blocks and finds them by trace ID or by TraceQL. It indexes no labels up front.

Design ideas

  • Object storage (S3, GCS, Azure, or local disk for labs) is the only required dependency in monolithic mode.
  • Columnar Parquet blocks: queries read only the columns they touch.
  • Receives OTLP (and Jaeger, Zipkin) directly.
  • Query by trace ID, TraceQL search, or TraceQL metrics.
  • Query from Grafana (Tempo data source) or tempo-cli.

Tempo 3 (2026) architecture

  • Microservices mode: distributors write spans to a Kafka-compatible log; block-builders consume and write blocks to object storage; live-stores serve recent data.
  • Ingesters, the scalable single binary mode and the compactor target are removed.
  • The metrics-generator consumes from Kafka.
  • Monolithic mode runs without Kafka (the labs use this).
  • TraceQL metrics GA and on by default.

Upgrading from 2.x

  • Microservices installs need Kafka (or Redpanda) first.
  • tempo-cli migrate config converts 2.x configs.
  • Block format vParquet4 or later; 3.1 writes vParquet5, older blocks stay readable.
  • Helm charts moved to grafana-community/helm-charts.
  • Read the official upgrade guide for your exact version.
TraceQL

Spansets: selecting spans by what they did

A TraceQL query returns traces that contain matching spans. { … } selects a spanset; conditions use scoped attributes and intrinsics.

# all spans of the orders service
{ resource.service.name = "orders" }

# failed spans in payments
{ resource.service.name = "payments" && status = error }

# slow server spans on one route
{ span.http.route = "/orders" && kind = server && duration > 500ms }

# 5xx responses, any service
{ span.http.response.status_code >= 500 }

# whole-trace intrinsics
{ trace:rootService = "orders" && trace:duration > 1s }

Scopes and intrinsics

SyntaxMeans
resource.xResource attribute (service.name, k8s.pod.name)
span.xSpan attribute (http.route)
name, status, kind, durationSpan intrinsics (span:name, span:status, … also accepted)
status valueserror, ok, unset
kind valuesserver, client, producer, consumer, internal, unspecified
trace:duration, trace:rootName, trace:rootServiceTrace-level intrinsics

Unscoped .x matches either scope but checks both; prefer span. or resource. — it reads one column instead of two.

TraceQL

Structure, pipelines and aggregates

Spansets can be combined by their position in the tree, and filtered with aggregates per trace.

# orders calls payments and the payments side fails
{ resource.service.name = "orders" }
  >> { resource.service.name = "payments" && status = error }

# direct child only
{ name = "POST /orders" } > { kind = client }

# siblings: two operations under the same parent
{ name = "GET inventory" } ~ { name = "POST /charge" }

# both conditions, anywhere in the same trace
{ span.http.route = "/orders" } && { status = error }

# traces with more than 20 DB spans (N+1 hunt)
{ span.db.system.name != nil } | count() > 20

# traces whose orders spans average over 300 ms
{ resource.service.name = "orders" } | avg(duration) > 300ms

# show extra attributes in the result table
{ status = error } | select(span.http.route, resource.k8s.pod.name)

Operators

OperatorMeaning
>> / <<Descendant / ancestor
> / <Child / parent
~Sibling
!>>, !>, !~Negated structural forms
&&, || between {}Both / either spanset in the trace
| count(), avg(), min(), max(), sum()Aggregate per spanset
| select(...)Return extra attributes
TraceQL

TraceQL metrics: RED from traces, on demand

TraceQL metrics turn a spanset into a time series at query time — useful for questions your metrics never anticipated.

# request rate per service (server spans)
{ kind = server } | rate() by (resource.service.name)

# error rate per route in orders
{ resource.service.name = "orders" && status = error }
  | rate() by (span.http.route)

# p95 latency per route
{ resource.service.name = "orders" && kind = server }
  | quantile_over_time(duration, .95) by (span.http.route)

# counts and histograms
{ name = "POST /charge" } | count_over_time()
{ name = "POST /charge" } | histogram_over_time(duration)

# what is different about slow spans?
{ resource.service.name = "payments" }
  | compare({ duration > 500ms })

Know the limits

  • GA in Tempo 3 and enabled by default (no metrics-generator needed).
  • Computed from stored spans: after tail sampling, rates and error ratios are biased (section 34).
  • Range and cost are bounded by Tempo limits; long ranges are slow.
  • Use them to explore; promote what you need permanently to span metrics or app metrics.
  • Alerting on TraceQL metrics is still experimental.
Alternatives

Tempo, Jaeger and friends

All of these accept OTLP; the instrumentation in your apps does not change when you switch.

BackendStorageQueryNotes
Grafana TempoObject storage (Parquet blocks)TraceQL, TraceQL metricsCheap at scale; tight Grafana integration
Jaeger v2Cassandra, Elasticsearch/OpenSearch, Badger, memoryJaeger UI, search by tagsv2 is built on the OpenTelemetry Collector
AWS X-Ray / CloudWatchManagedTrace map, Transaction SearchOTLP via ADOT or the CloudWatch agent (ch 12)
Commercial APMsManagedVendor UIsSame OTLP data; watch per-span pricing

Keep the vendor decision in the Collector's exporters, never in application code.

Chapter 9 · Part 9.5

Correlating signals

Exemplars, trace IDs in logs and span metrics: one click from a spike on a graph to the request that caused it.

Correlation

The links you want, and what makes each work

Correlation is not magic: each link needs a shared key in both signals and a data source setting in Grafana.

Metrics (Prometheus)RED panel, alertTraces (Tempo)waterfallLogs (Loki)request logsexemplar: trace_id on a buckettrace → logs: query by trace_idderived field: trace_id → tracetrace → metrics / span metricsShared keystrace_id · service.name ↔ service/job · k8s.pod.name ↔ podWithout the same service naming across signals, links open the right tool on the wrong data.
Exemplars

Exemplars: a trace ID attached to a metric sample

An exemplar is one example observation (value, timestamp, labels such as trace_id) stored next to a histogram bucket or counter.

// orders: observe latency with an exemplar (client_golang)
sc := trace.SpanContextFromContext(r.Context())   // inside otelhttp
h := reqDuration.WithLabelValues(r.Method, route)
if sc.IsSampled() {
    h.(prometheus.ExemplarObserver).ObserveWithExemplar(
        elapsed.Seconds(),
        prometheus.Labels{"trace_id": sc.TraceID().String()},
    )
} else {
    h.Observe(elapsed.Seconds())
}

// exemplars are only exposed in OpenMetrics format
http.Handle("/metrics", promhttp.HandlerFor(reg,
    promhttp.HandlerOpts{EnableOpenMetrics: true}))

The chain that must be in place

  • App records exemplars and serves OpenMetrics (left).
  • Prometheus stores them: --enable-feature=exemplar-storage (kps: prometheus.prometheusSpec.enableFeatures: [exemplar-storage]).
  • Grafana Prometheus data source: an exemplar link with label trace_id → Tempo data source.
  • Panel: Exemplars toggle on in the query options.
  • Result: dots on the latency graph; click one → the trace.
  • With tail sampling, the linked trace may have been dropped: keep slow and error traces so the interesting dots resolve.
Correlation

Traces ↔ logs in Grafana

Logs carry trace_id (ch 6); Grafana uses it in both directions.

# Grafana data source provisioning (kps: grafana.additionalDataSources)
- name: Tempo
  type: tempo
  uid: tempo
  url: http://tempo.tracing.svc:3200
  jsonData:
    tracesToLogsV2:
      datasourceUid: loki
      spanStartTimeShift: -5m
      spanEndTimeShift: 5m
      filterByTraceID: true
      tags: [{key: service.name, value: service_name}]
    serviceMap: {datasourceUid: prometheus}
    nodeGraph: {enabled: true}
- name: Loki
  type: loki
  uid: loki
  url: http://loki.logging.svc:3100
  jsonData:
    derivedFields:
      # trace_id inside the JSON log line → Tempo link
      - name: TraceID
        matcherRegex: '"trace_id":"(\w+)"'
        url: '$${__value.raw}'
        datasourceUid: tempo
      # or, if Alloy stores trace_id as structured metadata:
      # - name: TraceID
      #   matcherType: label
      #   matcherRegex: trace_id
      #   url: '$${__value.raw}'
      #   datasourceUid: tempo

$$ escapes $ in provisioning files, which otherwise expand environment variables. The tags mapping renames a span resource attribute to the Loki label name used in the generated query.

Span metrics

Span metrics and service graphs from traces

Generate RED metrics from 100 % of spans before sampling, so dashboards stay correct while Tempo stores only a subset.

connectors:
  spanmetrics:
    histogram:
      explicit:
        buckets: [5ms, 25ms, 100ms, 250ms, 500ms, 1s, 2.5s]
    dimensions:
      - name: http.route
      - name: http.response.status_code
  servicegraph:
    latency_histogram_buckets: [10ms, 100ms, 500ms, 1s]
  forward/sample: {}

service:
  pipelines:
    traces:          # before sampling
      receivers: [otlp]
      exporters: [spanmetrics, servicegraph, forward/sample]
    traces/sampled:
      receivers: [forward/sample]
      processors: [tail_sampling, batch]
      exporters: [otlp/tempo]
    metrics/derived:
      receivers: [spanmetrics, servicegraph]
      exporters: [prometheusremotewrite]

What you get

  • traces_span_metrics_calls_total and a duration histogram (unit ms by default) per service.name, span.name, span.kind, status.code + your dimensions.
  • traces_service_graph_request_total, _failed_total and latency histograms per client/server pair: the service map.
  • The forward connector splits the pipeline: metrics see every span, Tempo only the sampled ones.
  • Prometheus must accept remote write: kps prometheus.prometheusSpec.enableRemoteWriteReceiver: true.
  • Alternative: Tempo's metrics-generator produces the same metrics server-side.
  • Every dimension multiplies series (ch 5): never add IDs.
Correlation

From alert to root cause in five clicks

The workflow chapter 13 builds on, with the links from this section.

1 Alert

  • OrdersHighLatency fires
  • p95 on /orders > 500 ms
→

2 RED panel

  • Latency spike since 12:03
  • errors flat, rate flat
→

3 Exemplar

  • Click a high dot
  • → trace 4bf92f…
→

4 Trace

  • payments fraud-check 240 ms
  • TraceQL: is it all traces?
→

5 Logs

  • Trace → logs
  • "fraud model cache miss"

Each step answers one question: is it real, where is it, which request, which span, why. If any link is missing, write down which one and fix the instrumentation after the incident.

Common mistakes

Tracing mistakes that hide the problem

Most "tracing does not work" tickets come from this list.

Instrumentation

  • Span names with IDs (GET /orders/8812): every span unique, nothing groups.
  • Missing propagation through a queue or a custom HTTP client: traces break in two.
  • Plain TraceIDRatioBased without ParentBased: half-traces.
  • Marking 4xx as Error on server spans: error rate inflated by bad clients.
  • Forgetting tp.Shutdown(ctx): last spans lost on exit.

Collector

  • No memory_limiter: OOM-kill under load, all buffered data lost.
  • Tail sampling with several replicas and no trace-ID routing.
  • Components defined but not added to a pipeline: silently unused.
  • Not scraping :8888: export failures go unseen.
  • Computing error rates from sampled traces.

Checks that catch them

  • debug exporter with verbosity: detailed in a test pipeline.
  • otelcol-contrib validate in CI.
  • TraceQL: { trace:rootService != "orders" && resource.service.name = "payments" } finds orphaned payments roots.
  • Alert on otelcol_exporter_send_failed_spans_total.
  • Compare span-metric rate to app-metric rate.
Self-check

Traces, Collector and Tempo: can you answer these?

Answer with the exact setting, operator or attribute.

Spans and sampling

  • How many spans does one HTTP call between two instrumented services create, and of which kinds?
  • Should a 404 set Error on a server span? On a client span?
  • What is the critical path, and why does optimising a parallel branch not help?
  • Why must every service use ParentBased?
  • Why do you need trace-ID routing in front of tail sampling replicas?

Collector

  • Which processor goes first, and which last? Why?
  • What is a connector? Name two.
  • When does k8sattributes with from: connection fail?
  • Which internal metric shows export failures?
  • Agent vs gateway: which one reads container log files?

Tempo and correlation

  • Write TraceQL for "orders traces where payments failed".
  • What changed in Tempo 3's write path, and which mode needs no Kafka?
  • List the four things needed for exemplars to work end to end.
  • How does Grafana jump from a log line to a trace?
  • Why generate span metrics before tail sampling?

Practice: Lab 9 in Hands-on labs — Tempo in tracing, the gateway exporting to it with tail sampling and span metrics, exemplars and trace↔logs links in Grafana.

Chapter 10

Dashboards: RED, USE and Grafana

Dashboards that answer questions fast: RED for every service, USE for every resource, Grafana layouts that guide the eye, and dashboards kept as code.

Chapter 10 · Part 10.1

RED dashboards for services

Rate, errors and duration per service and route: the first screen you open when a user says "it is slow".

RED

RED: three questions for every request-driven service

Rate, Errors, Duration (Tom Wilkie). They describe the work as the caller experiences it, so they line up with SLIs (the SRE deck).

Rate

  • Requests per second the service handles.
  • Split by route (and method) to see which endpoint changed.
  • Sudden drop is often worse news than a spike: callers gave up or traffic is routed elsewhere.

Errors

  • Failed requests per second, and as a ratio of all requests.
  • Define "failed" once: 5xx for servers; include timeouts.
  • Show the ratio on the overview, the absolute rate on drill-downs.

Duration

  • Latency distribution: p50, p90, p99 from histograms.
  • Never only the average: it hides the slow tail users feel.
  • Split successes and errors: fast failures make average latency look better.

RED is for services (things that handle requests). Resources such as CPUs, disks and connection pools get USE (section 39). Google's four golden signals = RED + saturation.

RED

RED queries for the shop app

All three from the two metrics in the shared conventions: http_requests_total and http_request_duration_seconds.

# Rate: requests per second per route
sum by (route) (rate(http_requests_total{job="$service"}[$__rate_interval]))

# Errors: ratio of 5xx (0–1, unit percentunit)
  sum(rate(http_requests_total{job="$service", code=~"5.."}[$__rate_interval]))
/
  sum(rate(http_requests_total{job="$service"}[$__rate_interval]))

# Errors per route, absolute
sum by (route) (rate(http_requests_total{job="$service", code=~"5.."}[$__rate_interval]))

# Duration: p50 / p95 / p99 per route (classic histogram)
histogram_quantile(0.95,
  sum by (le, route) (rate(http_request_duration_seconds_bucket{job="$service"}[$__rate_interval])))

# Duration with native histograms (if enabled, ch 2): no le label needed
histogram_quantile(0.95,
  sum by (route) (rate(http_request_duration_seconds{job="$service"}[$__rate_interval])))

# Share of requests faster than 300 ms (an SLI-style ratio; bucket must exist)
  sum(rate(http_request_duration_seconds_bucket{job="$service", le="0.3"}[$__rate_interval]))
/
  sum(rate(http_request_duration_seconds_count{job="$service"}[$__rate_interval]))

If the error ratio has no traffic, 0/0 gives no data, not zero. In Grafana, show "No data" as 0 only when you are sure that zero traffic is healthy.

RED

Percentiles, averages and heatmaps

Latency is a distribution. Pick the view that shows the shape, not just one number.

Percentiles

  • p50 shows the typical request, p99 the tail that 1 in 100 users hit.
  • histogram_quantile interpolates inside a bucket: the answer is only as good as your bucket bounds (ch 3).
  • Never average percentiles across instances: aggregate buckets with sum by (le) first, then take the quantile.

Averages

  • rate(..._sum) / rate(..._count) = mean latency.
  • Useful for capacity math (Little's law), misleading for user experience.
  • A bimodal service (cache hit 5 ms, miss 800 ms) averages to a latency nobody sees.

Heatmaps

  • Panel type Heatmap, query sum by (le) (increase(..._bucket[$__rate_interval])), format Heatmap.
  • Each column is a time slice, each row a bucket, colour = count.
  • Shows bimodal latency, new slow modes after a deploy, and tail changes at a glance.
  • Native histograms render directly as heatmaps.
RED

Anatomy of a service overview dashboard

Top row answers "is it healthy?", the middle "where?", the bottom "what changed and what does it depend on?".

Shop / Service overview — variables: $service, $routeRequest ratestat + sparklineError ratiostat, thresholds 1 % / 5 %p95 latencystat, threshold 500 msSaturationCPU vs limit, pods readyRate by routetime seriesErrors by route and codetime series, stackedLatency p50/p95/p99time series + exemplarsLatency heatmapbuckets over timeDependenciespayments RED (client side)Logs (errors)Loki panel, filtered by $serviceAnnotations: deployments on every panel. Links: Tempo search for $service, Pod dashboard (USE).
RED

Dependencies: see the callee from the caller

A service is only as fast as its slowest dependency. Measure calls from the client side too.

Client-side metrics

  • Instrument outgoing calls: otelhttp.NewTransport emits http.client.request.duration (ch 8).
  • Or use span metrics from CLIENT spans (ch 9): span_kind="SPAN_KIND_CLIENT".
  • Client latency − server latency = network, DNS, pool waits.
  • Errors seen only on the client side: timeouts, connection refused, TLS.
# orders → payments, from orders' CLIENT spans
# (span metrics connector, ch 9; duration in ms by default)
histogram_quantile(0.95, sum by (le) (
  rate(traces_span_metrics_duration_milliseconds_bucket{
    service_name="orders",
    span_kind="SPAN_KIND_CLIENT"}[$__rate_interval])))

# service graph: failed requests between two services
sum by (client, server) (
  rate(traces_service_graph_request_failed_total[$__rate_interval]))

Label names produced by the span metrics connector (service_name, span_kind, span_name, status_code) are derived from the OTel attribute names; check them in Prometheus before hard-coding queries.

Chapter 10 · Part 10.2

USE dashboards for resources

Utilization, saturation and errors for every resource: where requests wait when RED says "slow".

USE

USE: three questions for every resource

Brendan Gregg's method: for every resource, check utilization, saturation and errors. Resources include CPUs, memory, disks, NICs — and software resources such as thread pools and connection pools.

Utilization

  • Fraction of time (or capacity) the resource was busy.
  • 100 % CPU is not an error by itself; a queue behind it is.
  • Averages over 1 min hide short bursts: check shorter windows or max.

Saturation

  • Work that waits because the resource is busy: run queue, swap, I/O queue, throttling, pending pool requests.
  • Best early warning: saturation rises before latency explodes.
  • Linux PSI (pressure stall information) measures it directly.

Errors

  • Device and operation errors: disk I/O errors, NIC drops, OOM kills, pool timeouts.
  • Often only in logs or counters you never graph.
  • Any non-zero rate deserves a look.

Workflow: when RED shows a problem, walk USE for every resource on the request path, starting with the one closest to the slow span (ch 9).

USE

USE for nodes with node_exporter

One row per resource. Queries use node_exporter metrics installed by kps (ch 5).

ResourceUtilizationSaturationErrors
CPU1 - avg by (instance) (rate(node_cpu_seconds_total{mode="idle"}[5m]))rate(node_pressure_cpu_waiting_seconds_total[5m]); or node_load1 ÷ CPU count— (rare; check dmesg/MCE)
Memory1 - node_memory_MemAvailable_bytes / node_memory_MemTotal_bytesrate(node_pressure_memory_waiting_seconds_total[5m]); rate(node_vmstat_pgmajfault[5m])OOM kills: increase(node_vmstat_oom_kill[1h])
Diskrate(node_disk_io_time_seconds_total[5m])rate(node_disk_io_time_weighted_seconds_total[5m]) (≈ avg queue length); rate(node_pressure_io_waiting_seconds_total[5m])filesystem node_filesystem_device_error
Networkbytes/s ÷ link speed: rate(node_network_receive_bytes_total[5m]) vs node_network_speed_bytesrate(node_network_receive_drop_total[5m]), ..._transmit_drop_totalrate(node_network_receive_errs_total[5m]), ..._transmit_errs_total
Filesystem1 - node_filesystem_avail_bytes / node_filesystem_size_bytespredict_linear(node_filesystem_avail_bytes[6h], 24*3600) < 0read-only remounts: node_filesystem_readonly

PSI metrics need a kernel with PSI enabled (most modern distros) and the node_exporter pressure collector (on by default on Linux).

USE

USE for containers and pods

Kubernetes adds two limits a node view cannot show: CPU quota (throttling) and memory limit (OOM kill).

# CPU utilization vs request, per pod (cAdvisor + kube-state-metrics)
sum by (pod) (rate(container_cpu_usage_seconds_total{namespace="shop", container!=""}[$__rate_interval]))
/
sum by (pod) (kube_pod_container_resource_requests{namespace="shop", resource="cpu"})

# CPU saturation: share of CFS periods that were throttled
sum by (pod) (rate(container_cpu_cfs_throttled_periods_total{namespace="shop", container!=""}[$__rate_interval]))
/
sum by (pod) (rate(container_cpu_cfs_periods_total{namespace="shop", container!=""}[$__rate_interval]))

# Memory utilization vs limit (working set is what the OOM killer compares)
sum by (pod) (container_memory_working_set_bytes{namespace="shop", container!=""})
/
sum by (pod) (kube_pod_container_resource_limits{namespace="shop", resource="memory"})

# Errors: OOM kills and restarts
max by (pod, container) (kube_pod_container_status_last_terminated_reason{namespace="shop", reason="OOMKilled"})
increase(kube_pod_container_status_restarts_total{namespace="shop"}[1h])

container!="" drops the pod-level cgroup series that cAdvisor also exports; without it, sums double-count.

USE

Software resources: pools, queues and runtimes

The most common saturation point in a service is not the CPU — it is a pool or a queue inside the process.

Resources to expose

  • DB connection pool: in use, idle, max, wait count and wait time (Go sql.DBStats).
  • HTTP client pools and goroutine/thread counts.
  • Work queues: depth, age of the oldest item, consumer lag.
  • Runtime: GC pause time, heap in use (Go collector go_* metrics).
// Go: export database/sql pool stats (client_golang)
import "github.com/prometheus/client_golang/prometheus/collectors"

reg.MustRegister(collectors.NewDBStatsCollector(db, "orders"))
// go_sql_open_connections, go_sql_in_use_connections,
// go_sql_wait_count_total, go_sql_wait_duration_seconds_total ...

# saturation: average wait per request for a connection
rate(go_sql_wait_duration_seconds_total{db_name="orders"}[5m])
/
rate(go_sql_wait_count_total{db_name="orders"}[5m])

Pool utilization near 100 % with growing wait time explains "high latency, low CPU" incidents — a classic USE finding that RED alone cannot explain.

USE

Cluster and capacity views

Above pods: is the cluster itself out of room? These panels belong on a platform dashboard, not on each service.

# requested vs allocatable CPU, cluster-wide
sum(kube_pod_container_resource_requests{resource="cpu"})
/
sum(kube_node_status_allocatable{resource="cpu"})

# pods that cannot be scheduled (saturation of the cluster)
sum(kube_pod_status_phase{phase="Pending"})

# nodes not ready (errors)
count(kube_node_status_condition{condition="Ready", status="true"} == 0)

Reading them

  • Scheduling is based on requests, not usage: a cluster can be "full" at 30 % CPU usage.
  • Usage far below requests → over-requesting (waste).
  • Usage above requests with throttling → limits too low or noisy neighbours.
  • Pending pods + requests near 100 % → add nodes or right-size (autoscaler logs).
  • kps ships "Kubernetes / Compute Resources / …" dashboards that cover most of this.
Chapter 10 · Part 10.3

Grafana dashboard design

Layouts, variables, units and panels that make a dashboard readable at 3 a.m.

Design

Start from the audience and the question

A dashboard is a tool for a decision. Name the question it answers before adding the first panel.

DashboardAudienceQuestionShape
Service overview (RED)On-call, service teamIs my service healthy? Where is it hurting?Stats on top, breakdowns below, links to drill-downs
Resource / pod (USE)On-call, platformWhich resource is saturated?One row per resource: U, S, E side by side
Platform / clusterPlatform teamDo we have capacity? Is the control plane OK?Cluster totals, nodes, pending pods
Business / productProduct, managementAre orders flowing?Few big numbers, long time ranges
Debug / explorationWhoever is investigatingAnythingUse Explore, not a permanent dashboard

Hierarchy beats volume: overview → service → pod, linked together. A 60-panel "everything" dashboard is slow to load and slow to read.

Design

Variables make one dashboard serve every service

Variables become part of every query. Use query variables from label values, chained from broad to narrow.

# Variables (Dashboard settings → Variables)
datasource  type: Data source (prometheus)
namespace   label_values(kube_pod_info, namespace)
service     label_values(
              http_requests_total{namespace="$namespace"}, job)
route       label_values(
              http_requests_total{job="$service"}, route)
            multi-value + "Include All"

# Using them in queries
sum by (route) (rate(http_requests_total{
  job="$service", route=~"$route"}[$__rate_interval]))

# Built-ins
$__rate_interval  # max(4×scrape, step + scrape)
$__interval       # step from time range and width
$__range          # whole selected range

Rules of thumb

  • Multi-value variables need =~ in the matcher, not =.
  • Use $__rate_interval for every rate()/increase() in panels: it never drops below 4 scrape intervals, so no gaps.
  • Set the scrape interval on the Prometheus data source (default 15s) so $__rate_interval is correct.
  • Chain variables so each one filters the next.
  • Use the data source variable to point the same dashboard at another cluster.
Design

Panels: choose by the question, not by habit

Each panel type answers a different kind of question.

PanelUse it forSettings that matter
Time seriesTrends and comparisons over timeUnit, min 0, legend as table with last/max, series overrides
StatOne number with a threshold colourReduce (last, mean), thresholds, sparkline
Gauge / bar gaugeValue against a known max (utilization)Min/max set explicitly
HeatmapLatency distributionsData format: heatmap buckets; le as bucket
TableTop-N lists, current state per podInstant query, transformations, cell colouring
State timelineUp/down or phases over timeValue mappings
Logs / Traces / Node graphLoki lines, a Tempo trace, a service graphData source specific

Always set the unit (reqps, percentunit, s, bytes). A latency of 0.25 without a unit is ambiguous; 250 ms is not.

Design

Thresholds, annotations and links

Context turns a line into an answer: what is bad, what changed, and where to go next.

Thresholds

  • Match alert thresholds (ch 4): if the alert fires at 5 %, colour red at 5 %.
  • Show thresholds as lines or regions on time series.
  • Base error and latency thresholds on SLOs (the SRE deck), not on "looks high".

Annotations

  • Deploys: from a Prometheus query, e.g. changes(kube_deployment_status_observed_generation{namespace="shop"}[2m]) > 0.
  • Or pushed from CI via the Grafana annotations API.
  • Alert state changes and incidents as annotations.
  • "Did it start with a deploy?" answered in one glance.

Links

  • Dashboard links: overview → service → pod, carrying variables and time range.
  • Data links on series: open Tempo search or Loki logs for that route and time.
  • Exemplars on latency panels (ch 9).
  • Runbook link in the dashboard description.
Grafana 13

Dynamic dashboards (Grafana 13)

Grafana 13 (Apr 2026) turned dynamic dashboards on by default: a new layout engine, editing experience and dashboard structure.

What changed

  • Every new and existing dashboard uses the new layout engine automatically.
  • Layouts: rows, tabs, and auto-grid next to the classic custom grid; groups can be nested.
  • Conditional rendering: show or hide panels and groups based on variable values or whether a panel has data.
  • Dashboard schema v2 behind it: layout and elements stored separately (section 41).

Using it well

  • Tabs per concern: Overview · Dependencies · Resources · Logs.
  • Hide the "database" row when $service has no database.
  • Auto-grid for repeated panels per route or pod.
  • Keep the top row identical across services so on-call knows where to look.

Before you migrate

  • Test old JSON dashboards in a staging Grafana first.
  • Dashboards provisioned from files or Git keep working; check the stored format your tooling expects.
  • Read "What's new in Grafana 13" and the upgrade notes for your exact minor version.
Design

Dashboard anti-patterns

Every one of these has cost someone time in an incident.

Content

  • Averages only, no percentiles.
  • Counters graphed raw (ever-increasing lines) instead of rate().
  • sum before rate, or avg of percentiles across pods.
  • No units; y-axis auto-scaled so a 0.1 % blip looks like an outage.
  • Panels nobody can explain ("what is x_ratio_2?").

Structure

  • One giant dashboard with 80 panels and 40 queries per load.
  • Copy-pasted dashboards per service that drift apart.
  • Hard-coded job="orders" instead of variables.
  • No links between levels; on-call guesses where to go next.
  • Edited in the UI only; nobody knows who changed what.

Fixes

  • RED top row on every service dashboard, generated from one template.
  • $__rate_interval, units and thresholds as defaults in the template.
  • Dashboards as code with review (section 41).
  • Review dashboards after each incident: which panel helped, which was noise?
Chapter 10 · Part 10.4

Dashboards as code

Provisioning, the JSON model, Git Sync and generators: dashboards reviewed and versioned like any other code.

As code

Four ways to keep dashboards in Git

They differ in who edits (UI or code) and in what is the source of truth.

ApproachSource of truthEdit inGood for
File provisioningJSON files on disk / ConfigMapsCode (UI saves blocked by default)Kubernetes installs, kps sidecar
Generators (Grafonnet, Foundation SDK)Code that renders JSONCodeMany similar dashboards from one template
Terraform grafana providerTerraform state + JSONCodeGrafana Cloud / managed Grafana, folders and permissions too
Git Sync (GA in Grafana 13)Git repositoryUI → commit or pull requestTeams that want UI editing with Git review
As code

Provisioning with the kps sidecar

kube-prometheus-stack runs a sidecar next to Grafana that loads every ConfigMap labelled grafana_dashboard: "1".

# dashboards/shop-red.json → ConfigMap with the label
kubectl -n monitoring create configmap shop-red \
  --from-file=shop-red.json=dashboards/shop-red.json \
  --dry-run=client -o yaml \
| kubectl label --local -f - grafana_dashboard=1 -o yaml \
| kubectl apply -f -

# relevant kps values (defaults shown)
grafana:
  sidecar:
    dashboards:
      enabled: true
      label: grafana_dashboard
      labelValue: "1"
    datasources:
      enabled: true
      label: grafana_datasource
      labelValue: "1"

How it behaves

  • The sidecar watches ConfigMaps and writes the JSON into Grafana's provisioning folder.
  • Grafana reloads; the dashboard appears in a few seconds.
  • Provisioned dashboards cannot be overwritten from the UI: "Save" asks you to export JSON instead.
  • Keep uid fixed in the JSON so links and bookmarks survive updates.
  • ConfigMaps are limited to about 1 MiB: fine for dashboards, not for huge generated ones.
  • Data sources work the same way with label grafana_datasource: "1", or via grafana.additionalDataSources.
As code

The dashboard JSON model

A classic (v1) dashboard is one JSON document: metadata, variables, and a list of panels with grid positions and queries.

{
  "uid": "shop-red",
  "title": "Shop / Service RED",
  "tags": ["shop", "red"],
  "time": {"from": "now-1h", "to": "now"},
  "templating": {"list": [{
    "name": "service", "type": "query",
    "datasource": {"type": "prometheus", "uid": "prometheus"},
    "query": "label_values(http_requests_total, job)",
    "refresh": 2
  }]},
  "panels": [{
    "type": "timeseries", "title": "Requests/s by route",
    "gridPos": {"x": 0, "y": 0, "w": 12, "h": 8},
    "datasource": {"type": "prometheus", "uid": "prometheus"},
    "fieldConfig": {"defaults": {"unit": "reqps"}},
    "targets": [{"refId": "A", "legendFormat": "{{route}}", "expr":
"sum by (route) (rate(http_requests_total{job=\"$service\"}[$__rate_interval]))"}]
  }]
}

Notes

  • uid is the stable identity; id is per-instance and should be omitted in files.
  • datasource.uid: kps provisions its Prometheus data source with uid prometheus.
  • refresh: 2 = refresh variable values on time range change.
  • Grafana migrates older JSON (schemaVersion) on load; export from your Grafana version to see the current shape.
  • Schema v2 (Grafana 13) separates elements (panels) from layout and is served as a Kubernetes-style resource; v1 JSON is still accepted for provisioning — verify for your version.
As code

Generate dashboards instead of copying them

When ten services need the same RED dashboard, write the template once.

Options

  • Grafana Foundation SDK: typed builders for Go, TypeScript, Python, Java and PHP that emit dashboard JSON.
  • Grafonnet: Jsonnet library, generated from the same schemas.
  • A small script in any language: dashboards are just JSON (Lab 10 does this in Python).
  • Terraform grafana_dashboard with config_json = file(...) to push results.
# Lab 10 style: one template, many services
def red_dashboard(service):
    return {
        "uid": f"red-{service}",
        "title": f"Shop / {service} RED",
        "panels": [
            ts("Requests/s", rate_q(service), "reqps", 0),
            ts("Error ratio", err_q(service), "percentunit", 8),
            ts("p95 latency", p95_q(service), "s", 16),
        ],
    }

for svc in ["orders", "payments"]:
    write(f"dashboards/red-{svc}.json", red_dashboard(svc))

Generate, review the diff in a pull request, then apply (ConfigMap, API, Terraform or Git Sync). Generated JSON is an artifact: do not hand-edit it.

As code

Git Sync: edit in the UI, review in Git

Git Sync (GA in Grafana 13) connects folders to a Git repository. Saving a dashboard becomes a commit or a pull request.

How it works

  • Connect a repository (GitHub, GitLab, Bitbucket or any Git server; GitHub App authentication supported).
  • Dashboards and folders are stored as files in the repo.
  • In the UI, "Save" writes to a branch and can open a pull request.
  • Merged changes sync back into Grafana.
  • Later 13.x releases added verified commits, dashboard import, root-level sync and GitHub Enterprise support.

When to choose it

  • People like editing panels in the UI, and you want review.
  • Dashboards are mostly hand-made, not generated.
  • You already gate other changes through pull requests.
  • Not a fit for heavy templating: use a generator instead.

Watch out

  • Two sources of truth (sidecar ConfigMaps and Git Sync) for the same folder will fight.
  • Decide per folder who owns it.
  • Check which features need Grafana Cloud or Enterprise for your setup in the current docs.
As code

Review checklist for a dashboard pull request

Treat a dashboard like code: it has users, and a bad one costs time during incidents.

Correctness

  • Every rate() uses $__rate_interval (or a window ≥ 4 × scrape).
  • sum by (le) before histogram_quantile.
  • Units and thresholds set; thresholds match the alerts.
  • Error ratio and latency split by route, not only total.

Usability

  • Fixed uid, clear title, tags, description with runbook link.
  • Variables chained and used in every query.
  • Top row answers "healthy or not" in 5 seconds.
  • Links to the next level and to Tempo/Loki.

Cost

  • Number of queries per load (each panel = at least one).
  • No unbounded by (pod) on large namespaces in overview panels.
  • Expensive queries backed by recording rules (ch 4).
  • Default time range short (1–6 h); long ranges on purpose only.
Common mistakes

Dashboard mistakes that mislead on-call

Each one makes a healthy system look broken or a broken one look healthy.

Queries

  • avg(histogram_quantile(...)) across pods instead of quantile of summed buckets.
  • rate(x[1m]) with a 30 s scrape interval: gaps and spikes.
  • Error ratio built from separately rounded panels.
  • Missing container!="" in cAdvisor sums: doubled usage.
  • Memory from container_memory_usage_bytes (includes cache) instead of working set.

Design

  • No units, no thresholds, auto-scaled axes.
  • Average latency only; tail invisible.
  • RED and USE mixed on one crowded screen.
  • Dashboards edited by hand in production, then lost on reinstall.
  • Two dashboards with the same uid in different ConfigMaps.

Checks

  • Compare a panel with a hand-written query in Explore.
  • Inject a fault (FAIL_RATE, EXTRA_LATENCY_MS) and confirm every panel reacts.
  • Open the dashboard on a phone-sized window: is the top row readable?
  • Dashboard review after each incident.
Self-check

Dashboards: can you answer these?

Answer with the exact query, panel setting or variable.

RED

  • Write the error-ratio query for $service.
  • Why is averaging p95 across pods wrong, and what is right?
  • When is a heatmap better than a p99 line?
  • How do you show orders → payments latency from the orders side?
  • What does a drop in request rate tell you?

USE

  • Name U, S and E for CPU on a node and for a container.
  • What does container_cpu_cfs_throttled_periods_total measure?
  • Why can a cluster be "full" at 30 % CPU usage?
  • Which metric does the OOM killer effectively compare to the limit?
  • Give a software resource and its saturation metric.

Grafana

  • What is $__rate_interval and why use it?
  • Which label makes the kps sidecar load a ConfigMap?
  • What changed with dynamic dashboards in Grafana 13?
  • Generator vs Git Sync: when do you choose each?
  • Why must uid stay fixed?

Practice: Lab 10 in Hands-on labs — generate RED and USE dashboards for the shop app as code and load them through the kps sidecar.

Chapter 11

CloudWatch metrics and alarms

The AWS-native metrics store: how a CloudWatch metric is identified, published and queried, how alarms evaluate, and the curated views built on top.

Chapter 11 · Part 11.1

CloudWatch metrics

Namespaces, dimensions, resolution, statistics, EMF, metric math, Metrics Insights and the 2026 OpenTelemetry + PromQL path.

Mental model

CloudWatch next to Prometheus: same ideas, different mechanics

If you know Prometheus (ch 2–4), map each concept before learning the AWS details. The differences are where people get surprised.

ConceptPrometheusCloudWatch
Series identityMetric name + label setNamespace + metric name + exact dimension set
How data arrivesPull: server scrapes /metricsPush: AWS services, PutMetricData, EMF log lines, CloudWatch agent, OTLP (2026)
Query languagePromQLMetric math, SEARCH(), Metrics Insights (SQL); PromQL for OTel metrics (2026)
Pre-computed seriesRecording rulesMetric math in alarms and dashboards (computed at read time); metric filters on logs
Alert evaluationAlerting rule with for:Alarm: period × evaluation periods, M out of N datapoints
Notification routingAlertmanager (group, inhibit, silence)Alarm actions → SNS / Lambda / EventBridge; composite alarms for suppression
RetentionConfigured per server (often 15 d) + remote storage15 months, rolled up to coarser periods as data ages
Cost driverYour servers and storagePer custom metric, per API call, per alarm metric, per GB of logs

Rule of thumb: CloudWatch is the default for AWS resource metrics (you get them for free and cannot scrape them); Prometheus is the default for Kubernetes and application metrics. Most teams run both (ch 12, part 4).

Anatomy

What identifies a CloudWatch metric

A metric is a time-ordered set of datapoints. Change any dimension name or value and you are writing to a different metric.

NamespaceAWS/Lambda, ObsLab, …Metric nameLatency, Errors, InvocationsDimensions (0–30)name=value pairs: service=ordersDatapointstimestamp, value or statistic set, unitOne metric= unique combinationObsLab / Latency {service=orders, route=POST /orders}ObsLab / Latency {service=orders}ObsLab / Latency {} ← three different metrics

Rules that matter

  • Namespaces starting with AWS/ are reserved for AWS services.
  • Up to 30 dimensions per metric.
  • A query must name the exact dimension set: {service} does not aggregate {service, route} for you.
  • No wildcard on dimensions in a plain GetMetricData stat; use SEARCH() or Metrics Insights.
  • Units (Milliseconds, Count, Bytes, …) are metadata; CloudWatch does not convert between them.
  • Every unique combination is billed as a separate custom metric.
Resolution and retention

Datapoints get coarser as they age

CloudWatch keeps metrics for 15 months, but not at the resolution you wrote them.

Retention by period

Datapoint periodAvailable for
< 60 s (high resolution)3 hours
60 s (1 minute)15 days
300 s (5 minutes)63 days
3600 s (1 hour)455 days (15 months)

Resolution

  • Standard resolution: 1-minute granularity (the default for custom metrics).
  • High resolution: StorageResolution=1, down to 1 second; alarms can then use 10 s or 30 s periods (or multiples of 60 s).
  • Many AWS services publish 1-minute metrics; EC2 basic monitoring is 5-minute, detailed monitoring 1-minute (paid).
  • Data older than 3 h is aggregated to 1 min, older than 15 d to 5 min, older than 63 d to 1 h.

Things you cannot do

  • Delete a metric: it stops appearing when no new data arrives for 15 months.
  • Rename a metric or a dimension in place: publish under the new name and update consumers.
  • Backfill far into the past: timestamps up to two weeks old (and up to two hours in the future) are accepted.
  • Get a 1-minute view of last quarter: by then only 1-hour datapoints are left.
Statistics

Statistics, percentiles and statistic sets

You choose the statistic when you read, not when you write — as long as CloudWatch has the raw values.

Common statistics

  • SampleCount, Sum, Average, Minimum, Maximum.
  • pNN percentiles: p50, p90, p99, p99.9.
  • Trimmed and windsorized: TM(10%:90%), WM, IQM (inter-quartile mean).
  • Range counts and sums: PR(:300) (share of values ≤ 300), TC, TS.
  • Statistic names are case-sensitive in the API (p90, not P90).

Statistic sets vs raw values

  • PutMetricData can send raw Values/Counts arrays or a pre-aggregated statistic set (SampleCount, Sum, Minimum, Maximum).
  • Percentiles need the raw values: a metric published only as statistic sets has no pNN.
  • EMF and the CloudWatch agent publish values, so percentiles work.
  • Percentiles are per metric: you cannot average p99s across dimensions and get a p99.

Pick the right one

  • Latency → p50 + p90/p99, never only Average (one slow call hides in the mean).
  • Errors, requests → Sum per period; rate = Sum / PERIOD(m1).
  • Saturation (queue depth, CPU) → Maximum or Average depending on burstiness.
  • SampleCount of a latency metric = request count for free.
  • Same reasoning as RED (ch 10): rate, errors, duration distribution.
Publishing

PutMetricData: the direct API

Simple and synchronous. Fine for scripts and low volume; for applications prefer EMF or an agent so you do not block request paths on API calls.

# one datapoint, two dimensions (shorthand)
aws cloudwatch put-metric-data --namespace ObsLab \
  --metric-name Latency --unit Milliseconds --value 87 \
  --dimensions service=orders,route="POST /orders"

# several values in one call (raw values keep percentiles)
cat > md.json <<'EOF'
[{"MetricName": "Latency", "Unit": "Milliseconds",
  "Dimensions": [{"Name": "service", "Value": "orders"},
                 {"Name": "route", "Value": "POST /orders"}],
  "Values": [42, 87, 120, 950], "Counts": [10, 5, 2, 1],
  "StorageResolution": 60}]
EOF
aws cloudwatch put-metric-data --namespace ObsLab \
  --metric-data file://md.json

# read it back: what does CloudWatch know?
aws cloudwatch list-metrics --namespace ObsLab \
  --metric-name Latency

What to know

  • Values + Counts = a compact histogram: value 42 seen 10 times.
  • Metrics appear in list-metrics a few minutes after the first write.
  • Each call is billed as an API request; each unique metric as a custom metric.
  • IAM: cloudwatch:PutMetricData; restrict with condition key cloudwatch:namespace.
  • The SDKs batch and retry; still, a CloudWatch outage must never fail a customer request.
  • Do not put IDs (order_id, user_id) in dimensions: every value is a new billed metric.
Embedded Metric Format

EMF: one log line that is also a metric

Write structured JSON to a log; CloudWatch Logs extracts the metrics asynchronously. High-cardinality context stays in the log, not in dimensions.

{
  "_aws": {
    "Timestamp": 1794200000000,
    "CloudWatchMetrics": [{
      "Namespace": "ObsLab",
      "Dimensions": [["service", "route"]],
      "Metrics": [
        {"Name": "Latency",
         "Unit": "Milliseconds"},
        {"Name": "Errors",
         "Unit": "Count"},
        {"Name": "Requests",
         "Unit": "Count"}]
    }]
  },
  "service": "orders",
  "route": "POST /orders",
  "Latency": 87.4, "Errors": 0,
  "Requests": 1,
  "level": "INFO",
  "msg": "order created",
  "order_id": "8c1f…",
  "trace_id": "4bf92f35…"
}

How it works

  • _aws.Timestamp in epoch milliseconds.
  • Each name in Dimensions and Metrics must exist as a root-level key with a value.
  • Dimensions is a list of dimension sets: [["service","route"],["service"]] publishes both aggregations.
  • Values can be arrays (several samples in one line).
  • Extraction happens in Lambda automatically (function stdout), or through the CloudWatch agent / Fluent Bit, or PutLogEvents with header x-amzn-logs-format: json/emf.
  • Fields not referenced (order_id, trace_id) stay searchable in Logs Insights (ch 12).

Why teams like it

  • No synchronous API call in the hot path.
  • One write gives a metric for alarms and the raw event for debugging.
  • Libraries: aws-embedded-metrics (Node, Python, Java, .NET), Powertools for AWS Lambda Metrics.
  • Cost: you pay for log ingestion and for the extracted custom metrics.
  • A malformed line is silently just a log: verify with list-metrics (Lab 11).
Querying

GetMetricData, metric math and SEARCH

Metric math combines up to hundreds of series at read time: the CloudWatch answer to simple PromQL arithmetic.

cat > q.json <<'EOF'
[{"Id": "req", "ReturnData": false,
  "MetricStat": {"Metric": {"Namespace": "ObsLab",
     "MetricName": "Requests", "Dimensions": [
       {"Name": "service", "Value": "orders"},
       {"Name": "route", "Value": "POST /orders"}]},
   "Period": 60, "Stat": "Sum"}},
 {"Id": "err", "ReturnData": false,
  "MetricStat": {"Metric": {"Namespace": "ObsLab",
     "MetricName": "Errors", "Dimensions": [
       {"Name": "service", "Value": "orders"},
       {"Name": "route", "Value": "POST /orders"}]},
   "Period": 60, "Stat": "Sum"}},
 {"Id": "error_pct", "Label": "error %",
  "Expression": "100 * FILL(err, 0) / req"}]
EOF
aws cloudwatch get-metric-data --metric-data-queries file://q.json \
  --start-time $(date -u -v-1H +%FT%TZ) --end-time $(date -u +%FT%TZ)
# Linux: date -u -d '-1 hour' +%FT%TZ

Metric math toolbox

  • Arithmetic on series: m1 / m2 * 100.
  • FILL(m, 0) for gaps (no errors → no datapoints).
  • IF(cond, a, b), RATE(m), DIFF(m), PERIOD(m).
  • METRICS() = all series in the request; SUM(METRICS()).
  • SEARCH('{ObsLab,service,route} MetricName="Errors"', 'Sum', 60) finds every matching series, like a regex selector.
  • Alarms accept metric math, but not SEARCH(); use a Metrics Insights query instead (part 2).
Metrics Insights

SQL over metrics: top-N and group-by without knowing every dimension value

Metrics Insights queries the most recent metrics with a SQL dialect. It is how you ask "which route has the most errors?" across many series.

-- errors per route for the orders service
SELECT SUM(Errors)
FROM SCHEMA(ObsLab, route, service)
WHERE service = 'orders'
GROUP BY route
ORDER BY SUM() DESC
LIMIT 5

-- slowest Lambda functions by average duration
SELECT AVG(Duration)
FROM SCHEMA("AWS/Lambda", FunctionName)
GROUP BY FunctionName
ORDER BY AVG() DESC
LIMIT 10

Reading the query

  • SCHEMA(ns, dim1, dim2) matches metrics with exactly those dimensions; plain FROM ObsLab matches any dimension set.
  • GROUP BY + ORDER BY + LIMIT = PromQL topk(5, sum by (route)(…)).
  • Aggregations: AVG, SUM, MIN, MAX, COUNT; no percentiles.
  • Use in the console (Metrics → Query), in dashboards, in GetMetricData (Expression = the SQL) and in alarms.
  • An alarm on a Metrics Insights query follows new resources automatically (new routes, new functions).
  • Check the current look-back window and limits in the docs before relying on it for long ranges.
2026

Native OpenTelemetry metrics and PromQL in CloudWatch

Announced in public preview in April 2026 and then generally available (AWS announcement mid-2026). Check region availability and pricing on the day you use it.

What it adds

  • A regional OTLP endpoint: Collectors and SDKs send OTel metrics as they are.
  • Metrics keep their OTel attributes as labels (not limited to 30 dimensions).
  • PromQL queries over these metrics and over AWS-vended metrics enriched with resource attributes.
  • Query Studio in the console: write PromQL, build dashboards, create alarms.
  • A Prometheus-compatible query API: Grafana (or AMG) reads it as a Prometheus data source with SigV4.

How it differs from classic custom metrics

  • Billed per GB ingested with 15 months of storage included, instead of per metric-month.
  • Two account-level settings must be enabled first (resource tags on telemetry and OTel enrichment for AWS metrics).
  • Classic metrics, alarms and EMF keep working unchanged.
  • The preview ran in five Regions (not Tokyo); verify ap-northeast-1 support before planning on it.

When to choose it

  • You already instrument with OpenTelemetry (ch 8) and want AWS-managed storage.
  • Your team thinks in PromQL and does not want to learn metric math.
  • High-cardinality OTel attributes that would explode a per-metric bill.
  • Alternatives: Amazon Managed Service for Prometheus (ch 12) or self-managed Prometheus (ch 5).

The deck treats this as new: the labs use classic EMF metrics, which work in every Region. Read the "CloudWatch OpenTelemetry Metrics" and "PromQL querying" pages of the CloudWatch user guide for the current endpoint, limits and settings.

Cost

What drives a CloudWatch metrics bill

Prices change and differ by Region: learn the drivers, then read the pricing page. The free tier is small and does not cover high-resolution data.

Drivers

  • Custom metrics: each unique namespace + name + dimension set, prorated per hour while it receives data.
  • API requests: GetMetricData (charged per metric requested), PutMetricData, ListMetrics.
  • Alarms: per alarm metric; high-resolution and anomaly-detection alarms cost more (an anomaly alarm counts several metrics).
  • Dashboards beyond the free allowance.
  • EMF: log ingestion and storage plus the extracted custom metrics.
  • Detailed monitoring on EC2 (1-minute) and similar per-resource options.

Habits that keep it small

  • Dimension values from a short, fixed list (route templates, not URLs).
  • Publish one dimension set, not every combination.
  • Third-party dashboards that poll GetMetricData every 10 s are a classic surprise: check request counts in Cost Explorer.
  • Delete unused alarms; consolidate with Metrics Insights alarms.
  • Tag and review with the cost dimensions in ch 13.
Chapter 11 · Part 11.2

Alarms

How an alarm evaluates, what missing data does, and the four alarm types: static, metric math, anomaly detection and composite.

Evaluation

Period, evaluation periods and datapoints to alarm

An alarm looks at the most recent N periods and goes to ALARM when M of them breach. This is the CloudWatch version of for: (ch 4).

Errors (Sum per 60 s period), threshold ≥ 3, evaluation periods N = 5, datapoints to alarm M = 31ok4breach0ok5breach3breach2ok6breachwindow at period 5: breaches = 3 of 5 → ALARMwindow at period 6: breaches = 3 of 5 (periods 2–6) → stays ALARMOKbelow thresholdALARMM of N breachingINSUFFICIENT_DATAnot enough datapointsActions run on statetransitions only, notevery evaluation.

--period 60 --evaluation-periods 5 --datapoints-to-alarm 3 tolerates two good minutes inside a bad stretch, so the alarm does not flap. With M = N it behaves like a strict for: 5m.

Missing data

TreatMissingData decides what "no datapoint" means

EMF error metrics with value 0 are published, but metrics that only appear on failure (or a function that stops being invoked) leave gaps.

SettingA missing period counts asUse for
missing (default)Ignored when enough other datapoints exist; all missing → INSUFFICIENT_DATAMetrics that should always report (CPU)
notBreachingGood (within threshold)Error counts that are only published when errors happen
breachingBad (over threshold)Heartbeats: "no data means dead" (e.g. a job that must report every 5 min)
ignoreNot evaluated: the alarm keeps its current stateSparse metrics where you want no state flip from gaps

Percentile alarms on low traffic: --evaluate-low-sample-count-percentile ignore keeps the state when there are too few samples for a meaningful percentile; evaluate always evaluates.

Static threshold

A static alarm from the CLI

The everyday alarm: one metric, one statistic, one threshold. The SNS topic is the bridge to email, chat or a pager.

TOPIC=$(aws sns create-topic --name obs-lab-alerts \
          --query TopicArn --output text)
aws sns subscribe --topic-arn "$TOPIC" \
  --protocol email --notification-endpoint you@example.com

aws cloudwatch put-metric-alarm \
  --alarm-name obs-lab-orders-errors \
  --alarm-description "orders: >=3 errors/min in 3 of 5 min. Runbook: <url>" \
  --namespace ObsLab --metric-name Errors \
  --dimensions Name=service,Value=orders "Name=route,Value=POST /orders" \
  --statistic Sum --period 60 \
  --evaluation-periods 5 --datapoints-to-alarm 3 \
  --threshold 3 --comparison-operator GreaterThanOrEqualToThreshold \
  --treat-missing-data notBreaching \
  --alarm-actions "$TOPIC" --ok-actions "$TOPIC"

aws cloudwatch describe-alarms --alarm-names obs-lab-orders-errors \
  --query 'MetricAlarms[0].[StateValue,StateReason]'

Checklist for every alarm

  • Name says service and symptom: obs-lab-orders-errors.
  • Description holds the runbook link (it appears in the notification).
  • Dimensions copied from list-metrics, not typed from memory: a typo means INSUFFICIENT_DATA forever (or OK forever with notBreaching).
  • --ok-actions too, so on-call sees recovery.
  • Test it: aws cloudwatch set-alarm-state --state-value ALARM --state-reason test fires the actions once.
  • Percentiles: --extended-statistic p90 instead of --statistic.
Metric math alarm

Alarm on a ratio: error percentage

Counts scale with traffic; ratios do not. Error % is the CloudWatch way to write an availability-style alarm.

aws cloudwatch put-metric-alarm \
  --alarm-name obs-lab-orders-error-pct \
  --evaluation-periods 5 --datapoints-to-alarm 3 \
  --threshold 5 --comparison-operator GreaterThanThreshold \
  --treat-missing-data notBreaching \
  --alarm-actions "$TOPIC" \
  --metrics '[
   {"Id":"req","ReturnData":false,"MetricStat":{"Metric":{
     "Namespace":"ObsLab","MetricName":"Requests","Dimensions":[
     {"Name":"service","Value":"orders"},
     {"Name":"route","Value":"POST /orders"}]},
     "Period":60,"Stat":"Sum"}},
   {"Id":"err","ReturnData":false,"MetricStat":{"Metric":{
     "Namespace":"ObsLab","MetricName":"Errors","Dimensions":[
     {"Name":"service","Value":"orders"},
     {"Name":"route","Value":"POST /orders"}]},
     "Period":60,"Stat":"Sum"}},
   {"Id":"pct","Label":"error %","ReturnData":true,
    "Expression":"IF(req >= 10, 100 * FILL(err,0) / req, 0)"}]'

Details

  • Exactly one entry has ReturnData: true: the series the alarm evaluates.
  • IF(req >= 10, …, 0): no paging on 1 error out of 2 requests at night.
  • All metrics in one alarm must use the same period.
  • Multi-resource version: a Metrics Insights query as the Expression (with Period) — one alarm covers every route.
  • For SLO burn-rate alarms (several windows), see the SRE deck; Application Signals SLOs (part 3) generate them for you.
Anomaly detection

Alarms on a learned band instead of a fixed number

CloudWatch trains a model on the metric's history (trend, hourly and weekly seasonality) and alarms when values leave the band.

aws cloudwatch put-metric-alarm \
  --alarm-name obs-lab-orders-latency-anomaly \
  --evaluation-periods 3 --datapoints-to-alarm 3 \
  --comparison-operator GreaterThanUpperThreshold \
  --threshold-metric-id band \
  --treat-missing-data notBreaching \
  --metrics '[
   {"Id":"m1","ReturnData":true,"MetricStat":{"Metric":{
     "Namespace":"ObsLab","MetricName":"Latency","Dimensions":[
     {"Name":"service","Value":"orders"},
     {"Name":"route","Value":"POST /orders"}]},
     "Period":60,"Stat":"p90"}},
   {"Id":"band","Label":"expected p90",
    "Expression":"ANOMALY_DETECTION_BAND(m1, 2)"}]'

How to use it well

  • ANOMALY_DETECTION_BAND(m1, 2): width in standard deviations; larger = fewer alarms.
  • Comparison: GreaterThanUpperThreshold, LessThanLowerThreshold or LessThanLowerOrGreaterThanUpperThreshold.
  • The model uses up to two weeks of history; a brand-new metric gives a poor band at first.
  • Exclude known bad windows (incidents, load tests) from training so the model does not learn them as normal.
  • Good for: traffic drops, latency drift, seasonal business metrics.
  • Poor for: hard limits (disk full, error budget) — use static thresholds there.
Composite alarms

Composite alarms: page on the combination, not on every symptom

A composite alarm has a rule over other alarms' states. Child alarms can stay silent (no actions) while the composite pages once.

orders-errorsstatic, M of Norders-latency-anomalyanomaly bandmaintenance-windowsuppressor alarmorders-pageALARM(a) AND ALARM(b)ActionsSuppressorSNSpage
aws cloudwatch put-composite-alarm \
  --alarm-name obs-lab-orders-page \
  --alarm-rule 'ALARM("obs-lab-orders-errors") AND
               ALARM("obs-lab-orders-latency-anomaly")' \
  --alarm-actions "$TOPIC"

# rule language
#   ALARM("x"), OK("x"), INSUFFICIENT_DATA("x")
#   AND, OR, NOT, parentheses, TRUE, FALSE
# suppression (all three together):
#   --actions-suppressor <alarm name or ARN>
#   --actions-suppressor-wait-period 120
#   --actions-suppressor-extension-period 300

Typical design: per-symptom alarms without actions (they still show on dashboards), one composite per service that notifies, and a suppressor alarm for "dependency down" or maintenance so downstream services do not all page at once.

Actions

What an alarm can trigger

Actions run when the state changes. Every state change is also an EventBridge event, which is the most flexible hook.

Alarm actions

  • SNS topic: email, SMS, HTTPS, chat integrations (Amazon Q Developer in chat applications), Lambda subscribers.
  • Lambda function invoked directly by the alarm.
  • EC2 actions: stop, terminate, reboot, recover (per-instance metric alarms).
  • Auto Scaling policies (step scaling; target tracking creates its own alarms).
  • Systems Manager: create an OpsItem or start an Incident Manager incident.
# EventBridge rule pattern: every alarm
# named obs-lab-* that enters ALARM
{
  "source": ["aws.cloudwatch"],
  "detail-type":
    ["CloudWatch Alarm State Change"],
  "detail": {
    "state": {"value": ["ALARM"]},
    "alarmName":
      [{"prefix": "obs-lab-"}]
  }
}
# targets: Lambda, Step Functions,
# SQS, an API destination (webhook)

Operational habits

  • Separate topics per severity (page, ticket), like Alertmanager receivers (ch 4).
  • Alarms are regional: create them in the Region of the metric.
  • Manage alarms as code (CloudFormation, CDK, Terraform), reviewed like rules (ch 13).
  • Review alarms that never fired and alarms that fire weekly without action.
Chapter 11 · Part 11.3

Dashboards and curated views

CloudWatch dashboards, Container Insights, Application Signals, Synthetics and the CloudWatch agent.

Dashboards

CloudWatch dashboards as JSON

A dashboard is a JSON document of widgets. Keep it in Git and deploy it with put-dashboard or IaC, the same discipline as Grafana (ch 10).

{"widgets": [
 {"type": "metric", "x": 0, "y": 0, "width": 12, "height": 6,
  "properties": {"title": "orders latency",
   "region": "ap-northeast-1",
   "view": "timeSeries", "period": 60,
   "metrics": [
    ["ObsLab", "Latency", "service", "orders", "route", "POST /orders",
     {"stat": "p50"}],
    ["...", {"stat": "p90"}]]}},
 {"type": "log", "x": 12, "y": 0, "width": 12, "height": 6,
  "properties": {"title": "recent errors",
   "region": "ap-northeast-1",
   "view": "table",
   "query": "SOURCE '/obs-lab/orders' | filter level = 'ERROR' | limit 20"}},
 {"type": "alarm", "x": 0, "y": 6, "width": 24, "height": 3,
  "properties": {"title": "alarms", "alarms": [
   "arn:aws:cloudwatch:ap-northeast-1:<acct>:alarm:obs-lab-orders-page"
  ]}}
]}

Features worth knowing

  • "..." repeats the previous metric's namespace, name and dimensions.
  • Widgets: metric, log (Logs Insights query), alarm status, text (Markdown), explorer, custom (Lambda-backed).
  • Dashboard variables switch a dimension value (e.g. service) from a dropdown.
  • Widgets can read other Regions and, with cross-account observability, other accounts (ch 12).
  • Automatic dashboards per service (Lambda, DynamoDB, …) are a free starting point.
  • Deploy: aws cloudwatch put-dashboard --dashboard-name obs-lab --dashboard-body file://dash.json.
Container Insights

Container Insights with enhanced observability for EKS

Curated metrics and logs for clusters, nodes, pods and containers, installed as an EKS add-on. CloudWatch's answer to kube-prometheus-stack dashboards (ch 5).

What you get

  • Cluster → namespace → workload → pod → container drill-down with health and performance views.
  • Metrics like CPU and memory utilization against requests and limits, restarts, network, and control-plane metrics.
  • Container logs to CloudWatch Logs through the bundled Fluent Bit.
  • Integration with Application Signals for application-level views.
  • Mid-2026: OpenTelemetry-based Container Insights metrics and dashboards for EKS, queryable with PromQL (verify availability).
# EKS: install the observability add-on
aws eks create-addon \
  --cluster-name <cluster> \
  --addon-name \
  amazon-cloudwatch-observability

# the node role (or the agent's Pod
# Identity / IRSA role) needs the
# managed policy
#   CloudWatchAgentServerPolicy

# check
kubectl -n amazon-cloudwatch get pods
# cloudwatch-agent-*  (DaemonSet)
# fluent-bit-*        (DaemonSet)

Cost and fit

  • Billing is per observation (enhanced observability) or per metric (older mode) plus logs: estimate with a small cluster first.
  • Container log volume is usually the bigger line item: set retention and filters (part 1 of ch 12).
  • ECS has its own Container Insights (also with an enhanced mode).
  • Running kube-prometheus-stack already? Pick one source of truth for dashboards and alerts per cluster.
Application Signals

Application Signals: RED metrics, service map and SLOs from OpenTelemetry

AWS's application performance monitoring (APM) layer. Instrumentation is OpenTelemetry-based (ADOT SDKs and the CloudWatch agent).

What it shows

  • Services discovered from traces and their operations (route templates).
  • Standard metrics per service and operation: Latency, Error (4xx-type), Fault (5xx-type), in the ApplicationSignals namespace.
  • Application map: callers, dependencies, AWS resources.
  • Links from a metric spike to correlated traces (via Transaction Search) and logs.

SLOs

  • Define an SLI on a service operation (latency or availability) or on any CloudWatch metric.
  • Attainment, error budget remaining and burn rate are computed for you.
  • Create burn-rate alarms on the SLO: concepts in the SRE deck.
  • Request-based or period-based SLIs.

Enabling it

  • One-time per account: aws application-signals start-discovery (or the console button).
  • EKS: the amazon-cloudwatch-observability add-on can auto-instrument Java, Python, .NET and Node.js workloads via annotations.
  • ECS/EC2: CloudWatch agent + ADOT SDK as sidecar or on the host.
  • Lambda: enable in the function's monitoring settings (adds the AWS OTel layer).
  • Go has no zero-code agent here: instrument with the OTel SDK (ch 8) and send OTLP.
Synthetics and RUM

Outside-in signals: Synthetics canaries and RUM

Black-box checks (ch 1) catch what in-process metrics cannot: DNS, TLS, CDN, login flows and the real user's browser.

CloudWatch Synthetics

  • Canaries are scheduled scripts (Node.js or Python runtimes; browser steps via Puppeteer, Playwright or Selenium).
  • Blueprints: heartbeat, API canary, broken-link checker, visual monitoring, GUI workflow.
  • Publish SuccessPercent and Duration metrics in CloudWatchSynthetics; alarm on them.
  • Run inside a VPC to test private endpoints.
  • Artifacts (screenshots, HAR files) go to S3: set a lifecycle rule.

CloudWatch RUM

  • A JavaScript snippet in your web app sends page loads, errors and Web Vitals.
  • Sessions link to X-Ray traces for the backend calls.
  • Sample sessions to control cost.
  • Prometheus equivalent: blackbox_exporter for probes (ch 2); no direct equivalent for RUM.

Where they fit

  • Synthetics = availability SLI from the user's side (SRE deck).
  • Use one canary per critical user journey, not per endpoint.
  • Alarm on 2 failures in a row from 2+ locations to avoid paging on one bad network path.
CloudWatch agent

The CloudWatch agent: host metrics, StatsD, logs and OTLP

One agent for EC2, on-premises servers and containers. It publishes custom metrics, ships logs, and can receive OpenTelemetry and X-Ray data.

{
  "agent": {"metrics_collection_interval": 60},
  "metrics": {
    "namespace": "ObsLab/Host",
    "append_dimensions": {"InstanceId": "${aws:InstanceId}"},
    "metrics_collected": {
      "mem":  {"measurement": ["mem_used_percent"]},
      "disk": {"measurement": ["used_percent"],
               "resources": ["/"]},
      "procstat": [{"exe": "orders",
                    "measurement": ["cpu_usage", "memory_rss"]}],
      "statsd": {"service_address": ":8125"}
    }
  },
  "logs": {"logs_collected": {"files": {"collect_list": [{
     "file_path": "/var/log/orders/*.log",
     "log_group_name": "/obs-lab/orders",
     "log_stream_name": "{instance_id}",
     "retention_in_days": 7}]}}}
}

Know these

  • Memory and disk usage are not EC2 default metrics: you need the agent.
  • procstat monitors individual processes; statsd and collectd accept app metrics.
  • append_dimensions adds instance context; avoid adding high-cardinality dimensions.
  • Config wizard: amazon-cloudwatch-agent-config-wizard; store config in SSM Parameter Store and roll out with Systems Manager.
  • IAM: CloudWatchAgentServerPolicy on the instance role.
  • Recent versions can collect traces from OpenTelemetry or X-Ray SDKs (ch 12) — the replacement path for the X-Ray daemon.
Common mistakes

CloudWatch metrics and alarm mistakes

Each one has caused either a silent outage or a surprise bill.

Metrics

  • order_id or user_id as an EMF dimension → thousands of billed custom metrics.
  • Publishing only statistic sets, then asking for p99.
  • Querying {service} and expecting it to sum {service, route} series.
  • Averaging latency; never looking at p90/p99.
  • A dashboard tool polling GetMetricData every few seconds.

Alarms

  • Dimension typo → alarm stuck in INSUFFICIENT_DATA (or OK with notBreaching) forever, nobody notices.
  • treatMissingData: missing on an error count that only appears on failure.
  • Paging on raw error counts instead of error % with a minimum traffic guard.
  • Every symptom alarm notifies → five pages for one incident; use composite alarms.
  • No OK action, no runbook in the description.

Checks that catch them

  • list-metrics after every new EMF field; copy dimensions from its output.
  • Alarm on INSUFFICIENT_DATA for critical alarms, or use describe-alarms --state-value INSUFFICIENT_DATA in a weekly review.
  • set-alarm-state drill for each paging alarm.
  • Cost Explorer grouped by usage type (CW:MetricMonitorUsage, CW:Requests).
Self-check

CloudWatch metrics and alarms: can you answer these?

Answer with the exact setting, field or expression.

Metrics

  • What three things identify a metric, and how many dimensions can it have?
  • How long are 1-minute datapoints kept, and what happens after that?
  • Why can a metric have no p99 at all?
  • Write the EMF JSON for Latency and Errors with dimensions service and route.
  • What does the 2026 OTel metrics feature change about billing and querying?

Alarms

  • Explain period 60, evaluation periods 5, datapoints to alarm 3.
  • Which TreatMissingData for a heartbeat metric, and which for an error count?
  • Why can an alarm not use SEARCH(), and what do you use instead?
  • When is anomaly detection the wrong choice?
  • Write a composite rule that pages only when errors and latency both alarm.

Views

  • What does Container Insights need to be installed on EKS?
  • Which three standard metrics does Application Signals publish?
  • Which host metrics require the CloudWatch agent on EC2?
  • Where do Synthetics canaries fit compared with RED metrics?
  • How do you put a Logs Insights query on a dashboard?

Practice: Lab 11 in Hands-on labs — a Lambda that emits EMF, a metric filter, static, anomaly and composite alarms, and a dashboard in ap-northeast-1.

Chapter 12

CloudWatch logs, traces and AWS managed services

CloudWatch Logs and Logs Insights, tracing on AWS after the X-Ray SDKs, and the managed open-source path: Amazon Managed Service for Prometheus, Amazon Managed Grafana and cross-account observability.

Chapter 12 · Part 12.1

CloudWatch Logs

Log groups and classes, retention, metric and subscription filters, data protection and Live Tail.

Model

Log groups, log streams, log events

Everything is regional. A log group is the unit of retention, encryption, class, access control and billing.

ProducersLambda function stdoutLoggingConfig.LogGroupECS awslogs / FireLensEKS: Fluent Bit (Container Insights)CloudWatch agentEC2, on-premises filesOTLP logs endpointCollector / SDK, HTTP + SigV4Vended logsVPC Flow Logs, Route 53, …CloudWatch LogsLog group /obs-lab/ordersclass · retention · KMS · policiesLog streamone per source (instance, container)Log eventstimestamp + message (≤ 1 MB event)ConsumersLogs InsightsQL · PPL · SQL (part 2)Metric filters→ CloudWatch metricsSubscription filtersKinesis, Firehose, LambdaLive Tailinteractive streamingExport to S3one-off export tasks

Name groups by owner and purpose (/obs-lab/orders, /aws/lambda/<fn>, /aws/eks/<cluster>/cluster) so IAM policies, retention rules and Logs Insights queries can target prefixes.

Log classes

Standard vs Infrequent Access

The class is chosen when the log group is created and cannot be changed later. Infrequent Access trades features for a lower ingestion price.

CapabilityStandardInfrequent Access
Ingestion priceBaselineAbout half of Standard per GB
Storage, Logs Insights scanSameSame
Logs Insights QLYesYes, except pattern, diff, unmask
OpenSearch PPL / SQLYesYes (since Mar 2026)
Data protection (masking)YesYes (since Mar 2026)
Metric filters, EMF, field indexes, facetsYesNo
Subscription filters, Live TailYesNo
Export to S3YesYes
GetLogEvents / FilterLogEventsYesNo (query with Logs Insights)

Choosing

  • Standard for application logs you alarm on, stream elsewhere, tail during incidents or extract EMF metrics from.
  • Infrequent Access for logs you keep for audit or occasional forensics: debug dumps, verbose access logs, compliance copies.
  • Container Insights and Lambda Insights do not ingest into IA groups.
  • Create the IA group up front: aws logs create-log-group --log-group-name /obs-lab/orders-ia --log-group-class INFREQUENT_ACCESS.
  • Now also called Essentials (API accepts both values; responses switch to ESSENTIALS from Dec 2026). IA has gained features over time: check the docs table.
Retention and encryption

Retention, KMS and the "never expire" default

New log groups keep data forever unless you set retention. That default is the single largest source of silent CloudWatch Logs cost.

# set retention (allowed values: 1, 3, 5, 7, 14, 30, 60, 90,
# 120, 150, 180, 365, 400, 545, 731, 1096, 1827, 2192,
# 2557, 2922, 3288, 3653 days)
aws logs put-retention-policy \
  --log-group-name /obs-lab/orders --retention-in-days 7

# find groups that never expire, biggest first
aws logs describe-log-groups \
  --query 'logGroups[?!retentionInDays].[logGroupName,storedBytes]' \
  --output text | sort -k2 -n -r | head

# encrypt with a customer managed key (key policy must
# allow the logs.<region>.amazonaws.com principal)
aws logs associate-kms-key --log-group-name /obs-lab/orders \
  --kms-key-id arn:aws:kms:ap-northeast-1:<acct>:key/<id>

Practice

  • Set retention in IaC for every group, including the ones AWS services create (/aws/lambda/…, /aws/eks/…).
  • Typical tiers: 7–30 days for app logs in CloudWatch, longer copies in S3 via Firehose if audit needs them.
  • Logs are always encrypted at rest; KMS adds key control and CloudTrail evidence of decrypts.
  • storedBytes shows where the storage money goes.
  • Lambda creates /aws/lambda/<fn> on first invocation without retention — set LoggingConfig.LogGroup to a group you created, or create the default group first.
Metric filters

Metric filters: count things in logs

A metric filter turns matching log events into a CloudWatch metric. It only sees events ingested after it was created.

# count ERROR events from JSON logs
aws logs put-metric-filter --log-group-name /obs-lab/orders \
  --filter-name orders-error-logs \
  --filter-pattern '{ $.level = "ERROR" }' \
  --metric-transformations metricName=ErrorLogs,\
metricNamespace=ObsLab,metricValue=1,defaultValue=0

# sum a numeric field instead of counting events
#   metricValue=$.duration_ms

# test a pattern against sample lines before creating it
aws logs test-metric-filter \
  --filter-pattern '{ $.status >= 500 }' \
  --log-event-messages '{"status":502,"level":"ERROR"}' \
                       '{"status":201,"level":"INFO"}'
# → matches[0].eventMessage shows only the 502 line

Filter pattern syntax

StyleExample
TermsERROR · "payment failed" · ?ERROR ?WARN (either)
JSON{ $.level = "ERROR" && $.route = "POST /orders" }
JSON numeric{ $.duration_ms > 500 }
Space-delimited[ip, id, user, ts, req, status=5*, size]
Regex (subset)%timeout|deadline%

JSON and space-delimited patterns can add up to three dimensions from fields, but then defaultValue is not allowed, and high-cardinality fields multiply custom metrics. For metrics from structured app logs, EMF (ch 11) is usually the better tool.

Subscription filters

Streaming logs out: subscription filters and export

Subscription filters deliver matching events in near real time. Export tasks copy a time range once. Subscription filters do not work on Infrequent Access groups; export tasks do.

Subscription filters

  • Destinations: Kinesis Data Streams, Amazon Data Firehose, Lambda, and (console) OpenSearch Service via a managed Lambda.
  • Up to five subscription filters per log group.
  • Payloads are gzip-compressed, base64-encoded batches.
  • Cross-account: a destination in the central account with a destination policy.
  • Account-level subscription filter policy (put-account-policy --policy-type SUBSCRIPTION_FILTER_POLICY) covers all groups, with an exclusion list.
# stream errors to Firehose
# → S3 / SIEM
FH=arn:aws:firehose:\
ap-northeast-1:<acct>:\
deliverystream/obs-lab-logs
aws logs put-subscription-filter \
  --log-group-name /obs-lab/orders \
  --filter-name to-firehose \
  --filter-pattern \
    '{ $.level = "ERROR" }' \
  --destination-arn $FH \
  --role-arn arn:aws:iam::<acct>:\
role/cwl-to-firehose

# one-off export to S3
# (times in epoch ms)
aws logs create-export-task \
  --log-group-name /obs-lab/orders \
  --from 1794000000000 \
  --to   1794086400000 \
  --destination obs-lab-archive

Pick by need

  • Continuous archive or SIEM → Firehose to S3 / third party.
  • React to a pattern (ticket, auto-remediation) → Lambda.
  • Several consumers, replay → Kinesis Data Streams.
  • Occasional copy for an investigation → export task (one active export task per account at a time).
  • Remember: the subscription pays ingestion once, then the destination's own costs.
Data protection

Masking sensitive data at ingestion

A data protection policy finds sensitive values with managed data identifiers and masks them in every read path. Seeing the raw value needs logs:Unmask.

cat > dp.json <<'EOF'
{"Name": "obs-lab-pii", "Version": "2021-06-01",
 "Statement": [
  {"Sid": "audit",
   "DataIdentifier": [
     "arn:aws:dataprotection::aws:data-identifier/EmailAddress"],
   "Operation": {"Audit": {"FindingsDestination": {}}}},
  {"Sid": "redact",
   "DataIdentifier": [
     "arn:aws:dataprotection::aws:data-identifier/EmailAddress"],
   "Operation": {"Deidentify": {"MaskConfig": {}}}}]}
EOF
aws logs put-data-protection-policy \
  --log-group-identifier /obs-lab/orders \
  --policy-document file://dp.json

# the event is stored; reads show ****************
# a principal with logs:Unmask can run in Logs Insights:
#   fields @timestamp, unmask(@message)

What it is and is not

  • Applies to events ingested after the policy is set.
  • Managed identifiers cover emails, credit cards, keys, many country-specific IDs; custom identifiers take a regex.
  • The audit statement reports findings (optionally to another log group, S3 or Firehose).
  • An account-level policy can cover all groups.
  • It is a safety net. The control is still not logging secrets in the first place (ch 6: redaction and allowlists).
  • Masking adds per-GB scanning cost on ingestion.
Live Tail

Live Tail: tail -f for CloudWatch Logs

Interactive streaming of new events from up to 10 log groups, with a filter pattern. Billed per minute of session; close it when done.

# CLI v2 streams to your terminal (Ctrl-C to stop)
aws logs start-live-tail \
  --log-group-identifiers \
    arn:aws:logs:ap-northeast-1:<acct>:log-group:/obs-lab/orders \
  --log-event-filter-pattern '{ $.level = "ERROR" }'

# older way, still useful for scripting:
aws logs tail /obs-lab/orders --follow \
  --filter-pattern '{ $.status >= 500 }' --format short

Use it for

  • Watching a deploy or a canary roll out.
  • Confirming a fix: errors stop arriving.
  • Following one request: filter on order_id or trace_id.
  • Not available on Infrequent Access groups.
  • --log-group-identifiers takes ARNs (without the trailing :*).
  • Sessions time out after a few hours; the console highlights matching terms.
Chapter 12 · Part 12.2

Logs Insights

Logs Insights QL, OpenSearch PPL and SQL, facets and field indexes, and how scanning drives cost.

Logs Insights QL

The pipeline query language

Commands are joined with | and run left to right. JSON fields are discovered automatically and addressed with dots (http.status).

Commands

CommandPurpose
fieldsChoose or compute fields: fields @timestamp, route, duration_ms
filterKeep matching events: filter level = "ERROR" and duration_ms > 500
statsAggregate: stats count(*), pct(duration_ms, 95) by route, bin(5m)
sort / limitOrder and cap the result (limit up to 10,000)
parseExtract fields from text with a glob or regex
dedupKeep the first event per value: dedup order_id
patternCluster messages into recurring patterns
diffCompare patterns with the previous equal-length period
anomalyFlag unusual patterns (pairs with pattern)

Built-in fields and functions

  • @timestamp, @message, @logStream, @log, @ingestionTime.
  • Lambda adds @requestId, @duration, @billedDuration, @maxMemoryUsed on REPORT lines.
  • Aggregations: count, count_distinct, sum, avg, min, max, pct, earliest, latest.
  • Strings: strcontains, concat, replace, toupper, regex like / =~.
  • ispresent(field), coalesce(a, b), datefloor(@timestamp, 1h).
  • Comments start with #.
Worked queries

Answering real questions about the shop

All against /obs-lab/orders with the slog schema from ch 6 (level, msg, route, status, duration_ms, order_id, trace_id).

# 1. error rate per route over time
fields route, status
| stats sum(status >= 500) as errors, count(*) as total,
        errors * 100 / total as error_pct by route, bin(5m)

# 2. latency distribution per route
filter ispresent(duration_ms)
| stats pct(duration_ms, 50) as p50, pct(duration_ms, 95) as p95,
        pct(duration_ms, 99) as p99, count(*) as n by route

# 3. the 10 slowest requests, with trace IDs to open in traces
fields @timestamp, route, duration_ms, trace_id
| sort duration_ms desc | limit 10
# 4. everything one request logged
filter order_id = "8c1f4c2e-…" or trace_id = "4bf92f35…"
| sort @timestamp asc

# 5. text logs: parse with a glob pattern
parse @message "* payment failed after *ms: *" as lvl, ms, reason
| stats count(*) by reason

# 6. top error messages, grouped into patterns
filter level = "ERROR"
| pattern msg

# 7. Lambda cold starts and memory headroom
filter @type = "REPORT"
| stats count(*) as invocations, max(@maxMemoryUsed) as maxMem,
        sum(ispresent(@initDuration)) as coldStarts by bin(1h)

Query 3 is the bridge to tracing: copy a trace_id into Transaction Search or Tempo (ch 9). That link only exists if your logger writes trace_id (ch 6).

PPL and SQL

OpenSearch PPL and SQL in Logs Insights

Pick the language in the console or with --query-language CWLI | PPL | SQL. PPL feels like a pipeline; SQL is the familiar database syntax.

# OpenSearch PPL: errors by route
fields route, status, duration_ms
| where status >= 500
| stats count() as errors,
        avg(duration_ms) as avg_ms by route
| sort - errors
| head 5

-- OpenSearch SQL: log group goes in FROM
SELECT route, COUNT(*) AS n,
       AVG(duration_ms) AS avg_ms
FROM `/obs-lab/orders`
WHERE status >= 500
GROUP BY route ORDER BY n DESC LIMIT 5
-- several groups:
-- FROM `logGroups(logGroupIdentifier:
--   ['/obs-lab/orders','/obs-lab/payments'])`

When to use which

  • Logs Insights QL: fastest to write; pattern, diff, anomaly; the only one with large-result pagination.
  • PPL: if you already know OpenSearch; eval, top, rare, fillnull, JSON functions, join and subqueries.
  • SQL: joins between log groups (orders ↔ payments by trace_id), familiar syntax.
  • With SQL the log groups are named inside the query, not in --log-group-names.
  • PPL and SQL have their own concurrency limit (15 concurrent queries).
  • Grafana's CloudWatch data source supports all three.
Running queries

Queries from the CLI: start, poll, read

Logs Insights is asynchronous. Scripts start a query, poll its status, then read the results (also how runbooks and bots use it).

END=$(date +%s); START=$((END - 3600))
QID=$(aws logs start-query \
  --log-group-names /obs-lab/orders \
  --start-time $START --end-time $END \
  --query-string 'filter level = "ERROR"
    | stats count(*) as errors by route' \
  --query queryId --output text)

# status: Scheduled → Running → Complete
aws logs get-query-results --query-id "$QID" \
  --query '[status, results]'
# ["Complete", [[{"field":"route","value":"POST /orders"},
#                {"field":"errors","value":"14"}]]]

# PPL: same call, different language
aws logs start-query --query-language PPL \
  --log-group-names /obs-lab/orders \
  --start-time $START --end-time $END \
  --query-string 'fields route | where status >= 500
    | stats count() by route'

Practical limits

  • A query can run for up to 60 minutes before it times out.
  • Results are kept for 7 days (get-query-results again by ID).
  • Save queries (put-query-definition) so the team shares them.
  • Add a query to a dashboard as a log widget (ch 11).
  • The SOURCE command in the query can select groups by prefix or class instead of listing names.
  • statistics.bytesScanned in the result is what you pay for.
Cost and speed

Scanning is the bill: field indexes and facets

Logs Insights charges per GB scanned. The fastest, cheapest query is the one that reads the fewest bytes.

Read less

  • Narrow time range first; then fewer log groups.
  • Field indexes (put-index-policy) on fields you filter by often (order_id, trace_id, status): queries skip events that cannot match.
  • Use filterIndex to force an index-only lookup.
  • Facets (Dec 2025) show value counts for indexed fields across groups and let you filter by data source and type, not only by log group.
  • Log less: debug-level noise is paid for at ingestion and again at every query.
# index the fields we search by
aws logs put-index-policy \
  --log-group-identifier \
    /obs-lab/orders \
  --policy-document \
  '{"Fields": ["order_id",
     "trace_id", "route"]}'

# a query that uses the index
filterIndex order_id = "8c1f…"
| fields @timestamp, msg

Compared with Loki

  • Loki indexes only labels and greps chunks (ch 7); Logs Insights scans fields without a schema.
  • Both punish broad time ranges.
  • Loki label rules (low cardinality) map to CloudWatch field indexes (high cardinality is fine there).
  • Loki cost is your storage and queriers; CloudWatch cost is per GB ingested + stored + scanned.
Chapter 12 · Part 12.3

Traces on AWS

X-Ray concepts, the move to OpenTelemetry, trace ID formats, Transaction Search and Application Signals.

X-Ray concepts

X-Ray vocabulary, mapped to OpenTelemetry

X-Ray is still the trace backend on AWS. What changed is how you instrument and ship data to it.

X-Ray termMeaningOpenTelemetry equivalent (ch 8–9)
TraceAll segments sharing one trace IDTrace
SegmentWork done by one service for a requestServer/consumer span (root span of a service)
SubsegmentA call or block inside a segment (DB, HTTP, AWS SDK)Child span (client/internal)
AnnotationsIndexed key-value pairs, usable in filter expressionsSpan attributes chosen for indexing
MetadataNon-indexed key-values for contextOther span attributes
Trace map / service mapGraph of services and edges with healthService graph (Tempo, ch 9)
Sampling rulesReservoir per second + fixed rate (default 1 req/s + 5%)Samplers; xray remote sampler
GroupsSaved filter expression with its own metricsTraceQL metrics query (ch 9)

Filter expression examples: service("orders") AND fault = true, responsetime > 0.5, annotation.order_type = "express".

Timeline

X-Ray SDKs and daemon: maintenance now, end of support in 2027

AWS made OpenTelemetry the primary instrumentation for X-Ray. Existing SDK-instrumented apps keep working; new work should start on OTel.

X-Ray SDKs (all languages) and the X-Ray daemon25 Feb 2026maintenance modeOnly critical bug and security fixes;no new features.25 Feb 2027end of supportNo more releases. AWS says X-Raykeeps accepting existing SDK traces.Path forwardOpenTelemetry SDK / ADOT → OTLP → X-Ray

Migration map

OldNew
X-Ray SDK in the appOpenTelemetry SDK or ADOT distribution (auto-instrumentation where available)
X-Ray daemon (UDP 2000)CloudWatch agent or an OpenTelemetry Collector exporting OTLP to X-Ray
X-Amzn-Trace-Id propagationW3C traceparent (plus the xray propagator while mixed)
AnnotationsSpan attributes (index selected ones)
Lambda active tracingStill available; add OTel (Application Signals) for in-function spans
Trace IDs

Two ID formats and two headers

Mixed fleets break traces at the boundary where one side only understands the other format. Know both.

# X-Ray trace ID: version - epoch seconds (hex) - 96 random bits
1-5759e988-bd862e3fe1be46a994272793

# W3C trace ID (32 hex) — the same trace written for OTel:
5759e988bd862e3fe1be46a994272793

# HTTP headers
X-Amzn-Trace-Id: Root=1-5759e988-bd862e3fe1be46a994272793;
                 Parent=53995c3f42cd8ad8;Sampled=1
traceparent: 00-5759e988bd862e3fe1be46a994272793-53995c3f42cd8ad8-01

What to configure

  • X-Ray accepts W3C-format trace IDs: a random first 32 bits are fine; you no longer need the epoch prefix.
  • Inside AWS (ALB, API Gateway, Lambda) the X-Amzn-Trace-Id header is still what services add and forward.
  • OTel SDKs: set propagators to tracecontext,baggage,xray while some services still use X-Ray headers.
  • Log the 32-hex form (trace_id) so Logs Insights, Loki and Tempo all match it (ch 6).
  • Converting: insert dashes after the 8th hex character and prefix 1-.
Transaction Search

Transaction Search: every span as a structured log

With Transaction Search, X-Ray stores spans in CloudWatch Logs (group aws/spans) and indexes a sample for trace summaries. Required for the OTLP traces endpoint.

# 1) allow X-Ray to write spans to CloudWatch Logs
A=<acct>; L=arn:aws:logs:ap-northeast-1:$A:log-group
cat > ts.json <<EOF
{"Version":"2012-10-17","Statement":[{
 "Sid":"TransactionSearchXRayAccess","Effect":"Allow",
 "Principal":{"Service":"xray.amazonaws.com"},
 "Action":"logs:PutLogEvents",
 "Resource":["$L:aws/spans:*",
             "$L:/aws/application-signals/data:*"],
 "Condition":{
  "ArnLike":{"aws:SourceArn":"arn:aws:xray:ap-northeast-1:$A:*"},
  "StringEquals":{"aws:SourceAccount":"$A"}}}]}
EOF
aws logs put-resource-policy --policy-name TransactionSearchXRay \
  --policy-document file://ts.json

# 2) send segments to CloudWatch Logs instead of X-Ray storage
aws xray update-trace-segment-destination --destination CloudWatchLogs
aws xray get-trace-segment-destination
# {"Destination": "CloudWatchLogs", "Status": "ACTIVE"}  (~10 min)

# 3) share of spans indexed as trace summaries (1% is the free tier)
aws xray update-indexing-rule --name Default \
  --rule '{"Probabilistic": {"DesiredSamplingPercentage": 1}}'

What you gain

  • All ingested spans are searchable (Logs Insights on aws/spans, the Transaction Search console), not only sampled trace summaries.
  • Span-level analytics: group by attributes, find the slow customer or the failing dependency.
  • Pricing moves to CloudWatch Logs ingestion of spans plus indexing above the free share.
  • Prerequisite for the X-Ray OTLP endpoint (next slide) and for Application Signals' trace links.
  • Reports exist of the trace map looking different after switching: verify your views in a sandbox first.
OTLP to AWS

Shipping OpenTelemetry to CloudWatch: agent or Collector, SigV4, HTTP

CloudWatch exposes OTLP/HTTP endpoints per Region. Requests are signed with SigV4, so a Collector needs the sigv4auth extension (or use the CloudWatch agent / ADOT Collector).

orders, paymentsOTel SDK (ch 8)OTLP gRPCCollector / CW agentbatch · resource · sigv4authxray.<region>…/v1/tracesneeds Transaction Searchlogs.<region>…/v1/logsx-aws-log-group / -stream headersmonitoring.<region>…OTel metrics (2026, check Region)HTTP only (no gRPC) on the AWS side; SigV4 service names xray, logs.sigv4auth ships in the contrib and ADOT Collector distributions.Credentials: default AWS chain (instance role, Pod Identity, env).One Collector can fan out the same traces to Tempo and X-Ray (ch 9).
extensions:
  sigv4auth/xray: {region: ap-northeast-1, service: xray}
  sigv4auth/logs: {region: ap-northeast-1, service: logs}
exporters:
  otlphttp/xray:
    traces_endpoint:
      https://xray.ap-northeast-1.amazonaws.com/v1/traces
    auth: {authenticator: sigv4auth/xray}
  otlphttp/logs:
    logs_endpoint:
      https://logs.ap-northeast-1.amazonaws.com/v1/logs
    headers: {x-aws-log-group: /obs-lab/orders,
              x-aws-log-stream: otel}
    auth: {authenticator: sigv4auth/logs}
service:
  extensions: [sigv4auth/xray, sigv4auth/logs]
  pipelines:   # receivers [otlp], processors [batch]
    traces: {exporters: [otlphttp/xray], …}
    logs:   {exporters: [otlphttp/logs], …}
Application Signals in practice

From instrumentation to an SLO, end to end

Application Signals is the managed APM layer over the same OTel data. It shines where you can use auto-instrumentation.

Instrument

  • EKS: add-on + workload annotation (Java, Python, .NET, Node.js).
  • Lambda: enable Application Signals (adds the AWS OTel layer).
  • Go and others: OTel SDK → CloudWatch agent.
→

Discover

  • aws application-signals start-discovery (once per account).
  • Services and operations appear with Latency / Error / Fault.
→

Investigate

  • Service page → operation → correlated spans (Transaction Search).
  • Jump to logs with the same trace ID.
→

Commit

  • Create an SLO on an operation (e.g. POST /orders p99 < 300 ms, 99.5%).
  • Burn-rate alarms → SNS (SRE deck).

Compared with the open-source path in this deck: Application Signals ≈ span metrics + service graph + SLO tooling (ch 9, SRE deck), managed for you, priced per signal volume.

Chapter 12 · Part 12.4

AWS managed open source and cross-account

Amazon Managed Service for Prometheus, Amazon Managed Grafana, cross-account observability, and how to choose.

AMP

Amazon Managed Service for Prometheus (AMP)

A Prometheus-compatible, multi-AZ store and query engine (based on Cortex). You bring the collector; AMP stores, evaluates rules and routes alerts.

Pieces

  • Workspace: the tenant; remote-write and query endpoints, rules and an alert manager.
  • Ingestion: Prometheus remote_write with SigV4, the OTel Collector, Grafana Alloy, or an AWS managed scraper for EKS (agentless).
  • Query: PromQL over SigV4 at /api/v1/query — Amazon Managed Grafana or any Grafana with SigV4.
  • Rules: rule groups namespaces (Prometheus rule YAML), recording and alerting.
  • Alert manager: Alertmanager config; receivers are SNS topics.
# create a workspace
WS=$(aws amp create-workspace \
  --alias obs-lab \
  --query workspaceId --output text)
URL=https://aps-workspaces.\
ap-northeast-1.amazonaws.com\
/workspaces/$WS

# Prometheus remote_write
remote_write:
  - url: <URL>/api/v1/remote_write
    sigv4:
      region: ap-northeast-1

# rules (same YAML as ch 4)
aws amp create-rule-groups-namespace \
  --workspace-id $WS --name shop \
  --data fileb://rules/shop.yaml

Fit

  • Long-term, HA Prometheus storage without running Thanos or Mimir (ch 5).
  • Billing by samples ingested, storage and query samples processed: high cardinality costs money here too (ch 5, ch 13).
  • Managed scrapers: created per EKS cluster (aws amp create-scraper), scrape config is base64 YAML, access through EKS access entries.
  • AMP also has a Prometheus-compatible query API for tools other than Grafana.
AMG

Amazon Managed Grafana (AMG)

Managed Grafana workspaces with AWS authentication and data sources pre-wired. The dashboards and skills from ch 10 transfer directly.

What AWS manages

  • Grafana servers, upgrades (major versions on a lag behind open-source Grafana), HA.
  • Users through IAM Identity Center or SAML; roles Admin / Editor / Viewer.
  • Data sources with SigV4: AMP, CloudWatch (QL, PPL, SQL, Metrics Insights), X-Ray, OpenSearch, Athena, Timestream, …
  • Service-managed IAM roles per data source, or your own role.
  • VPC connectivity for private data sources.

What you still own

  • Dashboards as code: the HTTP API, Terraform provider and provisioning still apply (ch 10).
  • Folder and team permissions.
  • Alert rules and contact points if you use Grafana alerting.
  • Licensing choices (Enterprise plugins are an add-on).
  • Version differences: check that the features you depend on exist in the AMG version.

Versus self-managed Grafana

  • Pick AMG when identity integration and zero ops matter more than the newest features.
  • Pick self-managed (kps Grafana, ch 5) for labs, latest Grafana 13 features (Git Sync), or full plugin freedom.
  • Grafana Cloud is a third option with its own pricing and stack.
Cross-account observability

One monitoring account for many source accounts (OAM)

CloudWatch cross-account observability links source accounts to a sink in a monitoring account. Metrics, logs, traces and Application Signals data become visible there without copying.

Source accounts (prod, staging)Linkresource types + label templateMetricsLog groupsTracesApp SignalsOAM linkMonitoring accountSinksink policy: allowed accounts / OUDashboards, alarms, Logs Insights,Transaction Search across accounts
# monitoring account
SINK=$(aws oam create-sink --name obs-sink \
        --query Arn --output text)
aws oam put-sink-policy --sink-identifier $SINK \
  --policy file://sink-policy.json  # allow source accounts / OU

# each source account
aws oam create-link --label-template '$AccountName' \
  --sink-identifier $SINK \
  --resource-types AWS::CloudWatch::Metric \
    AWS::Logs::LogGroup AWS::XRay::Trace

Set it up with AWS Organizations (a StackSet or the console wizard) so new accounts link automatically. Sharing is per Region: create a sink and links in every Region you operate in.

Choosing

CloudWatch, open source or both?

There is no single right answer; there is a clear default per signal and per team. Decide explicitly and write it down (ch 13).

NeedCloudWatch-nativeOpen source (self or AWS managed)
AWS service metrics (ALB, RDS, Lambda)Built in; no choice neededPull via CloudWatch data source or exporters (adds API cost)
Kubernetes and app metricsContainer Insights, EMF, OTel metrics + PromQL (2026)Prometheus + kps (ch 5), AMP for long term
LogsCloudWatch Logs, Logs Insights, IA classLoki (ch 7): cheaper at scale, label discipline required
TracesX-Ray + Transaction Search, Application SignalsTempo (ch 9) with TraceQL
DashboardsCloudWatch dashboardsGrafana or AMG, can read CloudWatch too
AlertingAlarms → SNS / EventBridgeAlertmanager (ch 4), AMP alert manager
Multi-cloud or on-premPossible with the agent; AWS-centricNatural fit
Ops effortLowestHighest self-managed; medium with AMP/AMG

Common hybrid: OpenTelemetry everywhere (ch 8), Prometheus/Loki/Tempo for engineering debugging, CloudWatch alarms for AWS resources, and Grafana reading both. OTel keeps the switch cheap if you change your mind.

Common mistakes

CloudWatch Logs and tracing mistakes

These show up on bills, in audits and in incidents where the trace stops at a service boundary.

Logs

  • Log groups left on "never expire".
  • Creating an IA group, then needing a metric filter or Live Tail (class cannot change).
  • Logs Insights over 30 days and 50 groups to find one request — use field indexes and a narrow window.
  • Secrets or emails in logs, "fixed" only by masking.
  • Metric filter with a high-cardinality dimension.

Traces

  • Starting new services on the X-Ray SDK in 2026.
  • Mixed X-Amzn-Trace-Id and traceparent without the xray propagator → broken traces.
  • Sending OTLP to the X-Ray endpoint before enabling Transaction Search → the endpoint rejects the spans.
  • Indexing 100% of spans "just in case".
  • Logging without trace_id, so logs and traces cannot be joined.

Managed services

  • AMP remote write from many clusters without dropping unused series (cost follows samples).
  • One AMP workspace per team with no shared conventions.
  • Cross-account links created in one Region only.
  • Assuming AMG runs the newest Grafana features.
Self-check

CloudWatch logs, traces and managed services: can you answer these?

Answer with the exact command, field or setting.

Logs

  • Which features do you lose with the Infrequent Access class, and can you switch later?
  • Write a metric filter that counts JSON events with status >= 500.
  • Name three subscription filter destinations and when to use each.
  • Who can see a masked value, and how?
  • Which Logs Insights QL query gives p95 latency per route?

Queries and traces

  • When would you use SQL instead of Logs Insights QL?
  • What does a field index change about cost?
  • What happens on 25 Feb 2026 and 25 Feb 2027 for X-Ray SDKs?
  • Convert 5759e988bd862e3fe1be46a994272793 to an X-Ray trace ID.
  • Which two steps enable Transaction Search?

Managed and cross-account

  • How does Prometheus authenticate to AMP?
  • Where do AMP alerts go?
  • What is an OAM sink and what is a link?
  • Which signals would you keep in CloudWatch in a hybrid setup, and why?
  • What must a Collector have to send OTLP to CloudWatch?

Practice: Lab 12 in Hands-on labs — Logs Insights in three languages, Live Tail and masking, an IA log group, and OpenTelemetry traces to Transaction Search (optional: AMP remote write from kind).

Chapter 13

Operating observability

Use the stack under pressure: a repeatable debugging loop, alerts people trust, a bill you can explain, and monitoring for the monitoring itself.

Chapter 13 · Part 13.1

Debugging workflow

From a page to a root cause in minutes: alert → RED dashboard → exemplar → trace → logs → change events.

The loop

One path through all three signals

Each step narrows the search space. Metrics tell you that and where, traces tell you which hop, logs tell you why.

Alertsymptom, with runbookRED dashboardwhich service, routeExemplarone slow requestTrace (Tempo)which span is slowLogs (Loki)same trace_id: the error textContext you check at every stepChange eventsdeploys, config, flagsDependenciesDB, payments, DNS, cloudResources (USE)CPU throttling, OOM, diskScopeone pod, zone, tenant?Write each hypothesis down, then pick the query that would prove it wrong. Stop when one survives.

The loop only works if the signals share keys: service.name / job, namespace, pod, and trace_id in logs (ch 6, ch 8, ch 9).

Method

Hypotheses, not hunches

Under pressure people jump to the last outage's cause. A short written loop keeps the investigation honest.

1 · Frame the symptom

  • What do users see? (errors, slowness, wrong data)
  • Since when? Compare with the same time yesterday or last week.
  • How big? Percentage of requests, which routes, which tenants.
  • Is it getting worse, stable, or recovering?

2 · Narrow the scope

  • Split by route, code, pod, node, zone, version.
  • If one pod: suspect the pod or node. If all pods: suspect a shared dependency or a change.
  • Compare a slow trace with a normal trace of the same route.
  • Look for what changed: deploy annotations, config, traffic shape.

3 · Prove, mitigate, record

  • One query per hypothesis; keep the links in the incident channel.
  • Mitigate first (roll back, scale, fail over), root-cause second.
  • Verify recovery on the same panel that showed the symptom.
  • Timeline and evidence go into the postmortem (SRE deck).
Worked incident

Payments latency: the page

Shop app (ch 0). At 14:05 the alert below fires. Nothing was deployed to orders.

# alert as received (Alertmanager → Slack)
[FIRING:1] ShopHighLatencyP95  severity=page
service: orders   namespace: shop
summary: p95 latency of orders is 0.93s (threshold 0.5s)
runbook: https://runbooks.example.com/shop/latency

# rule behind it (PrometheusRule, ch 4)
- alert: ShopHighLatencyP95
  expr: |
    histogram_quantile(0.95,
      sum by (le) (rate(
        http_request_duration_seconds_bucket{job="orders"}[5m]
      ))) > 0.5
  for: 5m
  labels: {severity: page, service: orders}

What we know at minute 0

  • Symptom: users wait almost 1 s for checkout (POST /orders).
  • It is a symptom alert on the user-facing service, not a cause alert — good.
  • No error alert fired: requests are slow, not failing (yet).
  • Open the runbook link; it points to the shop RED dashboard (ch 10).
  • Hypotheses to test: (a) orders is CPU-throttled, (b) a dependency is slow, (c) traffic spike.
Worked incident

Step 1 — RED dashboard: where is it slow?

Rate is flat, errors are flat, duration is up — and only on the route that calls payments.

# Rate: no traffic spike → hypothesis (c) rejected
sum by (route) (rate(http_requests_total{job="orders"}[5m]))

# Errors: unchanged
sum(rate(http_requests_total{job="orders",code=~"5.."}[5m]))
  / sum(rate(http_requests_total{job="orders"}[5m]))

# Duration by route: only POST /orders is slow
histogram_quantile(0.95, sum by (le, route) (
  rate(http_request_duration_seconds_bucket{job="orders"}[5m])))

# payments, as seen by its own server metrics
histogram_quantile(0.95, sum by (le) (
  rate(http_request_duration_seconds_bucket{job="payments"}[5m])))

# USE for orders: throttling? → hypothesis (a) rejected
sum by (pod) (rate(container_cpu_cfs_throttled_periods_total{
  namespace="shop", container="orders"}[5m]))
/ sum by (pod) (rate(container_cpu_cfs_periods_total{
  namespace="shop", container="orders"}[5m]))

Reading the panels

  • GET /orders/{id} p95 is normal: orders itself is healthy.
  • POST /orders p95 jumped from ~0.12 s to ~0.93 s at 14:01.
  • payments server-side p95 is ~0.85 s: the time is spent inside payments.
  • Throttling ratio for orders is near 0.
  • Surviving hypothesis: (b) payments is slow. Next: one concrete slow request.
Worked incident

Step 2 — exemplar to trace

An exemplar is a real request (with its trace_id) attached to a histogram bucket. Click it and you leave aggregates behind.

In Grafana

  • Duration panel → enable Exemplars on the Prometheus query.
  • Diamonds appear on the graph; hover one in the slow band.
  • Its labels include trace_id; the data source link opens the trace in Tempo (ch 9).
  • Needs: Prometheus exemplar storage, the client exposing exemplars (OpenMetrics), and a Tempo link on the data source.
# the trace as a waterfall (simplified)
orders    POST /orders           942 ms
├─ orders    SELECT orders (db)    6 ms
└─ orders    POST /charge        921 ms  client
   └─ payments POST /charge      918 ms  server
      ├─ payments validate card    2 ms
      └─ (no child spans)       ~910 ms  ← time
                                  inside the handler

What the trace adds: client ≈ server duration, so the network is not the problem; no child spans in the gap, so the time is inside the payments handler, not in a database or downstream call. Next: check k8s.pod.name and service.version on the spans — one pod or all?

Worked incident

Step 3 — confirm the scope with TraceQL

One trace is an anecdote. TraceQL tells you whether it is every request, one pod, or one version.

# all slow payments server spans in the last 30 min
{ resource.service.name = "payments" && kind = server
  && duration > 500ms }

# which pods? (TraceQL metrics, ch 9)
{ resource.service.name = "payments" && kind = server }
  | quantile_over_time(duration, .95) by (resource.k8s.pod.name)

# did errors start too?
{ resource.service.name = "payments" && status = error }
  | rate()

Result

  • Every payments pod is slow: not a bad node or a single pod.
  • Version unchanged: not a new release of payments.
  • Error rate still near zero.
  • Remaining question: what changed in payments at 14:01 without a new version? → logs and change events.
Worked incident

Step 4 — logs for the same trace, then the change

The trace_id in structured logs (ch 6) joins the trace to the log lines that explain it.

# logs of one slow request (Loki, LogQL)
{namespace="shop"} | json | trace_id="4bf92f3577b34da6a3ce929d0e0e4736"

# payments startup lines since 14:00 — config at boot
{namespace="shop", app="payments"} | json | msg=~"starting.*"

# what changed in Kubernetes?
kubectl -n shop rollout history deploy/payments
kubectl -n shop get deploy payments \
  -o jsonpath='{.spec.template.spec.containers[0].env}'

Root cause and fix

  • Startup log: starting payments extra_latency_ms=800 fail_rate=0 at 14:00.
  • Rollout history: revision 7 changed only the env var EXTRA_LATENCY_MS (a test knob left on).
  • Same image, new revision: why the "version" looked unchanged.
  • Mitigate: kubectl -n shop rollout undo deploy/payments.
  • Verify: POST /orders p95 back under 0.15 s on the same RED panel; alert resolves.
  • Follow-up: deploy annotations for config changes too, not only image changes.

Lab 13 replays this incident on your cluster, then adds errors (FAIL_RATE) to see how the picture changes.

Change events

Make "what changed?" a query, not a Slack search

Most incidents follow a change. Put changes on the same timeline as the symptoms.

Sources of change

  • Deploys and rollbacks (Argo CD sync, kubectl rollout).
  • Config and feature-flag changes.
  • Infrastructure: node pool upgrades, autoscaling events.
  • External: cloud provider incidents, dependency releases.

Ways to record them

  • Grafana annotations via the HTTP API (POST /api/annotations) from CI/CD.
  • A metric per deploy, e.g. kube_deployment_status_observed_generation from kube-state-metrics.
  • Kubernetes events shipped to Loki (an event exporter, or Alloy loki.source.kubernetes_events).
  • Argo CD notifications to a webhook that writes annotations.
# annotation from a pipeline step
curl -s -X POST \
  "$GRAFANA_URL/api/annotations" \
  -H "Authorization: Bearer $TOKEN" \
  -H 'Content-Type: application/json' \
  -d '{"tags":["deploy","payments"],
   "text":"payments rev 7"}'
# no "time" field → now
Chapter 13 · Part 13.2

Alert quality and on-call hygiene

Every page must be urgent, actionable and owned. Everything else is a ticket or a dashboard.

Alert quality

What makes an alert worth waking up for

An alert is a request for a human to act now. If no action is possible or needed, it should not page.

A good page

  • Urgent: users are affected now or will be soon.
  • Actionable: the responder can do something specific.
  • Symptom-based: error rate, latency, saturation of the user-facing path.
  • Owned: routed by a team/service label to one rotation.
  • Explained: summary, description with {{ $value }}, runbook_url.

A bad page

  • CPU > 80% on one node with no user impact.
  • Fires and resolves within minutes, several times a day (flapping).
  • Duplicates another alert for the same root cause.
  • Nobody knows what to do; the runbook is empty.
  • Goes to a shared channel nobody owns.

Destination by urgency

SituationGoes to
Users affected nowPage (severity: page)
Will hurt within days (disk 85%, cert expiry)Ticket (severity: ticket)
Interesting, not actionableDashboard only
Cause behind a firing symptomInhibited, shown as context
Alert quality

Measuring noise with the data you already have

Prometheus and Alertmanager record their own alerting activity. Review it weekly.

# which alerts spent the most time firing (last 7 days)
# (samples ≈ minutes firing × evaluations per minute)
sort_desc(
  count by (alertname) (
    count_over_time(ALERTS{alertstate="firing"}[7d])))

# currently firing, by severity
count by (severity) (ALERTS{alertstate="firing"})

# notifications sent and failed, per integration
sum by (integration) (increase(alertmanager_notifications_total[7d]))
sum by (integration) (
  increase(alertmanager_notifications_failed_total[7d]))

# alerts received by Alertmanager
sum(increase(alertmanager_alerts_received_total[7d]))

Weekly alert review

  • List every page of the week with its outcome: acted / no action needed / duplicate.
  • Target: every page led to an action. Fix or demote the rest.
  • Flapping → raise for:, use keep_firing_for:, or alert on a longer window.
  • Duplicates → group better (group_by) or add an inhibition rule (ch 4).
  • No runbook → write one before the next on-call shift.
  • Paging volume per shift is an SRE health metric (SRE deck).
Runbooks

A runbook template that is actually used

Linked from runbook_url, short enough to read at 3 a.m., with copy-paste queries.

# ShopHighLatencyP95 — runbook

## What it means
p95 latency of `orders` > 0.5 s for 5 min. Users wait at checkout.

## First 5 minutes
1. RED dashboard "Shop / Services", route panel.
2. Is `payments` slow too? (dependency panel)
3. Open an exemplar → trace in Tempo.
4. Logs for that trace_id in Loki.
5. Grafana annotations: any deploy/config change?

## Mitigations
- Recent change → `kubectl -n shop rollout undo deploy/<svc>`
- CPU throttling → raise limits / scale out

## Escalation
payments team: #payments-oncall

Rules for runbooks

  • One per alert; the alert name is the title.
  • Start with impact and the first queries, not with architecture.
  • Links must be deep: dashboard with variables set, Explore with the query filled.
  • Keep them in Git next to the rules (rules/, runbooks/) and review them in the same PR.
  • Update them in every postmortem that touched the alert.
Ownership

Route by ownership labels, not by alert name

Put team (or service) on every alert; the routing tree stays small as alerts multiply.

route:
  receiver: default-ticket
  group_by: [alertname, namespace, service]
  routes:
    - matchers: ['severity="page"', 'team="payments"']
      receiver: payments-pager
    - matchers: ['severity="page"']
      receiver: platform-pager
    - matchers: ['severity="ticket"']
      receiver: ticket-queue
inhibit_rules:
  # a down target makes its latency alerts noise
  - source_matchers: ['alertname="TargetDown"']
    target_matchers: ['severity="page"']
    equal: [namespace, service]

Practices

  • Set team in the PrometheusRule (static label) or derive it from namespace ownership.
  • Test the tree before merging: amtool config routes test --config.file=alertmanager.yml severity=page team=payments.
  • Keep a catch-all receiver so an unlabelled alert is never dropped silently.
  • Use silences with a comment and an expiry during maintenance; never delete rules to "silence".
  • Hand-offs: each rotation reviews its own noise (previous slide).
Chapter 13 · Part 13.3

Cost and cardinality governance

Every label, log line and span costs money. Decide what to keep on purpose.

Cost drivers

What drives the bill for each signal

Cost scales with different units per signal. Know the unit before you try to cut.

SignalMain cost unitTypical blow-upPrimary control
Metrics (Prometheus, AMP, Mimir)Active series × retention; samples ingestedUnbounded label (user_id, raw path, pod churn)Label hygiene, relabel drop, recording rules
Logs (Loki, CloudWatch Logs)GB ingested, GB stored, GB scanned by queriesDebug logs in prod, chatty health checks, stack traces per requestLevels, sampling, drop at the agent, retention tiers
Traces (Tempo, X-Ray)Spans ingested and stored100% sampling of high-QPS routesHead/tail sampling (ch 9), drop health-check spans
CloudWatch metricsCustom metrics per month (each name + dimension set), API requestsHigh-cardinality dimensions in EMFFewer dimensions, EMF only for what you alert on
CloudWatch Logs InsightsGB scanned per queryWide time ranges over all log groupsNarrow log groups and time, facets, saved queries

Prices differ by Region and change; read the current pricing pages before estimating. The shape of the curve matters more than the number.

Cardinality

Finding the series that cost you

Find the top metrics first, then the label that explodes inside them.

# total active series in the head block
prometheus_tsdb_head_series

# top 10 metric names by series count (expensive: run rarely)
topk(10, count by (__name__) ({__name__=~".+"}))

# which label explodes inside one metric?
count(count by (route) (http_requests_total{job="orders"}))
count(count by (pod)   (http_requests_total{job="orders"}))

# series per scrape target
topk(10, scrape_series_added)
topk(10, scrape_samples_post_metric_relabeling)
# the same from the TSDB status API
kubectl -n monitoring port-forward svc/prometheus-operated 9090 &
curl -s localhost:9090/api/v1/status/tsdb \
  | jq '.data.seriesCountByMetricName[:10]'
curl -s localhost:9090/api/v1/status/tsdb \
  | jq '.data.labelValueCountByLabelName[:10]'

# offline, on a copy of the data directory
promtool tsdb analyze /prometheus

A route label with thousands of values usually means the middleware records the raw path (/orders/8812) instead of the route template (/orders/{id}). Fix it in code (ch 2).

Controls

Where to cut, from source to storage

Cutting early is cheapest; cutting late is safest for teams you do not control. Use both.

In the application

  • Route templates, not raw paths; bounded enums for status and method.
  • No IDs in metric labels: put them in logs and trace attributes.
  • Histogram buckets chosen on purpose, or native histograms (ch 2).
  • Log at info by default; debug behind a flag.

In the pipeline

  • Prometheus metric_relabel_configs (drop, labeldrop) or metricRelabelings in a ServiceMonitor.
  • Prometheus sample_limit / label_limit per scrape job to stop surprises.
  • Collector filter and transform processors: drop health checks, delete attributes.
  • Alloy loki.process stage.drop for noisy lines; tail sampling for traces.

In storage

  • Retention per signal and per tier (raw short, aggregated long).
  • Recording rules so dashboards query few, pre-aggregated series.
  • Loki limits: max_streams_per_user, ingestion_rate_mb (ch 7).
  • CloudWatch Logs retention on every log group; Infrequent Access class for logs you rarely query.
Governance

Budgets and guardrails per team

Cost control is a social process backed by limits. Make usage visible before you enforce it.

Make it visible

  • Series per namespace or team: count by (namespace) ({__name__=~".+"}) (run rarely, or record it).
  • Log bytes per app: LogQL sum by (app) (bytes_over_time({namespace=~".+"}[1h])).
  • Spans per service from span metrics or Collector self-metrics.
  • A monthly report per team with trend and top offenders.

Guardrails

  • Per-tenant limits in Mimir/Loki/Tempo, or per-job sample_limit in Prometheus.
  • CI check on new metrics: name, unit suffix, label set reviewed (promtool check metrics).
  • Default log level info; debug needs an expiry.
  • Sampling policy owned by the platform team, overridable per service.

Trade-offs to state explicitly

  • Dropping data loses questions you have not asked yet.
  • Sampling traces breaks "find this exact request" unless errors and slow traces are always kept.
  • Shorter retention breaks month-over-month comparisons: keep aggregates longer.
Chapter 13 · Part 13.4

Meta-monitoring and observability as code

Who watches the watchers, and how to keep the whole setup reproducible.

Meta-monitoring

Monitor the monitoring pipeline

If Prometheus stops evaluating rules or Alertmanager cannot send, every other alert is silently dead.

ComponentSignal to watchExample
Prometheus scrapingTargets down, scrape errors, limits hitup == 0, prometheus_target_scrapes_exceeded_sample_limit_total
Prometheus rulesRule evaluation failures, slow groupsincrease(prometheus_rule_evaluation_failures_total[5m]) > 0
Prometheus → AlertmanagerNotifications dropped or erroringprometheus_notifications_dropped_total, prometheus_notifications_errors_total
AlertmanagerFailed notifications, cluster sizealertmanager_notifications_failed_total, alertmanager_cluster_members
Config reloadsBad config silently not appliedprometheus_config_last_reload_successful == 0, alertmanager_config_last_reload_successful == 0
LokiRejected or discarded linesloki_discarded_samples_total by reason
OTel CollectorRefused or failed telemetry, full queuesotelcol_receiver_refused_spans_total, otelcol_exporter_send_failed_spans_total, otelcol_exporter_queue_size

kube-prometheus-stack already ships many of these rules (e.g. PrometheusRuleFailures, AlertmanagerFailedToSendAlerts, TargetDown). Check they route somewhere real.

Dead man's switch

The Watchdog alert: proving the pipeline is alive

An alert that always fires. If the external receiver stops hearing it, the whole chain is broken.

PrometheusWatchdog: vector(1)Alertmanagerroute alertname=WatchdogExternal heartbeat serviceoutside the clusterPages on-callif no heartbeat in N minevery repeat_intervalsilence = alarmCatches: Prometheus down, rules not evaluated, Alertmanager down or misrouted, network egress broken, receiver credentials expired.kube-prometheus-stack defines `Watchdog` in its general rules and routes it to the "null" receiver by default — re-route it.The heartbeat service must run outside the cluster and the cloud account it watches.
routes:
  - matchers: ['alertname="Watchdog"']
    receiver: heartbeat
    group_wait: 0s
    group_interval: 1m
    repeat_interval: 1m     # heartbeat every minute
receivers:
  - name: heartbeat
    webhook_configs:
      - url: https://<heartbeat-service>/ping/<id>
Meta-monitoring

Self-alerts for the shop stack

A small PrometheusRule that covers the gaps the defaults do not know about: your own jobs and your own pipeline.

apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: shop-meta
  namespace: monitoring
  labels: {release: kps}          # picked up by kps
spec:
  groups:
    - name: shop-meta
      rules:
        - alert: ShopTargetMissing
          expr: absent(up{job="orders"} == 1)
          for: 5m
          labels: {severity: page, team: platform}
          annotations:
            summary: no healthy orders target is scraped
        - alert: CollectorExportFailing
          expr: |
            sum(rate(otelcol_exporter_send_failed_spans_total[5m])) > 0
          for: 10m
          labels: {severity: ticket, team: platform}

Why each one

  • absent(...) fires when the series is missing entirely — up == 0 cannot, because a target removed by service discovery produces no up series at all.
  • Collector export failures mean traces are lost while every app looks healthy.
  • Ticket, not page, for pipeline degradation that does not hurt users yet.
  • Self-metric names depend on the Collector version: check its :8888/metrics endpoint before writing the rule.
As code

Observability as code: one repo, reviewed like software

Everything that defines what you see and what pages you lives in Git and is tested in CI.

obs-lab/
├── apps/                    # instrumented services (ch 2, 6, 8)
├── deploy/
│   ├── kps-values.yaml      # Prometheus, AM, Grafana
│   ├── loki-values.yaml     # Loki + Alloy (ch 7)
│   ├── tempo-values.yaml    # Tempo (ch 9)
│   └── otel-gateway.yaml    # Collector config (ch 9)
├── rules/                   # PrometheusRule + promtool tests
│   ├── shop-red.yaml
│   └── shop-red.test.yaml
├── alertmanager/            # routing tree + amtool tests
├── dashboards/              # JSON / schema v2 (ch 10)
└── runbooks/                # one .md per alert

CI checks

  • promtool check rules and promtool test rules on every PR (ch 4).
  • amtool check-config and amtool config routes test for routing.
  • Collector: otelcol validate --config=… (or the distribution binary's validate).
  • Alloy: alloy fmt to check syntax.
  • Dashboards: lint JSON, keep UIDs stable; Grafana 13 Git Sync or provisioning (ch 10).
  • Deploy with GitOps (GitOps deck); no hand edits in the UI that are not committed.
Maturity

A simple maturity path

Use it to decide the next investment, not as a score.

StageYou haveNext step
1 · ReactiveHost metrics, logs on nodes, users report outagesCentral metrics and logs; up and error-rate alerts
2 · Service metricsRED per service, USE per node, dashboards per teamRecording rules, runbooks, routing by team
3 · CorrelatedStructured logs with trace_id, traces with sampling, exemplarsOne-click alert → trace → logs; change annotations
4 · SLO-drivenSLIs, error budgets, burn-rate paging (SRE deck)Alert review, cost governance, meta-monitoring
5 · PlatformSelf-service instrumentation, golden-path defaults, per-team budgetsEverything as code; continuous verification (game days)

Most teams get the biggest win from stage 2 → 3: shared keys across signals turn hours of guessing into minutes.

Common mistakes

Operating mistakes that cost the most

Each one has caused long outages or large bills somewhere.

Debugging

  • Starting from logs with a full-text search instead of narrowing with metrics first.
  • Trusting averages: p50 looks fine while p99 is on fire.
  • Stopping at the first plausible cause without checking it explains the whole symptom.
  • No change events on dashboards: "nothing changed" when a config did.

Alerting

  • Paging on causes (CPU, memory) instead of symptoms.
  • No runbook_url; alerts without an owner.
  • Watchdog left routed to "null": nobody notices Alertmanager is down.
  • Muting a noisy alert forever instead of fixing or deleting it.

Cost and pipeline

  • IDs and raw paths in metric labels.
  • Debug logging left on in production.
  • 100% trace sampling on high-QPS endpoints.
  • CloudWatch log groups with no retention.
  • No alerts on Collector or Loki rejections: data is lost quietly.
Self-check

Operating observability: can you answer these?

Answer out loud with the exact query, label or setting.

Debugging loop

  • Which signal answers "that", "where", "which hop" and "why"?
  • How do you tell "payments is slow" from "the network to payments is slow" in a trace?
  • What three things must be true for exemplars to appear in Grafana?
  • How did the worked incident change behavior without a new image?

Alerts

  • What makes an alert a page rather than a ticket?
  • Which query shows the alerts that spent the most time firing last week?
  • Why route by team instead of by alertname?
  • What does the Watchdog alert protect against, and where must its receiver run?

Cost and meta-monitoring

  • How do you find which label makes a metric explode?
  • Name one control each in the app, the pipeline and storage.
  • Why absent(up{job="orders"} == 1) rather than up == 0?
  • Which Collector and Loki metrics reveal silently dropped data?

Practice: Lab 13 in Hands-on labs — a game day on the obs cluster: inject latency and errors into payments, follow alert → dashboard → exemplar → trace → logs, wire up the Watchdog, and clean up a cardinality problem.

Capstone

The shop app, fully observable

Everything in the deck on one cluster, plus a CloudWatch mirror. Build it after Lab 12; Lab 13 is its game day.

kind cluster obsloadgencurl loop, ~5 req/sordersmetrics · logs · tracespaymentsFAIL_RATE, EXTRA_LATENCY_MSPrometheus (kps)rules, exemplarsAlloy → LokiJSON logs, trace_idotel-gatewaytail samplingTempoTraceQLAlertmanagerroutes by team, WatchdogGrafana 13RED · USE dashboards, exemplar → trace → logs linksAWS ap-northeast-1Lambda obs-lab-ordersEMF → namespace ObsLabCloudWatch Logs/obs-lab/orders · Logs InsightsAlarm → SNS obs-lab-alertsp95 latency, error count

Same ideas, two stacks: RED metrics, structured logs and alarms on symptoms work the same in Prometheus and in CloudWatch.

Capstone

Build order and acceptance checklist

Done means every box is ticked and proven with a query or a command, not a screenshot.

Build order

  • 1 · kind cluster obs, shop app, loadgen (Lab 0).
  • 2 · client_golang metrics and ServiceMonitors with release: kps (Labs 2, 5).
  • 3 · Recording rules, alerts with runbooks, promtool tests (Labs 3, 4).
  • 4 · slog JSON logs with trace_id; Alloy → Loki (Labs 6, 7).
  • 5 · OpenTelemetry SDK → otel-gateway → Tempo, with tail sampling (Labs 8, 9).
  • 6 · RED/USE dashboards as code, exemplar and log links (Lab 10).
  • 7 · Lambda obs-lab-orders with EMF, Logs Insights queries, alarm to SNS (Labs 11, 12).
  • 8 · Watchdog to a heartbeat, meta-alerts, cardinality review (Lab 13).

Acceptance

  • promtool test rules passes in CI; every alert has runbook_url and team.
  • Injecting EXTRA_LATENCY_MS=800 pages within ~7 min; rollout undo resolves it.
  • From the latency panel: exemplar → Tempo trace → Loki lines in three clicks.
  • No metric label has more than ~100 values (route uses templates).
  • Loki labels are only namespace, app, container (and similar); trace_id is not a label.
  • amtool config routes test alertname=Watchdog → the heartbeat receiver.

Stretch goals

  • SLO recording rules and multi-window burn-rate alerts (SRE deck).
  • Remote write to Amazon Managed Service for Prometheus with SigV4.
  • Span metrics and service graph connectors in the Collector.
  • Grafana Git Sync for the dashboards/ folder.
Cheat sheet

PromQL: everyday queries

Counters: rate first, then aggregate. Histograms: sum by (le) of the bucket rate, then histogram_quantile.

# request rate per route
sum by (route) (rate(http_requests_total{job="orders"}[5m]))

# error ratio (5xx / all)
sum(rate(http_requests_total{job="orders",code=~"5.."}[5m]))
  / sum(rate(http_requests_total{job="orders"}[5m]))

# p95 latency per route (classic histogram)
histogram_quantile(0.95, sum by (le, route) (
  rate(http_request_duration_seconds_bucket{job="orders"}[5m])))
# p95 with a native histogram (no _bucket, no le)
histogram_quantile(0.95, sum by (route) (
  rate(http_request_duration_seconds{job="orders"}[5m])))

# average latency
sum(rate(http_request_duration_seconds_sum[5m]))
  / sum(rate(http_request_duration_seconds_count[5m]))

# add the node to per-pod CPU (many-to-one join)
sum by (pod) (rate(container_cpu_usage_seconds_total{
    namespace="shop", container!=""}[5m]))
  * on (pod) group_left (node)
    max by (pod, node) (kube_pod_info{namespace="shop"})
Cheat sheet

PromQL: syntax at a glance

Selectors pick series, functions transform them, operators combine them.

ConstructMeaning
{job="orders", code=~"5.."}Selector; =, !=, =~, !~ (regexes are fully anchored)
[5m], offset 1d, @ end()Range; shift back in time; pin the evaluation time
rate / irate / increasePer-second average over the range / from the last two samples / total increase over the range
sum by (l) / sum without (l)Keep only / drop the listed labels
topk(5, …), quantile(0.9, …)Top series; quantile across series (not over time)
on (l) group_left (x)Match on l; many left to one right; copy label x from the right
and, or, unlessSet operations on label sets
> bool 0.5Return 0/1 instead of filtering
max_over_time(x[1h])Range function on a gauge
max_over_time(rate(x[5m])[1h:1m])Subquery: 1 h of 1-min steps of an inner expression
absent(up{job="orders"})1 when no such series exists (for "missing" alerts)
predict_linear(x[6h], 4*3600)Linear forecast 4 h ahead (gauges)
$__rate_intervalGrafana variable: a safe rate window for the scrape interval
Cheat sheet

Recording rules, alerting rules and Alertmanager

Recording rule names follow level:metric:operations. Every alert needs severity, an owner and a runbook_url.

groups:
  - name: shop-red
    interval: 30s
    rules:
      - record: job_route:http_requests:rate5m
        expr: |
          sum by (job, route) (
            rate(http_requests_total[5m]))
      - alert: ShopHighErrorRate
        expr: |
          sum(rate(http_requests_total{
                job="orders",code=~"5.."}[5m]))
          / sum(rate(http_requests_total{
                job="orders"}[5m])) > 0.05
        for: 5m
        keep_firing_for: 5m
        labels: {severity: page, team: shop}
        annotations:
          summary: '5xx {{ $value | humanizePercentage }}'
          runbook_url: https://runbooks.example.com/shop/errors
route:
  receiver: default
  group_by: [alertname, namespace]
  group_wait: 30s        # defaults: 30s / 5m / 4h
  group_interval: 5m
  repeat_interval: 4h
  routes:
    - matchers: ['severity="page"']
      receiver: pager
      continue: false
inhibit_rules:
  - source_matchers: ['severity="page"']
    target_matchers: ['severity="ticket"']
    equal: [alertname, namespace]
receivers:
  - name: default
  - name: pager
    webhook_configs:
      - url: https://pager.example.com/hook
Cheat sheet

promtool and amtool

Validate in CI; test routing before you merge; silence with a comment and an expiry.

CommandWhat it does
promtool check config prometheus.ymlValidate the server config (and the rule files it references)
promtool check rules rules.yamlValidate rule syntax
promtool test rules tests.yamlUnit-test rules with input_series and expected alerts or values
promtool query instant http://localhost:9090 'up'Run a query from the shell
promtool tsdb analyze <data-dir>Cardinality report from a TSDB directory
amtool check-config alertmanager.ymlValidate config and templates
amtool config routes test --config.file=am.yml severity=pageWhich receiver gets an alert with these labels?
amtool config routes show --config.file=am.ymlPrint the routing tree
amtool alert query alertname=X --alertmanager.url=http://localhost:9093List firing alerts
amtool silence add alertname=X --duration=2h --comment="maintenance"Create a silence
amtool silence query / amtool silence expire <id>List / remove silences
Cheat sheet

LogQL: queries

Select streams with labels (cheap), filter lines (cheap), then parse (costly). Metric queries wrap a log query in a range function.

# log queries
{namespace="shop", app="orders"} |= "error"
{namespace="shop"} != "healthz" |~ "timeout|refused"
{app="orders"} | json | status >= 500 and duration_ms > 300
{app="orders"} | json | trace_id="4bf92f3577b34da6a3ce929d0e0e4736"
{app="orders"} | logfmt | level="error"
{app="orders"} | json | line_format "{{.route}} {{.status}} {{.msg}}"
{app="orders"} | json | __error__=""        # drop parse failures
# metric queries
sum by (app) (rate({namespace="shop"} |= "error" [5m]))
sum by (app) (count_over_time({namespace="shop"}[1m]))
sum by (app) (bytes_over_time({namespace="shop"}[1h]))
topk(5, sum by (route) (count_over_time(
  {app="orders"} | json | status >= 500 [15m])))
quantile_over_time(0.95,
  {app="orders"} | json | unwrap duration_ms [5m]
) by (route)

Labels in these examples follow the lab's Alloy setup (ch 7): namespace, app, container. Fields such as trace_id and status come from parsing the JSON line, never from labels.

Cheat sheet

LogQL: syntax at a glance

A log query is a pipeline read left to right; every stage after the selector runs on each line.

PartNotes
{label="v"}Stream selector, required; =, !=, =~, !~
|= != |~ !~Line filters: contains / does not contain / regex / not regex
| json | logfmtExtract fields as labels for this query only
| pattern "<a> <_> <b>" / | regexp "…"Extract fields from unstructured lines
| status >= 500Label filter after parsing (numbers, durations, bytes, strings)
| line_format / | label_formatRewrite the line with a template / rename or set labels
| drop a, b / | keep a, bRemove / keep only these labels
| unwrap duration_msUse a numeric field as the sample value
rate, count_over_time, bytes_over_timeLines per second / lines in range / bytes in range
sum_over_time, avg_over_time, quantile_over_timeRange functions over unwrapped values
logcli query '{app="orders"}' --since=1hShell client; set LOKI_ADDR=http://localhost:3100
Cheat sheet

TraceQL: queries

A query selects spansets with { … }, then pipes them through aggregates, select or metrics functions.

# by service, status, duration
{ resource.service.name = "payments" && status = error }
{ resource.service.name = "orders" && duration > 500ms }
{ span.http.route = "/orders"
  && span.http.response.status_code >= 500 }

# unscoped attribute (span or resource)
{ .deployment.environment.name = "lab" }

# structural: payments spans below orders spans
{ resource.service.name = "orders" }
  >> { resource.service.name = "payments" }
# aggregates per trace, then filter
{ resource.service.name = "orders" } | count() > 10
{ } | avg(duration) > 1s

# return extra attributes with the results
{ status = error }
  | select(span.http.route, resource.k8s.pod.name)

# TraceQL metrics (time series from spans)
{ resource.service.name = "payments" } | rate()
{ kind = server }
  | quantile_over_time(duration, .95) by (resource.service.name)
{ status = error } | count_over_time() by (span.http.route)
Cheat sheet

TraceQL: syntax at a glance

Span intrinsics are bare (duration) or span:-scoped; trace intrinsics use trace:; attributes are resource., span. or unscoped ..

ElementMeaning
resource.x / span.x / .xResource attribute / span attribute / either
duration, name, status, kindSpan intrinsics
trace:duration, trace:rootName, trace:rootServiceTrace-level intrinsics
status = error | ok | unsetSpan status values
kind = server | client | producer | consumer | internalSpan kinds
&&, || inside { }Conditions on one span
>> / > / ~Descendant / child / sibling spansets
| count(), | avg(duration)Aggregate per trace, then filter with a comparison
| select(…)Add attributes to the returned spans
| rate(), | count_over_time(), | quantile_over_time()TraceQL metrics (GA in Tempo 3)
Cheat sheet

OpenTelemetry Collector config

Define components once, wire them into pipelines. In processors, memory_limiter goes first and batch late.

receivers:
  otlp:
    protocols:
      grpc: {endpoint: 0.0.0.0:4317}
      http: {endpoint: 0.0.0.0:4318}
processors:
  memory_limiter: {check_interval: 1s,
    limit_percentage: 80, spike_limit_percentage: 25}
  k8sattributes: {}
  filter/health:
    error_mode: ignore
    traces:
      span: ['attributes["url.path"] == "/healthz"']
  tail_sampling:
    decision_wait: 10s
    policies:
      - {name: errors, type: status_code,
         status_code: {status_codes: [ERROR]}}
      - {name: slow, type: latency,
         latency: {threshold_ms: 500}}
      - {name: rest, type: probabilistic,
         probabilistic: {sampling_percentage: 10}}
  batch: {}
exporters:
  otlp/tempo:
    endpoint: tempo.tracing.svc:4317
    tls: {insecure: true}
  otlphttp/loki:
    endpoint: http://loki.logging.svc:3100/otlp
  debug: {verbosity: basic}
connectors:
  spanmetrics: {}
service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [memory_limiter, k8sattributes,
                   filter/health, tail_sampling, batch]
      exporters: [otlp/tempo, spanmetrics]
    metrics:
      receivers: [spanmetrics]
      exporters: [debug]
    logs:
      receivers: [otlp]
      processors: [memory_limiter, k8sattributes, batch]
      exporters: [otlphttp/loki]
# check: otelcol-contrib validate --config=config.yaml

k8sattributes needs RBAC to read pods; tail sampling needs every span of a trace on the same Collector (ch 9). Loki 3 accepts OTLP logs at /otlp. Send span metrics to Prometheus with an exporter of your choice (ch 9).

Cheat sheet

CloudWatch Logs Insights and CLI

Logs Insights bills per GB scanned: pick few log groups and a short time range first.

# Logs Insights QL (JSON fields are discovered)
fields @timestamp, level, route, status, duration_ms
| filter level = "ERROR" and route = "/orders"
| sort @timestamp desc
| limit 50

filter status >= 500
| stats count(*) as errors by bin(5m)

stats pct(duration_ms, 95) as p95 by route

parse @message "status=* duration=*ms" as st, dur
| stats count(*) by st

pattern @message          # cluster similar lines

filter trace_id = "4bf92f3577b34da6a3ce929d0e0e4736"

filter @type = "REPORT"   # Lambda reports
| stats pct(@duration, 99) as p99ms
LG=/obs-lab/orders
aws logs tail $LG --since 15m --follow
aws logs tail $LG --since 1h \
  --filter-pattern '{ $.level = "ERROR" }'

QID=$(aws logs start-query --log-group-name $LG \
  --start-time $(($(date +%s) - 3600)) \
  --end-time $(date +%s) \
  --query-string 'stats count(*) by bin(5m)' \
  --query queryId --output text)
aws logs get-query-results --query-id $QID

aws logs put-retention-policy --log-group-name $LG \
  --retention-in-days 14
aws logs put-metric-filter --log-group-name $LG \
  --filter-name orders-5xx \
  --filter-pattern '{ $.status >= 500 }' \
  --metric-transformations \
  metricName=Orders5xx,metricNamespace=ObsLab,metricValue=1

aws cloudwatch describe-alarms --state-value ALARM
aws cloudwatch set-alarm-state --alarm-name <name> \
  --state-value ALARM --state-reason test  # test actions
Lab setup

Before you start

Labs 0–10 and 13 run on a local kind cluster and are free. Labs 11–12 use a small AWS sandbox in ap-northeast-1.

Machine, accounts and conventions

  • Docker Desktop, OrbStack or colima with 6 GB RAM and 4 CPUs; kind cluster obs (1 control plane + 2 workers).
  • AWS sandbox, AWS CLI v2, export AWS_REGION=ap-northeast-1, a budget alarm before Lab 11.
  • Repo obs-lab/ (local Git is enough); Go module obs-lab; images shop/orders:lab, shop/payments:lab loaded with kind load docker-image.
  • Helm release kps in monitoring; every ServiceMonitor, PodMonitor and PrometheusRule carries label release: kps.
  • Port-forwards: Prometheus 9090, Alertmanager 9093, Grafana 3000, Loki 3100, Tempo 3200, orders 8080.
  • Each lab = Build, Verify, Break it, Clean up. Later labs extend earlier ones: keep the cluster.
brew install kind kubectl helm go jq hey \
  prometheus alertmanager logcli awscli

kind version
kubectl version --client
helm version            # v3.x or v4.x
go version              # go1.25+
promtool --version      # ships with prometheus
amtool --version        # ships with alertmanager
logcli --version
aws --version           # aws-cli/2.x

helm repo add prometheus-community \
  https://prometheus-community.github.io/helm-charts
helm repo add grafana \
  https://grafana.github.io/helm-charts
helm repo add grafana-community \
  https://grafana-community.github.io/helm-charts
helm repo add open-telemetry \
  https://open-telemetry.github.io/opentelemetry-helm-charts
helm repo update
Lab 0 · Build (1/7)

Setup: the payments service

Goal: a dependency whose error rate and latency you control with two environment variables. No telemetry yet: later labs add it.

mkdir -p obs-lab/apps/{orders,payments} obs-lab/deploy
cd obs-lab && go mod init obs-lab
cat > apps/payments/main.go <<'EOF'
package main

import (
	"encoding/json"
	"log/slog"
	"math/rand/v2"
	"net/http"
	"os"
	"strconv"
	"time"
)

func envFloat(k string) float64 {
	v, _ := strconv.ParseFloat(os.Getenv(k), 64)
	return v
}

func main() {
	h := slog.NewJSONHandler(os.Stdout, nil)
	log := slog.New(h)
	failRate := envFloat("FAIL_RATE")     // 0..1
	extra := envFloat("EXTRA_LATENCY_MS") // added delay
	mux := http.NewServeMux()
	mux.HandleFunc("GET /healthz",
		func(http.ResponseWriter, *http.Request) {})
	charge := func(w http.ResponseWriter, r *http.Request) {
		var req struct {
			OrderID string  `json:"order_id"`
			Amount  float64 `json:"amount"`
		}
		err := json.NewDecoder(r.Body).Decode(&req)
		if err != nil {
			http.Error(w, "bad request", 400)
			return
		}
		// base latency 20-80 ms plus the injected delay
		ms := 20 + rand.Float64()*60 + extra
		time.Sleep(time.Duration(ms * float64(time.Millisecond)))
		if rand.Float64() < failRate {
			log.Error("charge failed", "order_id", req.OrderID)
			http.Error(w, "card declined", 502)
			return
		}
		log.Info("charged", "order_id", req.OrderID,
			"amount", req.Amount)
	}
	mux.HandleFunc("POST /charge", charge)
	log.Info("payments listening", "addr", ":8080")
	err := http.ListenAndServe(":8080", mux)
	log.Error("server stopped", "err", err)
	os.Exit(1)
}
EOF
Lab 0 · Build (2/7)

Setup: the orders service (1/2)

Goal: an API that keeps orders in memory and calls payments synchronously, so payment failures and latency surface on POST /orders.

cat > apps/orders/main.go <<'EOF'
package main

import (
	"bytes"
	"encoding/json"
	"fmt"
	"log/slog"
	"net/http"
	"os"
	"sync"
	"sync/atomic"
	"time"
)

type Order struct {
	ID     string  `json:"id"`
	Item   string  `json:"item"`
	Amount float64 `json:"amount"`
}

var (
	log    = slog.New(slog.NewJSONHandler(os.Stdout, nil))
	client = &http.Client{Timeout: 2 * time.Second}
	payURL = "http://payments:8080/charge"
	seq    atomic.Int64
	mu     sync.RWMutex
	orders = map[string]Order{}
)
func create(w http.ResponseWriter, r *http.Request) {
	var o Order
	if err := json.NewDecoder(r.Body).Decode(&o); err != nil {
		http.Error(w, "bad request", 400)
		return
	}
	o.ID = fmt.Sprintf("o-%d", seq.Add(1))
	body, _ := json.Marshal(map[string]any{
		"order_id": o.ID, "amount": o.Amount})
	resp, err := client.Post(payURL, "application/json",
		bytes.NewReader(body))
	if err == nil {
		resp.Body.Close()
		if resp.StatusCode != http.StatusOK {
			err = fmt.Errorf("payments: %s", resp.Status)
		}
	}
	if err != nil {
		log.Error("payment failed", "order_id", o.ID,
			"err", err)
		http.Error(w, "payment failed", 502)
		return
	}
	mu.Lock()
	orders[o.ID] = o
	mu.Unlock()
	w.WriteHeader(http.StatusCreated)
	json.NewEncoder(w).Encode(o)
}
Lab 0 · Build (3/7)

Setup: the orders service (2/2) and a first run

Goal: finish orders, then run both services on your laptop before any container or cluster is involved.

func get(w http.ResponseWriter, r *http.Request) {
	mu.RLock()
	o, ok := orders[r.PathValue("id")]
	mu.RUnlock()
	if !ok {
		http.NotFound(w, r)
		return
	}
	json.NewEncoder(w).Encode(o)
}

func main() {
	if v := os.Getenv("PAYMENTS_URL"); v != "" {
		payURL = v
	}
	mux := http.NewServeMux()
	mux.HandleFunc("POST /orders", create)
	mux.HandleFunc("GET /orders/{id}", get)
	mux.HandleFunc("GET /healthz",
		func(http.ResponseWriter, *http.Request) {})
	log.Info("orders listening", "addr", ":8080")
	err := http.ListenAndServe(":8080", mux)
	log.Error("server stopped", "err", err)
	os.Exit(1)
}
EOF
gofmt -l . ; go vet ./...   # no output = good

# both listen on :8080: run payments in a container
docker run --rm -d --name pay -p 9001:8080 \
  -v "$PWD":/src -w /src golang:1.27 \
  go run ./apps/payments

# go run would leave the server running after kill
go build -o /tmp/orders ./apps/orders
PAYMENTS_URL=http://localhost:9001/charge \
  /tmp/orders &

curl -s -X POST localhost:8080/orders \
  -d '{"item":"pen","amount":3}'
# {"id":"o-1","item":"pen","amount":3}
curl -s localhost:8080/orders/o-1
# {"id":"o-1","item":"pen","amount":3}

# stop both
kill %1; docker rm -f pay

Method patterns ("POST /orders") and r.PathValue need Go 1.22+; the images build with golang:1.27.

Lab 0 · Build (4/7)

Setup: images and the kind cluster

Goal: one Dockerfile for both services, a three-node cluster, and the images loaded into it (no registry needed).

cat > Dockerfile <<'EOF'
# syntax=docker/dockerfile:1
FROM golang:1.27 AS build
ARG APP
WORKDIR /src
COPY go.* ./
RUN go mod download
COPY apps ./apps
RUN CGO_ENABLED=0 go build -trimpath \
    -o /out/app ./apps/${APP}

FROM gcr.io/distroless/static-debian12:nonroot
COPY --from=build /out/app /app
EXPOSE 8080
ENTRYPOINT ["/app"]
EOF

cat > deploy/kind.yaml <<'EOF'
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
name: obs
nodes:
  - role: control-plane
  - role: worker
  - role: worker
EOF
kind create cluster --config deploy/kind.yaml
kubectl get nodes
# obs-control-plane   Ready   control-plane
# obs-worker          Ready   <none>
# obs-worker2         Ready   <none>

for a in orders payments; do
  docker build --build-arg APP=$a -t shop/$a:lab .
  kind load docker-image shop/$a:lab --name obs
done

# after any code change later in the deck:
#   docker build --build-arg APP=orders \
#     -t shop/orders:lab .
#   kind load docker-image shop/orders:lab --name obs
#   kubectl -n shop rollout restart deploy/orders

COPY go.* ./ copies go.mod now and go.sum once Lab 2 adds dependencies; go mod download then caches them in their own layer.

Lab 0 · Build (5/7)

Setup: kube-prometheus-stack

Goal: Prometheus, Alertmanager, Grafana, kube-state-metrics and node-exporter in monitoring. Control-plane scrapes that kind cannot serve are switched off, so the lab starts without false alerts.

cat > deploy/kps-values.yaml <<'EOF'
# kind binds these components to 127.0.0.1:
# scraping them from Prometheus only fails
kubeControllerManager: {enabled: false}
kubeScheduler: {enabled: false}
kubeEtcd: {enabled: false}
kubeProxy: {enabled: false}
prometheus:
  prometheusSpec:
    retention: 7d  # selectors stay default:
                   # objects need release: kps
EOF

# pin the chart; every later "helm upgrade kps"
# reuses this version (add the export to ~/.zshrc)
helm search repo kube-prometheus-stack   # CHART VERSION
export KPS_VER=<that version>
helm install kps \
  prometheus-community/kube-prometheus-stack \
  -n monitoring --create-namespace \
  --version $KPS_VER -f deploy/kps-values.yaml

kubectl -n monitoring get pods -w  # ~2 min
kubectl -n monitoring get svc
# kps-grafana                              80/TCP
# kps-kube-prometheus-stack-alertmanager   9093/TCP
# kps-kube-prometheus-stack-prometheus     9090/TCP
# ...

# one terminal each (or append & to background)
NS="-n monitoring"
kubectl $NS port-forward \
  svc/kps-kube-prometheus-stack-prometheus 9090
kubectl $NS port-forward \
  svc/kps-kube-prometheus-stack-alertmanager 9093
kubectl $NS port-forward svc/kps-grafana 3000:80

# Grafana: user admin, password from the secret
kubectl $NS get secret kps-grafana \
  -o jsonpath='{.data.admin-password}' \
  | base64 -d; echo

Service names are derived from the release name. If a port-forward fails, use the names that kubectl -n monitoring get svc prints.

Lab 0 · Build (6/7)

Setup: the shop manifests

Goal: one Deployment and one Service per app, generated by a loop so both stay identical. The Service port is named http: ServiceMonitors (Lab 2) refer to it by name.

echo 'apiVersion: v1
kind: Namespace
metadata: {name: shop}' > deploy/shop.yaml
# only payments reads the two env vars
for a in orders payments; do
cat >> deploy/shop.yaml <<EOF
---
apiVersion: apps/v1
kind: Deployment
metadata: {name: $a, namespace: shop, labels: {app: $a}}
spec:
  replicas: 2
  selector: {matchLabels: {app: $a}}
  template:
    metadata: {labels: {app: $a}}
    spec:
      containers:
        - name: $a
          image: shop/$a:lab
          imagePullPolicy: IfNotPresent
          env:
            - {name: FAIL_RATE, value: "0"}
            - {name: EXTRA_LATENCY_MS, value: "0"}
          ports: [{name: http, containerPort: 8080}]
          readinessProbe:
            httpGet: {path: /healthz, port: http}
          resources:
            requests: {cpu: 50m, memory: 32Mi}
            limits: {memory: 64Mi}
# (still inside the same heredoc and loop)
---
apiVersion: v1
kind: Service
metadata: {name: $a, namespace: shop, labels: {app: $a}}
spec:
  selector: {app: $a}
  ports: [{name: http, port: 8080, targetPort: http}]
EOF
done
kubectl apply -f deploy/shop.yaml
kubectl -n shop get deploy,svc
# deployment.apps/orders     2/2
# deployment.apps/payments   2/2
# service/orders     ClusterIP  ...  8080/TCP
# service/payments   ClusterIP  ...  8080/TCP
Lab 0 · Build (7/7)

Setup: the load generator

Goal: steady traffic through orders → payments, so every later lab has data to look at.

cat >> deploy/shop.yaml <<'EOF'
---
apiVersion: apps/v1
kind: Deployment
metadata: {name: loadgen, namespace: shop}
spec:
  replicas: 1
  selector: {matchLabels: {app: loadgen}}
  template:
    metadata: {labels: {app: loadgen}}
    spec:
      containers:
        - name: loadgen
          image: curlimages/curl:8.11.1
          command: ["/bin/sh", "-c"]
          args:
            - |
              while true; do
                curl -s -o /dev/null -X POST \
                  -d '{"item":"book","amount":42}' \
                  http://orders:8080/orders
                sleep 0.2
              done
EOF
kubectl apply -f deploy/shop.yaml
kubectl -n shop rollout status deploy/orders
kubectl -n shop get pods
# loadgen-…    1/1 Running
# orders-…     1/1 Running   (x2)
# payments-…   1/1 Running   (x2)

kubectl -n shop logs deploy/payments --tail=2
# {"time":"…","level":"INFO","msg":"charged",
#  "order_id":"o-…","amount":42}


# your own request through the Service
kubectl -n shop port-forward svc/orders 8080 &
curl -s -X POST localhost:8080/orders \
  -d '{"item":"pen","amount":3}'
# {"id":"o-…","item":"pen","amount":3}

curlimages/curl:8.11.1 is an example tag; any recent tag works. Each orders replica numbers its own orders, so IDs repeat across the two replicas: fine for a lab.

Lab 0 · Verify, break it, clean up

A running stack with traffic and no false alarms

Everything later in the deck observes this setup. Make sure it is healthy before you add telemetry.

Verify

  • kubectl -n shop get pods: 2 orders, 2 payments, 1 loadgen Running.
  • kubectl -n shop logs deploy/orders --tail=3: JSON lines from slog.
  • Prometheus → Status → Target health: kubelet, node-exporter, kube-state-metrics, apiserver all UP.
  • Alertmanager shows only Watchdog (always firing by design, ch 13) and possibly InfoInhibitor.
  • Grafana → Dashboards: "Kubernetes / Compute Resources / Namespace (Pods)" shows shop.

Break it

  • Errors: kubectl -n shop set env deploy/payments FAIL_RATE=0.3 → orders logs payment failed. Only logs show it: no metric exists yet (Lab 2).
  • Latency: EXTRA_LATENCY_MS=2500 → the 2 s client timeout fires and orders returns 502. Reset both to 0.
  • Scrape gap: remove kubeEtcd: {enabled: false}, helm upgrade with the same values, and watch etcd targets go DOWN and alerts fire. Restore it.
# keep the cluster: every later
# lab builds on it

# pause the load when not studying
kubectl -n shop scale \
  deploy/loadgen --replicas=0

# full teardown (only at the
# very end of the deck)
# kind delete cluster --name obs
Lab 1 · Build (1/2)

Observe the shop with what already exists (1/2): black-box and logs

Goal: answer "is it broken, and how badly?" for the shop app using only an outside probe and the logs Lab 0 writes. Note what you cannot answer.

kubectl -n shop port-forward svc/orders 8080 &  # if not running

# black-box: status codes and latency of 50 requests
probe() {
  for i in $(seq 50); do
    curl -s -o /dev/null -w '%{http_code} %{time_total}\n' \
      -X POST -d '{"item":"pen","amount":3}' \
      localhost:8080/orders
  done | awk '{n[$1]++; t+=$2; if ($2>m) m=$2}
    END {for (c in n) print "code", c, n[c];
         printf "avg %.3fs  max %.3fs\n", t/NR, m}'
}
probe
# code 201 50
# avg 0.060s  max 0.095s

kubectl -n shop set env deploy/payments FAIL_RATE=0.2 # 20%
kubectl -n shop rollout status deploy/payments
probe
# code 201 41
# code 502 9
# logs: what do the services write?
kubectl -n shop logs deploy/orders --tail=3
# {"time":"…","level":"ERROR","msg":"payment failed",
#  "order_id":"o-…","err":"payments: 502 Bad Gateway"}

# with -l, kubectl logs defaults to --tail=10 per pod!
L="-n shop -l app=payments --since=2m --tail=-1"
kubectl logs $L | jq -r .msg | sort | uniq -c
#  480 charged
#  121 charge failed

# error ratio from payments logs: 121 / 601 ≈ 20%
kubectl logs $L | jq -s 'map(select(.msg|test("^charge")))
  | (map(select(.msg=="charge failed"))|length) / length'
# 0.2013…

# orders logs only failures: no total, no route,
# no duration → no error RATIO and no latency here

Keep a note of every question you could not answer from these signals. Lab 2 adds the metrics that answer them.

Lab 1 · Build (2/2)

Observe the shop (2/2): USE from Kubernetes metrics, and a series inventory

Goal: see what kube-prometheus-stack already collects about the shop pods, how many series exist, and write the shop's signal plan.

# Prometheus UI: http://localhost:9090 (port-forward from Lab 0)

# CPU used per pod (cores)
sum by (pod) (rate(container_cpu_usage_seconds_total{
  namespace="shop", container!=""}[5m]))

# memory working set per pod (bytes) vs the 64Mi limit
sum by (pod) (container_memory_working_set_bytes{
  namespace="shop", container!=""})

# restarts and availability
sum by (pod) (kube_pod_container_status_restarts_total{
  namespace="shop"})
kube_deployment_status_replicas_available{namespace="shop"}

# app metrics?  → "Empty query result" (none yet)
{job=~"orders|payments"}
# series inventory from the TSDB status API
P=localhost:9090/api/v1/status/tsdb
curl -s $P | jq '.data.headStats.numSeries'
# e.g. 60000 (varies with versions and node count)
curl -s $P | jq -r '.data.seriesCountByMetricName[:5][]
  | "\(.value)\t\(.name)"'
# top metrics by series count (bucket metrics lead)

# signal plan: copy the table from ch 1 and fill in
mkdir -p docs && cat > docs/signal-plan.md <<'EOF'
| Question | Signal | Query / source | Alert? |
|---|---|---|---|
| Are orders failing? | metric (R,E) | TODO in Lab 2 | page |
| Are orders slow?    | metric (D)   | TODO in Lab 2 | page |
| Is payments the cause? | metric + trace | … | no |
| Pods sized right?   | USE (k8s)    | queries above | ticket |
EOF

container!="" drops the pod-level cgroup series that cAdvisor also exports, which would otherwise double-count each pod.

Lab 1 · Verify, break it, clean up

Black-box sees the symptom; USE does not

The point of this lab: resource metrics and failure-only logs cannot give you RED. That gap is what app instrumentation fills.

Verify

  • probe with FAIL_RATE=0.2 shows ~20% 502s; the payments log ratio agrees.
  • The CPU, memory and restart queries return one series per shop pod (2 orders, 2 payments, loadgen).
  • {job=~"orders|payments"} returns nothing — no app metrics yet.
  • docs/signal-plan.md lists at least 6 questions, each with a signal and an alert decision.
  • You can explain why orders logs alone cannot give an error ratio.

Break it

  • Latency, not CPU: kubectl -n shop set env deploy/payments FAIL_RATE=0 EXTRA_LATENCY_MS=1500 → probe avg jumps to ~1.5 s, yet the CPU and memory queries barely move. Waiting is not utilization: a symptom with no USE cause.
  • Timeouts: EXTRA_LATENCY_MS=2500 → every order is a 502 after 2 s (client timeout). Black-box sees it; USE still looks healthy.
  • Silent logs: run kubectl logs -n shop -l app=payments | wc -l without --tail=-1 → only 10 lines per pod. Your "error ratio" from logs was wrong without it.
# reset the knobs
kubectl -n shop set env \
  deploy/payments \
  FAIL_RATE=0 EXTRA_LATENCY_MS=0

# stop the port-forward
kill %1

# keep docs/signal-plan.md:
# Lab 2 fills in the TODOs
Lab 2 · Build (1/4)

Instrument with client_golang: the metric definitions

Goal: one shared package gives RED metrics for every route of both services. It lives under apps/ so the Lab 0 Dockerfile copies it.

mkdir -p apps/internal/metrics
cat > apps/internal/metrics/metrics.go <<'EOF'
package metrics

import (
	"net/http"
	"strconv"
	"strings"
	"time"

	"github.com/prometheus/client_golang/prometheus"
	"github.com/prometheus/client_golang/prometheus/promauto"
	"github.com/prometheus/client_golang/prometheus/promhttp"
)

var requests = promauto.NewCounterVec(prometheus.CounterOpts{
	Name: "http_requests_total",
	Help: "HTTP requests by method, route template and code.",
}, []string{"method", "route", "code"})
var duration = promauto.NewHistogramVec(prometheus.HistogramOpts{
	Name:    "http_request_duration_seconds",
	Help:    "HTTP request latency.",
	Buckets: []float64{.01, .025, .05, .1, .25, .3, .5, 1, 2.5, 5},
	// also expose a native histogram (protobuf scrape)
	NativeHistogramBucketFactor:     1.1,
	NativeHistogramMaxBucketNumber:  100,
	NativeHistogramMinResetDuration: time.Hour,
}, []string{"method", "route"})

// recorder remembers the status code the handler wrote
type recorder struct {
	http.ResponseWriter
	code int
}

func (r *recorder) WriteHeader(c int) {
	r.code = c
	r.ResponseWriter.WriteHeader(c)
}
# (file continues on the next slide)

The 0.3 s bucket is deliberate: it is the latency SLO threshold used in later chapters, so "share under 300 ms" is exact.

Lab 2 · Build (2/4)

Instrument with client_golang: middleware and /metrics

Goal: finish the package. r.Pattern (Go 1.23+) is set by the ServeMux, so route is the template, never the raw path.

// Middleware wraps the whole mux; the mux sets r.Pattern.
func Middleware(next http.Handler) http.Handler {
	return http.HandlerFunc(func(w http.ResponseWriter,
		r *http.Request) {
		start := time.Now()
		rec := &recorder{ResponseWriter: w, code: 200}
		next.ServeHTTP(rec, r)
		route := "unmatched" // 404 / 405
		if _, p, ok := strings.Cut(r.Pattern, " "); ok {
			route = p // "GET /orders/{id}" → "/orders/{id}"
		}
		requests.WithLabelValues(r.Method, route,
			strconv.Itoa(rec.code)).Inc()
		duration.WithLabelValues(r.Method, route).
			Observe(time.Since(start).Seconds())
	})
}

// Handler serves /metrics (OpenMetrics → exemplars, ch 9).
func Handler() http.Handler {
	return promhttp.HandlerFor(prometheus.DefaultGatherer,
		promhttp.HandlerOpts{EnableOpenMetrics: true})
}
EOF
go get github.com/prometheus/client_golang@v1.25.0
go mod tidy
go vet ./...

# what the middleware produces (tested):
#   GET /orders/o-1   → route="/orders/{id}", code="200"
#   GET /orders/o-2   → same series, value 2
#   GET /nope         → route="unmatched", code="404"
#
# http_requests_total{code="200",method="GET",
#   route="/orders/{id}"} 2
# http_requests_total{code="404",method="GET",
#   route="unmatched"} 1

client_golang v1.25.0 was current when this was written; any newer v1 works. go.sum now exists, so the Dockerfile's COPY go.* ./ picks it up.

Lab 2 · Build (3/4)

Wire it in, add business metrics, and scrape with a ServiceMonitor

Goal: both services expose /metrics; kube-prometheus-stack scrapes them as job="orders" and job="payments" (the job label defaults to the Service name).

// apps/orders/main.go — add to imports:
//   "obs-lab/apps/internal/metrics"
//   "github.com/prometheus/client_golang/prometheus"
//   "github.com/prometheus/client_golang/prometheus/promauto"
var ordersCreated = promauto.NewCounter(prometheus.CounterOpts{
	Name: "orders_created_total",
	Help: "Orders created successfully.",
})
// in create(), just before w.WriteHeader(http.StatusCreated):
	ordersCreated.Inc()
// in main():
	mux.Handle("GET /metrics", metrics.Handler())
	err := http.ListenAndServe(":8080", metrics.Middleware(mux))

// apps/payments/main.go — same imports, then:
var chargeAmount = promauto.NewHistogram(prometheus.HistogramOpts{
	Name:    "payments_charge_amount",
	Help:    "Charged amount per successful charge.",
	Buckets: prometheus.ExponentialBuckets(1, 2, 12),
})
// in charge(), after log.Info("charged", ...):
	chargeAmount.Observe(req.Amount)
// in main(): the same two lines as orders
gofmt -l . ; go vet ./...
for a in orders payments; do
  docker build --build-arg APP=$a -t shop/$a:lab .
  kind load docker-image shop/$a:lab --name obs
done
kubectl -n shop rollout restart deploy/orders deploy/payments

cat > deploy/shop-monitor.yaml <<'EOF'
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: shop
  namespace: shop
  labels: {release: kps}       # matched by kps
spec:
  selector:
    matchExpressions:
      - {key: app, operator: In, values: [orders, payments]}
  endpoints:
    - port: http               # Service port NAME
      path: /metrics
      interval: 15s
EOF
kubectl apply -f deploy/shop-monitor.yaml

# Prometheus → Status → Target health, after ~1 min:
# serviceMonitor/shop/shop/0   4/4 up
Lab 2 · Build (4/4)

Native histograms and a blackbox probe in a local Prometheus

Goal: write raw scrape_configs yourself (no Operator), scrape native and classic histograms side by side, and probe /healthz from outside.

cd local-prom   # after writing prometheus.yml (right)
kubectl -n shop port-forward --address 0.0.0.0 svc/orders 8080 &
echo 'modules: {http_2xx: {prober: http, timeout: 5s}}' \
  > blackbox.yml
docker network create obs-lab
docker run -d --name blackbox --network obs-lab \
  --add-host=host.docker.internal:host-gateway \
  -v "$PWD/blackbox.yml:/etc/blackbox_exporter/config.yml" \
  prom/blackbox-exporter:v0.27.0
docker run -d --name prom --network obs-lab -p 9091:9090 \
  --add-host=host.docker.internal:host-gateway \
  -v "$PWD/prometheus.yml:/etc/prometheus/prometheus.yml" \
  prom/prometheus:v3.15.1
# UI http://localhost:9091 (9090 = kps); after 5 min:
histogram_quantile(0.99, sum by (le) (rate(        # classic
  http_request_duration_seconds_bucket{job="orders-local"}[5m])))
histogram_quantile(0.99, sum(rate(                 # native
  http_request_duration_seconds{job="orders-local"}[5m])))
histogram_fraction(0, 0.3, sum(rate(               # share ≤ 0.3 s
  http_request_duration_seconds{job="orders-local"}[5m])))
mkdir -p local-prom && cat > local-prom/prometheus.yml <<'EOF'
global: {scrape_interval: 15s}
scrape_configs:
  - job_name: orders-local
    scrape_native_histograms: true         # protobuf first
    always_scrape_classic_histograms: true # keep _bucket
    sample_limit: 2000
    static_configs: [{targets: ['host.docker.internal:8080']}]
    metric_relabel_configs:
      - {source_labels: [__name__], regex: go_gc_.*, action: drop}
  - job_name: blackbox-http
    metrics_path: /probe
    params: {module: [http_2xx]}
    static_configs:
      [{targets: ['http://host.docker.internal:8080/healthz']}]
    relabel_configs:
      - {source_labels: [__address__], target_label: __param_target}
      - {source_labels: [__param_target], target_label: instance}
      - {target_label: __address__, replacement: blackbox:9115}
EOF
docker run --rm -v "$PWD/local-prom:/c" --entrypoint promtool \
  prom/prometheus:v3.15.1 check config /c/prometheus.yml

Image tags are examples; any Prometheus ≥ 3.8 supports scrape_native_histograms. Docker would create a missing bind-mount source as a directory, hence "write the file first".

Lab 2 · Verify, break it, clean up

RED for real, then break the scrape and the labels

From here on every lab queries these metrics for both services.

Verify

  • kps: sum by (job, route, code) (rate(http_requests_total[5m])) shows both services at ~5 req/s.
  • orders_created_total on localhost:8080/metrics grows with the loadgen.
  • Local: probe_success{job="blackbox-http"} is 1; native and classic p99 agree within a bucket.
  • Lab 1 signal-plan TODOs filled in.

Break it

  • Errors: FAIL_RATE=0.2 on payments → sum(rate(http_requests_total{job="orders",code="502"}[5m])) / sum(rate(http_requests_total{job="orders"}[5m])) ≈ 0.2.
  • Selector: delete the release: kps label from the ServiceMonitor → after a minute the orders/payments targets vanish (gone, not DOWN). Restore it.
  • Cardinality: set route = r.URL.Path, rebuild, request /orders/o-1 … /orders/o-300 → count(http_requests_total{job="orders"}) grows by ~300; scrape_series_added spikes. Revert.
  • Classic off: drop always_scrape_classic_histograms, docker restart prom → no new _bucket samples.
# local Prometheus only
docker rm -f prom blackbox
docker network rm obs-lab
kill %1   # port-forward
cd ..
# keep the metrics code and
# deploy/shop-monitor.yaml
kubectl -n shop set env \
  deploy/payments FAIL_RATE=0
Lab 3 · Build (1/3)

PromQL on the shop app: RED queries through the HTTP API

Goal: answer rate, errors and duration questions for orders and payments from the command line. Needs Labs 0–2 (metrics scraped as job="orders" / job="payments").

kubectl -n monitoring port-forward svc/prometheus-operated 9090 &
# helper: instant query → "labels  value"
q() {
  curl -sG localhost:9090/api/v1/query \
    --data-urlencode "query=$1" | jq -r '.data.result[]
    | "\(.metric | del(.__name__) | tostring)  \(.value[1])"'
}

# 0) both jobs scraped? (1 per pod)
q 'up{namespace="shop"}'

# 1) RATE: requests/s per job and route
q 'sum by (job, route) (
     rate(http_requests_total{namespace="shop"}[5m]))'
# {"job":"orders","route":"/orders"}  4.9…  (loadgen ≈ 5/s)

# 2) ERRORS: 5xx share per job (≈ 0 or empty for now)
q 'sum by (job) (rate(
     http_requests_total{namespace="shop",code=~"5.."}[5m]))
   / sum by (job) (rate(
     http_requests_total{namespace="shop"}[5m]))'
# 3) DURATION: which buckets does the app expose?
q 'count by (le) (
     http_request_duration_seconds_bucket{job="orders"})'

# 4) p99 per job and route
q 'histogram_quantile(0.99, sum by (job, route, le) (rate(
     http_request_duration_seconds_bucket{namespace="shop"}[5m])))'

# 5) mean latency from _sum / _count
q 'sum by (job) (rate(
     http_request_duration_seconds_sum{namespace="shop"}[5m]))
   / sum by (job) (rate(
     http_request_duration_seconds_count{namespace="shop"}[5m]))'

# 6) business metric: orders created per minute
q 'sum(rate(orders_created_total[5m])) * 60'

# 7) week-over-week: empty, the cluster is younger
q 'sum(rate(http_requests_total{job="orders"}[5m] offset 1w))'

Run each query in the Prometheus UI (Table tab) too and compare labels before and after each sum by. route values are route templates (/orders/{id}), not raw paths.

Lab 3 · Build (2/3)

Joins, range queries and injected failures

Copy node placement onto pod usage, count points in a range query, then use the payments knobs and watch the RED queries move.

# 8) JOIN: CPU cores per shop pod, plus its node
q 'sum by (namespace, pod) (rate(
     container_cpu_usage_seconds_total{
       namespace="shop", container!=""}[5m]))
   * on (namespace, pod) group_left (node) kube_pod_info'
# {"namespace":"shop","node":"obs-worker","pod":"orders-…"}  0.003

# 9) memory per pod vs limit (empty if no limit is set)
q 'sum by (pod) (container_memory_working_set_bytes{
     namespace="shop", container!=""})
   / sum by (pod) (kube_pod_container_resource_limits{
     namespace="shop", resource="memory"})'

# 10) range query: points in the last 30 min at step 60s
S=$(( $(date +%s) - 1800 ))
curl -sG localhost:9090/api/v1/query_range \
  --data-urlencode \
    'query=sum(rate(http_requests_total{job="orders"}[5m]))' \
  --data-urlencode "start=$S" \
  --data-urlencode "end=$(date +%s)" \
  --data-urlencode step=60 \
  | jq '.data.result[0].values | length'
# 31
# A) errors: 20% of charges fail
kubectl -n shop set env deploy/payments FAIL_RATE=0.2
kubectl -n shop rollout status deploy/payments
# wait 2–3 minutes, then
q 'sum by (job, code) (
     rate(http_requests_total{namespace="shop"}[5m]))'
# payments 5xx ≈ 20% of its traffic;
# orders shows how the failure propagates upstream

# B) latency: +300 ms on every charge
kubectl -n shop set env deploy/payments \
  FAIL_RATE=0 EXTRA_LATENCY_MS=300
q 'histogram_quantile(0.99, sum by (job, le) (rate(
     http_request_duration_seconds_bucket{namespace="shop"}[5m])))'
# both p99 values rise by ≈ 0.3 s (orders waits for payments)

# C) presence checks
q 'absent(up{job="orders"})'           # empty = good
q 'absent(up{job="does-not-exist"})'   # {"job":"does-not-exist"} 1
Lab 3 · Build (3/3)

Reproduce the sum-then-rate trap

Two pods, one rollout restart: the correct query stays flat, the wrong one spikes.

# correct: rate per pod, then sum
GOOD='sum(rate(http_requests_total{job="orders"}[5m]))'
# wrong: sum raw counters, then rate (needs a subquery)
BAD='rate(sum(http_requests_total{job="orders"})[5m:15s])'

R=$(kubectl -n shop get deploy orders \
      -o jsonpath='{.spec.replicas}')
kubectl -n shop scale deploy/orders --replicas=2
sleep 120
kubectl -n shop rollout restart deploy/orders

# watch both for ~5 minutes
for i in $(seq 10); do
  echo "good=$(q "$GOOD" | awk '{print $2}')" \
       "bad=$(q "$BAD" | awk '{print $2}')"
  sleep 30
done
# good stays ≈ 5; bad jumps far above 5 after the
# restart: the summed counter dropped, rate() saw a reset
# D) subquery: peak 5m rate in the last 30 minutes
q 'max_over_time(
     sum(rate(http_requests_total{job="orders"}[5m]))[30m:1m])'

# resolution omitted → global evaluation_interval
q 'max_over_time(
     sum(rate(http_requests_total{job="orders"}[5m]))[30m:])'

# reset everything
kubectl -n shop set env deploy/payments \
  FAIL_RATE=0 EXTRA_LATENCY_MS=0
kubectl -n shop scale deploy/orders --replicas=$R

Also graph GOOD and BAD in the Graph tab over the last 15 minutes: the BAD spike is what a dashboard built that way shows after every deploy.

Lab 3 · Verify, break it, clean up

Queries that answer the questions

You should be able to explain every number you saw, including the wrong one.

Verify

  • Query 1 shows ≈ 5 req/s on orders /orders (the loadgen rate).
  • With FAIL_RATE=0.2, ≈ 20% of payments responses are 5xx within a few minutes.
  • With EXTRA_LATENCY_MS=300, p99 rises by ≈ 0.3 s; precision depends on the bucket containing it.
  • The join returns one row per shop pod with a node label.
  • absent(up{job="orders"}) is empty while the app runs.

Break it

  • **Drop le:** histogram_quantile(0.99, sum by (job) (rate(…_bucket[5m]))) → empty result (series without le are skipped). Explain why.
  • Too short window: rate(http_requests_total[15s]) with a 15s or 30s scrape → empty or flickering.
  • Rate on a gauge: rate(container_memory_working_set_bytes[5m]) returns numbers that mean nothing.
  • Bad join: * on (namespace) group_left kube_pod_info → "many-to-many matching not allowed".
  • Restart: the BAD query of step 3 spikes; GOOD does not.
# knobs back to normal?
kubectl -n shop set env \
  deploy/payments --list
# FAIL_RATE=0
# EXTRA_LATENCY_MS=0
kubectl -n shop get deploy

# keep everything for Lab 4:
# its recording rules use
# the same series

# stop the port-forward
# with Ctrl-C
Lab 4 · Build (1/4)

Rules as code: recording rules and alerts in a plain file

Goal: recording rules and alerts for the shop app in Git, proven by promtool before anything touches the cluster. Needs Labs 0–3.

mkdir -p rules && cat > rules/shop.rules.yml <<'EOF'
groups:
  - name: shop-red
    rules:
      - record: job_route:http_requests:rate5m
        expr: |
          sum by (job, route) (
            rate(http_requests_total{namespace="shop"}[5m]))
      - record: job:http_requests:rate5m
        expr: sum by (job) (job_route:http_requests:rate5m)
      - record: job:http_request_errors:rate5m
        expr: |
          sum by (job) (rate(
            http_requests_total{namespace="shop",code=~"5.."}[5m]))
      - record: job:http_request_errors:ratio_rate5m
        expr: job:http_request_errors:rate5m / job:http_requests:rate5m
      - record: job:http_request_duration_seconds:p99_5m
        expr: |
          histogram_quantile(0.99, sum by (job, le) (rate(
            http_request_duration_seconds_bucket{namespace="shop"}[5m])))
EOF
cat >> rules/shop.rules.yml <<'EOF'
  - name: shop-alerts
    rules:
      - alert: ShopHighErrorRate
        expr: |
          job:http_request_errors:ratio_rate5m > 0.05
          and on (job) job:http_requests:rate5m > 0.5
        for: 5m
        labels: {severity: critical, team: shop}
        annotations:
          summary: "{{ $labels.job }}: {{ $value | humanizePercentage }} 5xx"
      - alert: ShopHighLatencyP99
        expr: job:http_request_duration_seconds:p99_5m > 0.5
        for: 10m
        labels: {severity: warning, team: shop}
        annotations:
          summary: "{{ $labels.job }} p99 {{ $value | humanizeDuration }}"
      - alert: ShopJobAbsent
        expr: absent(up{job="orders"})
        for: 5m
        labels: {severity: critical, team: shop, namespace: shop}
        annotations: {summary: "No orders targets at all"}
EOF
promtool check rules rules/shop.rules.yml
#   SUCCESS: 8 rules found
Lab 4 · Build (2/4)

Unit-test the rules with promtool

A 10% error ratio fires ShopHighErrorRate only after for: 5m (pending at 3m); a vanished target fires ShopJobAbsent.

cat > rules/shop.test.yml <<'EOF'
rule_files: [shop.rules.yml]   # evaluation_interval defaults to 1m
tests:
  - interval: 1m        # 6 errors + 54 ok per minute = 10%, 1 req/s
    input_series:
      - series: 'http_requests_total{namespace="shop",job="orders",code="500"}'
        values: '0+6x20'
      - series: 'http_requests_total{namespace="shop",job="orders",code="200"}'
        values: '0+54x20'
    promql_expr_test:
      - expr: job:http_request_errors:ratio_rate5m
        eval_time: 10m
        exp_samples:
          - labels: 'job:http_request_errors:ratio_rate5m{job="orders"}'
            value: 0.1
    alert_rule_test:
      - {eval_time: 3m, alertname: ShopHighErrorRate, exp_alerts: []}
      - eval_time: 10m
        alertname: ShopHighErrorRate
        exp_alerts:
          - exp_labels: {job: orders, severity: critical, team: shop}
            exp_annotations: {summary: "orders: 10% 5xx"}
EOF
cat >> rules/shop.test.yml <<'EOF'
  - interval: 1m        # target gone after 4 minutes
    input_series:
      - series: 'up{job="orders"}'
        values: '1x4 _x15'
    alert_rule_test:
      - eval_time: 15m
        alertname: ShopJobAbsent
        exp_alerts:
          - exp_labels:
              job: orders
              severity: critical
              team: shop
              namespace: shop
            exp_annotations:
              summary: No orders targets at all
EOF

cd rules && promtool test rules shop.test.yml && cd ..
# Unit Testing:  shop.test.yml
#   SUCCESS

# now make it fail on purpose: change value: 0.1 → 0.2,
# rerun, read the diff promtool prints, change it back

Why 15m in the second test: the last up sample is at 4m; with the 5m lookback, absent() becomes true just after 9m, then for: 5m → firing from ~14m.

Lab 4 · Build (3/4)

Load the rules and a webhook sink into the cluster

Wrap the tested file in a PrometheusRule with label release: kps, and deploy a tiny service that prints every Alertmanager notification.

# generate the CR from the tested file (no hand copy)
mkdir -p deploy
{ cat <<'EOF'
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: shop-rules
  namespace: shop
  labels:
    release: kps
spec:
EOF
  sed 's/^/  /' rules/shop.rules.yml
} > deploy/shop-rules.yaml
kubectl apply -f deploy/shop-rules.yaml
kubectl apply -f deploy/alert-sink.yaml   # written on the right

# loaded? (the config reload takes up to ~1 min)
kubectl -n monitoring port-forward svc/prometheus-operated 9090 &
sleep 60
curl -s localhost:9090/api/v1/rules | jq -r \
  '.data.groups[] | select(.name|startswith("shop")) | .name'
# shop-red
# shop-alerts
cat > deploy/alert-sink.yaml <<'EOF'
apiVersion: apps/v1
kind: Deployment
metadata: {name: alert-sink, namespace: monitoring}
spec:
  selector: {matchLabels: {app: alert-sink}}
  template:
    metadata: {labels: {app: alert-sink}}
    spec:
      containers:
      - name: sink
        image: python:3.12-alpine
        command: ["python", "-u", "-c"]
        args:
        - |
          import http.server as h, json
          class H(h.BaseHTTPRequestHandler):
            def do_POST(s):
              b = json.loads(s.rfile.read(int(s.headers['Content-Length'])))
              names = [a['labels']['alertname'] for a in b['alerts']]
              print(s.path, b['status'], names)
              s.send_response(200); s.end_headers()
          h.HTTPServer(('', 8080), H).serve_forever()
---
apiVersion: v1
kind: Service
metadata: {name: alert-sink, namespace: monitoring}
spec: {selector: {app: alert-sink}, ports: [{port: 8080}]}
EOF
Lab 4 · Build (4/4)

Route alerts with Alertmanager, testing the routes first

Critical shop alerts go to /critical, other shop alerts to /warning, Watchdog is dropped, and a missing job inhibits the rest.

mkdir -p alertmanager && cat > alertmanager/alertmanager.yml <<'EOF'
route:
  receiver: "null"
  group_by: [alertname, job]
  group_wait: 10s
  group_interval: 1m
  repeat_interval: 1h
  routes:
    - {receiver: "null", matchers: [alertname = "Watchdog"]}
    - {receiver: shop-critical, matchers: [team="shop", severity="critical"]}
    - {receiver: shop-warning, matchers: [team = "shop"]}
inhibit_rules:
  - source_matchers: [alertname = "ShopJobAbsent"]
    target_matchers: [alertname != "ShopJobAbsent"]
    equal: [team]
receivers:
  - name: "null"
  - name: shop-critical
    webhook_configs:
      - url: http://alert-sink.monitoring.svc:8080/critical
  - name: shop-warning
    webhook_configs:
      - url: http://alert-sink.monitoring.svc:8080/warning
EOF
C=alertmanager/alertmanager.yml
amtool check-config $C
amtool config routes test --config.file=$C \
  --verify.receivers=shop-critical team=shop severity=critical
amtool config routes test --config.file=$C \
  team=shop severity=warning               # shop-warning

# wrap the tested file as Helm values for kps
{ echo 'alertmanager:'; echo '  config:'
  sed 's/^/    /' $C; } > deploy/am-values.yaml

# same chart version as Lab 0; keep the other values
helm upgrade kps prometheus-community/kube-prometheus-stack \
  -n monitoring --reuse-values -f deploy/am-values.yaml \
  --version $KPS_VER

kubectl -n monitoring port-forward svc/alertmanager-operated 9093 &
export AM=http://localhost:9093

# fire a real alert: 30% of charges fail
kubectl -n shop set env deploy/payments FAIL_RATE=0.3
kubectl -n monitoring logs -f deploy/alert-sink
# after ~6–7 minutes:
# /critical firing ['ShopHighErrorRate']

Lists in Helm values replace the chart defaults instead of merging: this config replaces the chart's default routes, receivers and inhibit rules, which is why Watchdog and "null" are defined again here.

Lab 4 · Verify, break it, clean up

From a failing charge to a webhook call

Follow one alert through every state, then break loading, muting and routing on purpose.

Verify

  • Prometheus → Alerts: ShopHighErrorRate goes pending, then firing after 5 minutes.
  • Query ALERTS{team="shop"} and read the alertstate label.
  • alert-sink prints /critical firing ['ShopHighErrorRate'] once per group, not once per evaluation.
  • Set FAIL_RATE=0: the sink prints resolved (webhooks send resolved by default).
  • amtool alert query --alertmanager.url=$AM team=shop lists active shop alerts.

Break it

  • Lost label: kubectl -n shop label prometheusrule shop-rules release- → the shop groups disappear from /api/v1/rules, with no error anywhere. Put it back.
  • Silence: amtool silence add --alertmanager.url=$AM --author=$USER --duration=15m --comment=lab job=payments → no notification while it holds.
  • Inhibition: scale orders to 0 → after ~6 min ShopJobAbsent reaches /critical; other shop alerts show as inhibited in the UI. Scale back.
  • Bad route: change severity = "critical" to "crit" and rerun the --verify.receivers test: it fails before deploy.
kubectl -n shop set env \
  deploy/payments FAIL_RATE=0
kubectl -n shop scale \
  deploy/orders --replicas=2   # Lab 0 count

# keep shop-rules: ch 10 and
# Lab 13 use the recording rules

# remove only the sink
kubectl delete -f \
  deploy/alert-sink.yaml

# stop port-forwards
kill %1 %2 2>/dev/null
Lab 5 · Build (1/4)

Operator internals and a cardinality bomb

Goal: see how the Operator selects objects and renders config, then prepare a target that exports 20,000 series.

# 1) what the Operator manages and what it selects
kubectl api-resources --api-group=monitoring.coreos.com
P=kps-kube-prometheus-stack-prometheus
kubectl -n monitoring get prometheus $P \
  -o jsonpath='{.spec.podMonitorSelector}{"\n"}'
# {"matchLabels":{"release":"kps"}}

# 2) the config the Operator rendered (never edit it)
kubectl -n monitoring get secret prometheus-$P \
  -o jsonpath='{.data.prometheus\.yaml\.gz}' \
  | base64 -d | gunzip | grep job_name
# - job_name: serviceMonitor/shop/orders/0 ...

# 3) baseline (Prometheus port-forward on 9090)
curl -s localhost:9090/api/v1/query \
  --data-urlencode 'query=prometheus_tsdb_head_series' \
  | jq -r '.data.result[0].value[1]'
# note the number, e.g. 41000

# 4) write the bomb (right). It sets Content-Type
#    explicitly: Prometheus 3 rejects scrapes without
#    a valid one (unless fallback_scrape_protocol is set)
# deploy/bomb.yaml — one label with 20,000 values
apiVersion: apps/v1
kind: Deployment
metadata: {name: bomb, namespace: shop, labels: {app: bomb}}
spec:
  selector: {matchLabels: {app: bomb}}
  template:
    metadata: {labels: {app: bomb}}
    spec:
      containers:
        - name: bomb
          image: python:3.13-alpine
          ports: [{name: metrics, containerPort: 9100}]
          command: [python, -c]
          args:
            - |
              import http.server
              rows = [f'bomb_requests_total{{user_id="u{i}"}} 1'
                      for i in range(20000)]
              body = ("\n".join(rows) + "\n").encode()
              class H(http.server.BaseHTTPRequestHandler):
                  def do_GET(self):
                      self.send_response(200)
                      self.send_header("Content-Type",
                                       "text/plain; version=0.0.4")
                      self.end_headers()
                      self.wfile.write(body)
              http.server.HTTPServer(("", 9100), H).serve_forever()
Lab 5 · Build (2/4)

Scrape the bomb, find it, stop it at the door

Goal: find the culprit with the cheap tools (TSDB status, scrape metrics), then compare a drop rule with a hard limit.

# PodMonitor (append to deploy/bomb.yaml)
cat >> deploy/bomb.yaml <<'EOF'
---
apiVersion: monitoring.coreos.com/v1
kind: PodMonitor
metadata: {name: bomb, namespace: shop, labels: {release: kps}}
spec:
  selector: {matchLabels: {app: bomb}}
  podMetricsEndpoints: [{port: metrics, interval: 30s}]
EOF
kubectl apply -f deploy/bomb.yaml
sleep 60     # repeat step 3: about +20,000

# top metrics and labels from the head block (cheap)
curl -s localhost:9090/api/v1/status/tsdb | jq '.data |
  {top_metrics: .seriesCountByMetricName[:3],
   top_labels:  .labelValueCountByLabelName[:3]}'
# bomb_requests_total 20000 … / user_id 20000 …

# which target sends it (Prometheus UI)
topk(5, scrape_samples_post_metric_relabeling)
# {pod="bomb-…", namespace="shop", …}  20000
# Control A — drop the metric after the scrape
kubectl -n shop patch podmonitor bomb --type=json -p '[
 {"op":"add",
  "path":"/spec/podMetricsEndpoints/0/metricRelabelings",
  "value":[{"sourceLabels":["__name__"],
            "regex":"bomb_.*","action":"drop"}]}]'
# after ~1 min:
#   scrape_samples_scraped{pod=~"bomb-.*"}          20000
#   scrape_samples_post_metric_relabeling{…}            0

# Control B — a hard limit instead of a drop rule
kubectl -n shop patch podmonitor bomb --type=json -p '[
 {"op":"remove",
  "path":"/spec/podMetricsEndpoints/0/metricRelabelings"},
 {"op":"add","path":"/spec/sampleLimit","value":5000}]'
# after ~1 min:
#   up{pod=~"bomb-.*"}                                  0
#   increase(
#     prometheus_target_scrapes_exceeded_sample_limit_total[5m])
#   > 0;  Target health: "sample limit exceeded" 

prometheus_tsdb_head_series falls only after the next head compaction (up to about 2–3 hours): dropped series stay in memory until then. scrape_samples_post_metric_relabeling shows the effect immediately.

Lab 5 · Build (3/4)

CPU throttling and an OOMKill

Goal: produce the two classic container failures and find them with cAdvisor and kube-state-metrics queries.

# CPU: a busy loop with a 100m limit
kubectl -n shop create deploy burner \
  --image=busybox:1.37 -- sh -c 'while :; do :; done'
kubectl -n shop set resources deploy/burner \
  --requests=cpu=50m --limits=cpu=100m

# memory: allocate 64 MiB with a 32 MiB limit
kubectl -n shop run hog --image=python:3.13-alpine \
  --restart=Always --overrides='{"spec":{"containers":[
   {"name":"hog","image":"python:3.13-alpine",
    "command":["python","-c",
     "b=b\"x\"*(64*1024*1024)\nimport time; time.sleep(3600)"],
    "resources":{"limits":{"memory":"32Mi"}}}]}}'
kubectl -n shop get pod hog -w
# OOMKilled → CrashLoopBackOff
# throttling ratio (Prometheus UI, after ~5 min)
sum by (pod) (rate(container_cpu_cfs_throttled_periods_total{
                   pod=~"burner-.*"}[5m]))
/ sum by (pod) (rate(container_cpu_cfs_periods_total{
                   pod=~"burner-.*"}[5m]))
# → close to 1: throttled in almost every period

# CPU used: pinned at ~0.1 cores (the limit)
sum by (pod) (rate(container_cpu_usage_seconds_total{
                   pod=~"burner-.*", container!=""}[5m]))

# the OOMKill, as kube-state-metrics sees it
kube_pod_container_status_last_terminated_reason{pod="hog"}
# → {reason="OOMKilled", …} 1
increase(kube_pod_container_status_restarts_total{pod="hog"}[15m])

# bundled alerts that should fire in ~15–20 min (for: 15m):
#   CPUThrottlingHigh (burner), KubePodCrashLooping (hog)
Lab 5 · Build (4/4)

Remote write a curated subset

Goal: a second Prometheus acts as "long-term storage"; kps sends it only the series you keep.

# a second Prometheus that accepts remote write
kubectl -n monitoring create deploy prom-receiver \
  --image=quay.io/prometheus/prometheus:v3.5.3 \
  --port=9090 -- /bin/prometheus \
  --config.file=/etc/prometheus/prometheus.yml \
  --storage.tsdb.path=/prometheus \
  --web.enable-remote-write-receiver
kubectl -n monitoring expose deploy prom-receiver \
  --port=9090

# kps sends only what we keep
cat > deploy/kps-remote-write.yaml <<'EOF'
prometheus:
  prometheusSpec:
    remoteWrite:
      - name: receiver
        url: http://prom-receiver.monitoring:9090/api/v1/write
        writeRelabelConfigs:
          - sourceLabels: [__name__]
            regex: "up|http_requests_total"
            action: keep
EOF
helm upgrade kps \
  prometheus-community/kube-prometheus-stack \
  -n monitoring --reuse-values --version $KPS_VER \
  -f deploy/kps-remote-write.yaml

kubectl -n monitoring port-forward \
  svc/prom-receiver 9091:9090 &

# on the receiver (localhost:9091):
count by (__name__) ({__name__=~".+", prometheus!=""})
# → up, http_requests_total   (nothing else)

# on kps (localhost:9090): sending, and how far behind
rate(prometheus_remote_storage_samples_total[5m])
prometheus_remote_storage_highest_timestamp_in_seconds
 - ignoring(remote_name, url) group_right
   prometheus_remote_storage_queue_highest_sent_timestamp_seconds
# → a few seconds at most

Use the chart version you installed (helm list -n monitoring); --reuse-values keeps earlier values and adds these. The receiver has no volume: its data is lost when its pod restarts.

Lab 5 · Verify, break it, clean up

Operator, Kubernetes metrics, cardinality and remote write

You found 20,000 series without an expensive query, compared two controls, and sent a curated subset off the cluster.

Verify

  • The rendered config lists podMonitor/shop/bomb/0.
  • TSDB status shows bomb_requests_total and user_id at the top before Control A.
  • Throttling ratio for burner near 1; hog shows OOMKilled and a growing restart count.
  • Receiver series carry the external labels prometheus and prometheus_replica.
  • On kps the lag query stays near 0.

Break it

  • labeldrop trap: replace the bomb's limit with metricRelabelings: [{regex: user_id, action: labeldrop}] → 20,000 samples share one label set; rate(prometheus_target_scrapes_sample_duplicate_timestamp_total[5m]) > 0.
  • No release label: remove release: kps from the bomb PodMonitor → its job disappears from the rendered config.
  • Wrong URL: remote write to /api/v1/writ → 404, not retried: rate(prometheus_remote_storage_samples_failed_total[5m]) > 0, data dropped.
  • Too broad: remove writeRelabelConfigs → samples sent jump; on a paid backend, that is your bill.
kubectl -n shop delete deploy bomb burner
kubectl -n shop delete podmonitor bomb
kubectl -n shop delete pod hog
kubectl -n monitoring delete \
  deploy/prom-receiver svc/prom-receiver

# remove remote write again
cat > deploy/kps-no-rw.yaml <<'EOF'
prometheus:
  prometheusSpec:
    remoteWrite: []
EOF
helm upgrade kps \
  prometheus-community/kube-prometheus-stack \
  -n monitoring --reuse-values \
  --version $KPS_VER \
  -f deploy/kps-no-rw.yaml
Lab 6 · Build (1/4)

A shared logging package: logger and redaction

Goal: one package apps/internal/logx used by every service, so the shop log schema is enforced by code. It lives under apps/ because the Dockerfile copies only that directory.

mkdir -p apps/internal/logx
cat > apps/internal/logx/logx.go <<'EOF'
package logx

import (
	"context"
	"log/slog"
	"os"
	"strings"

	"go.opentelemetry.io/otel/trace"
)

var Level slog.LevelVar // INFO unless LOG_LEVEL says otherwise

var sensitive = map[string]bool{"password": true, "token": true,
	"authorization": true, "card_number": true, "cvv": true, "email": true}

func redact(_ []string, a slog.Attr) slog.Attr {
	if sensitive[strings.ToLower(a.Key)] {
		return slog.String(a.Key, "[REDACTED]")
	}
	return a
}
EOF
cat >> apps/internal/logx/logx.go <<'EOF'

// New returns the service logger and makes it the default.
func New(service string) *slog.Logger {
	_ = Level.UnmarshalText([]byte(os.Getenv("LOG_LEVEL")))
	h := slog.NewJSONHandler(os.Stdout, &slog.HandlerOptions{
		Level: &Level, ReplaceAttr: redact})
	l := slog.New(traceHandler{h}).With("service", service)
	slog.SetDefault(l)
	return l
}

// traceHandler adds trace_id/span_id when ctx has a valid span
// (from Lab 8 on; until then the fields are simply absent).
type traceHandler struct{ slog.Handler }

func (h traceHandler) Handle(ctx context.Context, r slog.Record) error {
	if sc := trace.SpanContextFromContext(ctx); sc.IsValid() {
		r.AddAttrs(slog.String("trace_id", sc.TraceID().String()),
			slog.String("span_id", sc.SpanID().String()))
	}
	return h.Handler.Handle(ctx, r)
}
EOF
Lab 6 · Build (2/4)

Request middleware

Goal: one wide request line per request with route, status and duration_ms; probes and scrapes only at DEBUG.

cat >> apps/internal/logx/logx.go <<'EOF'

func (h traceHandler) WithAttrs(a []slog.Attr) slog.Handler {
	return traceHandler{h.Handler.WithAttrs(a)}
}

func (h traceHandler) WithGroup(n string) slog.Handler {
	return traceHandler{h.Handler.WithGroup(n)}
}
EOF
cat > apps/internal/logx/middleware.go <<'EOF'
package logx

import (
	"log/slog"
	"net/http"
	"time"
)

type recorder struct {
	http.ResponseWriter
	status int
}

func (r *recorder) WriteHeader(code int) {
	r.status = code
	r.ResponseWriter.WriteHeader(code)
}
EOF
go get go.opentelemetry.io/otel/trace@latest && go mod tidy
cat >> apps/internal/logx/middleware.go <<'EOF'

// Middleware must wrap the ServeMux directly (it reads r.Pattern).
func Middleware(log *slog.Logger, next http.Handler) http.Handler {
	return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
		start := time.Now()
		rec := &recorder{ResponseWriter: w, status: http.StatusOK}
		next.ServeHTTP(rec, r)
		route := r.Pattern
		if route == "" {
			route = "unmatched"
		}
		lvl := slog.LevelInfo
		switch {
		case r.URL.Path == "/healthz" || r.URL.Path == "/metrics":
			lvl = slog.LevelDebug
		case rec.status >= 500:
			lvl = slog.LevelError
		case rec.status >= 400:
			lvl = slog.LevelWarn
		}
		log.LogAttrs(r.Context(), lvl, "request",
			slog.String("route", route),
			slog.Int("status", rec.status),
			slog.Int64("duration_ms",
				time.Since(start).Milliseconds()))
	})
}
EOF
go vet ./apps/internal/logx
Lab 6 · Build (3/4)

Tests that lock the schema

Goal: CI fails if a required key disappears or a sensitive key leaks.

cat > apps/internal/logx/logx_test.go <<'EOF'
package logx

import (
	"bytes"
	"encoding/json"
	"log/slog"
	"net/http"
	"net/http/httptest"
	"strings"
	"testing"
)

func newTest(buf *bytes.Buffer) *slog.Logger {
	h := slog.NewJSONHandler(buf,
		&slog.HandlerOptions{ReplaceAttr: redact})
	return slog.New(traceHandler{h}).With("service", "orders")
}

func TestRedaction(t *testing.T) {
	var buf bytes.Buffer
	newTest(&buf).Info("signup", "email", "a@example.com")
	if strings.Contains(buf.String(), "a@example.com") {
		t.Error("email leaked")
	}
}
EOF
cat >> apps/internal/logx/logx_test.go <<'EOF'

func TestRequestSchema(t *testing.T) {
	var buf bytes.Buffer
	mux := http.NewServeMux()
	mux.HandleFunc("POST /orders",
		func(w http.ResponseWriter, _ *http.Request) {
			w.WriteHeader(http.StatusBadGateway)
		})
	Middleware(newTest(&buf), mux).ServeHTTP(
		httptest.NewRecorder(),
		httptest.NewRequest("POST", "/orders", nil))
	var m map[string]any
	if err := json.Unmarshal(buf.Bytes(), &m); err != nil {
		t.Fatal(err)
	}
	for _, k := range []string{"time", "level", "msg",
		"service", "route", "status", "duration_ms"} {
		if _, ok := m[k]; !ok {
			t.Errorf("missing key %q", k)
		}
	}
	if m["route"] != "POST /orders" || m["level"] != "ERROR" {
		t.Errorf("route=%v level=%v", m["route"], m["level"])
	}
}
EOF
go test ./apps/internal/logx/     # ok
Lab 6 · Build (4/4)

Wire logx into orders and payments, then roll out

Goal: both services log the shared schema; business events carry order_id; every call passes the request context.

# apps/orders/main.go — edit these lines
import "obs-lab/apps/internal/logx"   // add to imports

log = logx.New("orders")  // was: slog.New(slog.NewJSONHandler(…))

// in create(): use the request context everywhere
log.ErrorContext(r.Context(), "payment failed",
	"order_id", o.ID, "err", err)
// new, after the order is stored:
log.InfoContext(r.Context(), "order created",
	"order_id", o.ID, "amount", o.Amount)

// in main(): the middleware wraps the mux DIRECTLY
err := http.ListenAndServe(":8080",
	logx.Middleware(log, mux))
// a Lab 2 metrics wrapper stays outside logx.Middleware

# apps/payments/main.go — same pattern
log := logx.New("payments")
log.ErrorContext(r.Context(), "charge failed",
	"order_id", req.OrderID)
log.InfoContext(r.Context(), "charged",
	"order_id", req.OrderID, "amount", req.Amount)
gofmt -l . ; go vet ./...  # remove unused imports if reported
for a in orders payments; do
  docker build --build-arg APP=$a -t shop/$a:lab .
  kind load docker-image shop/$a:lab --name obs
  kubectl -n shop rollout restart deploy/$a
done
kubectl -n shop rollout status deploy/orders
# request lines, typed fields
kubectl -n shop logs deploy/orders --since=1m \
  | jq -c 'select(.msg=="request")
      | {level, service, route, status, duration_ms}' \
  | head -2
# {"level":"INFO","service":"orders",
#  "route":"POST /orders","status":201,"duration_ms":64}

# business events have order_id
kubectl -n shop logs deploy/orders --since=1m \
  | jq -c 'select(.msg=="order created")
      | {order_id, amount}' | head -2

# top talkers by message
kubectl -n shop logs deploy/orders --since=5m \
  | jq -r .msg | sort | uniq -c | sort -rn

kubectl logs deploy/orders reads one pod of the Deployment; use -l app=orders --prefix for all replicas. trace_id stays absent until Lab 8 adds the OpenTelemetry SDK.

Lab 6 · Verify, break it, clean up

One schema, enforced by code

The logs are now ready for Loki (Lab 7) and for trace correlation (Lab 8).

Verify

  • go test ./apps/internal/logx/ passes.
  • Every orders line has time, level, msg, service; request lines add route, status (a number), duration_ms.
  • route is always a template (GET /orders/{id}), never /orders/o-123.
  • No /healthz lines at INFO: probes are DEBUG now.
  • With FAIL_RATE=0.3 on payments: orders logs ERROR request lines with status 502 and payment failed lines with order_id.

Break it

  • Volume: kubectl -n shop set env deploy/orders LOG_LEVEL=DEBUG → probe and scrape lines appear; compare --since=1m | wc -l before and after.
  • Leak: log "customer_email", "a@example.com" in create → not redacted: key-based redaction only knows email.
  • Non-JSON: add fmt.Println("starting") to main → jq stops with a parse error. Tolerant reading: jq -R 'fromjson? // {raw: .}'.
  • Wrong wrap order: put http.TimeoutHandler (it clones the request) between logx.Middleware and the mux → route becomes unmatched.
# keep logx: Labs 7–9 use it

# undo the break-it changes
kubectl -n shop set env \
  deploy/orders LOG_LEVEL-
kubectl -n shop set env \
  deploy/payments FAIL_RATE=0

# remove customer_email,
# fmt.Println and TimeoutHandler
# from the code, then rebuild,
# kind load and rollout restart
Lab 7 · Build (1/5)

Loki (monolithic) in logging

Goal: one Loki replica with filesystem storage, TSDB v13 schema, 3-day retention and a ruler that talks to the kps Alertmanager. Caches, gateway and canary are off to save memory.

cat > deploy/loki-values.yaml <<'EOF'
deploymentMode: Monolithic   # chart < 12.0.0: SingleBinary
loki:
  auth_enabled: false        # one tenant, named "fake"
  commonConfig: {replication_factor: 1}
  storage: {type: filesystem}
  schemaConfig:
    configs:
      - {from: "2024-04-01", store: tsdb, schema: v13,
         object_store: filesystem,
         index: {prefix: index_, period: 24h}}
  limits_config: {retention_period: 72h}
  compactor:
    retention_enabled: true
    delete_request_store: filesystem
  rulerConfig:
    storage: {type: local, local: {directory: /rules}}
    rule_path: /var/loki/rules-temp
    alertmanager_url: http://alertmanager-operated.monitoring:9093
    enable_api: true
singleBinary: {replicas: 1}
sidecar: {rules: {folder: /rules/fake}}  # rules of tenant "fake"
gateway: {enabled: false}
chunksCache: {enabled: false}
resultsCache: {enabled: false}
lokiCanary: {enabled: false}
test: {enabled: false}
EOF
helm repo add grafana-community \
  https://grafana-community.github.io/helm-charts
helm repo add grafana https://grafana.github.io/helm-charts
helm repo update

# key names move between chart majors: check yours
helm show values grafana-community/loki \
  | grep -n -E '^deploymentMode|^singleBinary:|^sidecar:'

helm upgrade --install loki grafana-community/loki \
  -n logging --create-namespace -f deploy/loki-values.yaml

kubectl -n logging get pods
# loki-0   2/2   Running    (loki + rules sidecar)

kubectl -n logging port-forward svc/loki 3100 &
curl -s localhost:3100/ready
# ready      (first ~15 s: "Ingester not ready: waiting …")
curl -s localhost:3100/loki/api/v1/status/buildinfo \
  | jq -r .version
# 3.7.x

curl -s localhost:3100/loki/api/v1/labels
# {"status":"success"}    (nothing ships logs yet)
Lab 7 · Build (2/5)

Alloy config: discover, process, push

Goal: each Alloy pod tails only the pods on its own node, parses the shop JSON, labels level and keeps trace_id as structured metadata.

cat > deploy/config.alloy <<'EOF'
discovery.kubernetes "pods" {
  role = "pod"
  selectors {
    role  = "pod"
    field = "spec.nodeName=" +
      coalesce(sys.env("HOSTNAME"), constants.hostname)
  }
}
discovery.relabel "pods" {
  targets = discovery.kubernetes.pods.targets
  rule {
    source_labels = ["__meta_kubernetes_namespace"]
    target_label  = "namespace"
  }
  rule {
    source_labels = ["__meta_kubernetes_pod_name"]
    target_label  = "pod"
  }
  rule {
    source_labels = ["__meta_kubernetes_pod_container_name"]
    target_label  = "container"
  }
  rule {
    source_labels = ["__meta_kubernetes_pod_label_app"]
    target_label  = "app"
  }
}
loki.source.kubernetes "pods" {
  targets    = discovery.relabel.pods.output
  forward_to = [loki.process.shop.receiver]
}
loki.process "shop" {
  stage.match {
    selector = "{namespace=\"shop\"}"
    stage.json {
      expressions = {level = "level", trace_id = "trace_id", ts = "time"}
    }
    stage.labels {
      values = {level = ""}
    }
    stage.structured_metadata {
      values = {trace_id = ""}
    }
    stage.timestamp {
      source = "ts"
      format = "RFC3339Nano"
    }
  }
  forward_to = [loki.write.default.receiver]
}
loki.write "default" {
  endpoint {
    url = "http://loki.logging:3100/loki/api/v1/push"
  }
}
EOF
Lab 7 · Build (3/5)

Install Alloy, add the data source, make some failures

Goal: logs flowing into Loki, Loki visible in Grafana, and payments failing 30 % of the time so there is something to find.

# config in its own ConfigMap: Helm never templates {{ }}
kubectl -n logging create configmap alloy-config \
  --from-file=config.alloy=deploy/config.alloy
helm upgrade --install alloy grafana/alloy -n logging \
  --set alloy.configMap.create=false \
  --set alloy.configMap.name=alloy-config \
  --set alloy.configMap.key=config.alloy
kubectl -n logging rollout status ds/alloy
# later edits: create … --dry-run=client -o yaml | kubectl apply -f -
# (the chart's config reloader picks them up)

# component graph and health: http://localhost:12345
kubectl -n logging port-forward ds/alloy 12345 &

cat > deploy/kps-loki.yaml <<'EOF'
grafana:
  additionalDataSources:
    - {name: Loki, type: loki, uid: loki,
       url: "http://loki.logging.svc:3100"}
EOF
helm upgrade kps prometheus-community/kube-prometheus-stack \
  -n monitoring --reuse-values --version $KPS_VER -f deploy/kps-loki.yaml

kubectl -n shop set env deploy/payments FAIL_RATE=0.3
# terminal client (or download it from the Loki releases)
brew install logcli
export LOKI_ADDR=http://localhost:3100

logcli labels
# app
# container
# level
# namespace
# pod
# service_name          (added by Loki, from app)

logcli series --analyze-labels '{namespace="shop"}'
# Total Streams:  ~10    (pods × levels)
# Unique Labels:  6
# Label Name    Unique Values  Found In Streams
# pod           5              …
# level         3              …

logcli query --since=10m --limit=3 -o raw \
  '{app="orders"} |= "payment failed"' | jq -c .
# {"time":"…","level":"ERROR","msg":"payment failed",
#  "service":"orders","order_id":"o-412",…}

Lab 9 replaces additionalDataSources with a list holding this Loki entry plus Tempo; keep uid: loki so links keep working.

Lab 7 · Build (4/5)

LogQL on the shop logs

Goal: run each query in Grafana → Explore → Loki (code mode) and explain its output before moving on.

# 1) failed payments, newest first
{namespace="shop", app="orders"} |= "payment failed"

# 2) request lines, typed filter, reshaped
{app="orders"} |= `"msg":"request"` | json
  | status >= 500
  | line_format "{{.route}} {{.status}} {{.duration_ms}}ms"
# POST /orders 502 71ms

# 3) level is already a label → the parsed one is renamed
{app="orders"} | json | keep level, level_extracted

# 4) find one order across both services
{namespace="shop"} |= "o-412" | json
  | line_format "{{.service}} {{.msg}}"
# payments charge failed
# orders payment failed
# 5) error lines per second per app
sum by (app) (
  rate({namespace="shop", level="ERROR"}[5m]))

# 6) share of 5xx requests (logs-as-RED)
sum(count_over_time({app="orders"}
      |= `"msg":"request"` | json | status >= 500 [5m]))
/
sum(count_over_time({app="orders"}
      |= `"msg":"request"` [5m]))
# ≈ 0.3

# 7) p95 request duration per route, from logs
quantile_over_time(0.95, {app="orders"}
  |= `"msg":"request"` | json | __error__=""
  | unwrap duration_ms [5m]) by (route)

# Query inspector → Stats: compare "bytes processed"
# for query 6 with and without the |= line filter
Lab 7 · Build (5/5)

A ruler alert from logs

Goal: a LogQL alerting rule delivered by the rules sidecar, evaluated by Loki, and visible in the kps Alertmanager.

cat > deploy/loki-rules.yaml <<'EOF'
apiVersion: v1
kind: ConfigMap
metadata:
  name: shop-log-rules
  namespace: logging
  labels: {loki_rule: "true"}   # watched by the sidecar
data:
  shop.yaml: |
    groups:
      - name: shop-logs
        interval: 1m
        rules:
          - alert: OrdersPaymentErrors
            expr: |
              sum(count_over_time(
                {namespace="shop", app="orders", level="ERROR"}
                  |= "payment failed" [5m])) > 10
            for: 1m
            labels: {severity: warning}
            annotations:
              summary: "orders: {{ $value }} payment failures in 5m"
          - alert: OrdersLogsMissing
            expr: absent_over_time({namespace="shop", app="orders"}[10m])
            for: 5m
            labels: {severity: warning}
EOF
kubectl apply -f deploy/loki-rules.yaml
kubectl -n logging exec loki-0 -c loki -- ls /rules/fake
# shop.yaml

# loaded by the ruler (re-sync within ~1 min)
curl -s localhost:3100/loki/api/v1/rules | head -3
# shop.yaml:               (namespace = file name)
#     - name: shop-logs
#       interval: 1m

# firing after ~2–3 min with FAIL_RATE=0.3
curl -s localhost:3100/prometheus/api/v1/alerts \
  | jq -r '.data.alerts[] | "\(.labels.alertname) \(.state)"'
# OrdersPaymentErrors firing

# in Alertmanager (port-forward 9093 from Lab 0)
amtool --alertmanager.url=http://localhost:9093 \
  alert query alertname=OrdersPaymentErrors

# troubleshooting
kubectl -n logging logs loki-0 -c loki-sc-rules | tail
kubectl -n logging logs loki-0 -c loki | grep -i ruler
Lab 7 · Verify, break it, clean up

Logs you can query, and the label that breaks Loki

Labs 8–10 build on this: keep Loki and Alloy running.

Verify

  • logcli labels lists namespace, app, container, pod, level, service_name — no order_id, no trace_id.
  • Query 2 shows only 5xx request lines; query 7 returns one series per route.
  • OrdersPaymentErrors fires in Alertmanager; with FAIL_RATE=0 it resolves within ~6 min.
  • Alloy UI (:12345) shows every component healthy.
  • kube-system lines arrive too, unparsed (outside stage.match).

Break it

  • Cardinality: add order_id = "order_id" to stage.json and order_id = "" to stage.labels → logcli series --analyze-labels '{app="orders"}' shows hundreds of streams within minutes. Revert it.
  • Empty selector: {app=~".*"} is rejected: a selector needs one matcher that cannot match empty.
  • Wrong parser: {app="orders"} | logfmt | status >= 500 → no results; | __error__!="" shows why.
  • Loki down: kubectl -n logging scale sts/loki --replicas=0 for 1 min → Alloy logs push errors and retries; scale back to 1 and look for lines from the gap.
# keep Loki + Alloy for Labs 8–10
kubectl -n shop set env \
  deploy/payments FAIL_RATE=0

# full removal (after Lab 10):
kubectl delete -f \
  deploy/loki-rules.yaml
helm uninstall alloy -n logging
helm uninstall loki -n logging
kubectl -n logging delete pvc \
  --all
kubectl delete ns logging
# and drop the Loki entry from
# grafana.additionalDataSources
Lab 8 · Build (1/5)

otel-gateway: a Collector that prints what it receives

Goal: a Collector in otel that accepts OTLP and writes every span to its own log (debug exporter). Tempo replaces the debug exporter in Lab 9.

cat > deploy/otel-gateway.yaml <<'EOF'
mode: deployment
fullnameOverride: otel-gateway   # Service otel-gateway.otel
replicaCount: 1
image:
  repository: otel/opentelemetry-collector-contrib
  # tag defaults to the chart's appVersion
config:
  # merged into the chart defaults, which already define
  # receivers.otlp (4317 gRPC, 4318 HTTP), memory_limiter,
  # batch, exporters.debug and the health_check extension
  exporters:
    debug: {verbosity: detailed}  # print every span
  service:
    pipelines:
      traces:
        receivers: [otlp]  # lists replace: drops jaeger, zipkin
EOF

helm repo add open-telemetry \
  https://open-telemetry.github.io/opentelemetry-helm-charts
helm repo update
helm upgrade --install otel-gateway \
  open-telemetry/opentelemetry-collector \
  -n otel --create-namespace -f deploy/otel-gateway.yaml
kubectl -n otel rollout status deploy/otel-gateway
kubectl -n otel get svc otel-gateway
# otel-gateway  ClusterIP  …  …,4317/TCP,4318/TCP,…
kubectl -n otel logs deploy/otel-gateway | grep -i ready
# … Everything is ready. Begin running and processing data.

# smoke test: one span by hand over OTLP/HTTP + JSON
cat > deploy/span.json <<'EOF'
{"resourceSpans": [{
  "resource": {"attributes": [{"key": "service.name",
    "value": {"stringValue": "curl"}}]},
  "scopeSpans": [{"spans": [{
    "traceId": "5b8efff798038103d269b633813fc60c",
    "spanId": "eee19b7ec3c1b174", "name": "smoke", "kind": 2,
    "startTimeUnixNano": "1762560000000000000",
    "endTimeUnixNano":   "1762560000250000000"}]}]}]}
EOF
kubectl -n otel port-forward svc/otel-gateway 4318 &
curl -s localhost:4318/v1/traces \
  -H 'Content-Type: application/json' -d @deploy/span.json
# {"partialSuccess":{}}            (HTTP 200)

kubectl -n otel logs deploy/otel-gateway \
  | grep -B4 -A1 'Name *: smoke'
#     Trace ID       : 5b8efff798038103d269b633813fc60c
#     Name           : smoke
#     Kind           : Server
Lab 8 · Build (2/5)

otelx: SDK setup in one function

Goal: a shared package (like logx in Lab 6), configured entirely from OTEL_* variables.

mkdir -p apps/internal/otelx && cat > apps/internal/otelx/setup.go <<'EOF'
package otelx

import (
	"context"
	"go.opentelemetry.io/otel"
	"go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc"
	"go.opentelemetry.io/otel/propagation"
	"go.opentelemetry.io/otel/sdk/resource"
	sdktrace "go.opentelemetry.io/otel/sdk/trace"
)

// Setup registers a TracerProvider and the W3C propagators.
func Setup(ctx context.Context) (func(context.Context) error, error) {
	res, err := resource.New(ctx,
		resource.WithFromEnv(), resource.WithTelemetrySDK())
	if err != nil {
		return nil, err
	}
	exp, err := otlptracegrpc.New(ctx) // OTEL_EXPORTER_OTLP_ENDPOINT
	if err != nil {
		return nil, err
	}
	tp := sdktrace.NewTracerProvider(
		sdktrace.WithBatcher(exp), sdktrace.WithResource(res))
	otel.SetTracerProvider(tp)
	otel.SetTextMapPropagator(propagation.NewCompositeTextMapPropagator(
		propagation.TraceContext{}, propagation.Baggage{}))
	return tp.Shutdown, nil
}
EOF

What each piece does

  • resource.WithFromEnv() reads OTEL_SERVICE_NAME and OTEL_RESOURCE_ATTRIBUTES (set in step 5).
  • otlptracegrpc.New reads OTEL_EXPORTER_OTLP_ENDPOINT; the http:// scheme means plaintext gRPC.
  • WithBatcher: spans are queued and sent in batches every 5 s, off the request path.
  • Default sampler ParentBased(AlwaysSample): every trace is kept for now; Lab 9 samples in the Collector instead.
  • The composite propagator makes otelhttp inject and extract traceparent and baggage.
  • The returned Shutdown flushes the queue; step 4 calls it on SIGTERM.
Lab 8 · Build (3/5)

otelx: server wrapper, route naming, traced client

Goal: one wrapper outside the handler chain, one inside it, and a client that propagates context.

cat > apps/internal/otelx/http.go <<'EOF'
package otelx

import (
	"net/http"
	"strings"
	"time"

	"go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp"
	"go.opentelemetry.io/otel/attribute"
	"go.opentelemetry.io/otel/trace"
)

func Handler(h http.Handler, service string) http.Handler {
	return otelhttp.NewHandler(h, service,
		otelhttp.WithFilter(func(r *http.Request) bool {
			return r.URL.Path != "/healthz" && r.URL.Path != "/metrics"
		}))
}

func RouteSpan(next http.Handler) http.Handler {
	return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
		next.ServeHTTP(w, r)
		if _, route, ok := strings.Cut(r.Pattern, " "); ok {
			span := trace.SpanFromContext(r.Context())
			span.SetName(r.Pattern)
			span.SetAttributes(attribute.String("http.route", route))
		}
	})
}

var Client = &http.Client{Timeout: 2 * time.Second,
	Transport: otelhttp.NewTransport(http.DefaultTransport)}
EOF

Why this shape

  • Handler is the outermost wrapper; RouteSpan wraps the mux, innermost.
  • otelhttp starts the SERVER span before routing, so it cannot know the route yet: "orders" is only the fallback name.
  • The mux fills r.Pattern while routing; RouteSpan runs after it and renames the span POST /orders with http.route=/orders — low-cardinality names (ch 8, section 31).
  • Recent otelhttp releases also read r.Pattern; doing it here keeps the result independent of the version you get.
  • WithFilter returning false skips tracing for probes and scrapes.
  • Client wraps the default transport: one CLIENT span per request, traceparent injected — but only for requests built with a context.
Lab 8 · Build (4/5)

Wire otelx into orders and payments

Goal: otelhttp outside the Lab 2 metrics and Lab 6 logging middleware, so both see the span; the call to payments carries the context; spans are flushed on SIGTERM.

go get go.opentelemetry.io/otel go.opentelemetry.io/otel/sdk \
  go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc \
  go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp
go mod tidy
// apps/orders/main.go — imports: "context", "os/signal", "syscall",
//   "obs-lab/apps/internal/otelx"
client = otelx.Client        // in the var block, was &http.Client{…}

// create(): the call to payments must carry the request context
req, _ := http.NewRequestWithContext(r.Context(), "POST",
	payURL, bytes.NewReader(body))
req.Header.Set("Content-Type", "application/json")
resp, err := client.Do(req)  // was client.Post(…)
// main(): before building the mux
ctx, stop := signal.NotifyContext(context.Background(),
	os.Interrupt, syscall.SIGTERM)
defer stop()
shutdown, err := otelx.Setup(ctx)
if err != nil {
	log.Error("otel setup failed", "err", err)
	os.Exit(1)
}
// main(): replace the old ListenAndServe block
h := otelx.Handler(
	metrics.Middleware(logx.Middleware(log, otelx.RouteSpan(mux))),
	"orders")
srv := &http.Server{Addr: ":8080", Handler: h}
go func() {
	<-ctx.Done()                   // SIGTERM from Kubernetes
	srv.Shutdown(context.Background())
}()
log.Info("orders listening", "addr", ":8080")
err = srv.ListenAndServe()       // http.ErrServerClosed on SIGTERM
log.Info("server stopped", "err", err)
c, cancel := context.WithTimeout(context.Background(),
	5*time.Second)
defer cancel()
shutdown(c)                      // flush the last batch of spans

// apps/payments/main.go: the same main() changes, with
// "payments" as the name. No client change: it calls nobody.
// optional, in payments charge(): business detail on the span
trace.SpanFromContext(r.Context()).SetAttributes(
	attribute.String("shop.order.id", req.OrderID))

Order matters: otelx.Handler(metrics(logx(RouteSpan(mux)))). Lab 6 traceHandler and Lab 9 exemplars read the span from the request context, so they must run inside otelhttp.

Lab 8 · Build (5/5)

Roll out and read one trace

Goal: the same trace ID on orders' SERVER and CLIENT spans and on payments' SERVER span, and in the request logs.

gofmt -l . ; go vet ./...

for a in orders payments; do
  docker build --build-arg APP=$a -t shop/$a:lab .
  kind load docker-image shop/$a:lab --name obs
  # set env changes the pod template → new pods, new image
  kubectl -n shop set env deploy/$a OTEL_SERVICE_NAME=$a \
    OTEL_RESOURCE_ATTRIBUTES=deployment.environment.name=lab \
    OTEL_EXPORTER_OTLP_ENDPOINT=http://otel-gateway.otel:4317
done
# after later code changes: rollout restart (Lab 0 convention)
kubectl -n shop rollout status deploy/orders
kubectl -n shop rollout status deploy/payments

# loadgen keeps calling orders; add some failures
kubectl -n shop set env deploy/payments FAIL_RATE=0.2

# request logs now carry the trace (Lab 6 traceHandler)
kubectl -n shop logs deploy/orders --since=1m \
  | jq -c 'select(.msg=="request")
      | {route, status, trace_id}' | head -2
# {"route":"POST /orders","status":201,
#  "trace_id":"7c3f9d0e5b1a4c2f8e6d3b2a1f0e9a1e"}
kubectl -n otel logs deploy/otel-gateway --since=1m \
  | grep -E 'service.name|Trace ID|Parent ID|^ +ID|Name +:|Kind' \
  | head -18
# one request, three spans (IDs shortened):
#      -> service.name: Str(payments)
#     Trace ID       : 7c3f…9a1e
#     Parent ID      : 5d2b…01c4     ← orders' CLIENT span
#     ID             : 9e40…77aa
#     Name           : POST /charge
#     Kind           : Server
#      -> service.name: Str(orders)
#     Trace ID       : 7c3f…9a1e
#     Parent ID      : 1a8f…e302     ← orders' SERVER span
#     ID             : 5d2b…01c4
#     Name           : POST           (older otelhttp: HTTP POST)
#     Kind           : Client
#     Trace ID       : 7c3f…9a1e
#     Parent ID      :                (root)
#     ID             : 1a8f…e302
#     Name           : POST /orders
#     Kind           : Server

# full detail of one trace: attributes, status, resource
kubectl -n otel logs deploy/otel-gateway --since=1m \
  | grep -A40 'Trace ID *: 7c3f' | less
Lab 8 · Verify, break it, clean up

Three spans, one trace — and four ways to split it

Run each break, find the symptom in the debug output or the app logs, explain it, then revert.

Verify

  • payments' SERVER span has orders' CLIENT span as parent; all three share one trace ID.
  • Resource shows service.name, deployment.environment.name=lab, telemetry.sdk.language=go.
  • Failed charges: payments SERVER span has status Error (502); so does orders SERVER.
  • No spans named GET /healthz or GET /metrics.
  • kubectl -n shop rollout restart deploy/orders → no export errors in the old pods' last logs.

Break it

  • No propagator: remove SetTextMapPropagator in setup.go → payments spans get new trace IDs and an empty Parent ID.
  • No context: go back to client.Post(…) → orders' CLIENT span becomes the root of a new trace; payments joins that one, not the order's.
  • Wrong port: set OTEL_EXPORTER_OTLP_ENDPOINT=http://otel-gateway.otel:4318 → export errors in the orders logs every few seconds; requests still return 201.
  • Collector down: kubectl -n otel scale deploy/otel-gateway --replicas=0 → the app keeps serving; spans are dropped once the SDK queue (2048) is full.
# revert the breaks, then:
kubectl -n otel scale \
  deploy/otel-gateway --replicas=1
kubectl -n shop set env \
  deploy/payments FAIL_RATE=0

# keep otel-gateway and the SDK:
# Lab 9 adds Tempo, tail
# sampling and span metrics

# full removal (after Lab 9):
helm uninstall otel-gateway -n otel
kubectl delete ns otel
kubectl -n shop set env \
  deploy/orders deploy/payments \
  OTEL_EXPORTER_OTLP_ENDPOINT-
Lab 9 · Build (1/3)

Tempo in tracing, and Grafana wired for traces

Goal: traces from the shop app land in Tempo (monolithic, local storage), and Grafana can jump between metrics, traces and logs.

# 1) Tempo, monolithic mode (no Kafka needed)
helm repo add grafana-community \
  https://grafana-community.github.io/helm-charts
helm repo update   # check keys/appVersion:
helm show values grafana-community/tempo | less
cat > deploy/tempo-values.yaml <<'EOF'
tempo:
  storage:
    trace:
      backend: local
  receivers:
    otlp:
      protocols:
        grpc: {endpoint: 0.0.0.0:4317}
        http: {endpoint: 0.0.0.0:4318}
EOF
helm upgrade --install tempo grafana-community/tempo \
  -n tracing --create-namespace -f deploy/tempo-values.yaml
kubectl -n tracing rollout status statefulset/tempo
kubectl -n tracing port-forward svc/tempo 3200 &
curl -s localhost:3200/ready       # → ready

# 2) kps: exemplars, remote write receiver, data
#    sources (file on the right), then:
helm upgrade kps prometheus-community/kube-prometheus-stack \
  -n monitoring --reuse-values --version $KPS_VER -f deploy/kps-tracing.yaml
# deploy/kps-tracing.yaml — the list below replaces
# Lab 7's; keep Lab 7's Loki uid/URL if they differ
prometheus:
  prometheusSpec:
    enableFeatures: [exemplar-storage]
    enableRemoteWriteReceiver: true
grafana:
  sidecar:
    datasources:
      exemplarTraceIdDestinations:
        {datasourceUid: tempo, traceIdLabelName: trace_id}
  additionalDataSources:
    - name: Tempo
      type: tempo
      uid: tempo
      url: http://tempo.tracing.svc:3200
      jsonData:
        tracesToLogsV2: {datasourceUid: loki,
          filterByTraceID: true,
          spanStartTimeShift: -5m, spanEndTimeShift: 5m}
        serviceMap: {datasourceUid: prometheus}
    - name: Loki
      type: loki
      uid: loki
      url: http://loki.logging.svc:3100
      jsonData:
        derivedFields:
          - {name: TraceID, datasourceUid: tempo,
             matcherRegex: '"trace_id":"(\w+)"',
             url: '$${__value.raw}'}
Lab 9 · Build (2/3)

otel-gateway: span metrics first, then tail sampling, then Tempo

Goal: RED metrics from 100 % of spans; Tempo keeps errors, slow traces and a 10 % baseline.

config:     # deploy/gateway-tracing.yaml
  connectors:
    spanmetrics:
      histogram: {explicit: {buckets:
        [5ms, 25ms, 100ms, 250ms, 500ms, 1s, 2.5s]}}
      dimensions: [{name: http.route}]
    servicegraph: {}
    forward/sample: {}
  processors:
    tail_sampling:
      decision_wait: 10s
      num_traces: 20000
      policies:
        - {name: errors, type: status_code,
           status_code: {status_codes: [ERROR]}}
        - {name: slow, type: latency,
           latency: {threshold_ms: 500}}
        - {name: baseline, type: probabilistic,
           probabilistic: {sampling_percentage: 10}}
  exporters:
    otlp/tempo: {endpoint: "tempo.tracing.svc:4317",
                 tls: {insecure: true}}
    prometheusremotewrite:
      endpoint: http://prometheus-operated.monitoring.svc:9090/api/v1/write
  service:
    pipelines:
      traces:
        receivers: [otlp]
        processors: [memory_limiter]
        exporters: [spanmetrics, servicegraph, forward/sample]
      traces/sampled:
        receivers: [forward/sample]
        processors: [tail_sampling, batch]
        exporters: [otlp/tempo]
      metrics/derived:
        receivers: [spanmetrics, servicegraph]
        processors: [batch]
        exporters: [prometheusremotewrite]

# apply (keeps image, mode and the rest of Lab 8's values)
helm upgrade otel-gateway open-telemetry/opentelemetry-collector \
  -n otel --reuse-values -f deploy/gateway-tracing.yaml
kubectl -n otel logs -l app.kubernetes.io/instance=otel-gateway \
  | grep -i -E 'error|Everything is ready'

# keep exactly ONE gateway replica: tail sampling needs all
# spans of a trace on the same instance (ch 9, section 34)
kubectl -n otel get deploy -l app.kubernetes.io/instance=otel-gateway

Lists in Helm values replace, maps merge: the traces pipeline above replaces Lab 8's (which exported to debug). memory_limiter and batch come from the chart's default config.

Lab 9 · Build (3/3)

Exemplars from orders, then inject a fault

Goal: latency samples carry the trace ID of a request, so Grafana shows clickable dots on the p95 graph.

// apps/orders/metrics.go — inside the otelhttp-wrapped handler
import (
    "github.com/prometheus/client_golang/prometheus"
    "github.com/prometheus/client_golang/prometheus/promhttp"
    "go.opentelemetry.io/otel/trace"
)

func observe(r *http.Request, route string, d time.Duration) {
    h := reqDuration.WithLabelValues(r.Method, route)
    sc := trace.SpanContextFromContext(r.Context())
    if sc.IsSampled() {
        h.(prometheus.ExemplarObserver).ObserveWithExemplar(
            d.Seconds(),
            prometheus.Labels{"trace_id": sc.TraceID().String()})
        return
    }
    h.Observe(d.Seconds())
}

// exemplars are only served in OpenMetrics format
mux.Handle("/metrics", promhttp.HandlerFor(reg,
    promhttp.HandlerOpts{EnableOpenMetrics: true}))
# rebuild and roll out orders
docker build --build-arg APP=orders -t shop/orders:lab . \
  && kind load docker-image shop/orders:lab --name obs \
  && kubectl -n shop rollout restart deploy/orders

# exemplars are exposed?
kubectl -n shop port-forward svc/orders 8080 &
curl -s -H 'Accept: application/openmetrics-text' \
  localhost:8080/metrics | grep -m2 'trace_id'
# http_request_duration_seconds_bucket{...,le="0.1"} 812 \
#   # {trace_id="4bf92f35..."} 0.043 1.762e+09

# inject errors and latency in payments
kubectl -n shop set env deploy/payments \
  FAIL_RATE=0.2 EXTRA_LATENCY_MS=400

# Grafana: localhost:3000 (port-forward svc/kps-grafana 3000:80)
# Explore → Prometheus, Exemplars on:
histogram_quantile(0.95, sum by (le) (
  rate(http_request_duration_seconds_bucket{job="orders"}[5m])))
Lab 9 · Verify, break it, clean up

Find the failing hop, then break the pipeline on purpose

Use TraceQL, exemplars and the log link together. Each break shows one rule from chapter 9.

Verify

  • Explore → Tempo: { resource.service.name = "orders" } >> { resource.service.name = "payments" && status = error } returns traces.
  • Click an exemplar dot on the p95 graph → a trace with a ~400 ms payments span.
  • In the trace, "Logs for this span" opens Loki filtered by the trace ID; in Loki, the TraceID link opens the trace.
  • Prometheus: sum by (status_code) (rate(traces_span_metrics_calls_total{service_name="payments"}[5m])) shows ~20 % errors.
  • Tempo holds far fewer OK traces than requests sent (10 % baseline).

Break it

  • Sample before counting: move tail_sampling into the first traces pipeline → span-metric error ratio jumps far above 20 %.
  • decision_wait too short: set 1s with EXTRA_LATENCY_MS=2000 → traces arrive in pieces, root spans missing.
  • Two replicas: scale the gateway to 2 → some traces lose spans or are decided twice.
  • No OpenMetrics: drop EnableOpenMetrics → /metrics shows no exemplars and the dots vanish.
# reset the fault knobs
kubectl -n shop set env deploy/payments \
  FAIL_RATE=0 EXTRA_LATENCY_MS=0

# keep Tempo + gateway for
# Labs 10 and 13. Remove Tempo:
helm uninstall tempo -n tracing
kubectl delete namespace tracing
# then point the gateway back to debug:
# re-apply Lab 8's values
# (without gateway-tracing.yaml)
Lab 10 · Build (1/2)

A dashboard generator: RED per service, USE per pod

Goal: dashboards are code. One Python script renders three dashboards from templates; nothing is built by hand in the UI.

# dashboards/gen.py — RED per service + one USE dashboard
import json, pathlib
DS = {"type": "prometheus", "uid": "prometheus"}   # kps default uid
OUT = pathlib.Path(__file__).parent / "out"
OUT.mkdir(exist_ok=True)
RI = "[$__rate_interval]"

def ts(title, expr, unit, x, y, legend="{{route}}"):
    return {"type": "timeseries", "title": title, "datasource": DS,
            "gridPos": {"x": x, "y": y, "w": 8, "h": 8},
            "fieldConfig": {"defaults": {"unit": unit}},
            "targets": [{"refId": "A", "expr": expr,
                         "legendFormat": legend}]}

def stat(title, expr, unit, x, red_at):
    steps = [{"color": "green", "value": None},
             {"color": "red", "value": red_at}]
    return {"type": "stat", "title": title, "datasource": DS,
            "gridPos": {"x": x, "y": 8, "w": 8, "h": 4},
            "fieldConfig": {"defaults": {"unit": unit, "thresholds":
                            {"mode": "absolute", "steps": steps}}},
            "targets": [{"refId": "A", "expr": expr}]}
def red(svc):
    s = f'job="{svc}", route=~"$route"'
    rate = f'sum by (route) (rate(http_requests_total{{{s}}}{RI}))'
    err = (f'sum by (route) (rate(http_requests_total'
           f'{{{s}, code=~"5.."}}{RI})) / {rate}')
    p95 = ('histogram_quantile(0.95, sum by (le, route) (rate('
           f'http_request_duration_seconds_bucket{{{s}}}{RI})))')
    tot_err = (f'sum(rate(http_requests_total{{{s}, code=~"5.."}}{RI}))'
               f' / sum(rate(http_requests_total{{{s}}}{RI}))')
    return {
        "uid": f"red-{svc}", "title": f"Shop / {svc} RED",
        "tags": ["shop", "red"], "time": {"from": "now-1h", "to": "now"},
        "templating": {"list": [{
            "name": "route", "type": "query", "datasource": DS,
            "query": f'label_values(http_requests_total{{job="{svc}"}}, route)',
            "multi": True, "includeAll": True, "refresh": 2}]},
        "panels": [ts("Requests/s", rate, "reqps", 0, 0),
                   ts("Error ratio", err, "percentunit", 8, 0),
                   ts("p95 latency", p95, "s", 16, 0),
                   stat("Error ratio (all routes)", tot_err,
                        "percentunit", 0, 0.05)]}
# use() is on the next slide
for name, d in [("red-orders", red("orders")),
                ("red-payments", red("payments")),
                ("use-shop", use())]:
    (OUT / f"{name}.json").write_text(json.dumps(d, indent=1))

The stat threshold (red at 5 %) should match the error-ratio alert from Lab 4.

Lab 10 · Build (2/2)

USE panels, then load everything through the kps sidecar

Goal: a USE dashboard for the shop pods, and all three dashboards loaded from labelled ConfigMaps.

# dashboards/gen.py (continued; define before the loop)
def use():
    ns = 'namespace="shop", container!=""'
    p = "{{pod}}"
    cpu = ('sum by (pod) (rate('
           f'container_cpu_usage_seconds_total{{{ns}}}{RI}))')
    thr = ('sum by (pod) (rate(container_cpu_cfs_throttled_periods_total'
           f'{{{ns}}}{RI})) / sum by (pod) (rate('
           f'container_cpu_cfs_periods_total{{{ns}}}{RI}))')
    mem = f'sum by (pod) (container_memory_working_set_bytes{{{ns}}})'
    rst = ('sum by (pod) (increase(kube_pod_container_status_'
           'restarts_total{namespace="shop"}[1h]))')
    return {"uid": "use-shop", "title": "Shop / USE (pods)",
            "tags": ["shop", "use"],
            "time": {"from": "now-1h", "to": "now"},
            "panels": [
                ts("CPU cores used (U)", cpu, "short", 0, 0, p),
                ts("CPU throttled ratio (S)", thr, "percentunit", 8, 0, p),
                ts("Memory working set (U)", mem, "bytes", 16, 0, p),
                ts("Restarts in 1h (E)", rst, "short", 0, 8, p)]}
# render and load
python3 dashboards/gen.py && ls dashboards/out
# red-orders.json  red-payments.json  use-shop.json

for f in dashboards/out/*.json; do
  n=$(basename "$f" .json)
  kubectl -n monitoring create configmap "dash-$n" \
    --from-file="$n.json=$f" --dry-run=client -o yaml \
  | kubectl label --local -f - grafana_dashboard=1 -o yaml \
  | kubectl apply -f -
done

# check through the Grafana API
kubectl -n monitoring port-forward svc/kps-grafana 3000:80 &
PW=$(kubectl -n monitoring get secret kps-grafana \
  -o jsonpath='{.data.admin-password}' | base64 -d)
curl -s -u "admin:$PW" 'localhost:3000/api/search?tag=shop' \
  | jq -r '.[].title'
# Shop / orders RED
# Shop / payments RED
# Shop / USE (pods)

# commit gen.py and out/*.json; review diffs in PRs

The throttling panel stays empty while shop pods have no CPU limit: CFS throttling only happens under a quota. The break step below adds one.

Lab 10 · Verify, break it, clean up

Dashboards that react to faults

Inject each fault and confirm the right panel moves. If a panel does not react, the panel is wrong.

Verify

  • kubectl -n shop set env deploy/payments FAIL_RATE=0.2 → orders "Error ratio" rises on /orders; the stat turns red above 5 %.
  • EXTRA_LATENCY_MS=400 → orders p95 rises by roughly 0.4 s on /orders only.
  • Choose $route = one route: every RED panel follows the variable.
  • USE: memory and CPU per orders/payments pod are drawn; restarts stay at 0.

Break it

  • Throttle it: kubectl -n shop set resources deploy/orders --limits=cpu=50m → throttled ratio climbs, p95 rises, CPU flat at 0.05 cores.
  • UI edit: change a panel and press Save → Grafana refuses (provisioned dashboard). Fix in gen.py instead.
  • Lost label: kubectl -n monitoring label cm dash-use-shop grafana_dashboard- → the dashboard disappears.
  • Wrong order: change p95 to histogram_quantile(0.95, rate(...)) without sum by (le, route) → one line per pod and instance; compare.
# reset faults and limits
kubectl -n shop set env deploy/payments \
  FAIL_RATE=0 EXTRA_LATENCY_MS=0
# remove the CPU limit: re-apply the
# original manifest
kubectl apply -f deploy/shop.yaml

# delete ONLY the lab ConfigMaps by name.
# Never: delete cm -l grafana_dashboard=1
# — that also deletes kps's built-in
# dashboards.
kubectl -n monitoring delete configmap \
  dash-red-orders dash-red-payments \
  dash-use-shop
Lab 11 · Build (1/4)

A Lambda that emits EMF: code, log group, role

Goal: CloudWatch metrics from structured logs, then alarms and a dashboard on them. Everything runs in ap-northeast-1.

# app.py — one EMF line per request, slog-style fields (ch 6)
import json, os, random, time, uuid
FAIL = float(os.environ.get("FAIL_RATE", "0"))

def handler(event, context):
    t0, route = time.time(), event.get("route", "POST /orders")
    fail = random.random() < FAIL
    time.sleep(random.uniform(0.02, 0.12) + (0.4 if fail else 0))
    ms = round((time.time() - t0) * 1000, 1)
    status = 502 if fail else 201
    print(json.dumps({"_aws": {
        "Timestamp": int(time.time() * 1000),
        "CloudWatchMetrics": [{"Namespace": "ObsLab",
          "Dimensions": [["service", "route"]],
          "Metrics": [{"Name": "Latency", "Unit": "Milliseconds"},
                      {"Name": "Errors", "Unit": "Count"},
                      {"Name": "Requests", "Unit": "Count"}]}]},
        "service": "orders", "route": route,
        "Latency": ms, "Errors": int(fail), "Requests": 1,
        "level": "ERROR" if fail else "INFO",
        "msg": "payment failed" if fail else "order created",
        "order_id": str(uuid.uuid4()), "status": status,
        "duration_ms": ms}))
    return {"status": status}
export AWS_REGION=ap-northeast-1
ACCT=$(aws sts get-caller-identity --query Account --output text)

# log group first, with retention (otherwise Lambda creates
# /aws/lambda/<fn> that never expires)
aws logs create-log-group --log-group-name /obs-lab/orders
aws logs put-retention-policy --log-group-name /obs-lab/orders \
  --retention-in-days 7

# execution role: logs + X-Ray (active tracing, used in Lab 12)
aws iam create-role --role-name obs-lab-orders-role \
  --assume-role-policy-document '{"Version":"2012-10-17",
   "Statement":[{"Effect":"Allow","Action":"sts:AssumeRole",
   "Principal":{"Service":"lambda.amazonaws.com"}}]}'
P=arn:aws:iam::aws:policy
aws iam attach-role-policy --role-name obs-lab-orders-role \
  --policy-arn $P/service-role/AWSLambdaBasicExecutionRole
aws iam attach-role-policy --role-name obs-lab-orders-role \
  --policy-arn $P/AWSXRayDaemonWriteAccess
Lab 11 · Build (2/4)

Deploy, generate traffic, add a metric filter and a static alarm

Step 2: ~10 minutes of traffic, then a metric filter and a static M-of-N alarm. LogFormat=Text keeps each print line intact, which EMF needs.

zip fn.zip app.py
aws lambda create-function --function-name obs-lab-orders \
  --runtime python3.13 --handler app.handler \
  --zip-file fileb://fn.zip --timeout 10 \
  --role arn:aws:iam::$ACCT:role/obs-lab-orders-role \
  --environment 'Variables={FAIL_RATE=0.05}' \
  --logging-config LogFormat=Text,LogGroup=/obs-lab/orders \
  --tracing-config Mode=Active
# "role cannot be assumed"? a new role needs a few seconds: retry

# traffic: second terminal, leave it running
while true; do
  aws lambda invoke --function-name obs-lab-orders \
    --cli-binary-format raw-in-base64-out \
    --payload '{"route":"POST /orders"}' /dev/null >/dev/null
done

# after a few minutes
aws cloudwatch list-metrics --namespace ObsLab \
  --query 'Metrics[].[MetricName,Dimensions[].Value]'
# metric filter: count ERROR events
aws logs put-metric-filter --log-group-name /obs-lab/orders \
  --filter-name orders-error-logs \
  --filter-pattern '{ $.level = "ERROR" }' \
  --metric-transformations metricName=ErrorLogs,\
metricNamespace=ObsLab,metricValue=1,defaultValue=0

TOPIC=$(aws sns create-topic --name obs-lab-alerts \
  --query TopicArn --output text)
aws sns subscribe --topic-arn "$TOPIC" --protocol email \
  --notification-endpoint you@example.com  # confirm the mail

# static: >= 3 errors/min in 3 of 5 minutes (no actions)
aws cloudwatch put-metric-alarm \
  --alarm-name obs-lab-orders-errors \
  --namespace ObsLab --metric-name Errors \
  --dimensions Name=service,Value=orders \
    "Name=route,Value=POST /orders" \
  --statistic Sum --period 60 --evaluation-periods 5 \
  --datapoints-to-alarm 3 --threshold 3 \
  --comparison-operator GreaterThanOrEqualToThreshold \
  --treat-missing-data notBreaching
Lab 11 · Build (3/4)

Anomaly and composite alarms, Metrics Insights

Step 3: an anomaly band on p90 latency, a composite alarm that is the only one that notifies, and a SQL query over the metrics.

cat > anom.json <<'EOF'
[{"Id": "m1", "ReturnData": true, "MetricStat": {
   "Metric": {"Namespace": "ObsLab", "MetricName": "Latency",
     "Dimensions": [{"Name": "service", "Value": "orders"},
                    {"Name": "route", "Value": "POST /orders"}]},
   "Period": 60, "Stat": "p90"}},
 {"Id": "band", "Expression": "ANOMALY_DETECTION_BAND(m1, 2)"}]
EOF
aws cloudwatch put-metric-alarm \
  --alarm-name obs-lab-orders-latency-anomaly \
  --evaluation-periods 3 --datapoints-to-alarm 3 \
  --comparison-operator GreaterThanUpperThreshold \
  --threshold-metric-id band --treat-missing-data notBreaching \
  --metrics file://anom.json

aws cloudwatch put-composite-alarm \
  --alarm-name obs-lab-orders-page \
  --alarm-rule 'ALARM("obs-lab-orders-errors") AND
    ALARM("obs-lab-orders-latency-anomaly")' \
  --alarm-actions "$TOPIC" --ok-actions "$TOPIC"
# top routes by errors (Metrics Insights)
Q="SELECT SUM(Errors) FROM SCHEMA(ObsLab, route, service) \
GROUP BY route ORDER BY SUM() DESC LIMIT 5"
jq -n --arg q "$Q" \
  '[{Id: "q1", Period: 300, Expression: $q}]' > mi.json
aws cloudwatch get-metric-data \
  --metric-data-queries file://mi.json \
  --start-time $(date -u -v-30M +%FT%TZ) \
  --end-time $(date -u +%FT%TZ) \
  --query 'MetricDataResults[].[Label,Values[0]]'
# Linux: date -u -d '-30 min' +%FT%TZ

# a second route = new custom metrics (the ch 11 cost lesson)
aws lambda invoke --function-name obs-lab-orders \
  --cli-binary-format raw-in-base64-out \
  --payload '{"route":"GET /orders/{id}"}' /dev/null
# run the query again: two rows

# the anomaly model needs history: with minutes of data the
# band is rough; in production give it days to weeks
Lab 11 · Build (4/4)

A dashboard as JSON

Step 4: latency percentiles, requests and errors, recent error logs and the composite alarm on one page, deployed from a file.

cat > dash.json <<EOF
{"widgets": [
 {"type":"metric","x":0,"y":0,"width":12,"height":6,
  "properties":{"title":"latency p50 / p90",
   "region":"ap-northeast-1","view":"timeSeries","period":60,
   "metrics":[["ObsLab","Latency","service","orders",
     "route","POST /orders",{"stat":"p50"}],
    ["...",{"stat":"p90"}]]}},
 {"type":"metric","x":12,"y":0,"width":12,"height":6,
  "properties":{"title":"requests and errors / min",
   "region":"ap-northeast-1","stat":"Sum","period":60,
   "metrics":[["ObsLab","Requests","service","orders",
     "route","POST /orders"],
    [".","Errors",".",".",".","."]]}},
 {"type":"log","x":0,"y":6,"width":16,"height":6,
  "properties":{"title":"recent errors",
   "region":"ap-northeast-1","view":"table",
   "query":"SOURCE '/obs-lab/orders' | filter level = 'ERROR' | limit 20"}},
 {"type":"alarm","x":16,"y":6,"width":8,"height":6,
  "properties":{"title":"page","alarms":[
   "arn:aws:cloudwatch:ap-northeast-1:$ACCT:alarm:obs-lab-orders-page"]}}
]}
EOF
aws cloudwatch put-dashboard --dashboard-name obs-lab \
  --dashboard-body file://dash.json
# {"DashboardValidationMessages": []}  ← empty = valid

# read it back (what IaC would store)
aws cloudwatch get-dashboard --dashboard-name obs-lab \
  --query DashboardBody --output text | jq .

# metrics array shorthand
#   "."   repeats the value at the same position
#         in the row above
#   "..." repeats everything up to the options object

# heredoc without quotes: $ACCT is expanded in dash.json

# console: CloudWatch → Dashboards → obs-lab
#   - set the time range to 1 h
#   - hover the p90 line, compare with the anomaly band
#     on the alarm's own graph
Lab 11 · Verify, break it, clean up

Alarms that fire once, for the right reason

Use the knobs to drive each alarm through its states and watch which ones notify.

Verify

  • list-metrics shows Latency, Errors, Requests with service and route, plus ErrorLogs.
  • The dashboard shows p50 near 70 ms and p90 near 110 ms with FAIL_RATE=0.05.
  • describe-alarms --alarm-name-prefix obs-lab → OK after a few minutes (the anomaly alarm can stay INSUFFICIENT_DATA until its model has data).
  • set-alarm-state --alarm-name obs-lab-orders-page --state-value ALARM --state-reason drill → exactly one email.

Break it

  • Errors: update-function-configuration --environment "Variables={FAIL_RATE=0.5}" → errors alarm, then the anomaly alarm (failed calls are slower), then one composite email.
  • Typo: copy the errors alarm with Value=order and default --treat-missing-data missing → INSUFFICIENT_DATA forever (with notBreaching it sits in OK and never pages); find it with describe-alarms --state-value INSUFFICIENT_DATA.
  • Broken EMF: drop "route" from the JSON root → logs arrive, metrics stop.
  • Silence: stop traffic, set --treat-missing-data breaching → it alarms on no data.
# stop the traffic loop, then
aws cloudwatch delete-alarms \
  --alarm-names obs-lab-orders-page
aws cloudwatch delete-alarms \
  --alarm-names obs-lab-orders-errors \
  obs-lab-orders-latency-anomaly
aws cloudwatch delete-dashboards \
  --dashboard-names obs-lab
aws logs delete-metric-filter \
  --log-group-name /obs-lab/orders \
  --filter-name orders-error-logs
aws sns delete-topic \
  --topic-arn "$TOPIC"
# keep Lambda, role, log group for
# Lab 12 (removed there). Custom
# metrics cannot be deleted; they
# expire after 15 months.
Lab 12 · Build (1/5)

Logs Insights in three languages

Goal: answer the same questions in Logs Insights QL, PPL and SQL against /obs-lab/orders. Restart the Lab 11 traffic loop for ~10 minutes first.

export AWS_REGION=ap-northeast-1
END=$(date +%s); START=$((END - 3600))

run() {  # $1 = CWLI | PPL, $2 = query string
  QID=$(aws logs start-query --query-language "$1" \
    --log-group-names /obs-lab/orders \
    --start-time $START --end-time $END \
    --query-string "$2" --query queryId --output text)
  until [ "$(aws logs get-query-results --query-id $QID \
     --query status --output text)" = Complete ]; do
    sleep 2; done
  aws logs get-query-results --query-id $QID \
    --query results --output text
}

# Logs Insights QL: latency percentiles per route
run CWLI 'filter ispresent(duration_ms)
  | stats pct(duration_ms, 50) as p50,
          pct(duration_ms, 95) as p95, count(*) as n by route'

# PPL: errors by route
run PPL 'fields route, status | where status >= 500
  | stats count() as errors by route'
# SQL: the log group is named in FROM, not in a flag
QID=$(aws logs start-query --query-language SQL \
  --start-time $START --end-time $END \
  --query-string "SELECT route, COUNT(*) AS n,
      AVG(duration_ms) AS avg_ms
    FROM \`/obs-lab/orders\`
    GROUP BY route ORDER BY n DESC" \
  --query queryId --output text)
sleep 5; aws logs get-query-results --query-id $QID

# recurring error messages (QL only)
run CWLI 'filter level = "ERROR" | pattern msg'

# Lambda platform lines: cold starts and memory
run CWLI 'filter @type = "REPORT"
  | stats count(*) as inv, max(@maxMemoryUsed) as mem,
    sum(ispresent(@initDuration)) as cold by bin(15m)'

# bytes scanned = what the last query cost
aws logs get-query-results --query-id $QID \
  --query statistics

Expected: p50 ≈ 70 ms for POST /orders; p95 near 120 ms or 450+ ms (5% of calls fail slowly); PPL errors ≈ 5% of n; QL and SQL agree on n.

Lab 12 · Build (2/5)

Live Tail and a data protection policy

Step 2: stream errors as they happen, then prove that a data protection policy masks an email address in every read path.

ACCT=$(aws sts get-caller-identity \
  --query Account --output text)
LG=arn:aws:logs:ap-northeast-1:$ACCT:\
log-group:/obs-lab/orders

# Live Tail: errors only (traffic must run)
aws logs start-live-tail \
  --log-group-identifiers $LG \
  --log-event-filter-pattern \
  '{ $.level = "ERROR" }'
# Ctrl-C after a few events:
# billed per minute of session

# simpler, scriptable alternative
aws logs tail /obs-lab/orders --follow \
  --filter-pattern '{ $.status >= 500 }' \
  --format short
cat > dp.json <<'EOF'
{"Name": "obs-lab-pii", "Version": "2021-06-01",
 "Statement": [
  {"Sid": "audit", "DataIdentifier": [
    "arn:aws:dataprotection::aws:data-identifier/EmailAddress"],
   "Operation": {"Audit": {"FindingsDestination": {}}}},
  {"Sid": "redact", "DataIdentifier": [
    "arn:aws:dataprotection::aws:data-identifier/EmailAddress"],
   "Operation": {"Deidentify": {"MaskConfig": {}}}}]}
EOF
aws logs put-data-protection-policy \
  --log-group-identifier /obs-lab/orders \
  --policy-document file://dp.json

# masking only applies to events ingested
# after the policy exists
Lab 12 · Build (3/5)

A leaked email, masked; an Infrequent Access group, limited

Step 3: write one bad log line by hand and read it back; then create an IA log group and hit its feature limits.

aws logs create-log-stream \
  --log-group-name /obs-lab/orders \
  --log-stream-name manual
M='{\"level\":\"INFO\",\"msg\":\"receipt sent\",'
M+='\"email\":\"alice@example.com\"}'
cat > ev.json <<EOF
[{"timestamp": $(date +%s)000, "message": "$M"}]
EOF
aws logs put-log-events \
  --log-group-name /obs-lab/orders \
  --log-stream-name manual \
  --log-events file://ev.json

aws logs tail /obs-lab/orders \
  --log-stream-names manual --since 5m
# … "email":"*****************"

# with logs:Unmask, in Logs Insights:
#   filter msg = "receipt sent"
#   | fields unmask(@message)
# Infrequent Access: cheaper ingestion, fewer features
aws logs create-log-group \
  --log-group-name /obs-lab/orders-ia \
  --log-group-class INFREQUENT_ACCESS

aws logs describe-log-groups \
  --log-group-name-prefix /obs-lab/orders-ia \
  --query 'logGroups[0].logGroupClass'
# "INFREQUENT_ACCESS"

# not supported on this class → expect an error
aws logs put-metric-filter \
  --log-group-name /obs-lab/orders-ia \
  --filter-name x --filter-pattern ERROR \
  --metric-transformations \
  metricName=X,metricNamespace=ObsLab,metricValue=1

# supported: Logs Insights (QL, PPL, SQL)
# the class can never be changed: pick it at creation
Lab 12 · Build (4/5)

OpenTelemetry traces into X-Ray with Transaction Search

Step 4: store spans in CloudWatch Logs, then send OTLP spans from a local contrib Collector signed with SigV4. Spans take a few minutes to become searchable.

cat > ts.json <<EOF
{"Version":"2012-10-17","Statement":[{
 "Sid":"TransactionSearchXRayAccess","Effect":"Allow",
 "Principal":{"Service":"xray.amazonaws.com"},
 "Action":"logs:PutLogEvents",
 "Resource":[
  "arn:aws:logs:ap-northeast-1:$ACCT:log-group:aws/spans:*",
  "arn:aws:logs:ap-northeast-1:$ACCT:log-group:\
/aws/application-signals/data:*"],
 "Condition":{"ArnLike":{"aws:SourceArn":
    "arn:aws:xray:ap-northeast-1:$ACCT:*"},
  "StringEquals":{"aws:SourceAccount":"$ACCT"}}}]}
EOF
aws logs put-resource-policy \
  --policy-name TransactionSearchXRay \
  --policy-document file://ts.json
aws xray update-trace-segment-destination \
  --destination CloudWatchLogs
aws xray get-trace-segment-destination  # wait: ACTIVE

# Lambda active tracing (Lab 11) also lands here
for i in 1 2 3; do aws lambda invoke \
  --function-name obs-lab-orders /dev/null; done
# col.yaml
extensions:
  sigv4auth: {region: ap-northeast-1, service: xray}
receivers:
  otlp: {protocols: {grpc: {endpoint: 0.0.0.0:4317}}}
exporters:
  otlphttp:
    traces_endpoint:
      https://xray.ap-northeast-1.amazonaws.com/v1/traces
    auth: {authenticator: sigv4auth}
service:
  extensions: [sigv4auth]
  pipelines:
    traces: {receivers: [otlp], exporters: [otlphttp]}

# profile creds → container (needs xray:PutTraceSegments,
# e.g. AWSXrayWriteOnlyAccess)
eval "$(aws configure export-credentials --format env)"
docker run -d --name obs-col -p 4317:4317 \
  -e AWS_ACCESS_KEY_ID -e AWS_SECRET_ACCESS_KEY \
  -e AWS_SESSION_TOKEN \
  -v $PWD/col.yaml:/etc/otelcol-contrib/config.yaml \
  otel/opentelemetry-collector-contrib:<version>

# 20 synthetic traces, service.name = orders
docker run --rm ghcr.io/open-telemetry/\
opentelemetry-collector-contrib/telemetrygen:<version> \
  traces --otlp-insecure --traces 20 --service orders \
  --otlp-endpoint host.docker.internal:4317
Lab 12 · Build (5/5)

Optional: query spans, and remote-write kind metrics to AMP

Step 5: spans are logs now, so Logs Insights works on them. Then (optional, small cost) send the kind cluster's shop metrics to an AMP workspace.

# spans are structured logs in aws/spans
run2() { aws logs start-query --log-group-names aws/spans \
  --start-time $START --end-time $(date +%s) \
  --query-string "$1" --query queryId --output text; }
QID=$(run2 'fields @timestamp, traceId, name, durationNano
  | sort durationNano desc | limit 10')
sleep 5; aws logs get-query-results --query-id $QID
# console: Application Signals → Transaction Search

# AMP workspace + a remote-write-only lab user
WS=$(aws amp create-workspace --alias obs-lab \
  --query workspaceId --output text)
aws iam create-user --user-name obs-lab-amp-writer
aws iam attach-user-policy \
  --user-name obs-lab-amp-writer --policy-arn \
  arn:aws:iam::aws:policy/AmazonPrometheusRemoteWriteAccess
aws iam create-access-key --user-name obs-lab-amp-writer
kubectl -n monitoring create secret generic amp-writer \
  --from-literal=accessKey=<AccessKeyId> \
  --from-literal=secretKey=<SecretAccessKey>
# amp-values.yaml for kps: keep only the shop jobs
prometheus:
  prometheusSpec:
    remoteWrite:
      - url: <AMP_URL>/api/v1/remote_write
        sigv4:
          region: ap-northeast-1
          accessKey: {name: amp-writer, key: accessKey}
          secretKey: {name: amp-writer, key: secretKey}
        writeRelabelConfigs:
          - sourceLabels: [job]
            regex: orders|payments
            action: keep
# <AMP_URL> = https://aps-workspaces.ap-northeast-1
#   .amazonaws.com/workspaces/<WS>  (one string)

helm upgrade kps prometheus-community/kube-prometheus-stack \
  -n monitoring --reuse-values --version $KPS_VER -f amp-values.yaml

# in Prometheus: samples flow, none fail
#   rate(prometheus_remote_storage_samples_total[5m])
#   rate(prometheus_remote_storage_samples_failed_total[5m])

Static keys are a lab-only shortcut for a cluster outside AWS. On EKS use Pod Identity or IRSA for Prometheus, or an AMP managed scraper — no keys at all.

Lab 12 · Verify, break it, clean up

Logs, masks and traces where you expect them

Each break produces a specific symptom: learn to recognise it.

Verify

  • QL and SQL agree on requests per route; PPL errors are ≈ 5% of them.
  • Live Tail shows only ERROR lines.
  • The manual line shows the email masked.
  • Destination is CloudWatchLogs / ACTIVE; orders spans appear in Transaction Search and aws/spans.
  • (Optional) AMP has up{job="orders"}.

Break it

  • **SQL plus --log-group-names** → rejected: SQL names groups in FROM.
  • IA limits: metric filter and Live Tail fail on /obs-lab/orders-ia; queries work.
  • **Destination back to XRay** → the Collector logs export errors from the traces endpoint (spans rejected).
  • **service: logs** in sigv4auth for traces → 403 signature errors (docker logs obs-col).
  • **AMP without keep** → remote-write samples jump: that is the bill.
docker rm -f obs-col
aws xray update-trace-segment-destination \
  --destination XRay
aws logs delete-resource-policy \
  --policy-name TransactionSearchXRay
for g in aws/spans /obs-lab/orders-ia \
  /obs-lab/orders; do
  aws logs delete-log-group \
    --log-group-name $g; done
aws lambda delete-function \
  --function-name obs-lab-orders
R=obs-lab-orders-role
P=arn:aws:iam::aws:policy
for p in service-role/\
AWSLambdaBasicExecutionRole \
  AWSXRayDaemonWriteAccess; do
  aws iam detach-role-policy \
  --role-name $R --policy-arn $P/$p
done
aws iam delete-role --role-name $R
# AMP: delete-workspace, drop
# remoteWrite from kps, delete the
# access key and the IAM user
Lab 13 · Build (1/4)

Game day: game-day alerts, then inject latency

Goal: replay the ch 13 incident on the obs cluster. Needs Labs 2–10 running (kps, Loki + Alloy, otel-gateway, Tempo, RED dashboard).

# 0) everything up?
for ns in monitoring logging tracing otel shop; do
  kubectl -n $ns get pods --no-headers | grep -v Running; done
# (no output from the loop = all Running)

# 1) port-forwards (one terminal each, or with &)
M="kubectl -n monitoring port-forward"
$M svc/prometheus-operated 9090 &
$M svc/alertmanager-operated 9093 &
$M svc/kps-grafana 3000:80 &

# 2) game-day latency rule (right): short "for" for a quick lab
kubectl apply -f rules/gameday.yaml
kubectl -n monitoring get prometheusrule shop-gameday

# 3) baseline p95 per route, before the change
promtool query instant http://localhost:9090 \
 'histogram_quantile(0.95, sum by (le, route) (rate(
   http_request_duration_seconds_bucket{job="orders"}[5m])))'

# 4) the "incident": a config-only change, same image
kubectl -n shop set env deploy/payments EXTRA_LATENCY_MS=800
kubectl -n shop rollout status deploy/payments
date   # write down the time: this is your change event
# rules/gameday.yaml
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: shop-gameday
  namespace: monitoring
  labels: {release: kps}    # required by kps default selectors
spec:
  groups:
    - name: shop-gameday
      rules:
        - alert: ShopHighLatencyP95
          expr: |
            histogram_quantile(0.95, sum by (le) (rate(
              http_request_duration_seconds_bucket{job="orders"}[5m]))) > 0.5
          for: 2m
          labels: {severity: critical, service: orders, team: shop}
          annotations:
            summary: 'orders p95 {{ $value | humanizeDuration }}'

The 5-minute rate window plus for: 2m means the alert needs roughly 4–7 minutes to fire. Use the wait to predict which panels will move.

Lab 13 · Build (2/4)

Game day: investigate like on-call

Follow the loop: alert → RED dashboard → exemplar → trace → logs → change. Write each hypothesis down before you run its query.

# 1) the alert
amtool --alertmanager.url=http://localhost:9093 \
  alert query alertname=ShopHighLatencyP95
# also: Prometheus UI → Alerts (pending → firing)

# 2) RED dashboard (Lab 10), service=orders:
#    rate flat? errors flat? which route is slow?
#    then service=payments: is its server p95 high too?

# 3) rule out throttling of orders (USE)
promtool query instant http://localhost:9090 \
 'sum by (pod) (rate(container_cpu_cfs_throttled_periods_total{
    namespace="shop", container="orders"}[5m]))'

# 4) exemplar → trace: Grafana Explore, Prometheus,
#    duration query, toggle "Exemplars", click a
#    diamond in the slow band → open the trace
#    No diamonds? is exemplar storage enabled?
kubectl -n monitoring get prometheus \
  -o jsonpath='{.items[0].spec.enableFeatures}'
# 5) TraceQL in Explore → Tempo: all pods or one?
{ resource.service.name = "payments" && kind = server
  && duration > 500ms }

{ resource.service.name = "payments" && kind = server }
  | quantile_over_time(duration, .95)
    by (resource.k8s.pod.name)

# 6) logs for the trace (Explore → Loki)
{namespace="shop"} | json | trace_id="<id from step 4>"

# 7) what changed?
kubectl -n shop rollout history deploy/payments
kubectl -n shop get deploy payments -o \
  jsonpath='{.spec.template.spec.containers[0].env}'

# 8) mitigate, verify on the SAME panel (~5 min)
kubectl -n shop rollout undo deploy/payments

# 9) round 2: errors instead of latency
kubectl -n shop set env deploy/payments FAIL_RATE=0.2
#  → Lab 4's ShopHighErrorRate (after ~6 min); Tempo: status = error
kubectl -n shop set env deploy/payments FAIL_RATE-

If resource.k8s.pod.name is empty, the k8sattributes processor is not adding it in otel-gateway (ch 9): group by resource.service.name instead and fix it after the game day.

Lab 13 · Build (3/4)

Watchdog to a heartbeat receiver

Make silence visible: the always-firing Watchdog must reach a receiver that would notice if it stopped.

# 1) Watchdog is always firing in kps — confirm
amtool --alertmanager.url=http://localhost:9093 \
  alert query alertname=Watchdog

# 2) where does it go today? (kps default: "null")
amtool --alertmanager.url=http://localhost:9093 \
  config routes test alertname=Watchdog

# 3) a stand-in heartbeat endpoint
kubectl -n monitoring create deployment heartbeat \
  --image=mccutchen/go-httpbin --port=8080
kubectl -n monitoring expose deployment heartbeat \
  --port=80 --target-port=8080

# 4) route Watchdog to it (config on the right)
helm upgrade kps \
  prometheus-community/kube-prometheus-stack \
  -n monitoring --reuse-values --version $KPS_VER \
  -f deploy/am-values.yaml

# 5) check: routes and notification counter
amtool --alertmanager.url=http://localhost:9093 \
  config routes test alertname=Watchdog   # heartbeat
# alertmanager/alertmanager.yml (Lab 4): Watchdog
# → heartbeat (go-httpbin stand-in) instead of "null"
route:
  ...
  routes:
    - receiver: heartbeat
      matchers: [alertname = "Watchdog"]
      repeat_interval: 1m      # keep pinging
    - {receiver: shop-critical, ...}   # unchanged
    - {receiver: shop-warning, ...}    # unchanged
receivers:
  - name: "null"
  - name: heartbeat
    webhook_configs:
      - url: http://heartbeat.monitoring.svc/anything
  # ... shop-critical and shop-warning unchanged

# test, then re-wrap exactly as in Lab 4
C=alertmanager/alertmanager.yml
amtool check-config $C
amtool config routes test --config.file=$C \
  alertname=Watchdog                    # heartbeat
{ echo 'alertmanager:'; echo '  config:'
  sed 's/^/    /' $C; } > deploy/am-values.yaml

Always change the one Alertmanager config file from Lab 4 and re-wrap it: a separate values file with its own alertmanager.config would replace Lab 4's routes and receivers (Helm replaces lists).

Lab 13 · Build (4/4)

Meta-alerts and a cardinality clean-up

Catch a missing target with absent(), then find the biggest metrics and drop what nobody uses.

# 1) meta-alerts from ch 13 (shop-meta, release: kps)
kubectl apply -f rules/shop-meta.yaml

# 2) test: no orders pods → ShopTargetMissing after 5m
kubectl -n shop get deploy orders \
  -o jsonpath='{.spec.replicas}'; echo   # remember N
kubectl -n shop scale deploy/orders --replicas=0
amtool --alertmanager.url=http://localhost:9093 \
  alert query alertname=ShopTargetMissing
kubectl -n shop scale deploy/orders --replicas=<N>

# 3) cardinality: biggest metric names
promtool query instant http://localhost:9090 \
  'topk(10, count by (__name__) ({__name__=~".+"}))'
curl -s localhost:9090/api/v1/status/tsdb \
  | jq '.data.seriesCountByMetricName[:5]'
# 4) series per shop job, and go_gc_* among them
promtool query instant http://localhost:9090 \
  'count by (job) ({job=~"orders|payments"})'
promtool query instant http://localhost:9090 \
  'count({__name__=~"go_gc_.*", job=~"orders|payments"})'

# 5) drop them at scrape time (ServiceMonitor shop, Lab 2)
kubectl -n shop get servicemonitor
kubectl -n shop patch servicemonitor shop \
  --type=json -p='[{"op":"add",
  "path":"/spec/endpoints/0/metricRelabelings",
  "value":[{"action":"drop",
            "sourceLabels":["__name__"],
            "regex":"go_gc_.*"}]}]'

# 6) after ~5 min the old series go stale
promtool query instant http://localhost:9090 \
  'count({__name__=~"go_gc_.*", job=~"orders|payments"})'
# → empty result
Lab 13 · Verify, break it, clean up

You found it, fixed it, and the pipeline watches itself

Write a five-line timeline: change, alert, diagnosis, mitigation, recovery — with the query that proved each step.

Verify

  • ShopHighLatencyP95: pending → firing → resolved, at times that match your date note.
  • A trace puts the time inside the payments server span; log lines share its trace_id.
  • rollout undo of the config-only revision restored p95.
  • alertmanager_notifications_total{integration="webhook"} grows every minute.
  • No go_gc_* series left for the shop jobs.

Break it

  • No label: drop release: kps from shop-gameday → the rules vanish from Prometheus → Rules, silently.
  • Bad expression: change job="orders" to job="order" → the alert can never fire. Only a promtool test rules unit test (ch 4) catches this.
  • Break the receiver: kubectl -n monitoring scale deploy/heartbeat --replicas=0 → alertmanager_notifications_failed_total rises; kps fires AlertmanagerFailedToSendAlerts.
  • Exploding label: make the app record raw paths (/orders/8812) → watch count(count by (route)(http_requests_total)) climb, then revert.
# undo game-day changes
kubectl -n shop set env deploy/payments \
  EXTRA_LATENCY_MS- FAIL_RATE-
kubectl delete -f rules/gameday.yaml
# keep shop-meta and the Watchdog route

# stop the port-forwards
pkill -f 'kubectl -n monitoring port-forward'

# end of the deck? remove everything:
kind delete cluster --name obs
Lab safety

Cost and cleanup

The kind labs are free apart from laptop resources. The AWS labs (11–12) cost little but bill per GB, per metric and per alarm: clean up when you finish.

Can cost money (AWS)

  • CloudWatch Logs: GB ingested and GB-month stored; Logs Insights per GB scanned.
  • Custom metrics (each name + dimension combination, per month) and GetMetricData API calls.
  • Alarms (anomaly detection and high-resolution alarms cost more than standard ones).
  • Application Signals, Transaction Search, Synthetics canaries, Container Insights if you enable them.
  • Amazon Managed Service for Prometheus (samples ingested, stored, queried) and Managed Grafana (per user), if you try them.

Habits

  • Budget alarm before Lab 11 (Lab 0).
  • Set a retention policy the moment you create a log group.
  • Keep EMF dimensions few: every new value is a new custom metric.
  • Query Logs Insights over 15 min to 1 h, not "last 4 weeks".
  • On the laptop: give Docker at least 6 GB RAM for the full stack; stop port-forwards when done.

Clean up

  • AWS: delete Lambda obs-lab-orders, log group /obs-lab/orders (and /aws/lambda/obs-lab-orders), alarms, metric filters, SNS topic obs-lab-alerts; any AMP workspace or Grafana workspace; delete the budget last.
  • Custom metrics cannot be deleted: they stop costing once no new data arrives and expire after 15 months.
  • Local: kind delete cluster --name obs; docker image prune.
Glossary

Metrics and alerting terms

Short anchors; explain each in your own words.

TermShort meaning
Time seriesMetric name + unique label set, with timestamped samples
CardinalityNumber of distinct series (or label values); drives memory and cost
Counter / gaugeOnly-increasing value (resets on restart) / value that goes up and down
HistogramObservations counted in buckets (le), plus _sum and _count
Native histogramHistogram with dynamic exponential buckets in one series (stable since Prometheus 3.8)
ExemplarSample attached to a metric pointing to one trace (trace_id)
Recording rulePrecomputed query saved as a new series
Remote writeProtocol to stream samples to another store (Mimir, Thanos, AMP)
TermShort meaning
REDRate, Errors, Duration — for request-driven services
USEUtilization, Saturation, Errors — for resources
Golden signalsLatency, traffic, errors, saturation
Symptom vs causeWhat users feel vs why it happens; page on symptoms
for / keep_firing_forPending time before firing / hold time after the condition clears
GroupingAlertmanager batching alerts with the same group_by labels into one notification
InhibitionSuppressing target alerts while a matching source alert fires
WatchdogAlways-firing alert used as a dead man's switch
Glossary

Logs, traces and AWS terms

Short anchors; explain each in your own words.

TermShort meaning
Structured logLog line as key/value fields (JSON, logfmt) with a stable schema
Stream (Loki)All lines with the same label set; the unit Loki indexes
Structured metadataPer-line key/values stored with Loki entries, not indexed as labels
SpanOne timed operation with attributes, status, events and links
TraceTree of spans sharing one trace ID
Context propagationPassing trace context across process boundaries (W3C traceparent)
ResourceAttributes describing the telemetry source (service.name, k8s.pod.name)
Head / tail samplingDecide at trace start / after the trace completes
OTLPOpenTelemetry protocol; gRPC 4317, HTTP 4318
TermShort meaning
CollectorOpenTelemetry pipeline: receivers → processors → exporters
AlloyGrafana's collector; replaces Promtail (EOL 2 Mar 2026) and Grafana Agent
Namespace / dimension (CloudWatch)Container for metrics / name-value pair identifying a metric
EMFEmbedded Metric Format: JSON log lines CloudWatch turns into metrics
Composite alarmAlarm whose state is a rule over other alarms
Logs InsightsCloudWatch log query engine (Logs Insights QL, PPL, SQL)
Application SignalsCloudWatch APM built on OpenTelemetry: services, SLOs, service map
ADOTAWS Distro for OpenTelemetry
OAMObservability Access Manager: sinks and links for cross-account observability
Further reading

Primary references

Versions move every month; when a slide and the docs disagree, the docs win.

Prometheus, Grafana, Loki, Tempo

  • Prometheus docs (PromQL, configuration, alerting) — prometheus.io/docs
  • Alertmanager configuration — prometheus.io/docs/alerting/latest/configuration/
  • Prometheus Operator — prometheus-operator.dev; Helm charts — github.com/prometheus-community/helm-charts
  • Loki and LogQL — grafana.com/docs/loki/latest/; Alloy — grafana.com/docs/alloy/latest/
  • Tempo docs, TraceQL and the 3.0 release notes — grafana.com/docs/tempo/latest/
  • Grafana (what's new in v13) — grafana.com/docs/grafana/latest/

OpenTelemetry, methods and AWS

  • OpenTelemetry docs and specification — opentelemetry.io/docs
  • Semantic conventions — opentelemetry.io/docs/specs/semconv/; W3C Trace Context — w3.org/TR/trace-context/
  • Google SRE book, "Monitoring Distributed Systems" — sre.google/sre-book/monitoring-distributed-systems/
  • USE method — brendangregg.com/usemethod.html; RED method — grafana.com/blog/2018/08/02/the-red-method-how-to-instrument-your-services/
  • CloudWatch metrics, alarms, EMF — docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/; Logs and Logs Insights — …/latest/logs/
  • CloudWatch OpenTelemetry metrics and PromQL (Jun 2026) — aws.amazon.com/about-aws/whats-new/2026/06/amazon-cloudwatch-otel-metrics/
  • X-Ray SDK and daemon end of support — docs.aws.amazon.com/xray/latest/devguide/xray-daemon-eos.html
  • Amazon Managed Service for Prometheus — docs.aws.amazon.com/prometheus/
1 / 1