Metrics, logs and traces you can actually debug with: Prometheus and PromQL, alerting with Alertmanager, structured logs in Loki, OpenTelemetry traces through the Collector into Tempo, RED and USE dashboards in Grafana, and the same ideas on Amazon CloudWatch — with a lab for every chapter.
Use ← → / Space to navigate · I contents · O overview · T theme · F fullscreen
Click any line to jump there · press I to return to this page
What observability is for, the stack this deck builds, the versions it covers, and how to study it.
Telemetry is only worth its cost if it shortens the path from "something is wrong" to "this change fixes it".
trace_id, not read end to end.Monitoring checks the failure modes you predicted. Observability lets you ask new questions without shipping new code.
trace_id).Read left to right: instrument, collect, store, query, act. The same shape applies to the open-source stack and to CloudWatch.
Chapter 13 closes the loop: a debugging workflow from alert to root cause, alert quality, cost and cardinality governance, and monitoring the monitoring.
These projects release monthly. When a slide and the official docs disagree, the docs win.
scrape_native_histograms); classic histograms are not converted.grafana-community/helm-charts (Loki: 16 Mar 2026).Chart and image versions in the labs are not pinned on the slides: run helm search repo <chart> and pin what you install in your own values files.
Small enough to read in one sitting, realistic enough to fail in interesting ways: a synchronous dependency with tunable errors and latency.
http_requests_total, http_request_duration_seconds, ServiceMonitors.trace_id, Loki and Alloy.Labs 0–10 and 13 run on a laptop with kind and cost nothing. Labs 11–12 use a small AWS sandbox with a budget alarm.
export AWS_REGION=ap-northeast-1 (Labs 11–12).kind, kubectl, helm, Go 1.25+ (the images build with golang:1.27).promtool and amtool (ship with the Prometheus and Alertmanager releases).logcli (Loki release), jq, hey (local load tests).k9s to watch the cluster.obs-lab/
├── go.mod # module obs-lab
├── Dockerfile # ARG APP
├── apps/
│ ├── orders/main.go
│ └── payments/main.go
├── deploy/
│ ├── kind.yaml
│ ├── kps-values.yaml
│ └── shop.yaml
├── rules/ # Lab 4
└── dashboards/ # Lab 10
namespaces:
monitoring kps (Lab 0)
shop orders, payments, loadgen
logging Loki, Alloy (Lab 7)
otel otel-gateway (Lab 8)
tracing Tempo (Lab 9)Read a part, run its lab, then answer the self-check out loud without notes.
Order matters: Prometheus (ch 2) before PromQL (ch 3); OpenTelemetry basics (ch 8) before the Collector and Tempo (ch 9); every signal before dashboards (ch 10) and the debugging workflow (ch 13).
What each signal answers, what it costs, how cardinality decides the bill, and the RED / USE methods that tell you what to measure.
Metrics, logs, traces, profiles and events: what each one is good at, and how they fit into one pipeline.
Monitoring checks the failures you predicted. Observability lets you ask new questions about failures you did not predict — without shipping new code.
Rule of thumb: metrics tell you something is wrong, traces tell you where, logs (and profiles) tell you why. Good tooling lets you jump between them with one click (ch 9).
Each signal trades detail for cost differently. You need all of them, in different volumes.
| Signal | What it is | Best at answering | Cost shape | In this deck |
|---|---|---|---|---|
| Metrics | Numeric samples aggregated over time, identified by name + labels | Is it broken? Trends, alerting, capacity | Per series (label combinations); cheap per event | Prometheus (ch 2–5), CloudWatch (ch 11) |
| Logs | Timestamped records of discrete events, ideally structured | Why? Details of one event, errors, audit | Per byte ingested and stored | slog, Loki (ch 6–7), CloudWatch Logs (ch 12) |
| Traces | Tree of spans: one request across services | Where? Which hop is slow or failing | Per span; usually sampled | OpenTelemetry, Tempo (ch 8–9), X-Ray (ch 12) |
| Profiles | Stack samples of CPU, memory, locks over time | Which function burns CPU or memory | Per sample; continuous profiling is low overhead | Mentioned only; OTel Profiles is Alpha (Mar 2026) |
| Events | Changes: deploys, config flips, scaling, incidents | What changed just before it broke? | Low volume, high value | Grafana annotations (ch 10), ch 13 |
"Three pillars" (metrics, logs, traces) is a useful starting point, not a law: what matters is that the signals share identity (service, trace ID) so you can correlate them.
Tools differ per stage; the stages do not. Most design decisions (sampling, dropping, labels) happen before storage.
A POST /orders that failed because payments timed out. The shared fields are what make the signals one story.
# METRIC — one sample in one series (aggregated, no request detail)
http_requests_total{job="orders",method="POST",route="/orders",code="502"} 17
# LOG — one structured line from orders (detail of this one event)
{"time":"2026-11-08T14:05:12.481Z","level":"ERROR","msg":"payment failed",
"service":"orders","route":"/orders","status":502,"duration_ms":2004,
"order_id":"o-81231","trace_id":"4bf92f3577b34da6a3ce929d0e0e4736",
"span_id":"00f067aa0ba902b7","error":"payments: context deadline exceeded"}
# TRACE — the request as a tree of spans (where the time went)
trace 4bf92f3577b34da6a3ce929d0e0e4736
└─ orders POST /orders 2004 ms status=ERROR
├─ orders validate order 3 ms
└─ payments POST /charge 2000 ms status=ERROR (timeout)service / job / service.name: the same service name everywhere.route: the template (/orders/{id}), never the raw path.trace_id in logs → open the trace (ch 9).trace_id to a metric sample → jump from a latency spike to one slow trace (ch 2, ch 9).Prometheus pulls; OpenTelemetry, Loki and CloudWatch mostly receive pushes. Both work; know what each gives you.
/metrics from each target on an interval.up == 0.PutMetricData).Open source on Kubernetes first, then the AWS managed equivalents. The concepts transfer; only the query language changes.
| Stage | Metrics | Logs | Traces | AWS managed |
|---|---|---|---|---|
| Instrument | client_golang, OTel SDK, exporters | log/slog (Go), OTel log bridge | OTel SDK, otelhttp | ADOT, CloudWatch agent, EMF |
| Collect / process | Prometheus scrape, Alloy, OTel Collector | Grafana Alloy (Promtail is EOL) | OTel Collector | CloudWatch agent, ADOT, AMP managed scrapers |
| Store | Prometheus TSDB; Thanos / Mimir long term | Loki | Tempo | CloudWatch, AMP, X-Ray |
| Query | PromQL | LogQL | TraceQL | Metrics Insights, PromQL, Logs Insights |
| Visualize / act | Grafana, Alertmanager | Grafana, Loki ruler | Grafana | CloudWatch dashboards and alarms, Managed Grafana |
Versions (Nov 2026): Prometheus 3.15, Loki 3.7, Alloy 1.21, Tempo 3.x, Grafana 13. See the versions slide in Orientation.
Where to put telemetry, and why one careless label can cost more than the whole cluster.
Black-box sees what users see; white-box sees why. Use both, for different jobs.
Instrument at boundaries: everything that enters, leaves, waits or retries. Internal functions only when profiling says so.
The server side of each hop (①) gives RED for that service; the client side (②) measures the dependency as the caller experiences it, including network and retries. They differ, and the difference is information.
A metric name is not one series. Every unique combination of label values is a separate series with its own memory and storage.
http_requests_total{method, route, code}
method: GET, POST → 2
route: /orders, /orders/{id}, /healthz → 3
code: 200, 201, 400, 404, 500, 502 → 6
worst case: 2 × 3 × 6 = 36 series per pod
× 3 pods (pod label from the scrape) = 108 series
# now add the raw path instead of the route:
path: /orders/o-1, /orders/o-2, ... → unbounded
→ 1 new series per order ID, forever (until retention)
# and a histogram multiplies again:
http_request_duration_seconds_bucket
(11 default buckets + "+Inf") = 12 series per combination
+ _sum + _count → 14 series per label combinationA label is a dimension you will aggregate or filter by. If its values are unbounded, it belongs in logs or traces, not metrics.
| Tempting label | Why it explodes | Do this instead |
|---|---|---|
user_id, order_id, email | One series per user or order | Put it in logs and span attributes; metrics stay aggregate |
Raw URL path /orders/o-81231 | IDs in the path | Label with the route template /orders/{id} |
| Error message text | Free text, often with IDs inside | A small enum reason="timeout"; full message in logs |
| Client IP, user agent | Unbounded | Logs; or bucket (country, browser family) |
| Timestamps, request IDs, trace IDs | Unique per request | Exemplars (one trace ID per sample, not a label) |
| Query string, SQL text | Unbounded | A normalized operation name (db.operation.name) |
Test before you ship: count the realistic number of values for every label and multiply. If the product is above a few thousand per service, stop and redesign.
Each technique loses something different. Choose per signal and per use case.
Know the unit you pay for. That unit is what you control.
| Signal | Unit that drives cost | Biggest lever | Typical trap |
|---|---|---|---|
| Metrics (self-hosted) | Active series × retention; query load | Drop unused metrics and labels at scrape (metric_relabel_configs) | Histograms with many buckets × many labels |
| Metrics (CloudWatch) | Custom metrics (each name + dimension combination), API calls; OTel metrics per GB ingested (2026) | Fewer dimensions; EMF instead of PutMetricData loops | Per-instance or per-pod dimensions |
| Logs | GB ingested + GB stored (+ GB scanned by queries) | Levels, sampling, dropping noisy lines at the agent | Debug logs left on; logging full request bodies |
| Traces | Spans ingested and stored | Sampling (head or tail) | 100% tracing of health checks |
| Queries | Data scanned (Logs Insights, Loki), series touched | Narrow time range and selectors first | Dashboards refreshing every 5 s over 30 days |
Chapter 13 turns this into governance: budgets per team, limits per tenant, and a monthly review of top series and top log streams.
Three checklists that decide what to measure first, so dashboards are designed instead of accumulated.
From Google's SRE book: if you can measure only four things about a user-facing system, measure these.
RED and USE below are two specializations: RED for services (requests), USE for resources (things requests consume). Together they cover the golden signals.
Tom Wilkie's method. Same three panels for every service, so any engineer can read any dashboard.
sum by (route) (rate(http_requests_total{job="orders"}[5m]))sum(rate(http_requests_total{job="orders",code=~"5.."}[5m])) / sum(rate(http_requests_total{job="orders"}[5m]))histogram_quantile(0.99, sum by (le) (rate(http_request_duration_seconds_bucket{job="orders"}[5m])))These queries are explained in ch 3 and turned into dashboards in ch 10. RED does not cover saturation: pair it with USE for the resources each service runs on.
Brendan Gregg's method. For each resource, check utilization, saturation and errors — in that order, and quickly.
| Resource | Utilization | Saturation | Errors |
|---|---|---|---|
| CPU (node) | 1 - avg(rate(node_cpu_seconds_total{mode="idle"}[5m])) | Load average / cores; run queue | Machine check errors (rare) |
| CPU (container) | Usage / limit (or / request) | CFS throttling: container_cpu_cfs_throttled_periods_total | — |
| Memory | Working set / limit | Swapping, page faults, reclaim | OOM kills (kube_pod_container_status_last_terminated_reason) |
| Disk | Busy time, space used % | I/O queue length, wait time | Device errors, filesystem read-only |
| Network | Bytes/s vs link capacity | Drops, retransmits, conntrack table full | Interface errors |
| Software: pools, queues | Connections in use / max | Waiters, queue depth, age of oldest message | Rejected or timed-out acquisitions |
Kubernetes queries for each row are in ch 5 and ch 10. The Linux A to Z deck covers the underlying tools (vmstat, iostat, sar).
Users feel symptoms. Causes are many, change over time, and often heal themselves.
POST /orders above budget.Alert design is ch 4; alert quality and on-call hygiene is ch 13; SLO-based paging is in the SRE deck.
An SLI is a carefully chosen RED measurement expressed as "good events / valid events". This deck builds the metrics; the SRE deck builds the objectives.
non-5xx requests / all requests.requests faster than 300 ms / all requests — needs a histogram bucket at 0.3 s.Write this table before writing instrumentation code. Lab 1 asks you to fill it in for the shop app.
| Question | Signal | Name / query (built later) | Alert? |
|---|---|---|---|
| Are orders failing? | Metric (R, E) | http_requests_total{job="orders",route="/orders",code=~"5.."} ratio | Page (SLO burn, SRE deck) |
| Are orders slow? | Metric (D) | http_request_duration_seconds p99 per route | Page if above SLO |
| Is payments the cause? | Metric + trace | payments RED; orders → payments span duration | No — dashboard |
| Which orders failed and why? | Logs | {namespace="shop", app="orders"} | json | status >= 500 | No |
| Is revenue flowing? | Business metric | rate(orders_created_total[5m]) vs last week | Ticket on large drop |
| Are pods healthy and sized? | USE (Kubernetes) | CPU throttling, memory vs limit, restarts | Ticket; page only on crash loop |
| Is the site reachable? | Black-box | probe_success for /healthz via blackbox_exporter | Page |
Each of these looks harmless on day one.
service.name, deployment.environment.name.Answer out loud, with an example from the shop app.
method (2) × route (3) × code (6) produce per pod?order_id never be a metric label? Where should it go?Practice: Lab 1 in Hands-on labs — observe the shop app with only black-box checks, logs and Kubernetes metrics, write its signal plan, and see why RED needs app metrics.
How Prometheus stores time series, the four metric types and how to instrument with them, scraping and relabeling, and the exporters around it.
A pull-based server with a local time-series database, a rule engine and an HTTP API — and what a "series" really is.
One binary does discovery, scraping, storage, rule evaluation and querying. Everything else plugs in around it.
Prometheus is deliberately single-node: each server is independent and reliable on its own. Scale and long-term storage come from sharding and remote write (ch 5).
Change any label value and you have a different series. Each series is a stream of (timestamp, value) samples.
# exposition format (what /metrics returns)
# HELP http_requests_total HTTP requests handled.
# TYPE http_requests_total counter
http_requests_total{method="POST",route="/orders",code="201"} 1027
http_requests_total{method="POST",route="/orders",code="502"} 17
http_requests_total{method="GET",route="/orders/{id}",code="200"} 4410
# after the scrape, Prometheus adds target labels:
http_requests_total{job="orders", instance="10.244.1.7:8080",
namespace="shop", pod="orders-7d9c...", method="POST",
route="/orders", code="502"} @1762610712.481 17
# the metric name is itself a label:
{__name__="http_requests_total", job="orders", ...}__name__; matches [a-zA-Z_:][a-zA-Z0-9_:]* by convention — Prometheus 3 also accepts any UTF-8 (quoted).job, instance and whatever relabeling adds.__ are reserved for internal use.Recent data lives in memory with a write-ahead log; older data is in immutable, compacted blocks on disk.
--storage.tsdb.retention.time (default 15d).--storage.tsdb.retention.size (e.g. 50GB; oldest blocks dropped first).--storage.tsdb.path (default data/).A query at time t takes, for each series, the newest sample at or before t — if it is not too old and not marked stale.
--query.lookback-delta).absent() (ch 3).Prometheus 3.0 shipped in Nov 2024; 3.15 is current and 3.5 is the long-term support line.
| Change | Since | What it means for you |
|---|---|---|
| New web UI (Mantine-based) | 3.0 | Query, targets, rules, TSDB status; the old UI is gone |
| UTF-8 metric and label names | 3.0 | {"http.server.request.duration", "service.name"="orders"} — OTel names can be stored unescaped |
| OTLP receiver | 3.0 | --web.enable-otlp-receiver → push OTLP metrics to /api/v1/otlp/v1/metrics |
| Remote Write 2.0 | 3.0 (experimental) | Smaller payloads, metadata, exemplars and created timestamps in one protocol |
scrape_classic_histograms renamed always_scrape_classic_histograms | 3.0 | Update scrape configs when upgrading from 2.x |
| Range selectors are left-open | 3.0 | [5m] excludes a sample exactly at t − 5m; edge cases in rate and *_over_time changed slightly |
| Native histograms stable | 3.8 | Opt-in per scrape job with scrape_native_histograms: true; the old feature flag is a no-op since 3.9 |
Check the release notes before relying on anything newer than this table; experimental features need --enable-feature=... and may change.
One YAML file for scraping and rules; flags for storage and features. Validate every change with promtool.
# prometheus.yml
global:
scrape_interval: 15s # default 1m
scrape_timeout: 10s # default 10s, must be <= interval
evaluation_interval: 15s # rules; default 1m
external_labels: # added when data leaves this server
cluster: obs
rule_files:
- /etc/prometheus/rules/*.yml
alerting:
alertmanagers:
- static_configs: [{targets: ['alertmanager:9093']}]
scrape_configs:
- job_name: prometheus
static_configs: [{targets: ['localhost:9090']}]# validate before deploying
promtool check config prometheus.yml
# Checking prometheus.yml
# SUCCESS: prometheus.yml is valid prometheus config file syntax
# run (container)
docker run -p 9090:9090 -v $PWD:/etc/prometheus \
prom/prometheus:v3.15.1 \
--config.file=/etc/prometheus/prometheus.yml \
--storage.tsdb.retention.time=15d \
--web.enable-lifecycle
# reload without restart
kill -HUP <pid>
curl -X POST localhost:9090/-/reload # needs --web.enable-lifecycle
# health and readiness
curl localhost:9090/-/healthy
curl localhost:9090/-/readyWith the Prometheus Operator (ch 5) you never edit this file: the operator generates it from ServiceMonitor, PodMonitor and PrometheusRule objects and reloads Prometheus for you.
Counter, gauge, histogram, summary — and native histograms — with naming rules and Go code for the shop app.
The type is a contract with the query language: it decides which functions make sense.
| Type | Semantics | Examples | Query with |
|---|---|---|---|
| Counter | Only goes up; resets to 0 on process restart | http_requests_total, orders_created_total, node_cpu_seconds_total | rate(), increase() — never the raw value |
| Gauge | Goes up and down; a current value | go_goroutines, node_memory_MemAvailable_bytes, queue depth, temperature | Raw value, avg_over_time, max_over_time, deriv, predict_linear |
| Histogram | Counts observations into buckets + sum + count | http_request_duration_seconds, payments_charge_amount | histogram_quantile() on rate() of buckets; ratios per bucket |
| Summary | Client-side quantiles + sum + count | Legacy Go GC pause summaries, some libraries | Read quantiles as-is; cannot aggregate quantiles across instances |
Info-style metrics are gauges with value 1 that carry metadata as labels, e.g. go_info{version="go1.27.1"}, kube_pod_info{node=...}; you join them with other series (ch 3).
Two subtle facts that explain most beginner query bugs.
rate() and increase() detect the drop and compensate.max - min of a counter yourself: a reset breaks it.orders_created_total, not orders_in_progress.code="500" at 0) so a first error is a visible increase, not a new series.http_requests_in_flight._over_time functions summarize gauges over a window (ch 3)._total).Each bucket counts observations less than or equal to its upper bound le. Buckets are counters, so you rate() them.
# TYPE http_request_duration_seconds histogram
http_request_duration_seconds_bucket{route="/orders",le="0.05"} 812
http_request_duration_seconds_bucket{route="/orders",le="0.1"} 950
http_request_duration_seconds_bucket{route="/orders",le="0.25"} 1003
http_request_duration_seconds_bucket{route="/orders",le="0.3"} 1010
http_request_duration_seconds_bucket{route="/orders",le="0.5"} 1018
http_request_duration_seconds_bucket{route="/orders",le="1"} 1022
http_request_duration_seconds_bucket{route="/orders",le="2.5"} 1044
http_request_duration_seconds_bucket{route="/orders",le="+Inf"} 1044
http_request_duration_seconds_sum{route="/orders"} 71.3
http_request_duration_seconds_count{route="/orders"} 1044
# 1010 of 1044 requests took <= 300 ms → 96.7%
# +Inf bucket == _count; _sum/_count = mean (68 ms)prometheus.DefBuckets: .005 … 10 s (11 buckets) — fine for web latency.Summaries compute quantiles inside each process. That is precise for one process and useless for the fleet.
| Property | Histogram (classic) | Summary |
|---|---|---|
| Aggregate across pods | Yes: sum bucket rates, then compute the quantile | No: averaging p99s is mathematically meaningless |
| Choose quantile after the fact | Yes, any quantile at query time | No, fixed in code (e.g. 0.5, 0.9, 0.99) |
| Accuracy | Limited by bucket width | Configurable error, exact per process |
| Client cost | Cheap: increment a counter | Higher: streaming quantile estimation |
| Server cost | One series per bucket | One series per quantile |
| SLO "fraction under X" | Exact if X is a bucket boundary | Not possible |
Default to histograms. Native histograms (next slide) remove most of the bucket-design pain.
Stable since Prometheus 3.8, still opt-in. Buckets are exponential and sparse, and the whole histogram is a single series.
_sum + _count.NativeHistogramBucketFactor (Go) — the client then exposes both classic and native.scrape_native_histograms: true in the scrape config; native histograms need the protobuf scrape format, which is then negotiated first.always_scrape_classic_histograms: true.histogram_quantile(0.99, sum(rate(http_request_duration_seconds[5m]))) — no _bucket, no le (more functions in ch 3).histogram_fraction estimates within the bucket; classic buckets at the threshold are exact.In this deck the shop app exposes both forms; Lab 2 scrapes native histograms with a local Prometheus so you can compare queries side by side.
Names are an API: dashboards, alerts and other teams depend on them. Follow the conventions and they explain themselves.
<namespace>_<what>_<unit>[_total]: http_request_duration_seconds, orders_created_total._total; _count, _sum, _bucket are reserved for histograms and summaries.requests_by_route_total is wrong).sum by or filter on.job, instance, pod) — Prometheus adds them.type="total" with its parts.http_requests_total{route="/orders/{id}"}http_requests_total{path="/orders/o-81231"}process_resident_memory_bytesmemory_usage_mbcache_requests_total{result="hit"}cache_hits_total + cache_requests_total that double countOne middleware gives RED for every route; r.Pattern (Go 1.23+) keeps route bounded. promauto registers on the default registry, which already has go_* and process_* collectors.
// apps/internal/metrics/metrics.go (module obs-lab)
var requests = promauto.NewCounterVec(prometheus.CounterOpts{
Name: "http_requests_total",
Help: "HTTP requests by method, route template and status code.",
}, []string{"method", "route", "code"})
var duration = promauto.NewHistogramVec(prometheus.HistogramOpts{
Name: "http_request_duration_seconds",
Help: "HTTP request latency.",
Buckets: []float64{.01, .025, .05, .1, .25, .3, .5, 1, 2.5, 5},
NativeHistogramBucketFactor: 1.1, // also expose native
NativeHistogramMaxBucketNumber: 100,
NativeHistogramMinResetDuration: time.Hour,
}, []string{"method", "route"})
type recorder struct {
http.ResponseWriter
code int
}
func (r *recorder) WriteHeader(c int) {
r.code = c; r.ResponseWriter.WriteHeader(c)
}// Middleware wraps the whole mux; the mux sets r.Pattern,
// e.g. "GET /orders/{id}", before calling the handler.
func Middleware(next http.Handler) http.Handler {
return http.HandlerFunc(func(w http.ResponseWriter,
r *http.Request) {
start := time.Now()
rec := &recorder{ResponseWriter: w, code: 200}
next.ServeHTTP(rec, r)
route := "unmatched" // 404/405
if _, p, ok := strings.Cut(r.Pattern, " "); ok {
route = p // "/orders/{id}"
}
requests.WithLabelValues(r.Method, route,
strconv.Itoa(rec.code)).Inc()
duration.WithLabelValues(r.Method, route).
Observe(time.Since(start).Seconds())
})
}
// apps/orders/main.go
mux.Handle("GET /metrics", promhttp.HandlerFor(prometheus.
DefaultGatherer, promhttp.HandlerOpts{EnableOpenMetrics: true}))
srv := &http.Server{Addr: ":8080",
Handler: metrics.Middleware(mux)}RED says the service works; business metrics say the business works. They are often the first to show a silent failure.
// apps/orders: count successful orders
var ordersCreated = promauto.NewCounter(prometheus.CounterOpts{
Name: "orders_created_total",
Help: "Orders created successfully.",
})
// in the POST /orders handler, after payments succeeded:
ordersCreated.Inc()
// apps/payments: distribution of charged amounts
var chargeAmount = promauto.NewHistogram(prometheus.HistogramOpts{
Name: "payments_charge_amount",
Help: "Charged amount per successful charge (currency units).",
Buckets: prometheus.ExponentialBuckets(1, 2, 12), // 1 … 2048
})
// in the POST /charge handler, after the charge:
chargeAmount.Observe(amount)ExponentialBuckets(start, factor, count) and LinearBuckets(start, width, count) generate bucket lists.Help.customer_id or currency with many values as labels.rate(orders_created_total[5m]) while RED looks fine = logic bug, not an outage — worth a ticket alert (ch 4).order_id) in logs (ch 6).An exemplar is one example observation (value + labels such as trace_id) stored next to a series. It is the bridge from a latency graph to one real trace.
// observe with an exemplar when the request is traced (ch 8)
sc := trace.SpanContextFromContext(r.Context())
obs := duration.WithLabelValues(r.Method, route)
if sc.IsSampled() {
obs.(prometheus.ExemplarObserver).ObserveWithExemplar(
secs, prometheus.Labels{"trace_id": sc.TraceID().String()})
} else {
obs.Observe(secs)
}
# OpenMetrics exposition (needs EnableOpenMetrics: true)
http_request_duration_seconds_bucket{route="/orders",le="2.5"} 1044 \
# {trace_id="4bf92f3577b34da6a3ce929d0e0e4736"} 2.004 1762610712.48--enable-feature=exemplar-storage (in-memory ring buffer, size set by storage.exemplars.max_exemplars)./api/v1/query_exemplars; Grafana shows them as dots on time series panels.How Prometheus finds targets, decides what to scrape, and rewrites labels before and after the scrape.
A scrape job is: where to find targets, how to talk to them, how to rewrite labels, and limits.
scrape_configs:
- job_name: shop-pods # becomes job="shop-pods" by default
scrape_interval: 15s # overrides global
scrape_timeout: 10s
metrics_path: /metrics # default
scheme: http # default
kubernetes_sd_configs: # WHERE: discovery
- role: pod
namespaces: {names: [shop]}
relabel_configs: # BEFORE scrape: targets
- source_labels: [__meta_kubernetes_pod_container_port_name]
regex: http
action: keep
- source_labels: [__meta_kubernetes_pod_label_app]
target_label: job
metric_relabel_configs: # AFTER scrape: samples
- source_labels: [__name__]
regex: go_gc_.*
action: drop
sample_limit: 5000 # LIMITS: fail scrape above
label_limit: 30scrape_interval: 1m; scrape_timeout: 10s; evaluation_interval: 1m.instance defaults to __address__ (host:port).__ are dropped after target relabeling (except they can be copied first).__param_<name> sets URL query parameters; __scheme__ and __metrics_path__ override scheme and path.ServiceMonitor produces exactly this kind of job (ch 5).up and the synthetic scrape metricsEvery scrape writes a few series about itself. They are your first stop when a graph is empty.
| Series | Meaning | Typical use |
|---|---|---|
up | 1 if the scrape succeeded, 0 if it failed | up == 0 alert per job; avg by (job) (up) |
scrape_duration_seconds | How long the scrape took | Close to scrape_timeout → slow exporter, will start failing |
scrape_samples_scraped | Samples exposed by the target | Spot a target that suddenly exposes 10× more |
scrape_samples_post_metric_relabeling | Samples kept after metric_relabel_configs | Check that your drop rules work |
scrape_series_added | New series created by this scrape | Churn detector: should be ~0 in steady state |
The Targets page (/targets) shows the last error per target ("context deadline exceeded", "connection refused", "sample limit exceeded"). Service discovery (/service-discovery) shows discovered labels before and after relabeling — including dropped targets.
Prometheus watches the API server and turns objects into targets with __meta_kubernetes_* labels you can relabel.
| Role | One target per | Useful meta labels | Use for |
|---|---|---|---|
node | Node (kubelet address) | __meta_kubernetes_node_name, _node_label_<name> | kubelet, cAdvisor |
pod | Declared container port (or pod IP if none) | _namespace, _pod_name, _pod_label_<name>, _pod_container_port_name, _pod_annotation_<name> | Apps without a Service; DaemonSets |
service | Service port | _service_name, _service_port_name | Black-box probing of Services |
endpoints | Endpoint address + port | service + pod labels | Apps behind a Service (classic) |
endpointslice | EndpointSlice endpoint + port | service + pod labels, _endpointslice_* | Same as endpoints; scales better |
ingress | Ingress path | _ingress_name, _ingress_path | Black-box probing of public URLs |
Meta label names are abbreviated above: each starts with __meta_kubernetes_ (e.g. __meta_kubernetes_pod_label_app). Label names are sanitized: app.kubernetes.io/name becomes ..._label_app_kubernetes_io_name.
Two different hooks with the same syntax. Mixing them up is the most common relabeling bug.
Dropping a metric in metric_relabel_configs saves storage and memory, but the target still exposes and Prometheus still parses it. Removing it at the source is cheaper when you own the code.
Each rule: join source_labels with separator (default ;), match against regex (fully anchored), then act.
| Action | Effect |
|---|---|
replace (default) | Set target_label to replacement (with $1 groups) |
keep / drop | Keep or drop the target / sample if regex matches |
keepequal / dropequal | Keep / drop if source equals target_label value |
labelmap | Copy labels whose names match regex |
labeldrop / labelkeep | Remove / retain labels by name |
hashmod | Set target_label to hash(source) mod modulus (sharding) |
lowercase / uppercase | Change case of the joined source value |
# copy all pod labels as-is
- action: labelmap
regex: __meta_kubernetes_pod_label_(.+)
# namespace and pod labels
- source_labels: [__meta_kubernetes_namespace]
target_label: namespace
- source_labels: [__meta_kubernetes_pod_name]
target_label: pod
# opt-in by annotation (prometheus.io/scrape=true)
- source_labels:
[__meta_kubernetes_pod_annotation_prometheus_io_scrape]
regex: "true"
action: keep
# drop a noisy label from samples (metric_relabel_configs)
- regex: pod_template_hash|controller_revision_hash
action: labeldrophonor_labels, limits and other safety settingsA target can expose labels that collide with the labels Prometheus attaches. And a broken target can flood you. Both are configurable.
job="x" and Prometheus wants job="shop".honor_labels: false: the scraped label is renamed exported_job.honor_labels: true: the scraped value wins — only for Pushgateway and federation, where the source labels are the truth.honor_timestamps: true (default) uses timestamps in the exposition if present.sample_limit: scrape fails (all samples rejected, up = 0) above N samples after relabeling.label_limit, label_name_length_limit, label_value_length_limit.target_limit: maximum targets per job after relabeling.body_size_limit: uncompressed response size.up == 0 and on scrapes failing because of limits (the target error says why).scrape_timeout < scrape_interval.scrape_series_added and prometheus_tsdb_head_series (ch 5).Metrics for software you do not own, black-box probes, batch jobs, OTLP pushes and federation.
An exporter translates a system's own stats into Prometheus metrics on scrape. It runs next to the system, not inside it.
9100; in kube-prometheus-stack it runs as a DaemonSet.node_cpu_seconds_total{mode} — CPU time per mode (counter).node_memory_MemAvailable_bytes, node_memory_MemTotal_bytes.node_filesystem_avail_bytes{mountpoint}, node_disk_io_time_seconds_total.node_network_receive_bytes_total{device}, node_load1.--collector.textfile.directory reads *.prom files written by cron jobs./metrics natively: CoreDNS, etcd, Argo CD, NGINX Ingress, Envoy.up metric loses meaning).Prometheus asks the exporter to probe a target; relabeling moves the real target into a URL parameter.
# blackbox.yml (exporter config)
modules:
http_2xx:
prober: http
timeout: 5s
http:
valid_status_codes: [] # empty = 2xx
preferred_ip_protocol: ip4
# prometheus.yml
- job_name: blackbox-http
metrics_path: /probe
params: {module: [http_2xx]}
static_configs:
- targets: ['http://orders.shop.svc:8080/healthz']
relabel_configs:
- source_labels: [__address__]
target_label: __param_target # ?target=...
- source_labels: [__param_target]
target_label: instance # keep the URL as instance
- target_label: __address__
replacement: blackbox:9115 # actually scrape the exporterprobe_success — 1 or 0: the symptom to alert on.probe_duration_seconds — total probe time.probe_http_status_code, probe_http_duration_seconds{phase} (resolve, connect, tls, processing, transfer).probe_ssl_earliest_cert_expiry — Unix time; alert at < 14 days: probe_ssl_earliest_cert_expiry - time() < 14*86400.up here means "the exporter answered", not "the target is up".Probe CRD generates this job (ch 5).For values you already have elsewhere (a queue length, a pool size), compute them at scrape time instead of updating gauges constantly.
type queueCollector struct {
depth *prometheus.Desc
q *Queue
}
func newQueueCollector(q *Queue) *queueCollector {
return &queueCollector{q: q, depth: prometheus.NewDesc(
"orders_queue_depth", // name
"Orders waiting for payment.", // help
[]string{"priority"}, nil)} // variable, const labels
}
func (c *queueCollector) Describe(ch chan<- *prometheus.Desc) {
ch <- c.depth
}
func (c *queueCollector) Collect(ch chan<- prometheus.Metric) {
for prio, n := range c.q.DepthByPriority() { // called on each scrape
ch <- prometheus.MustNewConstMetric(
c.depth, prometheus.GaugeValue, float64(n), prio)
}
}
// main: prometheus.MustRegister(newQueueCollector(q))Counter.Inc(), Histogram.Observe()): events in your own code./proc, an API.GaugeValue or CounterValue correctly: a counter from an external source must never go down except on reset.testutil.CollectAndCompare from client_golang/prometheus/testutil.A service-level batch job (a nightly backup) finishes before any scrape. It pushes its result; Prometheus scrapes the Pushgateway.
# end of a nightly job: push the result, grouped by job name
cat <<EOF | curl --data-binary @- \
http://pushgateway:9091/metrics/job/orders_backup
# TYPE backup_last_success_timestamp_seconds gauge
backup_last_success_timestamp_seconds $(date +%s)
# TYPE backup_duration_seconds gauge
backup_duration_seconds 312
EOF
# Prometheus scrape job for the gateway
- job_name: pushgateway
honor_labels: true # keep job="orders_backup" from the push
static_configs: [{targets: ['pushgateway:9091']}]
# alert: backup older than 26 hours
time() - backup_last_success_timestamp_seconds > 26*3600DELETE /metrics/job/<name>) or alert on the timestamp.up reflects the Pushgateway, not the jobs.Two ways to get metrics into a Prometheus without scraping the source directly.
# Prometheus 3: accept OTLP metrics pushes
prometheus --web.enable-otlp-receiver ...
# endpoint: http://prometheus:9090/api/v1/otlp/v1/metrics
# prometheus.yml
otlp:
promote_resource_attributes: # resource attrs → labels
- service.name
- deployment.environment.name
- k8s.namespace.name
# translation_strategy: UnderscoreEscapingWithSuffixes (default)
# http.server.request.duration → http_server_request_duration_seconds
storage:
tsdb:
out_of_order_time_window: 30m # recommended for pushed OTLP
# OTel SDK env: OTEL_EXPORTER_OTLP_METRICS_ENDPOINT=
# http://prometheus:9090/api/v1/otlp/v1/metricsservice.namespace/service.name → job, service.instance.id → instance; other resource attributes go to target_info unless promoted.up and no staleness markers./federate?match[]={job="orders"} exposes current series for another Prometheus to scrape. Fine for a few aggregated series (recording rules) in a global view; not for copying everything — large responses are slow and time out. Prefer remote write to Thanos, Mimir or AMP (ch 5).Each one has caused an outage of either the app or the monitoring.
latency_ms) instead of base units.metric_relabel_configs instead of relabel_configs.honor_labels: true on normal app jobs → apps overwrite job/instance.sample_limit → one bad deploy OOMs Prometheus.scrape_timeout equal to a 1 s interval.promtool check config and promtool check metrics in CI./metrics of every new service in review.up == 0 and on scrape_series_added spikes.Explain with the exact config field, function or metric.
route bounded?relabel_configs vs metric_relabel_configs: when does each run??target=?sample_limit?honor_labels: true correct?Practice: Lab 2 in Hands-on labs — instrument orders and payments with client_golang, scrape them with ServiceMonitors, then compare classic and native histograms in a local Prometheus with a blackbox probe.
The query language behind every Prometheus graph, alert and recording rule: selectors, rates, aggregation, joins, histograms and the traps that produce confident wrong answers.
What a PromQL expression returns, how series are selected, and how time enters a query.
Most PromQL errors ("expected type range vector") are type errors. Know what each expression returns before you nest it.
| Type | What it is | Example |
|---|---|---|
| Instant vector | One sample per series, all at the same evaluation time | http_requests_total{job="orders"} |
| Range vector | A window of samples per series, ending at the evaluation time | http_requests_total{job="orders"}[5m] |
| Scalar | A single floating-point number, no labels | 0.05, scalar(...), time() |
| String | A string literal; rarely used outside label_replace | "orders" |
rate, increase, avg_over_time take a range vector and return an instant vector.sum, max, …) take and return instant vectors.Both are evaluated "at" time t. An instant selector takes the latest sample within the lookback window; a range selector takes every sample inside the range.
Prometheus 3 made range selectors and the lookback window left-open: a sample exactly at t−5m is excluded. Lookback is set by --query.lookback-delta (default 5m).
A selector is a metric name plus label matchers. Regular expressions are RE2 and fully anchored.
# exact match and negative match
http_requests_total{job="orders", code!="200"}
# regex (anchored: "5.." matches 500..599, not "1500")
http_requests_total{job=~"orders|payments", code=~"5.."}
# negative regex
http_requests_total{route!~"/healthz|/metrics"}
# name as a label: every metric starting with http_
{__name__=~"http_.*", job="orders"}
# Prometheus 3: UTF-8 names must be quoted
{"http.server.request.duration", "http.route"="/orders"}
# INVALID: matches every series in the TSDB
{job=~".*"} # error: needs a matcher that can't be empty= equal, != not equal, =~ regex match, !~ regex no-match.{team=""} selects series without team.=~"orders" is the same as ="orders" (anchored), but slower.__name__; foo{} ≡ {__name__="foo"}.offset and @: move the evaluation timeoffset shifts relative to the evaluation time; @ pins an absolute time. Both apply to a single selector, not the whole expression.
# requests per second now vs exactly one week ago
sum(rate(http_requests_total{job="orders"}[5m]))
/
sum(rate(http_requests_total{job="orders"}[5m] offset 1w))
# negative offset (look "ahead" inside a past range query)
rate(http_requests_total[5m] offset -10m)
# pin to a Unix timestamp
http_requests_total{job="orders"} @ 1762560000
# the value at the end of the graph's range, for every step
topk(5, rate(http_requests_total[5m] @ end()))offset 1w).@ end().start() and end() refer to the query range; in an instant query both equal the evaluation time.[5m] offset 1w or @ … offset … attached to the selector.A graph is many instant evaluations: one per step between start and end. Alerts and recording rules are instant evaluations.
# instant query: evaluate once at "time" (default: now)
curl -sG localhost:9090/api/v1/query \
--data-urlencode 'query=sum by (job) (rate(http_requests_total[5m]))' \
| jq -r '.data.result[] | "\(.metric.job) \(.value[1])"'
# orders 5.03
# payments 5.03
# range query: evaluate at start, start+step, … end
curl -sG localhost:9090/api/v1/query_range \
--data-urlencode 'query=sum(rate(http_requests_total[5m]))' \
--data-urlencode "start=$(date -u -v-1H +%s)" \
--data-urlencode "end=$(date -u +%s)" \
--data-urlencode 'step=60' | jq '.data.result[0].values | length'
# 61
# from the CLI
promtool query instant http://localhost:9090 'up{namespace="shop"}'resultType is vector, matrix, scalar or string.[5m] is re-evaluated at every step: a step larger than the range skips data.step from the panel width ($__interval); a 30-day panel may step every few minutes."5.03") to keep float precision; NaN and +Inf are possible.date -v-1H is BSD/macOS; on Linux use date -u -d "-1 hour" +%s.Write queries where you can see intermediate results, then move them into rules and dashboards.
$__interval and $__rate_interval resolution.count(...) before sum(...) to know how many series you aggregate.promtool (ch 4).Counters only go up — except when they reset. Every useful counter query turns them into per-second rates.
http_requests_total is a running total since process start. Its absolute value depends on uptime, and it drops to zero on restart.
# raw value: grows forever, drops on restart
http_requests_total{job="orders", code="200"}
# per-second rate over 5m, reset-safe
rate(http_requests_total{job="orders", code="200"}[5m])
# total increase over 1h (= rate × 3600)
increase(orders_created_total[1h])
# how many times did the counter reset in 1h?
resets(http_requests_total{job="orders"}[1h])_total (ch 2).rate.increase.rate looks at all samples in the range, adds back any drop as a reset, and extrapolates to the window boundaries.
Because of extrapolation, increase() returns non-integers (e.g. 59.8 requests) even for integer counters. It is an estimate, not a count. Use rate/increase only on counters; on a gauge a decrease is misread as a reset.
rate vs irate vs increaseSame input, different questions. rate is the default for dashboards and the only sane choice for alerts.
| Function | Uses | Output | Use for | Avoid for |
|---|---|---|---|---|
rate(c[5m]) | All samples in the window | Average per-second rate over the window | Dashboards, alerts, recording rules | — |
irate(c[5m]) | Only the last two samples in the window | Instantaneous per-second rate | Zoomed-in, high-resolution graphs of fast-moving counters | Alerts (too spiky); long-range graphs (steps skip most data) |
increase(c[1h]) | All samples in the window | rate × window seconds | Human-readable totals: "orders in the last hour" | Summing many short windows to get a total (edge errors add up) |
delta(g[1h]) | First and last sample | Change of a gauge | Gauges such as queue length | Counters (no reset handling) |
deriv(g[10m]) | Linear regression of all samples | Per-second slope of a gauge | Trend of memory or disk usage | Counters |
rate needs at least two samples in the window, otherwise the series disappears from the result.
[range] be?Too short and series flicker in and out; too long and spikes vanish. The scrape interval sets the floor.
scrape_interval: 15s, use at least [1m]; [5m] is a common default.$__rate_intervalmax($__interval + scrape interval, 4 × scrape interval).rate/increase in dashboards (ch 10).[5m], [1h]) so the meaning never changes.…:rate5m (ch 4).rate(http_requests_total{job="orders"}[$__rate_interval]) # Grafana panel
rate(http_requests_total{job="orders"}[5m]) # alert / recording rule
rate first, then sumEach pod has its own counter that resets independently. Summing raw counters first creates a fake series whose "resets" are restarts of one pod.
# RIGHT: per-series rate, then aggregate
sum by (job) (rate(http_requests_total[5m]))
# WRONG (needs a subquery to even parse): sum, then rate
rate(sum by (job) (http_requests_total)[5m:30s])
# one pod restarts → the sum drops by that pod's total
# → rate sees a "reset" → huge false spike
# RIGHT: ratios are ratios of sums of rates
sum by (job) (rate(http_requests_total{code=~"5.."}[5m]))
/
sum by (job) (rate(http_requests_total[5m]))rate detects resets per series; once series are summed, a drop in one pod is indistinguishable from a reset of the total.rate results, never sums of raw counters (ch 4).avg(error_ratio)) gives a quiet pod the same weight as a busy one: divide sums instead.orders and shows the false spike side by side.Gauges go up and down: queue length, memory in use, free disk. Do not rate them; summarize them over time.
# memory trend (bytes per second) of shop pods
deriv(container_memory_working_set_bytes{namespace="shop",container!=""}[15m])
# will the root filesystem fill within 4 hours?
predict_linear(node_filesystem_avail_bytes{mountpoint="/"}[6h], 4 * 3600) < 0
# peak and average over a window
max_over_time(container_memory_working_set_bytes{namespace="shop"}[1h])
avg_over_time(kube_deployment_status_replicas_available{namespace="shop"}[1h])
# how often did a value change? (config reloads, flapping state)
changes(kube_pod_container_status_restarts_total{namespace="shop"}[1h])*_over_time familyavg_, min_, max_, sum_, count_, last_, stddev_, quantile_over_time.present_over_time and absent_over_time test presence (section 11).sum by aggregates across series at one time.predict_linear fits a line: good for disks, poor for bursty memory.Collapse series with by/without, then combine vectors with arithmetic, comparisons and joins.
by / withoutAggregations work across series at a single time. by keeps only the listed labels; without drops the listed labels and keeps the rest.
| Operator | Result |
|---|---|
sum, avg, min, max | Arithmetic across series |
count | Number of series in each group |
group | Value 1 per group (only the label set matters) |
stddev, stdvar | Spread across series |
topk(k, v), bottomk(k, v) | k largest / smallest series, original labels kept |
quantile(φ, v) | φ-quantile across series (not latency!) |
count_values("lbl", v) | Count series per distinct value |
limitk, limit_ratio | Experimental in 3.x (feature flag) |
# RPS per job and route (all other labels dropped)
sum by (job, route) (rate(http_requests_total[5m]))
# same, but keep everything except pod and instance
sum without (pod, instance) (rate(http_requests_total[5m]))
# how many shop pods are being scraped?
count by (job) (up{namespace="shop"})
# how many deployments run 1, 2, 3 … replicas?
count_values("replicas", kube_deployment_spec_replicas{namespace="shop"})
# containers per image in the shop namespace
count by (image) (kube_pod_container_info{namespace="shop"})without is safer in shared rules: labels added later (e.g. cluster) survive. by is clearer on dashboards.
topk and friends in graphs: a classic surprisetopk is evaluated at every step of a range query, so a graph of "top 5" can show many more than 5 lines.
bottomk and for sort (which has no effect in range queries).topk(5, … @ end()) then filter with and (below).topk(5, sum by (route) (rate(x[1h]))).# top 5 routes by traffic, chosen once at the end
sum by (route) (rate(http_requests_total[5m]))
and on (route)
topk(5, sum by (route) (rate(http_requests_total[1h] @ end())))
Operators work between scalars and vectors. Comparisons filter by default; add bool to get 0/1 instead.
# vector ∘ scalar: applied to every sample
container_memory_working_set_bytes{namespace="shop"} / 1024 / 1024
# comparison filters: only series above 0.5 s remain
job_route:http_request_duration_seconds:p99_5m > 0.5
# bool: keeps all series, value 1 (true) or 0 (false)
job_route:http_request_duration_seconds:p99_5m > bool 0.5
# precedence, highest first
# ^
# * / % atan2
# + -
# == != <= < >= >
# and unless
# or
a + b * c # = a + (b * c)
2 ^ 3 ^ 2 # ^ is right-associative: 2 ^ 9 = 512bool is required when comparing two scalars.+Inf (x/0) or NaN (0/0); every comparison with NaN is false, so a NaN ratio never fires an alert.on and ignoringFor a / b Prometheus pairs series whose label sets are identical. When they differ, tell it which labels to match on.
# error ratio per job: both sides have exactly {job}
sum by (job) (rate(http_requests_total{code=~"5.."}[5m]))
/
sum by (job) (rate(http_requests_total[5m]))
# without aggregation the left side has code="500",
# the right side has code="200" etc. → no pairs
rate(http_requests_total{code="500"}[5m])
/ ignoring (code)
sum without (code) (rate(http_requests_total[5m]))
# match only on job and route
sum by (job, route, code) (rate(http_requests_total{code=~"5.."}[5m]))
/ on (job, route) group_left
sum by (job, route) (rate(http_requests_total[5m]))on (l1, l2): match only on these labels.ignoring (l1): match on all labels except these.code from the left side, so it is many-to-one: it needs group_left (next slide).group_leftThe classic use: copy a label from an "info" metric (value always 1) onto a usage metric, e.g. which node a pod runs on.
# CPU cores used per pod, labelled with its node
sum by (namespace, pod) (
rate(container_cpu_usage_seconds_total{
namespace="shop", container!=""}[5m])
)
* on (namespace, pod) group_left (node)
kube_pod_info
# then roll up per node
sum by (node) (
sum by (namespace, pod) (
rate(container_cpu_usage_seconds_total{
container!=""}[5m]))
* on (namespace, pod) group_left (node)
kube_pod_info
)group_left = the left side has many series per match; group_right the reverse. Labels listed in the parentheses are copied from the "one" side. If the "one" side has duplicates (e.g. two kube-state-metrics replicas), the join fails with "found duplicate series" — aggregate it first with max by (namespace, pod, node).
and, or, unless: filter by presence, not valueSet operators match label sets like on(); the value always comes from the left side (or whichever side exists, for or).
# alert on error ratio only where there is real traffic
job:http_request_errors:ratio_rate5m > 0.05
and on (job)
sum by (job) (rate(http_requests_total[5m])) > 1
# series that exist on the left but not on the right:
# deployments with fewer available replicas than desired
kube_deployment_spec_replicas{namespace="shop"}
unless on (namespace, deployment)
(kube_deployment_spec_replicas == kube_deployment_status_replicas_available)
# fill a missing series with 0 (careful: no labels!)
sum(rate(http_requests_total{code=~"5.."}[5m])) or vector(0)a and b: series of a that have a match in b.a unless b: series of a with no match in b.a or b: all of a, plus series of b not matched in a.vector(0) has no labels: or vector(0) only works when the left side is aggregated to no labels too.on (…) to restrict which labels decide the match.Rate, Errors, Duration per service and route. These become recording rules in ch 4 and dashboard panels in ch 10.
# Rate: requests per second per service and route
sum by (job, route) (rate(http_requests_total{namespace="shop"}[5m]))
# Errors: share of 5xx responses per service
sum by (job) (rate(http_requests_total{namespace="shop", code=~"5.."}[5m]))
/
sum by (job) (rate(http_requests_total{namespace="shop"}[5m]))
# Duration: p99 latency per service and route (classic histogram)
histogram_quantile(0.99,
sum by (job, route, le) (rate(http_request_duration_seconds_bucket{namespace="shop"}[5m])))
# Duration: mean latency (sum / count)
sum by (job) (rate(http_request_duration_seconds_sum{namespace="shop"}[5m]))
/
sum by (job) (rate(http_request_duration_seconds_count{namespace="shop"}[5m]))
# Business: orders per minute, and charged amount per minute
sum(rate(orders_created_total[5m])) * 60
sum(rate(payments_charge_amount_sum[5m])) * 60
# Saturation of the service's pods: CPU used / CPU requested
sum by (pod) (rate(container_cpu_usage_seconds_total{namespace="shop", container!=""}[5m]))
/
sum by (pod) (kube_pod_container_resource_requests{namespace="shop", resource="cpu"})
Latency percentiles done right, nested time windows, missing-series checks and query cost.
histogram_quantile estimates a percentileA classic histogram is a set of cumulative counters, one per bucket upper bound le. The quantile is interpolated inside one bucket.
# p99 per route over the last 5 minutes
histogram_quantile(0.99,
sum by (job, route, le) (
rate(http_request_duration_seconds_bucket{job="orders"}[5m])
)
)
# keep le in the by() clause: without it the
# series have no le label and the result is emptyRules: rate the _bucket series, sum by (…, le), then histogram_quantile. If the quantile lands in the +Inf bucket, the result is the largest finite bound. Averaging percentiles across pods is meaningless; aggregate buckets instead.
The answer can only be as precise as the bucket containing the quantile. Choose buckets around your SLO threshold.
count by (le) (http_request_duration_seconds_bucket{job="orders"}).# share of requests faster than 300 ms
# (exact only if a bucket le="0.3" exists)
sum by (job) (rate(
http_request_duration_seconds_bucket{le="0.3"}[5m]))
/
sum by (job) (rate(
http_request_duration_seconds_count[5m]))
# Apdex-style score: satisfied ≤ 0.3 s,
# tolerating ≤ 1 s
(
sum(rate(http_request_duration_seconds_bucket{
job="orders", le="0.3"}[5m]))
+ sum(rate(http_request_duration_seconds_bucket{
job="orders", le="1"}[5m]))
) / 2
/ sum(rate(http_request_duration_seconds_count{
job="orders"}[5m]))
# le="1" includes the satisfied requests,
# so (S + (S + T)) / 2 = S + T / 2Stable since Prometheus 3.8 but opt-in (scrape_native_histograms). One series holds the whole distribution with exponential buckets: no _bucket, no le.
# p99: same function, no _bucket suffix, no le label
histogram_quantile(0.99,
sum by (job) (rate(http_request_duration_seconds{job="orders"}[5m])))
# request rate and mean latency from the same series
histogram_count(sum by (job) (rate(http_request_duration_seconds[5m])))
histogram_avg(sum by (job) (rate(http_request_duration_seconds[5m])))
# share of requests between 0 and 0.3 s
histogram_fraction(0, 0.3,
sum by (job) (rate(http_request_duration_seconds[5m])))
# spread
histogram_stddev(sum by (job) (rate(http_request_duration_seconds[5m])))histogram_fraction for any threshold, not only existing bucket bounds.NativeHistogramBucketFactor) and Prometheus must scrape them (ch 2).expr[range:resolution] evaluates expr at each resolution step inside range, producing a range vector you can feed to *_over_time.
# highest 5m request rate seen in the last hour
max_over_time(
sum by (job) (rate(http_requests_total{namespace="shop"}[5m]))[1h:1m]
)
# was the error ratio above 5% for at least 10 of the last 30 minutes?
count_over_time(
(job:http_request_errors:ratio_rate5m > 0.05)[30m:1m]
) >= 10
# resolution omitted → global evaluation_interval
avg_over_time(job:http_request_errors:ratio_rate5m[1h:])[1h:1m] = 60 evaluations per step of the outer query.max_over_time(rule[1h]) reads stored samples.rate a sum of counters (section 09).An expression over nothing returns nothing — not zero. Alerts on "error rate > 5%" stay silent when the target vanishes.
# 1 with labels {job="orders"} when no such series exists
absent(up{job="orders"})
# no sample at all in the last 10 minutes
absent_over_time(http_requests_total{job="orders"}[10m])
# target exists but scrape fails
up{namespace="shop"} == 0
# derive a label: instance "10.244.1.7:8080" → ip "10.244.1.7"
label_replace(up{namespace="shop"}, "ip", "$1", "instance", "(.*):.*")
# join two labels into one
label_join(up{namespace="shop"}, "where", "/", "namespace", "pod")absent() only copies labels from equality matchers in the selector; regex matchers give no labels.absent per critical job; up == 0 tells you which instance is down.label_replace(v, dst, replacement, src, regex): anchored regex; if it does not match, the series is returned unchanged.time(), timestamp(v), hour(), day_of_week() (UTC).time() - timestamp(up{job="orders"}).Cost ≈ series selected × samples per series × steps. Dashboards multiply it by panels and viewers.
rate(http_requests_total[5m]) over the whole cluster.[7d] on per-pod buckets.=~".*orders.*" on many series.job, namespace, route.pod and instance early.$__rate_interval and sensible panel resolutions.--query.timeout (default 2m) and --query.max-samples (default 50,000,000).query_log_file in the global config.prometheus_engine_query_duration_seconds.Each of these returns a number. That is what makes them dangerous.
sum then rate (false spikes on every restart).rate on a gauge (decreases read as resets).[30s] window with a 30s scrape interval → empty or flickering results.irate in alerts.increase.le before histogram_quantile → empty result.group_left against a metric that has duplicates.topk in a range graph showing more than k lines.error_ratio > 0.05 with no traffic: 0/0 = NaN, the alert never fires and never tells you why.absent() / up == 0 alert: a dead target looks healthy.or vector(0) on a labelled expression.Write the query first, then explain why it is correct.
{job=~"orders"} match, and why is {job=~".*"} rejected?increase() return 59.8 instead of 60?rate window be with a 30s scrape interval?rate come before sum?without over by?node label onto per-pod memory usage.a > bool 0.5 return, compared to a > 0.5?Practice: Lab 3 in Hands-on labs — RED queries on the shop app via the HTTP API, a node join, injected errors and latency, and the sum-then-rate trap during a restart.
Precompute what dashboards and alerts need, write alerts that wake people only for real symptoms, route them with Alertmanager, and test all of it before it ships.
Store the result of an expensive or shared expression as a new series.
A recording rule evaluates a PromQL expression on a schedule and writes the result back to the TSDB as a new metric.
Rules live in groups. Groups run in parallel; rules inside a group run in order, so later rules can use earlier results from the same evaluation.
# rules/shop.rules.yml
groups:
- name: shop-red
interval: 30s # default: global evaluation_interval
rules:
- record: job_route:http_requests:rate5m
expr: |
sum by (job, route) (
rate(http_requests_total{namespace="shop"}[5m]))
- record: job:http_requests:rate5m
expr: sum by (job) (job_route:http_requests:rate5m)
- record: job:http_request_errors:rate5m
expr: |
sum by (job) (rate(
http_requests_total{namespace="shop", code=~"5.."}[5m]))
- record: job:http_request_errors:ratio_rate5m
expr: job:http_request_errors:rate5m / job:http_requests:rate5m
- record: job_route:http_request_duration_seconds:p99_5m
expr: |
histogram_quantile(0.99, sum by (job, route, le) (rate(
http_request_duration_seconds_bucket{namespace="shop"}[5m])))
labels:
quantile: "0.99" # static labels added to the resultgroups[].name must be unique per file.interval: evaluation interval for the group.limit: max series a rule may produce (0 = no limit); exceeding it fails the rule.record: the new metric name (must be a valid metric name).expr: any PromQL expression returning an instant vector.labels: added or overwritten on every result series.rule_files: in prometheus.yml, or from PrometheusRule objects with the Operator (section 14).level:metric:operations conventionFrom the Prometheus docs: the name tells you which labels remain, which metric it came from, and what was done to it.
| Part | Meaning | Example |
|---|---|---|
| level | Labels kept by the aggregation, joined by _ | job_route, job, namespace |
| metric | Source metric name; drop _total after rate | http_requests |
| operations | Operations applied, newest first | rate5m, ratio_rate5m, p99_5m |
job_route:http_requests:rate5m — sum by (job, route) (rate(…[5m])).job:http_request_errors:ratio_rate5m — 5xx share per job over 5m.namespace_pod:container_cpu_usage:rate5m — CPU per pod.job:http_request_duration_seconds:p99_5m — p99 per job._seconds, _bytes).Colons are reserved for recording rules: never use them in metric names you instrument.
Rules are reused far from where they were written. The two common errors are averaging ratios and re-aggregating percentiles.
# RIGHT: per-job ratio from per-job sums
job:http_request_errors:ratio_rate5m =
job:http_request_errors:rate5m / job:http_requests:rate5m
# WRONG: average of per-route ratios
avg by (job) (job_route:http_request_errors:ratio_rate5m)
# a route with 1 request/hour weighs as much as /orders
# RIGHT: per-job p99 from buckets
histogram_quantile(0.99,
sum by (job, le) (rate(http_request_duration_seconds_bucket[5m])))
# WRONG: average or max of per-route p99s
avg by (job) (job_route:http_request_duration_seconds:p99_5m)
# Longer windows: build from rates, not from ratios
job:http_request_errors:ratio_rate1h =
sum by (job) (rate(http_requests_total{code=~"5.."}[1h]))
/ sum by (job) (rate(http_requests_total[1h]))le in recorded bucket rates if you need percentiles at a higher level.An alert is a PromQL condition plus a duration, labels for routing and annotations for the human.
Every series returned by expr is one alert instance. No series → no alert.
groups:
- name: shop-alerts
rules:
- alert: ShopHighErrorRate
expr: |
job:http_request_errors:ratio_rate5m{job=~"orders|payments"}
> 0.05
and on (job) job:http_requests:rate5m > 0.5
for: 5m
keep_firing_for: 5m
labels:
severity: critical
team: shop
annotations:
summary: >-
{{ $labels.job }}: {{ $value | humanizePercentage }}
of requests fail
description: >-
5xx ratio above 5% for 5 minutes on {{ $labels.job }}.
Check payments FAIL_RATE and recent deploys.
runbook_url: >-
https://runbooks.example.internal/shop/high-error-ratealert: name, becomes label alertname.expr: condition; the value is available as $value.for: how long the condition must hold before firing (pending state).keep_firing_for: keep firing this long after the condition clears (avoids flapping resolves; Prometheus ≥ 2.42).labels: used by Alertmanager for routing, grouping and inhibition — keep them few and stable.annotations: text for humans; templated; never used for routing.and on (job) … > 0.5 guard stops alerts on 1 failed request out of 2.Prometheus evaluates the rule every interval and sends active alerts to Alertmanager repeatedly; Alertmanager decides whom to notify and when.
Effective delay to the first notification ≈ rule interval + for + Alertmanager group_wait. With 30s + 5m + 30s, expect ~6 minutes. That is a design decision, not an accident.
Annotations use Go templates. Give the number, the scope, the likely cause and a link.
annotations:
summary: >-
p99 {{ $value | humanizeDuration }}
on {{ $labels.job }} {{ $labels.route }}
description: |
p99 latency above 500 ms for 10 minutes.
Cluster: {{ $externalLabels.cluster }}
Dashboard: https://grafana.example.internal/d/shop-red
?var-job={{ $labels.job }}
runbook_url: >-
https://runbooks.example.internal/shop/{{ $labels.alertname }}
# rendered:
# summary: p99 742ms on payments /charge$labels (or .Labels): labels of the alert instance.$value (or .Value): value of expr for this instance.$externalLabels, $externalURL.humanize, humanize1024, humanizeDuration, humanizePercentage, humanizeTimestamp, printf, query.$value is the value at the current evaluation; it changes over time — do not put it in labels (it would create a new alert every evaluation).Users feel symptoms (errors, latency, unavailability). Causes (CPU, disk, a pod restart) matter only when they lead to symptoms or will soon.
| Alert | Kind | Severity / action |
|---|---|---|
5xx ratio > 5% for 5m on orders | Symptom | critical → page now |
| p99 latency > 500 ms for 10m | Symptom | critical or warning, depending on the SLO |
up == 0 for all orders pods | Symptom (total outage of a dependency) | critical → page |
| Disk predicted full within 4 h | Imminent cause | warning → page in working hours or ticket |
| CPU > 90% on one node | Cause | Dashboard, not an alert (unless it breaks something) |
| One pod restarted once | Cause | Nothing; repeated CrashLoopBackOff → warning ticket |
| Prometheus cannot reach Alertmanager | Meta-monitoring (ch 13) | critical via an independent path |
Every page must be urgent, actionable and need a human. If the response is "wait and see", it is a ticket. Paging policy and SLO-based paging are covered in the SRE deck.
A noisy alert trains people to ignore the pager. These knobs remove most noise without hiding real problems.
for: filters short spikes; typical 2–15 m.rate window ≥ for / 2, so the condition is smooth.keep_firing_for: prevents resolve → fire → resolve cycles.unless).and on (job) job:http_requests:rate5m > 0.5.unless on (namespace) kube_namespace_labels{label_maintenance="true"} (needs that label exported).job), not on every pod, unless the pod matters on its own.up == 0 per job, and absent(up{job="orders"}) when the target disappears entirely.up plus scrape_samples_scraped == 0.prometheus_rule_evaluation_failures_total (ch 13).Watchdog (always firing) to prove the pipeline works end to end.Three symptom alerts and one presence alert. Lab 4 loads a trimmed version of this set (no ShopTargetDown) as a PrometheusRule.
- alert: ShopHighErrorRate
expr: job:http_request_errors:ratio_rate5m{job=~"orders|payments"} > 0.05 and on (job) job:http_requests:rate5m > 0.5
for: 5m
labels: {severity: critical, team: shop}
annotations: {summary: "{{ $labels.job }}: {{ $value | humanizePercentage }} 5xx"}
- alert: ShopHighLatencyP99
expr: |
histogram_quantile(0.99, sum by (job, le) (rate(http_request_duration_seconds_bucket{job=~"orders|payments"}[5m]))) > 0.5
for: 10m
labels: {severity: warning, team: shop}
annotations: {summary: "{{ $labels.job }} p99 {{ $value | humanizeDuration }}"}
- alert: ShopTargetDown
expr: up{job=~"orders|payments"} == 0
for: 2m
labels: {severity: warning, team: shop}
annotations: {summary: "{{ $labels.job }} pod {{ $labels.pod }} not scraped"}
- alert: ShopJobAbsent
expr: absent(up{job="orders"})
for: 5m
labels: {severity: critical, team: shop, namespace: shop} # absent() has no namespace
annotations: {summary: "No orders targets at all: deployment or ServiceMonitor gone"}
Threshold alerts ("5% errors") ignore how much failure your SLO can afford. Burn-rate alerts page on how fast the error budget is being spent.
# page: 2% of the 30-day budget in 1 hour (SLO 99.9%)
(
job:http_request_errors:ratio_rate1h{job="orders"} > (14.4 * 0.001)
and
job:http_request_errors:ratio_rate5m{job="orders"} > (14.4 * 0.001)
)SLIs, SLOs, error budgets and the full multiwindow, multi-burn-rate alert table are covered in the SRE deck. The recording rules from section 12 are its inputs.
Deduplicate, group, route, inhibit and silence alerts, then deliver notifications.
Prometheus only says "these alerts are active". Alertmanager turns that stream into a few notifications for the right people.
Prometheus re-sends active alerts periodically and sets an end time; if Alertmanager stops hearing about an alert, it resolves it. That is why a dead Prometheus looks like "all clear" — hence Watchdog (ch 13).
Alerts with equal values for the group_by labels go into one notification. Timers decide how quickly and how often that group is (re)sent.
route:
receiver: default
group_by: [alertname, namespace, job]
group_wait: 30s # wait for more alerts before the first message
group_interval: 5m # min time before sending an updated group
repeat_interval: 4h # re-send an unchanged, still-firing group
# group_by: ['...'] disables grouping (one message per alert)| Timer | Default | Effect |
|---|---|---|
group_wait | 30s | First notification of a new group is delayed to collect siblings |
group_interval | 5m | New or resolved alerts in an existing group wait at most this long |
repeat_interval | 4h | Reminder for a group with no changes; set ≥ group_interval |
Example: 3 orders pods go down within 20 s → one message listing 3 alerts after ~30 s. A 4th pod 2 minutes later → an updated message at the next group_interval tick, not immediately.
Every alert enters at the root and walks down. The first matching child wins unless it sets continue: true. Children inherit settings they do not override.
route:
receiver: default-slack # fallback for everything
group_by: [alertname, namespace]
routes:
- matchers: [alertname = "Watchdog"]
receiver: deadmans-switch
repeat_interval: 1m
- matchers: [team = "shop", severity = "critical"]
receiver: shop-pager
group_wait: 10s
continue: true # also try the next routes
- matchers: [team = "shop"]
receiver: shop-slack
routes:
- matchers: [severity = "info"]
receiver: "null" # drop
- matchers: [namespace =~ "kube-system|monitoring"]
receiver: platform-slack
receivers:
- name: "null" # no configs = drop
# … other receivers on the next slidematchers: list of label op "value" with =, !=, =~, !~; all must match (AND).match / match_re fields are deprecated; use matchers.amtool config routes test (section 15).shop-pager and shop-slack.A receiver is a named list of integrations. One receiver can notify several systems.
receivers:
- name: shop-slack
slack_configs:
- api_url_file: /etc/alertmanager/secrets/slack-url
channel: '#shop-alerts'
send_resolved: true
title: '{{ template "slack.shop.title" . }}'
text: '{{ template "slack.shop.text" . }}'
- name: shop-pager
pagerduty_configs:
- routing_key_file: /etc/alertmanager/secrets/pd-key # Events API v2
- name: lab-webhook
webhook_configs:
- url: http://alert-sink.monitoring.svc:8080/
send_resolved: true
- name: oncall-mail
email_configs:
- to: shop-oncall@example.com # needs global smtp_* settings*_file fields (api_url_file, routing_key_file) to keep secrets out of the config.send_resolved defaults differ: true for webhook and PagerDuty, false for Slack and email.status, alerts[] (labels, annotations, startsAt, endsAt), groupLabels, commonLabels, externalURL.Inhibition suppresses alerts that are consequences of another firing alert. Silences mute matching alerts for a time window, with an author and a comment.
inhibit_rules:
# if the whole job is gone, don't also send latency/error alerts
- source_matchers: [alertname = "ShopJobAbsent"]
target_matchers: [alertname != "ShopJobAbsent"]
equal: [team]
# critical hides warning of the same alert on the same job
- source_matchers: [severity = "critical"]
target_matchers: [severity = "warning"]
equal: [alertname, job]export AM=http://localhost:9093
# silence during planned work (default duration 1h)
amtool silence add --alertmanager.url=$AM \
--author="$USER" --duration=2h \
--comment="payments DB migration CHG-1234" \
job=payments
amtool silence query --alertmanager.url=$AM
amtool silence expire --alertmanager.url=$AM <silence-id>
# what is firing right now?
amtool alert query --alertmanager.url=$AM team=shopequal matters: without it, any critical alert anywhere would inhibit every warning in the company. Silences are stored in Alertmanager (replicated in the cluster) and expire on their own — always set a short duration and a comment.
Named time intervals (top-level time_intervals) can mute a route (mute_time_intervals) or make it active only then (active_time_intervals).
time_intervals:
- name: offhours
time_intervals:
- weekdays: ['saturday', 'sunday']
location: Asia/Tokyo
- times:
- start_time: '19:00'
end_time: '24:00'
- start_time: '00:00'
end_time: '09:00'
location: Asia/Tokyo
route:
routes:
- matchers: [team = "shop", severity = "warning"]
receiver: shop-slack
mute_time_intervals: [offhours] # warnings wait for office hours
- matchers: [team = "shop", severity = "critical"]
receiver: shop-pager # always activetimes, weekdays, days_of_month, months, years, location (IANA time zone; default UTC).mute_time_intervals still works but time_intervals is preferred.Default messages are generic. A short template with counts, summaries and links makes alerts readable on a phone.
{{/* /etc/alertmanager/config/shop.tmpl */}}
{{ define "slack.shop.title" -}}
[{{ .Status | toUpper }}
{{- if eq .Status "firing" }}:{{ .Alerts.Firing | len }}{{ end }}]
{{ .CommonLabels.alertname }} · {{ .CommonLabels.job }}
{{- end }}
{{ define "slack.shop.text" -}}
{{ range .Alerts -}}
• {{ .Annotations.summary }}
{{- with .Annotations.runbook_url }} <{{ . }}|runbook>{{ end }}
{{ end -}}
{{- end }}
# alertmanager.yml
templates:
- /etc/alertmanager/config/*.tmpl.Status: firing or resolved for the group..Alerts (with .Firing and .Resolved), each with .Labels, .Annotations, .StartsAt, .GeneratorURL..GroupLabels, .CommonLabels, .CommonAnnotations, .ExternalURL.toUpper, join, title, safeHtml.amtool template render before shipping.Run 2–3 replicas as a cluster. They gossip notification logs and silences, so a group is notified once even though every replica receives every alert.
--cluster.peer=<host>:9094 (TCP/UDP gossip).alertmanager.alertmanagerSpec.replicas: 3 creates the cluster; the Operator wires peers.storage:) so silences survive restarts.In kube-prometheus-stack, rules are PrometheusRule objects and Alertmanager config comes from Helm values or AlertmanagerConfig objects (ch 5).
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
name: shop-rules
namespace: shop
labels:
release: kps # matched by the default ruleSelector
spec:
groups:
- name: shop-red
rules:
- record: job:http_requests:rate5m
expr: |
sum by (job) (
rate(http_requests_total{namespace="shop"}[5m]))
- name: shop-alerts
rules:
- alert: ShopTargetDown
expr: up{job=~"orders|payments"} == 0
for: 2m
labels: {severity: warning, team: shop}kubernetes-*, node-*, prometheus-*, alertmanager-*) using severity critical/warning/info.Watchdog: always firing, route it to a dead man's switch.InfoInhibitor: suppresses info alerts unless a warning/critical fires in the same namespace.alertmanager.config in values (a full alertmanager.yml).AlertmanagerConfig CRs (namespaced): the Operator adds a namespace matcher so teams only route their own alerts; selected by alertmanagerConfigSelector.release: kps, a PrometheusRule is silently ignored.Rules are code: lint them, unit-test them, and test routing before anyone gets paged by a typo.
promtool and amtoolFast checks that catch syntax errors, bad PromQL, duplicate rules and broken Alertmanager config.
# rules: YAML structure + PromQL parse + duplicate rules
promtool check rules rules/shop.rules.yml
# Checking rules/shop.rules.yml
# SUCCESS: 9 rules found
# lint options: duplicate-rules (default), all, none
promtool check rules --lint=all rules/shop.rules.yml
# full Prometheus config including referenced rule files
promtool check config prometheus.yml
# Alertmanager config + templates
amtool check-config alertmanager.yml
# Checking 'alertmanager.yml' SUCCESS
# Found:
# - global config
# - route
# - 1 inhibit rules
# - 4 receivers
# - 1 templatesrelease: kps label).promtool test rules: feed synthetic series, assert resultsTests run the real rule engine against series you define, at times you choose. No Prometheus server needed.
# rules/shop.test.yml
rule_files:
- shop.rules.yml
evaluation_interval: 1m
tests:
- interval: 1m
input_series:
# 6 errors/min and 54 ok/min → 10% errors, 1 req/s
- series: >-
http_requests_total{namespace="shop",job="orders",code="500"}
values: '0+6x20'
- series: >-
http_requests_total{namespace="shop",job="orders",code="200"}
values: '0+54x20'
alert_rule_test:
- eval_time: 3m # still pending (for: 5m)
alertname: ShopHighErrorRate
exp_alerts: []
- eval_time: 10m
alertname: ShopHighErrorRate
exp_alerts:
- exp_labels:
{job: orders, severity: critical, team: shop}
# a rule with annotations also needs exp_annotationsinterval: spacing of the input samples.values uses expanding notation (next slide).promql_expr_test (not shown; used in Lab 4) checks any expression at an eval_time against exp_samples.alert_rule_test checks which alerts are firing (pending alerts are not listed).exp_labels must list all labels of the alert, including those from expr and labels:; alertname is implied.exp_annotations the test expects none, so list the rendered ones (exp_annotations: {summary: "…"}).promtool test rules rules/shop.test.yml → SUCCESS.A compact language to describe counters, gaps and staleness.
values notation| Notation | Expands to |
|---|---|
0+6x4 | 0 6 12 18 24 (start, +step, 4 more times) |
100-10x3 | 100 90 80 70 |
1x3 | 1 1 1 1 (shorthand for 1+0x3) |
1 2 _ 4 | _ = no sample at that step |
_x3 | three missing samples |
1 2 stale | staleness marker: series ends |
0+6x10 0+6x10 | counter reset in the middle |
for)._x10 and expect ShopJobAbsent.team, severity).Ask Alertmanager which receiver a label set would reach, then automate all checks on every pull request.
# show the tree
amtool config routes --config.file=alertmanager.yml
# which receivers get a critical shop alert?
amtool config routes test --config.file=alertmanager.yml \
team=shop severity=critical alertname=ShopHighErrorRate
# shop-pager,shop-slack
# fail if the result differs from what you expect
amtool config routes test --config.file=alertmanager.yml \
--verify.receivers=shop-pager,shop-slack \
team=shop severity=critical# .github/workflows/monitoring.yml (excerpt)
jobs:
rules:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@<sha>
- name: install promtool and amtool
run: |
# pin exact versions; download from the GitHub releases
./ci/install-prom-tools.sh
- run: promtool check rules rules/*.rules.yml
- run: promtool test rules rules/*.test.yml
- run: amtool check-config alertmanager/alertmanager.yml
- run: ./ci/test-routes.sh # amtool routes test casesWith the Operator, keep rules in plain rule files in Git, test them, and generate the PrometheusRule wrapper (Lab 4 does this with a heredoc). Tests on the CR YAML itself need the spec extracted first.
Most bad alerting is not missing alerts: it is alerts nobody trusts.
$value or a timestamp in alert labels → a new alert every evaluation.for: → pages on one bad scrape.NaN when traffic is zero) → either noise or silence.absent() / up == 0 alerts for the jobs you care about.group_by too fine (a message per pod) or too coarse (everything in one).equal:.Watchdog route: a dead pipeline looks like a quiet week.PrometheusRule without release: kps → never loaded, nobody notices.team label).Answer with the exact field, label or command.
job_route:http_requests:rate5m tell you about its expression?avg of per-route p99s wrong?keep_firing_for prevent?$value never be in labels?orders Deployment is deleted?group_wait, group_interval, repeat_interval control?equal: be set on inhibit rules?0+6x20 expand to?shop-pager?Practice: Lab 4 in Hands-on labs — recording rules and alerts for the shop app tested with promtool, loaded as a PrometheusRule, routed by Alertmanager to a webhook sink, then broken with a missing label, inhibition and a silence.
kube-prometheus-stack and the Operator CRDs, the metrics Kubernetes gives you for free, keeping cardinality under control, and running Prometheus highly available with long-term storage.
One Helm release that installs a full monitoring stack, and CRDs that turn scrape config into Kubernetes objects.
The chart prometheus-community/kube-prometheus-stack bundles the Prometheus Operator, its CRDs and a ready-to-use stack. In the labs the release is kps in namespace monitoring.
Watchdog alert: always firing, used as a dead man's switch (ch 13).kpskps-kube-prometheus-stack-prometheus.kps-kube-prometheus-stack-alertmanager.kps-grafana.prometheus-kps-kube-prometheus-stack-prometheus-0.kubectl -n monitoring get svc,prometheus,alertmanager.Chart names are derived from the release name and truncated; if you pick another release name, list the services instead of guessing them.
You never edit prometheus.yml. You create Kubernetes objects; the Operator renders the config, stores it in a Secret and the config-reloader sidecar reloads Prometheus.
The reloader calls Prometheus's /-/reload endpoint after the generated config changes, so most changes apply without a restart. Invalid rules are rejected by the Operator's admission webhook (if enabled) or reported in the Operator logs.
All are in the API group monitoring.coreos.com. Most are v1; PrometheusAgent and ScrapeConfig are v1alpha1, AlertmanagerConfig is v1alpha1/v1beta1 — check kubectl api-resources.
| CRD | Replaces in plain Prometheus | Typical owner |
|---|---|---|
Prometheus | The server Deployment/StatefulSet, flags, global config | Platform team |
PrometheusAgent | A Prometheus in agent mode (scrape + remote write only) | Platform team |
Alertmanager | The Alertmanager cluster | Platform team |
ServiceMonitor | A scrape_config that discovers targets through a Service's Endpoints | App team |
PodMonitor | A scrape_config that discovers Pods directly (no Service needed) | App team |
Probe | A blackbox_exporter scrape of URLs or Ingresses | App or platform team |
ScrapeConfig | Any raw scrape config: static targets, file, HTTP, EC2, Consul SD… | Platform team |
PrometheusRule | A rule file (recording + alerting rules) | App team |
AlertmanagerConfig | A namespaced sub-tree of the Alertmanager routing config | App team |
ThanosRuler | Thanos Ruler instances (rules over Thanos Query) | Platform team |
release labelThe Prometheus object selects monitors and rules by label. The chart's default makes it select only objects labeled release: <release name>.
# values.yaml (kube-prometheus-stack) — defaults
prometheus:
prometheusSpec:
serviceMonitorSelectorNilUsesHelmValues: true
podMonitorSelectorNilUsesHelmValues: true
ruleSelectorNilUsesHelmValues: true
probeSelectorNilUsesHelmValues: true
scrapeConfigSelectorNilUsesHelmValues: true
# → rendered Prometheus CR:
# serviceMonitorSelector:
# matchLabels: {release: kps}
# serviceMonitorNamespaceSelector: {} # all namespaces
# inspect what YOUR Prometheus selects
kubectl -n monitoring get prometheus \
kps-kube-prometheus-stack-prometheus \
-o jsonpath='{.spec.serviceMonitorSelector}{"\n"}'
# {"matchLabels":{"release":"kps"}}metadata.labels.release: kps on each ServiceMonitor, PodMonitor and PrometheusRule (what the labs do).…NilUsesHelmValues values to false → empty selectors → every monitor in every namespace is picked up.serviceMonitorSelector: {matchLabels: {prometheus: shared}}) and restrict namespaces with serviceMonitorNamespaceSelector.orders: every field that mattersA ServiceMonitor selects Services; Prometheus then scrapes each ready endpoint (pod IP) behind them. port refers to the Service port name, not a number.
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
name: orders
namespace: shop
labels:
release: kps # picked up by kps
spec:
selector:
matchLabels: {app: orders} # Service labels
namespaceSelector:
matchNames: [shop] # default: own namespace
endpoints:
- port: http # Service port NAME
path: /metrics
interval: 30s
scrapeTimeout: 10s
metricRelabelings: # after scrape
- sourceLabels: [__name__]
regex: go_gc_.*
action: drop
sampleLimit: 5000 # fail scrape above this
jobLabel: app # job = Service label "app"serviceMonitor/shop/orders/0 internally; the job label is the Service name unless jobLabel is set.namespace, service, pod, container, endpoint, instance (pod IP:port).relabelings run before the scrape (on discovered targets); metricRelabelings run after (on scraped samples).sampleLimit makes the whole scrape fail (up == 0) when exceeded — a hard cardinality guard.job="orders" and job="payments": the Services are named orders and payments.Use a PodMonitor for pods that have no Service (workers, batch, sidecars). Rules live next to the app as a PrometheusRule.
apiVersion: monitoring.coreos.com/v1
kind: PodMonitor # example: a queue worker
metadata: # (not part of the shop app)
name: worker
namespace: shop
labels: {release: kps}
spec:
selector:
matchLabels: {app: worker}
podMetricsEndpoints:
- port: metrics # container port NAME
interval: 30s
scrapeTimeout: 10sapiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
name: orders
namespace: shop
labels: {release: kps}
spec:
groups:
- name: orders.rules
rules:
- record: job_route:http_requests:rate5m
expr: sum by (job, route) (
rate(http_requests_total[5m]))
- alert: OrdersHighErrorRatio
expr: |
sum(rate(http_requests_total{job="orders",code=~"5.."}[5m]))
/ sum(rate(http_requests_total{job="orders"}[5m])) > 0.05
for: 10m
labels: {severity: page}
annotations:
summary: "orders 5xx ratio {{ $value | humanizePercentage }}"Rule groups from all selected PrometheusRules are merged into the Prometheus config. Group names must be unique per PrometheusRule; validate with promtool check rules in CI before the Operator sees them (ch 4).
Defaults are sized for a demo. These are the knobs that decide cost, durability and who can add monitors.
# deploy/kps-values.yaml
prometheus:
prometheusSpec:
retention: 10d # chart default 10d
retentionSize: 40GiB # delete oldest blocks first
replicas: 2 # HA pair (section 19)
externalLabels:
cluster: obs
scrapeInterval: 30s
evaluationInterval: 30s
resources:
requests: {cpu: 500m, memory: 2Gi}
limits: {memory: 4Gi}
storageSpec:
volumeClaimTemplate:
spec:
accessModes: [ReadWriteOnce]
resources: {requests: {storage: 50Gi}}
enforcedSampleLimit: 20000 # cap for every monitor
alertmanager:
alertmanagerSpec: {replicas: 3}
grafana:
adminPassword: change-me # use an existing SecretstorageSpec Prometheus uses emptyDir — every pod restart loses all data.cluster is the minimum.helm upgrade; follow the chart's upgrade notes for each major version.Where each Kubernetes metric comes from, and the queries that answer "is my pod starved, throttled or killed?".
Kubernetes metrics come from different components. Knowing the source tells you the label set and the meaning.
| Source | Scraped from | Answers | Example metrics |
|---|---|---|---|
| kubelet / cAdvisor | Each node's kubelet /metrics/cadvisor | What containers actually use | container_cpu_usage_seconds_total, container_memory_working_set_bytes, container_cpu_cfs_throttled_periods_total |
| kubelet | Kubelet /metrics, /metrics/probes | Volumes, pod lifecycle, probe results | kubelet_volume_stats_available_bytes, prober_probe_total |
| kube-state-metrics | One Deployment that watches the API | What the API declares (desired vs actual state) | kube_pod_status_phase, kube_deployment_status_replicas_available, kube_pod_container_resource_requests |
| node-exporter | DaemonSet, host namespaces | The node: CPU, memory, disk, network | node_cpu_seconds_total, node_memory_MemAvailable_bytes, node_filesystem_avail_bytes |
| Control plane | API server, etcd, scheduler, controller-manager | Is the cluster itself healthy? | apiserver_request_total, apiserver_request_duration_seconds, etcd_server_has_leader |
On managed control planes (EKS, GKE) etcd, scheduler and controller-manager are often not scrapeable; disable those ServiceMonitors in the chart values to avoid permanently-down targets.
kube-state-metrics turns object state into series. It does not measure usage; it reports what the API server knows.
# Deployments not at desired replicas
kube_deployment_spec_replicas
!= kube_deployment_status_replicas_available
# pods in Pending/Unknown/Failed (alert with `for: 15m`)
sum by (namespace, pod) (
kube_pod_status_phase{phase=~"Pending|Unknown|Failed"}
) > 0
# containers restarting in the last hour
increase(kube_pod_container_status_restarts_total[1h]) > 3
# last termination reason was OOMKilled
kube_pod_container_status_last_terminated_reason{reason="OOMKilled"} == 1
# containers in CrashLoopBackOff
kube_pod_container_status_waiting_reason{reason="CrashLoopBackOff"} == 1
# nodes not Ready
kube_node_status_condition{condition="Ready",status="true"} == 0--metric-labels-allowlist (e.g. pods=[team]) → kube_pod_labels.kube_pod_owner or kube_pod_info and group_left (ch 3).--shard, --total-shards) or watch its own scrape duration.CPU is a compressible resource: a container over its limit is throttled, not killed. Throttling shows up as latency, not as errors.
# CPU cores used per pod (cAdvisor)
sum by (namespace, pod) (
rate(container_cpu_usage_seconds_total{
container!="", namespace="shop"}[5m]))
# usage vs request (1.0 = using exactly the request)
sum by (namespace, pod) (
rate(container_cpu_usage_seconds_total{container!=""}[5m]))
/
sum by (namespace, pod) (
kube_pod_container_resource_requests{resource="cpu"})
# share of CFS periods that were throttled
sum by (namespace, pod, container) (
rate(container_cpu_cfs_throttled_periods_total[5m]))
/
sum by (namespace, pod, container) (
rate(container_cpu_cfs_periods_total[5m]))container!="" drops the pod-level cgroup series that cAdvisor also exports (otherwise you double count).GOMAXPROCS to match the limit (Go 1.25+ does this automatically from the cgroup limit), reduce parallelism.Memory is not compressible. A container above its memory limit is killed by the kernel OOM killer; the pod shows OOMKilled and restarts.
# working set per container (what the OOM decision is based on)
sum by (namespace, pod, container) (
container_memory_working_set_bytes{container!="", namespace="shop"})
# working set as a share of the limit
sum by (namespace, pod, container) (
container_memory_working_set_bytes{container!=""})
/
sum by (namespace, pod, container) (
kube_pod_container_resource_limits{resource="memory"}) > 0.9
# OOM kill events (cAdvisor counter, recent kubelets)
increase(container_oom_events_total{namespace="shop"}[1h]) > 0
# node memory actually available
node_memory_MemAvailable_bytes / node_memory_MemTotal_bytescontainer_memory_usage_bytes: usage includes page cache that the kernel can reclaim.GOMEMLIMIT to ~90% of the container limit so the GC works harder before the kernel kills the process.last_terminated_reason; alert on restarts + reason, not on the event alone.The kubernetes-mixin rules shipped by the chart already alert on most of these; read them before writing your own.
# node CPU utilization (0–1), excluding idle and iowait
1 - avg by (instance) (
rate(node_cpu_seconds_total{mode=~"idle|iowait"}[5m]))
# filesystem will be full within 4 hours
predict_linear(
node_filesystem_avail_bytes{fstype!~"tmpfs|overlay"}[6h],
4*3600) < 0
# API server error ratio by verb
sum by (verb) (rate(apiserver_request_total{code=~"5.."}[5m]))
/ sum by (verb) (rate(apiserver_request_total[5m]))
# API server p99 latency for mutating calls
histogram_quantile(0.99, sum by (le, verb) (rate(
apiserver_request_duration_seconds_bucket{
verb=~"POST|PUT|PATCH|DELETE"}[5m])))KubePodCrashLooping, KubePodNotReady, KubeDeploymentReplicasMismatch, KubeContainerWaiting, NodeFilesystemSpaceFillingUp, KubeCPUOvercommit.apiserver_request_* and throttling (429s), not etcd.Active series drive memory, CPU and cost. Find the expensive ones and stop them at the door.
Series count = product of distinct values of every label on a metric, summed over targets. One unbounded label multiplies everything else.
http_request_duration_seconds_bucket{method, route, code, le}user_id with 50,000 active users …/orders/8f2c…) instead of route templates (/orders/{id}).Put unbounded values in logs and traces (ch 6, ch 9), where they are cheap per event. Metrics are for aggregates over bounded label sets.
Start with the TSDB status page (Status → TSDB Status in the UI) — it is computed from the head block and costs almost nothing.
# total active series in the head block
prometheus_tsdb_head_series
# TSDB stats API: top metrics, labels, label pairs
curl -s localhost:9090/api/v1/status/tsdb | jq '.data |
{series: .headStats.numSeries,
topMetrics: .seriesCountByMetricName[:5],
topLabels: .labelValueCountByLabelName[:5]}'
# which jobs add the most series per scrape
topk(10, sum by (job) (scrape_samples_post_metric_relabeling))
# churn: new series per scrape, by job
topk(10, sum by (job) (rate(scrape_series_added[10m])))
# series per metric name (EXPENSIVE: touches every series)
topk(10, count by (__name__) ({__name__=~".+"}))seriesCountByMetricName: the metric to fix first.labelValueCountByLabelName: the label with the most distinct values — usually the culprit.scrape_samples_scraped vs …_post_metric_relabeling: how much your drop rules already remove.scrape_series_added high and steady = churn (new pods, rotating label values) even if the total looks flat.count by (__name__) query only on small servers or with a narrow selector; on a large server it can take seconds and lots of memory.Fix in the code first (route templates, no IDs). Until the fix ships, drop or rewrite at scrape time.
# ServiceMonitor endpoint: metricRelabelings (after scrape)
metricRelabelings:
# drop a whole metric
- sourceLabels: [__name__]
regex: http_client_request_duration_seconds_bucket
action: drop
# remove a label from every series
- regex: user_id
action: labeldrop
# collapse raw paths into one value
- sourceLabels: [path]
regex: /orders/.+
targetLabel: path
replacement: /orders/{id}
# hard limits per scrape (ServiceMonitor spec)
sampleLimit: 5000
labelLimit: 30
labelValueLengthLimit: 200labeldrop can create duplicate series (two series that only differed by that label) → scrape errors "duplicate sample for timestamp". Aggregate in the app instead, or drop the metric.up == 0 for that target. Alert on prometheus_target_scrapes_exceeded_sample_limit_total increasing.enforcedSampleLimit on the Prometheus CR caps every monitor, whatever teams set.filter and transform processors in the Collector (ch 9).Prometheus memory scales mainly with active head series and churn; disk scales with samples ingested per second × retention.
process_resident_memory_bytes / prometheus_tsdb_head_series gives your bytes per series (often a few KiB including overhead).rate(prometheus_tsdb_head_samples_appended_total[1h]) × retention seconds × ~2 bytes.--query.timeout (default 2m).--query.max-samples (default 50,000,000 samples loaded per query).--query.max-concurrency (default 20).spec.query fields on the Prometheus CR.Bytes-per-sample and bytes-per-series figures are rules of thumb; they change with Prometheus versions and label sizes. Always compute them from your own server's metrics.
Most "Prometheus is slow" tickets are one expensive dashboard or one heavy recording rule.
# prometheus.yml (global) — log every query to a file
global:
query_log_file: /prometheus/query.log
# Operator: spec.queryLogFile on the Prometheus CR
# rule groups that take longer than their interval
prometheus_rule_group_last_duration_seconds
> prometheus_rule_group_interval_seconds
# missed rule evaluations (group overran its interval)
increase(prometheus_rule_group_iterations_missed_total[1h]) > 0
# query latency by stage
prometheus_engine_query_duration_seconds{quantile="0.9"}
# scrape durations close to the timeout
max by (job) (scrape_duration_seconds) > 8job or namespace before regexes on other labels.[30d] with a 15 s step).scrapeTimeout only after fixing the exporter.One Prometheus is a single point of failure with local disk. Replicas, remote write and object storage fix both.
Prometheus has no clustering. HA means running two (or more) identical servers that scrape the same targets and send the same alerts.
prometheus_replica by default) so remote storage can deduplicate.replicas: 2 on the Prometheus CR (StatefulSet with two pods).replicaExternalLabelName (default prometheus_replica).replicas: 3 forms a gossip cluster automatically.Prometheus appends every sample to its WAL; remote write tails the WAL and sends batches over HTTP to any compatible receiver.
# Prometheus CR (or kps values under prometheusSpec)
remoteWrite:
- url: https://mimir.example.com/api/v1/push
name: longterm
queueConfig:
capacity: 10000
maxShards: 50
maxSamplesPerSend: 2000
writeRelabelConfigs: # only what you need off-cluster
- sourceLabels: [__name__]
regex: "up|job_.*:.*|http_request_duration_seconds_bucket"
action: keep
# AMP: add sigv4: {region: ap-northeast-1} and
# url .../workspaces/<id>/api/v1/remote_write
# health of the queue
rate(prometheus_remote_storage_samples_failed_total[5m])
prometheus_remote_storage_shards
# lag in seconds
prometheus_remote_storage_highest_timestamp_in_seconds
- ignoring(remote_name, url) group_right
prometheus_remote_storage_queue_highest_sent_timestamp_secondsmaxShards when the queue falls behind.protobuf_message: io.prometheus.write.v2.Request in remote_write when the receiver supports it.--web.enable-remote-write-receiver), Mimir, Thanos Receive, AMP, Cortex, VictoriaMetrics, many vendors.writeRelabelConfigs is the cheapest cost control for paid backends.A Prometheus started with --agent has no local querying, rules or alerting. It scrapes, writes a WAL and remote-writes. Ideal for edge clusters that report to a central store.
--agent (Prometheus 3); in the Operator, the PrometheusAgent CRD.mode: DaemonSet, behind a feature gate) to scrape only node-local pods.prometheus.scrape + prometheus.remote_write), the OTel Collector prometheus receiver, AMP managed collectors.prometheus_remote_storage_*) and the /agent UI page.All three keep Prometheus-compatible querying (PromQL) and put blocks in object storage. They differ in how data arrives and who operates it.
The question is less "which is best" than "who operates it, and how does data get there".
| Thanos sidecar | Mimir / Thanos Receive | AMP | |
|---|---|---|---|
| Data path | Upload completed blocks (pull for recent data) | Remote write push | Remote write push or managed scraping |
| You operate | Sidecar, store, querier, compactor | A distributed system (+ Kafka in newer Mimir designs) | Nothing server-side; pay per sample and query |
| Recent data | Queried live from each Prometheus (sidecar StoreAPI) | Held by ingesters until flushed to blocks | Managed |
| Alerting | Local Prometheus (or Thanos Ruler globally) | Mimir ruler + Alertmanager, or local Prometheus | AMP rules + alert manager → SNS |
| Long ranges | Downsampled blocks (5m, 1h) | Recording rules; query sharding and caching | Recording rules; managed query engine |
| Cost shape | Your compute + S3 | Your compute + S3 (+ Kafka if used) | Per sample ingested, stored and queried |
| Good for | Few clusters, keep local Prometheus | Many clusters, multi-tenant platform | AWS shops, no ops team for the backend |
Mixed setups are common: local Prometheus for alerting close to the workload, plus remote write of a curated subset to a central store for global dashboards and long retention.
The details you need to know before you operate either system.
--query.replica-label=prometheus_replica merges the two HA replicas at query time.cluster and __replica__); fails over when the elected replica stops sending.cluster, different replica label.X-Scope-OrgID, Thanos Receive reads THANOS-TENANT (configurable); limits per tenant (series, ingestion rate).shards: N on the Prometheus CR splits targets with hashmod; each shard sees only part of the data, so cross-shard queries need a global layer.Each of these shows up in real clusters within the first months.
release: kps → silently ignored.port: 8080 instead of the port name → no targets.storageSpec → all data lost on every restart.container!="" → double counting.container_memory_usage_bytes for OOM risk → false alarms from page cache.sampleLimit anywhere.keep list.Answer with the exact field, flag, label or query.
relabelings and metricRelabelings?sampleLimit do when exceeded?kube_pod_status_phase, and which exports container_cpu_usage_seconds_total?labeldrop?Practice: Lab 5 in Hands-on labs — inspect the Operator's selectors, find a cardinality bomb with the TSDB status API and stop it with metricRelabelings and sampleLimit, trigger CPU throttling and an OOMKill, then remote-write a curated subset to a second Prometheus.
Logs that machines can query and humans can still read: a field schema, slog in Go, trace correlation, safety and cost, and how logs leave a Kubernetes pod.
A log line is an event with fields. Treat its shape as an API that dashboards, alerts and on-call engineers depend on.
Only the structured forms can be filtered, aggregated and joined without fragile regular expressions.
2026-11-08 10:15:02 ERROR payment failed for order o-123 after 2013ms (502)avg(duration) without parsing.time=2026-11-08T10:15:02Z level=ERROR msg="payment failed" order_id=o-123 duration_ms=2013 status=502| logfmt (ch 7).{"time":"2026-11-08T10:15:02Z","level":"ERROR","msg":"payment failed","service":"orders","order_id":"o-123","route":"POST /orders","status":502,"duration_ms":2013}| json, OpenSearch.The message (msg) stays a short, constant string ("payment failed"). Everything that varies goes into fields. Constant messages are what you count and alert on.
A small, fixed set of keys used by every service. Agree on names once; dashboards and queries in ch 7, 9 and 12 rely on them.
| Key | Type | Example | Meaning |
|---|---|---|---|
time | RFC 3339 string | 2026-11-08T10:15:02.123Z | When the event happened (UTC, sub-second) |
level | string | INFO, WARN, ERROR, DEBUG | Severity (next slide) |
msg | string | payment failed | Constant event name, no variables inside |
service | string | orders | Same value as OTel service.name (ch 8) |
trace_id, span_id | hex string | 4bf92f3577b34da6a3ce929d0e0e4736 | Link to the trace (ch 9); absent when no span |
order_id | string | o-123 | Business key: lets support find "my order" in seconds |
route | string | POST /orders | Route template, never the raw path |
status | integer | 502 | HTTP status returned |
duration_ms | integer | 2013 | Request duration in milliseconds |
Align names with OpenTelemetry where a convention exists (service.name, http.route, http.response.status_code) if you export logs through OTel; the shop app keeps short JSON keys and maps them in the pipeline.
A level is a routing decision: who should look, and how soon. Use few levels, consistently.
| Level | Use it for | Not for | Typical production volume |
|---|---|---|---|
DEBUG | Details for developers while diagnosing | Anything someone needs in an incident | Off by default; switch on per service, temporarily |
INFO | Normal business events: request served, order created, job finished | Every loop iteration or cache hit | The bulk of volume; budget it |
WARN | Unexpected but handled: retry succeeded, fallback used, 4xx from a client | Errors you will never act on | Low; trend it |
ERROR | The operation failed and the caller saw it: 5xx, failed job, lost message | Expected outcomes (validation failures, 404) | Rare; each one should be explainable |
Do not page on log levels alone: alert on rates of a constant message or on metrics (ch 4, ch 7). Go's log/slog has exactly DEBUG (-4), INFO (0), WARN (4) and ERROR (8); levels in between are allowed but rarely worth it.
Logs are the most flexible and the most expensive signal per unit of information. Use each signal for what it is cheap at.
trace_id so you can jump between them (ch 9).A "wide event" approach (one rich structured log or span per request with many fields) replaces dozens of scattered log lines and is easier to query. The request log in section 21 is exactly that.
Other people build on your fields: dashboards, alerts, SIEM rules, support runbooks. Breaking them silently breaks those tools.
snake_case, stable, documented in the repo (docs/logging.md).status is always an integer).+00:00, equal to Z).apps/internal/logx).fmt.Println / log.Printf in services.The standard library's structured logger: handlers, attributes, context, and how to put trace IDs on every line.
log/slog: logger, handler, recordSince Go 1.21 the standard library has structured logging. A Logger builds Records; a Handler decides how to format and where to write them.
package main
import (
"log/slog"
"os"
)
func main() {
h := slog.NewJSONHandler(os.Stdout, &slog.HandlerOptions{
Level: slog.LevelInfo, // drop DEBUG
})
log := slog.New(h).With("service", "orders")
slog.SetDefault(log) // also captures log.Printf output
log.Info("order created", "order_id", "o-123", "amount", 42.5)
log.Debug("cache miss", "key", "o-123") // not written
log.Error("payment failed",
slog.String("order_id", "o-124"),
slog.Int("status", 502))
}{"time":"2026-11-08T10:15:02.123456+00:00","level":"INFO",
"msg":"order created","service":"orders",
"order_id":"o-123","amount":42.5}
{"time":"2026-11-08T10:15:02.130211+00:00","level":"ERROR",
"msg":"payment failed","service":"orders",
"order_id":"o-124","status":502}
# (each object is one line in reality; wrapped here)With returns a child logger whose attributes are pre-formatted once, so per-call cost stays low. The typed forms (slog.String, slog.Int, LogAttrs) avoid allocations and catch key/value mismatches at compile time; go vet also reports odd key/value pairs.
Four features that matter in services.
// groups → nested JSON
log.Info("charge", slog.Group("card",
slog.String("brand", "visa"),
slog.String("last4", "4242")))
// {"msg":"charge","card":{"brand":"visa","last4":"4242"}}
// LogValuer: a type decides how it is logged
type Token string
func (Token) LogValue() slog.Value {
return slog.StringValue("[REDACTED]")
}
log.Info("login", "token", Token("s3cr3t")) // "token":"[REDACTED]"
// change the level at runtime (e.g. from an admin endpoint)
var lvl slog.LevelVar // zero value = INFO
_ = lvl.UnmarshalText(
[]byte(os.Getenv("LOG_LEVEL")))
h := slog.NewJSONHandler(os.Stdout, &slog.HandlerOptions{
Level: &lvl,
AddSource: false, // true: "source":{file,line,…}
})
lvl.Set(slog.LevelDebug) // takes effect immediatelyUnmarshalText accepts DEBUG, INFO, WARN, ERROR (case-insensitive, optional offset like INFO+2); on error the level stays INFO.runtime.Callers per record; useful for ERROR, expensive for hot INFO paths.ReplaceAttr in HandlerOptions rewrites or removes any attribute (renaming keys, redaction by key) — section 22.A single "wide" INFO line per request with route, status and duration replaces scattered "start/end" lines.
// recorder wraps http.ResponseWriter and keeps the status code
func Middleware(log *slog.Logger, next http.Handler) http.Handler {
return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
start := time.Now()
rec := &recorder{ResponseWriter: w, status: http.StatusOK}
next.ServeHTTP(rec, r)
route := r.Pattern // set by ServeMux (Go 1.23+)
if route == "" {
route = "unmatched"
}
lvl := slog.LevelInfo
switch {
case rec.status >= 500:
lvl = slog.LevelError
case rec.status >= 400:
lvl = slog.LevelWarn
}
log.LogAttrs(r.Context(), lvl, "request",
slog.String("route", route),
slog.Int("status", rec.status),
slog.Int64("duration_ms",
time.Since(start).Milliseconds()))
})
}Pattern on the request it receives; a wrapper in between that clones the request (r.WithContext, http.TimeoutHandler) hides it.route is the template (GET /orders/{id}), so it is safe to index or aggregate.LogAttrs(r.Context(), …) passes the context, so a handler can add trace_id (next slide)./metrics scrapes are noise: skip them or log them at DEBUG.order created with order_id) are logged inside the handler, not here.trace_id and span_id on every lineA small handler wrapper reads the active span from the context. Every …Context log call then carries the IDs, and logs and traces link both ways (ch 9).
import "go.opentelemetry.io/otel/trace"
type traceHandler struct{ slog.Handler }
func (h traceHandler) Handle(ctx context.Context, r slog.Record) error {
if sc := trace.SpanContextFromContext(ctx); sc.IsValid() {
r.AddAttrs(
slog.String("trace_id", sc.TraceID().String()),
slog.String("span_id", sc.SpanID().String()))
}
return h.Handler.Handle(ctx, r)
}
// keep the wrapper when attributes or groups are added
func (h traceHandler) WithAttrs(a []slog.Attr) slog.Handler {
return traceHandler{h.Handler.WithAttrs(a)}
}
func (h traceHandler) WithGroup(n string) slog.Handler {
return traceHandler{h.Handler.WithGroup(n)}
}
log := slog.New(traceHandler{slog.NewJSONHandler(os.Stdout, nil)})
log.ErrorContext(ctx, "payment failed", "order_id", id)…Context methods (InfoContext, ErrorContext, LogAttrs(ctx, …)); log.Info has no context and no IDs.trace_id a Loki label or a metric label: it is unique per request (ch 7).Enabled is inherited from the embedded handler, so level filtering still works.An error log is useful only if it says what failed, for which entity, and why — once.
// wrap with context as the error travels up
if err != nil {
return fmt.Errorf("charge order %s: %w", id, err)
}
// log once, at the boundary that handles it
if err := svc.Create(ctx, o); err != nil {
log.ErrorContext(ctx, "create order failed",
"order_id", o.ID,
"err", err) // err.Error() as a string
http.Error(w, "payment failed", http.StatusBadGateway)
return
}
// {"level":"ERROR","msg":"create order failed","order_id":"o-124",
// "err":"charge order o-124: Post \"http://payments:8080/charge\":
// context deadline exceeded", ...}
// errors.Is / As still work on the wrapped error
if errors.Is(err, context.DeadlineExceeded) { /* timeout */ }%w and the entity ID; the final message reads like a stack of causes.err for the error string; add err_type or a stable error code if you want to aggregate by kind.span.RecordError, SetStatus) so the trace shows it (ch 9).Instead of (or in addition to) stdout, a bridge turns slog records into OTel log records and exports them over OTLP with the resource and trace context attached.
import (
"go.opentelemetry.io/contrib/bridges/otelslog"
"go.opentelemetry.io/otel/log/global"
)
// after the LoggerProvider is set up (ch 8):
// global.SetLoggerProvider(lp)
logger := otelslog.NewLogger("orders")
logger.InfoContext(ctx, "order created", "order_id", id)
// → OTLP log record:
// body: "order created"
// attributes: order_id=o-123
// trace_id/span_id from ctx, resource: service.name=ordersLogs are copied everywhere and kept for months. What you put in them is a security and budget decision.
Logs are replicated to agents, backends, exports, laptops and tickets. Assume everyone with log access can read every field.
Authorization headers.LogValuer (previous section).ReplaceAttr redacts by key as a safety net.Logging regulations and retention limits depend on your jurisdiction and contracts; involve your security or privacy team when defining the schema.
ReplaceAttrReplaceAttr is called for every attribute before it is written. Use it to rename keys and to redact known-sensitive keys anywhere, including inside groups.
var sensitive = map[string]bool{
"password": true, "token": true, "authorization": true,
"card_number": true, "cvv": true, "email": true,
}
func replace(groups []string, a slog.Attr) slog.Attr {
if sensitive[strings.ToLower(a.Key)] {
return slog.String(a.Key, "[REDACTED]")
}
return a
}
h := slog.NewJSONHandler(os.Stdout, &slog.HandlerOptions{
ReplaceAttr: replace,
})
slog.New(h).Info("signup", "email", "a@example.com", "plan", "pro")
// {"time":"…","level":"INFO","msg":"signup",
// "email":"[REDACTED]","plan":"pro"}msg or inside an error string passes through.%+v-style formatting is one string: redact by type (LogValuer) instead.time, level, msg, source) also pass through ReplaceAttr (with empty groups); do not drop them by accident.Most backends charge by ingested GB, and some by GB stored or scanned. Volume is driven by a few chatty lines, not by the average.
msg and service: usually 3–5 messages are most of the volume.Quick estimate: 200 bytes/line × 50 req/s × 86,400 s ≈ 864 MB/day ≈ 26 GB/month for one request log line per request. Check your backend's current price per GB before deciding.
Logs are not sampled by default. Sample only what you can afford to lose, and keep everything that explains a failure.
// keep all non-2xx; keep 1 in 10 successful request logs
type sampler struct{ n atomic.Uint64 }
func (s *sampler) keep(status int) bool {
if status >= 400 {
return true
}
return s.n.Add(1)%10 == 0
}
// in the middleware
if s.keep(rec.status) {
log.LogAttrs(ctx, lvl, "request", attrs...)
}
// better: decide per trace, so a sampled request has
// ALL its lines (and its trace) — use the trace flag
if trace.SpanContextFromContext(ctx).IsSampled() { … }loki.process with a sampling stage) or the OTel Collector filter/probabilistic sampler can sample without code changes.sample_rate: 10) so counts from logs can be scaled back.Different consumers, retention and integrity requirements. Keep them in separate streams.
| Application logs | Audit logs | Access logs | |
|---|---|---|---|
| Purpose | Debugging, operations | Who did what, when — accountability | Every request at the edge (LB, ingress, API gateway) |
| Examples | shop request logs, errors | Admin changed a price; Kubernetes API audit; CloudTrail | ALB access logs, NGINX ingress logs |
| Sampling | Allowed for success paths | Never | Sometimes (very high volume) |
| Retention | Days to weeks | Months to years (policy-driven) | Weeks to months |
| Integrity | Best effort | Tamper-evident, restricted write and delete | Usually stored in S3 |
| Access | Engineers | Security, compliance; few engineers | Engineers, security |
Do not build audit trails on the application log pipeline: a log level change or a sampling rule must never be able to delete an audit record.
How a line written to stdout becomes a file on the node, and how an agent ships it with Kubernetes metadata.
fmt.Fprintln(os.Stdout) to a backendContainers write to stdout/stderr. The container runtime writes them to files on the node; the kubelet rotates them; a node agent tails them.
kubectl logs reads the same files through the kubelet. If the file is rotated away or the pod is deleted, kubectl logs cannot show it — only your backend can.
The runtime prefixes every line. Agents must parse this CRI format before your JSON.
# /var/log/pods/shop_orders-…_<uid>/orders/0.log
2026-11-08T10:15:02.123456789Z stdout F {"level":"INFO",…}
2026-11-08T10:15:02.200000000Z stderr F panic: runtime error: …
2026-11-08T10:15:03.000000000Z stdout P {"msg":"huge line, part 1…
2026-11-08T10:15:03.000000000Z stdout F …part 2"}
# timestamp stream tag content
# tag F = full line, P = partial (line was split by the runtime)
# useful commands
kubectl -n shop logs deploy/orders --tail=20
kubectl -n shop logs <pod> --previous # crashed container
kubectl -n shop logs -l app=orders --since=10m --prefix
kubectl -n shop logs <pod> -c orders --timestampsP) chunks (containerd splits at 16 KiB by default); the agent must reassemble them (cri stage in Alloy does).containerLogMaxSize (default 10Mi) and containerLogMaxFiles (default 5) per container.--previous shows the last terminated container: the first place to look after a crash.The node-level DaemonSet is the default. The other two patterns solve specific problems.
| Pattern | How | Pros | Cons |
|---|---|---|---|
| Node agent (DaemonSet) | One agent per node tails /var/log/pods | Zero app changes; covers every container; one config | Only sees stdout/stderr; needs hostPath access; per-node resource use |
| Sidecar streaming | App writes to a file; a sidecar tails it to its own stdout | Works for legacy apps that only log to files | Duplicate storage on disk; more containers per pod |
| Sidecar agent | An agent container in the pod ships directly | Per-app config and credentials; works on Fargate-style nodes without DaemonSets | One agent per pod: cost and config sprawl |
| Direct export (OTLP) | App sends logs over OTLP to a Collector | Fully structured, trace context included | Logs lost if the app crashes before flushing; library needed per language |
EKS on Fargate has no DaemonSets: use its built-in Fluent Bit log router (configured with a ConfigMap) or a sidecar. CloudWatch side: ch 12.
Two jobs every node agent must do well.
loki.process → stage.multiline with firstline regex in Alloy).namespace, pod, container, node, and selected pod labels (app).pod into a primary label on huge clusters without thinking: each pod creates new streams.Each of these turns logs into either noise, a liability or a large bill.
msg ("payment failed for o-123") → impossible to count by message.status "502" vs 502).route.fmt.Println lines with JSON lines.trace_id: logs and traces cannot be joined.log.Info instead of log.InfoContext → IDs missing.kubectl logs for incidents: gone after rotation or pod deletion.Answer precisely; write the code for the Go questions.
msg be a constant string?status?log.Info and log.InfoContext for trace correlation?trace_id and span_id.r.Pattern?F and P tags mean?Practice: Lab 6 in Hands-on labs — add the shared logx package to orders and payments (schema, request middleware, trace-ready handler, redaction), then break it with DEBUG volume, a leaked secret and a non-JSON line.
A log store that indexes labels instead of text: how Loki stores streams, how Alloy ships them, how LogQL filters, parses and turns logs into metrics, and how to keep it healthy.
Streams, chunks and a small index: why Loki is cheap to run and why label design decides everything.
Loki stores log lines compressed in object storage and indexes only a small set of labels per stream. Queries select streams by label, then scan their lines in parallel.
{namespace="shop", app="orders", container="orders", level="INFO"} is one stream.|= "timeout" is a brute-force scan.trace_id, order_id, user_id) stay in the line or in structured metadata.Writes are replicated to ingesters and flushed as chunks; reads merge recent data from ingesters with flushed chunks from storage.
In monolithic mode all of these run in one process (-target=all); the boxes are still there, just in-process. RF = replication factor (3 by default, 1 in the lab).
Know these names: they appear in metrics, logs, Helm values and error messages.
| Component | Job | Notes |
|---|---|---|
| Distributor | Receives pushes, validates labels and limits, hashes each stream to ingesters | Stateless; first place 429 rate-limit errors come from |
| Ingester | Builds chunks in memory, writes a WAL, flushes chunks to storage, serves recent data | Stateful; replication factor 3 by default |
| Query frontend | Splits queries by time, shards them, queues them, caches results | Stateless; put it in front of queriers |
| Query scheduler | Holds the query queue between frontend and queriers | Optional; lets frontends scale independently |
| Querier | Executes LogQL over ingesters + storage, deduplicates replicas | Scale out for read load |
| Index gateway | Serves TSDB index lookups | Keeps index downloads off queriers |
| Compactor | Compacts index files, applies retention and deletes | Exactly one per cluster |
| Ruler | Evaluates LogQL alerting and recording rules | Sends alerts to Alertmanager, recorded series via remote write |
| Pattern ingester | Detects recurring line patterns | Feeds Grafana Logs Drilldown pattern views |
Ingesters keep one open chunk per stream, then cut and flush it. Many tiny streams mean many tiny chunks — the classic Loki performance problem.
chunk_target_size (1.5 MB compressed)…chunk_idle_period (30m) without new lines…max_chunk_age (2h).# schema_config: decides index type and layout
schema_config:
configs:
- from: "2024-04-01" # applies to data from this day
store: tsdb # index type (current)
object_store: s3 # filesystem in the lab
schema: v13 # needed for structured metadata
index:
prefix: index_
period: 24h # one index table per day
# never edit an existing entry: add a new one with
# a future "from" date to change schema or storeboltdb-shipper and schemas before v13 are legacy. New installs use tsdb + v13; structured metadata and OTLP ingestion depend on it.
Same binary, different -target. Pick by volume and by how much you want to operate.
| Mode | How it runs | Rough size (chart guidance) | Status |
|---|---|---|---|
| Monolithic (single binary) | One process, -target=all; filesystem or object storage | Up to a few tens of GB/day | Lab choice; simplest to run |
| Simple scalable | read, write, backend targets; object storage required | Up to about 1 TB/day | Deprecated; the chart notes it is removed in Loki 4 |
| Distributed (microservices) | Each component as its own Deployment/StatefulSet; object storage required | Above ~1 TB/day | Recommended for large clusters |
grafana-community/helm-charts since 16 Mar 2026.SingleBinary to Monolithic (old name still accepted).Structured metadata (Loki 3, schema v13) attaches key/value pairs to each line without creating streams: the right home for IDs you want to filter on.
| Field | Put it in | Why |
|---|---|---|
namespace, app, container | Label | Low cardinality, used in every selector |
level | Label (or rely on detected_level) | Handful of values |
trace_id, span_id | Structured metadata | Unique per request; filter with | trace_id="…" |
order_id, user_id | Line (JSON) or structured metadata | Unbounded cardinality |
route, status, duration_ms | Line (JSON), parsed at query time | Needed only by some queries |
pod | Label is acceptable | Churns on every rollout: watch stream counts |
/otlp/v1/logs.service.name → service_name, k8s.namespace.name → k8s_namespace_name.service_name for non-OTLP pushes and can add detected_level.Promtail is gone: Alloy discovers pods, tails their logs, processes lines and pushes streams to Loki.
Grafana Alloy is Grafana's OpenTelemetry Collector distribution with native Prometheus and Loki pipelines. It is the supported way to ship logs to Loki.
alloy convert --source-format=promtail --output=config.alloy promtail.yaml--bypass-errors to convert what it can and report the rest.forward_to, targets).Discover pods, turn metadata into labels, read the logs, process lines, push to Loki.
discovery.kubernetes__meta_kubernetes_* labels.discovery.relabelnamespace, pod, container, app.loki.source.kubernetesloki.processloki.write/loki/api/v1/push, retries.Alternative source: local.file_match + loki.source.file reading /var/log/pods/* from a hostPath (Promtail's classic model). Both work; the API source needs no node mounts but puts load on the API server at scale.
Each Alloy pod only discovers pods on its own node (HOSTNAME is set to the node name by the Alloy Helm chart), so no line is shipped twice.
discovery.kubernetes "pods" {
role = "pod"
selectors {
role = "pod"
field = "spec.nodeName=" +
coalesce(sys.env("HOSTNAME"), constants.hostname)
}
}
discovery.relabel "pods" {
targets = discovery.kubernetes.pods.targets
rule {
source_labels = ["__meta_kubernetes_namespace"]
target_label = "namespace"
}
rule {
source_labels = ["__meta_kubernetes_pod_name"]
target_label = "pod"
}
rule {
source_labels = ["__meta_kubernetes_pod_container_name"]
target_label = "container"
}
// app rule: continued on the right rule {
source_labels = ["__meta_kubernetes_pod_label_app"]
target_label = "app"
}
}
loki.source.kubernetes "pods" {
targets = discovery.relabel.pods.output
forward_to = [loki.process.shop.receiver]
}
loki.process "shop" {
// processing stages: next slide
forward_to = [loki.write.default.receiver]
}
loki.write "default" {
endpoint {
url = "http://loki.logging:3100/loki/api/v1/push"
}
}
// wiring: <component>.<label>.<export>
// discovery.relabel.pods.output -> targets
// loki.process.shop.receiver -> log receiverStages run in order on every line. The shop apps log slog JSON (ch 6); other namespaces pass through untouched.
loki.process "shop" {
stage.match {
selector = "{namespace=\"shop\"}"
stage.json {
expressions = {
level = "level",
trace_id = "trace_id",
ts = "time",
}
}
stage.labels {
values = { level = "" } // "" = same name
}
stage.structured_metadata {
values = { trace_id = "" }
}
stage.timestamp {
source = "ts"
format = "RFC3339Nano"
}
}
stage.drop {
expression = ".*GET /healthz.*"
drop_counter_reason = "healthz"
}
forward_to = [loki.write.default.receiver]
}stage.match: run nested stages only for lines whose stream matches a LogQL selector.stage.json: extracts fields into a temporary map (nothing is sent as a label yet).stage.labels: promotes level to a stream label (values INFO, WARN, ERROR).stage.structured_metadata: attaches trace_id per line, no new stream.stage.timestamp: uses the app's own time instead of read time.stage.drop: discards noise before it costs anything.stage.cri/stage.docker (file sources), stage.multiline, stage.logfmt, stage.regex, stage.static_labels, stage.label_drop, stage.output.Stream count ≈ product of label value counts. Keep it in the thousands per tenant, not the millions.
| Label | Values |
|---|---|
namespace | tens |
app / service_name | tens to hundreds |
container | a few per app |
level | 3–5 |
cluster, env | a few |
| Label | Why |
|---|---|
trace_id, request_id | One stream per request |
order_id, user_id, email | Unbounded, and PII in the index |
status, route, duration_ms | Multiply streams; parse at query time |
ip, url with query string | Unbounded |
filename per rotation | New stream on every rotation |
logcli series --analyze-labels).Choose by where logs live and which pipeline you already run.
| Source | Use when | Notes |
|---|---|---|
local.file_match + loki.source.file | DaemonSet reading /var/log/pods on the node | Needs a hostPath mount (alloy.mounts.varlog: true in the chart) and stage.cri to strip the CRI prefix |
loki.source.kubernetes | DaemonSet or clustered Deployment without host mounts | Uses the API server log endpoint; filter by node or enable clustering to avoid duplicates |
loki.source.podlogs | Teams declare what to collect with a PodLogs custom resource | Like a ServiceMonitor for logs |
loki.source.kubernetes_events | Ship Kubernetes Events as log lines | Run one instance (or clustering); events otherwise expire after about an hour |
otelcol.receiver.* → otelcol.exporter.otlphttp to /otlp | Apps or Collectors already emit OTLP logs | Resource attributes become labels/structured metadata (section 24) |
OTel Collector filelog receiver | You standardise on the Collector instead of Alloy | Export with otlphttp to Loki's OTLP endpoint |
Select streams, filter lines, parse fields, filter on them and reshape the output.
A stream selector, then stages joined by |. Each stage sees the output of the one before.
{namespace="shop", app="orders"} # 1 stream selector (index lookup: which chunks)
|= "payment failed" # 2 line filter (cheap substring scan)
| json # 3 parser (extracts every JSON key as a label)
| status >= 500 and duration_ms > 250 # 4 label filters (typed comparisons)
| line_format "{{.order_id}} {{.route}} {{.status}} {{.duration_ms}}ms" # 5 formatting
# sample result line (labels shown in Grafana's details pane):
# o-4182 /orders 502 311msrate(… [5m]) and returns series (section 27).logcli query in a terminal (section 28).Matchers work like PromQL. Line filters are substring or regex matches on the raw line.
| Matcher | Example |
|---|---|
= equal | {app="orders"} |
!= not equal | {namespace="shop", container!="istio-proxy"} |
=~ regex | {app=~"orders|payments"} |
!~ not regex | {namespace!~"kube-.*"} |
| Filter | Keeps lines that… |
|---|---|
|= "error" | contain the string (case-sensitive) |
!= "healthz" | do not contain it |
|~ "(?i)timeout|refused" | match the RE2 regex |
!~ "GET /metrics" | do not match |
|> "<_> level=ERROR <_>" | match a pattern (<_> = anything) |
!> "<_> level=DEBUG <_>" | do not match the pattern |
A selector needs at least one matcher that cannot match an empty value: {app=~".*"} alone is rejected. Chained filters are ANDed: |= "payment" != "retry".
Parsers add extracted labels to each line. Nothing is stored: you pay the parse cost on every query.
# json: every key (nested keys joined with _)
{app="orders"} | json
# {"level":"ERROR","msg":"payment failed",
# "order_id":"o-41","route":"/orders","status":502}
# -> labels level_extracted, msg, order_id, route, status
# json with only the fields you need (faster)
{app="orders"} | json status="status", route="route"
# logfmt: key=value lines
{app="legacy"} | logfmt
# pattern: positional, for fixed formats
{app="nginx"}
| pattern `<ip> - - [<_>] "<method> <uri> <_>" <code> <size>`
# regexp: named RE2 groups
{app="legacy"} | regexp `took (?P<ms>\d+)ms`
# keep only lines that parsed cleanly
{app="orders"} | json | __error__ = ""_extracted: with level already a label, | json yields level_extracted._ (http.status → http_status).__error__ (e.g. JSONParserErr) instead of dropping the line; filter with | __error__="".{app="orders"} | trace_id="4bf9…".| unpack reverses Alloy's stage.pack.After parsing, compare fields with types: numbers, durations and byte sizes are understood.
# typed comparisons
{app="orders"} | json | status >= 500
{app="orders"} | json | duration_ms > 250
{app="api"} | logfmt | latency > 250ms # "312ms", "1.2s"
{app="api"} | logfmt | size > 1MB
# boolean logic
{app="orders"} | json
| level_extracted = "ERROR" or status >= 500
# reshape the line
{app="orders"} | json
| line_format "{{.status}} {{.route}} {{.order_id}}"
# rename or create labels
{app="orders"} | json | label_format code=status
# keep or drop labels in the result
{app="orders"} | json | keep route, status
{app="orders"} | json | drop msg, time
# strip ANSI colour codes
{app="legacy"} | decolorizeroute = "/orders" and regex route =~ "/orders.*" work on any label.> sets __error__ for that line.line_format and label_format use Go templates plus functions such as ToUpper, trunc, div.keep/drop cut result size; useful before metric queries to reduce series labels.|= "\"status\":502" first, then parse."Show me failed payments for order o-4182 in the last hour."
{namespace=~".+"} | json | order_id="o-4182"max_query_series / query limits.{namespace="shop", app="orders"} |= "o-4182" | jsonorder_id were structured metadata: {app="orders"} | order_id="o-4182".trace_id.Watch the query statistics in Grafana (Query inspector → Stats): bytes processed and lines processed tell you whether the selector or the filter needs work.
Counting lines, aggregating extracted values, and alerting from logs with the ruler.
Wrap a log query in a range and an aggregation function; the result is a set of time series, just like PromQL.
| Function | Returns per stream |
|---|---|
count_over_time(q [5m]) | Number of lines in the window |
rate(q [5m]) | Lines per second |
bytes_over_time(q [5m]) | Bytes of log lines in the window |
bytes_rate(q [5m]) | Bytes per second |
absent_over_time(q [5m]) | 1 if there were no lines (for "logs stopped" alerts) |
# error lines per second per app
sum by (app) (
rate({namespace="shop", level="ERROR"}[5m])
)
# top 5 noisiest containers by bytes
topk(5,
sum by (namespace, container) (
bytes_rate({namespace=~".+"}[5m])
)
)
# 5xx responses per route, parsed from the line
sum by (route) (
count_over_time({app="orders"} | json
| status >= 500 [5m])
)Aggregate with sum by (…): without it you get one series per stream, including per pod, which is rarely what a panel needs.
unwrap turns an extracted label into the sample value, so you can compute sums, averages and quantiles from logs.
# p99 latency per route from duration_ms in the JSON
quantile_over_time(0.99,
{app="orders"} | json
| __error__ = ""
| unwrap duration_ms [5m]
) by (route)
# total charged amount per minute
sum(
sum_over_time({app="payments"} |= "charged"
| json | unwrap amount [1m])
)
# values like "312ms" or "1.5s": convert to seconds
avg_over_time({app="api"} | logfmt
| unwrap duration(latency) [5m]) by (route)
# "10MB" style values
sum_over_time({app="up"} | logfmt
| unwrap bytes(size) [5m])sum_over_time, avg_over_time, min_over_time, max_over_time, stddev_over_time, stdvar_over_time.quantile_over_time(φ, …): quantile over the unwrapped values in the window (sharded execution may approximate).first_over_time, last_over_time; rate / rate_counter on unwrapped values.quantile_over_time(…) by (route).__error__: filter them or the query errors.Useful when a component has no metrics (a vendor binary, a legacy job). For your own services, prefer the Prometheus metrics from ch 2.
# Rate: requests per second (one log line per request)
sum(rate({app="orders"} |= "\"route\"" [5m]))
# Errors: share of requests with status >= 500
sum(count_over_time({app="orders"} | json
| status >= 500 [5m]))
/
sum(count_over_time({app="orders"} | json
| status > 0 [5m]))
# Duration: p95 per route
quantile_over_time(0.95,
{app="orders"} | json | __error__ = ""
| unwrap duration_ms [5m]) by (route)Same rule file format as Prometheus (ch 4); only the expression language differs. Alerts go to the same Alertmanager.
# rules for tenant "fake" (auth disabled):
# /etc/loki/rules/fake/shop.yaml
groups:
- name: shop-logs
interval: 1m
rules:
- alert: OrdersPaymentErrors
expr: |
sum(count_over_time(
{namespace="shop", app="orders", level="ERROR"}
|= "payment failed" [5m])) > 10
for: 2m
labels:
severity: warning
annotations:
summary: "orders: {{ $value }} payment failures in 5m"
runbook_url: https://runbooks.example/shop/payments
- alert: OrdersLogsMissing
expr: absent_over_time({namespace="shop", app="orders"}[10m])
for: 5m
labels: {severity: warning}# Loki config (ruler block)
ruler:
storage:
type: local
local:
directory: /etc/loki/rules # <dir>/<tenant>/*.yaml
rule_path: /var/loki/rules-temp # scratch space
alertmanager_url: http://alertmanager-operated.monitoring:9093
enable_api: true
# check what the ruler loaded / what is firing
curl -s localhost:3100/loki/api/v1/rules
curl -s localhost:3100/prometheus/api/v1/alerts | jq
# with object storage instead of local:
# lokitool rules sync (or the ruler API) uploads
# rule groups per tenantA recording rule evaluates a LogQL metric query on a schedule and remote-writes the result as a Prometheus series. Dashboards then query cheap PromQL.
# Loki rule file
groups:
- name: shop-log-metrics
interval: 1m
rules:
- record: app:log_errors:rate5m
expr: |
sum by (app) (
rate({namespace="shop", level="ERROR"}[5m]))
# Loki config: where recorded series go
ruler:
wal:
dir: /var/loki/ruler-wal
remote_write:
enabled: true
clients:
prom: # any name; url is one line:
url: http://kps-kube-prometheus-stack-prometheus
.monitoring:9090/api/v1/writeprometheus.prometheusSpec.enableRemoteWriteReceiver: true.source="loki" label to tell them apart.level:metric:operations.Retention, limits, ingestion errors, the command line, and the metrics that tell you Loki is in trouble.
Retention is applied by the compactor. Without it, data lives as long as the bucket keeps it.
compactor:
working_directory: /var/loki/compactor
retention_enabled: true
retention_delete_delay: 2h # grace before deleting
delete_request_store: s3 # required; lab: filesystem
limits_config:
retention_period: 744h # 31 days, global default
retention_stream: # per-selector overrides
- selector: '{namespace="dev"}'
priority: 1
period: 72h
- selector: '{namespace="shop", level="DEBUG"}'
priority: 2
period: 24hpriority wins when several retention_stream selectors match.overrides:), reloadable without restart.retention_period: 0s = keep forever (the default). Decide explicitly./loki/api/v1/delete with a selector and time range.Defaults below are Loki 3.x limits_config defaults; Helm charts may override some.
| Limit | Default | What happens when exceeded |
|---|---|---|
ingestion_rate_mb / ingestion_burst_size_mb | 4 MB/s / 6 MB per tenant (global strategy: shared by all distributors) | Push rejected with HTTP 429 (rate_limited) |
max_global_streams_per_user | 5000 | New streams rejected (stream_limit); existing ones keep working |
max_label_names_per_series | 15 | Push rejected (400) |
max_line_size | 256KB | Line rejected (line_too_long) unless max_line_size_truncate |
reject_old_samples_max_age | 168h (1 week) | Old lines rejected (greater_than_max_sample_age) |
max_query_series | 500 | Metric query fails: too many series — aggregate more |
max_entries_limit_per_query | 5000 | Log query returns at most this many lines |
Raise limits deliberately per tenant in the runtime config. A team asking for more streams usually has a label problem, not a capacity problem.
Clients see HTTP errors; Loki counts every rejected line by reason. Alert on the counter, not on someone noticing a gap.
loki_discarded_samples_total{reason}| reason | Meaning | Usual fix |
|---|---|---|
rate_limited | Tenant over ingestion rate | Raise limit or cut noisy logs |
per_stream_rate_limit | One stream too hot | Add a label that splits it (e.g. pod) |
stream_limit | Too many active streams | Remove high-cardinality labels |
line_too_long | Line > max_line_size | Truncate or fix the logger |
greater_than_max_sample_age | Timestamp too old | Agent backlog or wrong timestamp parsing |
too_far_behind | Out-of-order beyond the window | Clock skew or bad timestamp stage |
unordered_writes: true) as long as a line is not older than the stream's newest line minus half of max_chunk_age (1h with defaults).too_far_behind: stage.timestamp parsing a wrong field, or an agent replaying an old backlog.loki_write_* metrics.Handy for scripts, incident notes and checking label cardinality.
export LOKI_ADDR=http://localhost:3100
logcli labels # label names
logcli labels app # values of one label
logcli series '{namespace="shop"}' # streams that match
# cardinality per label for a selector
logcli series --analyze-labels '{namespace="shop"}'
# Total Streams: 12
# Unique Labels: 5
# Label Name Unique Values Found In Streams
# pod 6 12
# level 3 12
# ...
logcli query --since=1h --limit=20 \
'{app="orders"} |= "payment failed" | json'
logcli query --tail '{namespace="shop", level="ERROR"}'
# metric query at one instant
logcli instant-query \
'sum by (app) (rate({namespace="shop"}[5m]))'--since, --from/--to (RFC3339) set the range; default is the last hour.--limit caps lines (default 30); --batch pages through large results.-o raw prints just the lines (pipe into jq); -o jsonl includes labels.--org-id sets the tenant when multi-tenancy is on./loki/api/v1/query_range, /loki/api/v1/labels, /loki/api/v1/series, /loki/api/v1/tail.Loki and Alloy expose Prometheus metrics; scrape them with a ServiceMonitor (release: kps) and alert on a few.
| Metric | Tells you |
|---|---|
loki_distributor_bytes_received_total | Ingest volume per tenant |
loki_discarded_samples_total | Lines rejected, by reason |
loki_ingester_memory_streams | Active streams held in ingesters |
loki_request_duration_seconds | Latency per route (push, query_range) |
loki_ingester_wal_* | WAL health and replay |
loki_write_* (Alloy) | Agent-side send errors and retries |
split_queries_by_interval) and sharding by stream run pieces in parallel.Most Loki outages and bills trace back to labels.
trace_id, order_id or status as labels → millions of streams, stream_limit rejections.too_far_behind rejections.{namespace=~".+"} with a 7-day range from a dashboard.level is already a label, then filtering | json | level="ERROR" (it is level_extracted).sum by → one series per stream, max_query_series errors.loki_discarded_samples_total.Answer with the exact component, label, stage or function.
trace_id go, and why not a label?|= "o-4182" before | json?| json produce level_extracted in the shop logs?duration_ms per route.__error__ contain after a failed parse, and how do you drop those lines?stream_limit rejections. What do you look at first?Practice: Lab 7 in Hands-on labs — Loki (monolithic) and Alloy in logging, a Grafana data source, LogQL on the shop logs and a ruler alert; then break it with a high-cardinality label.
One vendor-neutral way to produce traces, metrics and logs: the API/SDK split, OTLP, context propagation, resources and semantic conventions, and instrumenting the shop app in Go.
What OpenTelemetry is, how its pieces fit, and what is stable today.
A CNCF project: one specification, per-language APIs and SDKs, a wire protocol, a Collector, and shared attribute names.
tracer.Start, counter.Add).Libraries depend only on the API. The application owner installs the SDK once, at startup, and decides where data goes.
Sending straight from the SDK to a backend also works; a Collector in between lets you batch, enrich, sample, redact and fan out without redeploying apps.
This split is why a library can ship instrumentation without forcing a vendor or even an SDK on its users.
| Layer | Used by | Contains |
|---|---|---|
| API | Library authors, application code | Interfaces: TracerProvider, Tracer, Span, MeterProvider, instruments, propagators; no-op default |
| SDK | Application owner, once at startup | Samplers, span processors, metric readers, views, resource, exporters |
| Instrumentation | Application owner picks | Wrappers for frameworks and clients, built on the API |
| Exporter | Configured in the SDK | OTLP (default choice), Prometheus, console/stdout |
TracerProvider → Tracer → Span.MeterProvider → Meter → instruments: Counter, UpDownCounter, Histogram, Gauge, and async Observable* variants.LoggerProvider with a bridge API — you keep using slog/log4j/logging and a bridge forwards records with trace context.Protobuf messages over gRPC or HTTP. Every SDK, the Collector and most backends speak it.
| Transport | Default port | Paths / notes |
|---|---|---|
| gRPC | 4317 | Services for traces, metrics, logs on one port |
| HTTP/protobuf | 4318 | /v1/traces, /v1/metrics, /v1/logs |
| HTTP/JSON | 4318 | Same paths; handy for debugging with curl |
UNAVAILABLE, HTTP 429/502/503/504) with backoff.OTEL_EXPORTER_OTLP_PROTOCOL is http/protobuf; Go has separate otlptracegrpc and otlptracehttp packages instead./api/v1/otlp/v1/metrics), Loki (/otlp/v1/logs), Tempo, CloudWatch (ch 11–12).Endpoint scheme matters for gRPC exporters: http://otel-gateway.otel:4317 means plaintext, https://… means TLS.
Stability is per signal and per language. Check the status table for your SDK before relying on a feature.
| Area | Status | Notes |
|---|---|---|
| Traces (API, SDK, OTLP) | Stable | Mature in all major languages |
| Metrics (API, SDK, OTLP) | Stable | Exponential histograms map to Prometheus native histograms |
| Logs (data model, bridge API, SDK, OTLP) | Stable spec | Language SDK maturity varies; check yours |
| Profiles | Public Alpha (Mar 2026) | Not for production-critical use; protocol not frozen |
| Span events | Being phased out | Announced early 2026: emit events through the Logs API; existing span events keep working |
| Collector | Core modules 1.x; contrib 0.x | Component stability is per component (alpha/beta/stable) in its README |
| Semantic conventions | Per area | HTTP stable (v1.23.0, 2023); database client stable (2025); messaging, GenAI still in development |
Same ideas as ch 2 with a few extra concepts: temporality, views and exponential histograms.
| OTel instrument | Prometheus result |
|---|---|
Counter (monotonic) | counter (_total) |
UpDownCounter | gauge |
Gauge / ObservableGauge | gauge |
Histogram (explicit buckets) | classic histogram (_bucket, _sum, _count) |
Histogram (exponential aggregation) | native histogram |
deltatocumulative processor converts.OTEL_METRIC_EXPORT_INTERVAL).How a trace survives a network hop, a goroutine and a queue.
The caller injects its span context into request headers; the callee extracts it and starts a child span. Without both halves you get two unrelated traces.
In Go, otelhttp.NewTransport injects and otelhttp.NewHandler extracts — but only with a propagator registered via otel.SetTextMapPropagator. Forgetting it is the number one "broken traces" cause.
The default propagator in every OpenTelemetry SDK. Example from the W3C specification:
traceparent: 00-0af7651916cd43dd8448eb211c80319c-b7ad6b7169203331-01
│ │ │ │
│ trace-id (16 bytes, 32 hex) parent-id trace-flags (01 = sampled)
version (00) (8 bytes, 16 hex: the caller's span ID)
tracestate: congo=t61rcWkgMzE,rojo=00f067aa0ba902b7 # optional vendor-specific key=value listParentBased samplers honour it (ch 9).tracestate lets vendors carry extra data without breaking others; keep it small.X-B3-TraceId … or single b3 header.X-Amzn-Trace-Id: Root=1-…;Parent=…;Sampled=1 (ch 12).uber-trace-id), legacy.OTEL_PROPAGATORS=tracecontext,baggage,xray.Span context travels inside ctx. Every function that might create a span or call out must take ctx and pass it on.
// good: the child span and the outgoing request
// both inherit the request's trace
func create(w http.ResponseWriter, r *http.Request) {
ctx := r.Context() // set by otelhttp
ctx, span := tracer.Start(ctx, "reserve stock")
defer span.End()
req, _ := http.NewRequestWithContext(ctx,
"POST", payURL, body)
resp, err := client.Do(req) // traceparent injected
...
}
// background work that outlives the request:
// keep the trace link, drop the cancellation
go func(ctx context.Context) {
_, span := tracer.Start(ctx, "send receipt")
defer span.End()
...
}(context.WithoutCancel(ctx))// bad: each of these starts a NEW trace
ctx := context.Background() // context dropped
tracer.Start(context.TODO(), "x")
client.Post(payURL, "application/json", body)
// no ctx at all
http.NewRequest("POST", payURL, body)
// ctx-less request
// bad: span never ends -> never exported
_, span := tracer.Start(ctx, "charge")
if err != nil { return err } // leaked
span.End()
// rule: start, then immediately
// defer span.End()Other languages hide the context in thread-locals or async-locals (Java, Python, Node), so it usually "just works" — until you hand work to a thread pool or callback.
Queues break the request/response chain. Put the context into message headers on produce and extract it on consume.
prop := otel.GetTextMapPropagator()
// producer: PRODUCER span, inject into headers
ctx, span := tracer.Start(ctx, "publish orders",
trace.WithSpanKind(trace.SpanKindProducer))
headers := propagation.MapCarrier{}
prop.Inject(ctx, headers) // traceparent, baggage
msg.Headers = toKafkaHeaders(headers)
span.End()
// consumer: extract, then CONSUMER span
carrier := propagation.MapCarrier(fromKafkaHeaders(msg))
pctx := prop.Extract(context.Background(), carrier)
ctx, span = tracer.Start(pctx, "process orders",
trace.WithSpanKind(trace.SpanKindConsumer))
defer span.End()trace.WithLinks).messaging.system, messaging.destination.name, messaging.operation.type) are still in development; instrumentation libraries for Kafka, SQS and NATS set them for you.Baggage propagates application-defined pairs (baggage header) to every downstream service. It is not added to spans automatically.
// header on the wire
baggage: shop.tenant=acme,
shop.experiment=checkout-v2
// set at the edge
m, _ := baggage.NewMember(
"shop.tenant", "acme")
b, _ := baggage.New(m)
ctx = baggage.ContextWithBaggage(ctx, b)
// read anywhere downstream
t := baggage.FromContext(ctx).
Member("shop.tenant").Value()
// copy onto a span when useful
span.SetAttributes(attribute.String(
"shop.tenant", t))Propagation is a security and data-quality decision at the edge of your system.
traceparent, including one with sampled=1 on every request.traceparent.OTEL_PROPAGATORS (default tracecontext,baggage) selects propagators via env in SDKs that read it.b3 or xray so older services keep joining traces during migration.Who emitted the data, and standard names for what it describes.
A resource is a set of attributes describing the entity producing telemetry. It is attached once per provider and repeated on every span, metric and log.
| Attribute | Example | Why it matters |
|---|---|---|
service.name | orders | Required; default is unknown_service:<executable> — always set it |
service.version | 1.4.2 or git SHA | Compare before/after a deploy |
service.namespace | shop | Groups services; with Prometheus forms job="shop/orders" |
service.instance.id | pod UID or random UUID | Distinguishes replicas; becomes Prometheus instance |
deployment.environment.name | lab, prod | Renamed from deployment.environment in semconv 1.27 |
k8s.namespace.name, k8s.pod.name, k8s.deployment.name, k8s.node.name | shop, orders-7d9… | Usually added by the Collector k8sattributes processor (ch 9) |
cloud.provider, cloud.region, cloud.account.id | aws, ap-northeast-1 | Resource detectors (EC2, ECS, EKS) or the Collector |
telemetry.sdk.language, telemetry.sdk.version | go, 1.x | Added by the SDK |
Set identity in code or env; let detectors and the Collector add infrastructure attributes.
res, err := resource.New(ctx,
resource.WithFromEnv(), // OTEL_SERVICE_NAME,
// OTEL_RESOURCE_ATTRIBUTES
resource.WithTelemetrySDK(), // telemetry.sdk.*
resource.WithHost(), // host.name
resource.WithProcess(), // process.* (PID, runtime)
resource.WithAttributes(
semconv.ServiceVersion(version),
),
)
# Kubernetes Deployment: identity from env
env:
- name: OTEL_SERVICE_NAME
value: orders
- name: OTEL_RESOURCE_ATTRIBUTES
value: deployment.environment.name=lab,service.namespace=shopservice.*, deployment.environment.name — things only the team knows.k8sattributes: pod, namespace, deployment, node from the API server, matched by pod IP or UID.resourcedetection: env, system, ec2, eks, … on the node.override: false).HTTP conventions were declared stable in semconv v1.23.0. Instrumentation that still emits the old names can emit both during migration.
| Old (experimental) | Stable |
|---|---|
http.method | http.request.method |
http.status_code | http.response.status_code |
http.target | url.path + url.query |
http.url | url.full |
net.peer.name / net.host.name | server.address |
http.server.duration (ms) | http.server.request.duration (s) |
http.client.duration (ms) | http.client.request.duration (s) |
{method} {http.route}, e.g. POST /orders — low cardinality, never the raw path.{method}, optionally with a URL template.http.route is the template (/orders/{id}), url.path the actual path.OTEL_SEMCONV_STABILITY_OPT_IN=http (new names) or http/dup (both).Use an existing convention whenever one exists; dashboards and backends understand them.
| Area | Key attributes | Status |
|---|---|---|
| Database client | db.system.name, db.namespace, db.operation.name, db.query.text (sanitised) | Stable (2025) |
| Messaging | messaging.system, messaging.destination.name, messaging.operation.type | Development |
| RPC / gRPC | rpc.system, rpc.service, rpc.method | Check the semconv site |
| Exceptions | exception.type, exception.message, exception.stacktrace | Recorded by span.RecordError |
| Errors | error.type (e.g. timeout, 500) | Stable with HTTP |
shop.order.items, shop.customer.tier — never bare type or id.snake_case within a segment.When OTLP metrics land in Prometheus (OTLP receiver, or Collector → remote write), names and resources are translated.
UnderscoreEscapingWithSuffixes)| OpenTelemetry | Prometheus |
|---|---|
http.server.request.duration (s, histogram) | http_server_request_duration_seconds + _bucket/_sum/_count |
orders.created (Counter, unit {order}) | orders_created_total ({…} units dropped) |
attribute http.response.status_code | label http_response_status_code |
resource service.namespace + service.name | job="shop/orders" |
resource service.instance.id | instance |
| other resource attributes | target_info series (join with group_left, ch 3) |
# prometheus.yml (Prometheus 3)
otlp:
# copy selected resource attributes onto every series
promote_resource_attributes:
- service.namespace
- deployment.environment.name
- k8s.namespace.name
# keep UTF-8 names with dots (no _ escaping):
# translation_strategy: NoUTF8EscapingWithSuffixes
# start flag that enables the endpoint
# --web.enable-otlp-receiver
# push to: /api/v1/otlp/v1/metricsThe Go SDK in practice: setup, HTTP in and out, manual spans, configuration, and zero-code options for other languages.
Create the resource, the exporter and the provider; register the provider and the propagator globally; return the shutdown function.
import (
"context"
"go.opentelemetry.io/otel"
"go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc"
"go.opentelemetry.io/otel/propagation"
"go.opentelemetry.io/otel/sdk/resource"
sdktrace "go.opentelemetry.io/otel/sdk/trace"
)
func setupOTel(ctx context.Context) (func(context.Context) error, error) {
res, err := resource.New(ctx,
resource.WithFromEnv(), resource.WithTelemetrySDK())
if err != nil {
return nil, err
}
exp, err := otlptracegrpc.New(ctx) // OTEL_EXPORTER_OTLP_ENDPOINT
if err != nil {
return nil, err
}
tp := sdktrace.NewTracerProvider(
sdktrace.WithBatcher(exp), sdktrace.WithResource(res))
otel.SetTracerProvider(tp)
otel.SetTextMapPropagator(propagation.NewCompositeTextMapPropagator(
propagation.TraceContext{}, propagation.Baggage{}))
return tp.Shutdown, nil
}WithBatcher uses the batch span processor: queue 2048, batch 512, 5 s delay (spec defaults, tunable with OTEL_BSP_*).ParentBased(AlwaysSample); the Go SDK also reads OTEL_TRACES_SAMPLER / _ARG.SetTracerProvider, every tracer is a no-op: no error, no data.SetTextMapPropagator, spans exist but each service starts its own trace.sdkmetric.NewMeterProvider(sdkmetric.WithReader(sdkmetric.NewPeriodicReader(exp))) + otel.SetMeterProvider.Wrap the whole handler chain once, outermost, so logging and metrics middleware (ch 2, ch 6) run inside the span. Wrap the client transport to create CLIENT spans and inject headers.
// server: otelhttp outermost, mux innermost
handler := otelhttp.NewHandler(
metrics.Middleware(logx.Middleware(log, routeSpan(mux))),
"orders", // fallback span name
otelhttp.WithFilter(func(r *http.Request) bool {
return r.URL.Path != "/healthz" && r.URL.Path != "/metrics"
}))
// mux sets r.Pattern ("POST /orders"); name the span after it
func routeSpan(next http.Handler) http.Handler {
return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
next.ServeHTTP(w, r)
if _, route, ok := strings.Cut(r.Pattern, " "); ok {
span := trace.SpanFromContext(r.Context())
span.SetName(r.Pattern)
span.SetAttributes(attribute.String("http.route", route))
}
})
}
// client: requests made with ctx join the trace
client = &http.Client{Timeout: 2 * time.Second,
Transport: otelhttp.NewTransport(http.DefaultTransport)}
req, _ := http.NewRequestWithContext(ctx, "POST", payURL, body)
resp, err := client.Do(req)POST /orders with http.request.method, http.route, http.response.status_code, url.path, server.address.trace_id in logs, exemplars on histograms (ch 9).WithFilter returns false).traceparent injected.r.Pattern themselves (otelhttp.WithRouteTag was deprecated and later removed); routeSpan keeps the result independent of the version.http.server.request.duration through the global MeterProvider (no-op if none is set).Library spans show HTTP; your spans show what the request was doing.
var tracer = otel.Tracer("obs-lab/orders")
func create(w http.ResponseWriter, r *http.Request) {
ctx := r.Context()
span := trace.SpanFromContext(ctx) // the otelhttp span
...
span.SetAttributes(
attribute.String("shop.order.id", o.ID),
attribute.Int("shop.order.items", 1))
ctx, pay := tracer.Start(ctx, "charge payment")
defer pay.End()
if err := charge(ctx, o); err != nil {
pay.RecordError(err) // exception.* event
pay.SetStatus(codes.Error, "charge failed")
http.Error(w, "payment failed", 502)
return
}
...
}charge payment, not charge o-4182.Error only for real failures; leave Unset for success (Ok is for explicit overrides).RecordError stores the error as an exception event; with span events being phased out, check what your SDK version emits.Defined by the specification so the same Deployment settings work across languages. Support varies slightly per SDK.
| Variable | Example | Effect |
|---|---|---|
OTEL_SERVICE_NAME | orders | Sets service.name (wins over OTEL_RESOURCE_ATTRIBUTES) |
OTEL_RESOURCE_ATTRIBUTES | deployment.environment.name=lab | Extra resource attributes, comma-separated |
OTEL_EXPORTER_OTLP_ENDPOINT | http://otel-gateway.otel:4317 | Base endpoint for all signals (_TRACES_ENDPOINT etc. per signal) |
OTEL_EXPORTER_OTLP_PROTOCOL | grpc, http/protobuf | Transport, in SDKs that pick it at runtime |
OTEL_TRACES_SAMPLER / _ARG | parentbased_traceidratio / 0.25 | Head sampling (ch 9) |
OTEL_PROPAGATORS | tracecontext,baggage | Propagators (default shown) |
OTEL_TRACES_EXPORTER, OTEL_METRICS_EXPORTER, OTEL_LOGS_EXPORTER | otlp, console, none | Exporter choice in agents and autoconfigure SDKs |
OTEL_METRIC_EXPORT_INTERVAL | 60000 | Metric export period in ms |
OTEL_SDK_DISABLED | true | Turn the SDK off without a code change |
Agents and wrappers load instrumentation libraries at startup and configure the SDK from OTEL_* variables.
| Language | How | Notes |
|---|---|---|
| Java | -javaagent:opentelemetry-javaagent.jar (e.g. via JAVA_TOOL_OPTIONS) | Most complete: hundreds of libraries |
| Python | opentelemetry-bootstrap -a install then opentelemetry-instrument python app.py | Installs instrumentations matching your packages |
| Node.js | node --require @opentelemetry/auto-instrumentations-node/register app.js | Must load before other modules |
| .NET | OpenTelemetry .NET Automatic Instrumentation (CLR profiler) | Also a NuGet-based option |
| Go | eBPF-based: OpenTelemetry Go auto-instrumentation and OBI (OpenTelemetry eBPF Instrumentation) | Needs elevated privileges; covers common libraries; compile-time instrumentation is a separate SIG effort |
Zero-code gets you HTTP, RPC and DB spans quickly. Business attributes, custom spans and good span names still need a few lines of code.
The Operator manages Collectors (OpenTelemetryCollector CR) and injects language agents into pods via an Instrumentation CR and an annotation.
apiVersion: opentelemetry.io/v1alpha1
kind: Instrumentation
metadata:
name: default
namespace: shop
spec:
exporter:
endpoint: http://otel-gateway.otel:4318
propagators: [tracecontext, baggage]
sampler:
type: parentbased_traceidratio
argument: "0.25"
resource:
resourceAttributes:
deployment.environment.name: lab
---
# in the Deployment's pod template
metadata:
annotations:
instrumentation.opentelemetry.io/inject-java: "true"
# inject-python, inject-nodejs, inject-dotnet,
# inject-go (+ otel-go-auto-target-exe: /app)JAVA_TOOL_OPTIONS / PYTHONPATH / NODE_OPTIONS and OTEL_* env (service name from the workload, k8s resource attributes).Most "OTel does not work" tickets are one of these.
otel.SetTracerProvider: all spans are no-ops, silently.service.name unset → unknown_service:app in every backend.context.Background() or client.Post in the request path → broken parent chain.defer span.End() on early returns.GET /orders/o-41): unbounded names break span metrics.Error on 404s from servers.http.status_code) and new names across services without http/dup during migration.id, type).deployment.environment after instrumentation moved to deployment.environment.name.http.server.request.duration is seconds, not ms.Answer with the exact API, header, attribute or variable.
00-0af7…319c-b7ad…3331-01: what are the three fields after the version?context.Background() in a handler break a trace?http.status_code? For http.server.duration?GET /orders/o-41 be called?service.namespace and service.name appear in Prometheus?OTEL_SEMCONV_STABILITY_OPT_IN=http/dup do?Practice: Lab 8 in Hands-on labs — the otel-gateway Collector with a debug exporter, orders and payments instrumented with the Go SDK and otelhttp, and one trace spanning both; then break propagation and watch the trace split.
Read a trace like a profiler for a distributed request, decide which traces to keep, run the Collector as the telemetry router, query Tempo with TraceQL, and connect traces to metrics and logs.
Kinds, status, attributes, events and links: what a span records and how a waterfall is read.
A span is one timed operation. A trace is the tree of spans that share one trace ID.
# one span from the shop app, as the SDK exports it (simplified)
trace_id: 4bf92f3577b34da6a3ce929d0e0e4736
span_id: 00f067aa0ba902b7
parent_span_id: a3ce929d0e0e4736 # empty on root
name: "POST /charge" # route, not URL
kind: SPAN_KIND_SERVER
start, end: 01.120 s, 01.388 s # 268 ms
status: ERROR "card declined"
attributes:
http.request.method: POST
http.route: /charge
http.response.status_code: 502
resource: # who produced it (ch 8)
service.name: payments
deployment.environment.name: lab
events: [ exception {exception.type, …} ]
links: []Unset, Ok or Error — the error flag queries filter on.An HTTP call between two services produces two spans: a CLIENT span in the caller and a SERVER span in the callee.
| Kind | Meaning | Shop example |
|---|---|---|
SERVER | Handles an incoming synchronous request | orders handling POST /orders |
CLIENT | Makes an outgoing synchronous request | orders calling payments |
PRODUCER | Hands a message to a broker or queue | publish order.created |
CONSUMER | Processes a message from a broker | worker handling order.created |
INTERNAL | In-process work (the default) | validateOrder() |
otelhttp) sets it for you.Status is not the HTTP status code. The semantic conventions define the mapping so dashboards and TraceQL agree.
Unset (default): the instrumentation made no judgment.Error: the operation failed; add a description.Ok: an explicit "this succeeded" set by the application; it overrides later Error from instrumentation. Use rarely.5xx → Error; 4xx → left Unset (the client sent a bad request, the server did its job).4xx and 5xx → Error.Error on the client span.otelhttp applies this mapping for you.// Go: record a failure on the current span
import (
"go.opentelemetry.io/otel/codes"
"go.opentelemetry.io/otel/trace"
)
span := trace.SpanFromContext(ctx)
if err != nil {
span.RecordError(err) // event
span.SetStatus(codes.Error,
"charge failed")
return err
}Attributes describe the whole operation; events mark moments inside it; links connect spans that are related but not parent and child.
http.route, db.system.name, messaging.system).shop.order.items, shop.customer.tier.exception, retry, cache miss.span.RecordError(err) adds an exception event.trace.WithLinks(...).Total latency is decided by the longest chain of sequential work, not by the sum of all spans.
Speeding up GET inventory (off the critical path) saves nothing; fixing fraud-check (240 of 412 ms) does.
Most latency problems have a shape. Learn the shapes and a trace tells you the fix.
| Shape in the waterfall | What it usually means | Next step |
|---|---|---|
| Staircase of identical short DB/HTTP spans | N+1 queries or a call inside a loop | Batch the call; check the code path in code.function attributes |
| One long child span covering most of the parent | A slow dependency | Open the dependency's SERVER span; check its own children |
| Gap before the first child | Work before any call: lock wait, GC, CPU, a missing span | Add a manual span or check profiles |
| CLIENT span much longer than its SERVER child | Network, DNS, connection pool exhaustion, queueing | Check client pool metrics and DNS spans |
| Repeated CLIENT spans with errors, then success | Retries | Count retries; check timeouts and backoff |
| Wide fan-out, total = slowest branch | Tail latency amplification | Hedge, cache or cut the fan-out |
Keep the traces that explain problems, pay for fewer of the boring ones, and keep the decision consistent across services.
Tracing every request of a busy service is expensive to export, store and query. The question is which traces you can afford to lose.
Use ParentBased everywhere so a service never breaks a trace that an upstream service decided to keep.
// Go SDK: 10 % of new traces, follow the parent otherwise
import sdktrace "go.opentelemetry.io/otel/sdk/trace"
tp := sdktrace.NewTracerProvider(
sdktrace.WithSampler(
sdktrace.ParentBased(sdktrace.TraceIDRatioBased(0.10)),
),
sdktrace.WithBatcher(exporter),
sdktrace.WithResource(res),
)
# the same with environment variables (no code change)
OTEL_TRACES_SAMPLER=parentbased_traceidratio
OTEL_TRACES_SAMPLER_ARG=0.10| Sampler | Behaviour |
|---|---|
AlwaysOn / always_on | Record and export everything (SDK default is parentbased_always_on) |
AlwaysOff / always_off | Nothing |
TraceIDRatioBased(r) | Keep a fraction, decided from the trace ID, so every service agrees |
ParentBased(root) | Follow the parent's sampled flag; use root only for new traces |
A service that uses plain TraceIDRatioBased with a different ratio than its callers creates broken traces: some spans kept, their parents dropped. Wrap it in ParentBased.
The tail_sampling processor (contrib) waits for a trace's spans, then evaluates policies. A trace is kept if any policy says sample (unless a drop policy matches).
processors:
tail_sampling:
decision_wait: 10s # wait for late spans of a trace
num_traces: 50000 # traces held in memory
expected_new_traces_per_sec: 100
policies:
- name: keep-errors
type: status_code
status_code: {status_codes: [ERROR]}
- name: keep-slow
type: latency
latency: {threshold_ms: 500}
- name: keep-checkout-route
type: string_attribute
string_attribute: {key: http.route, values: [/orders]}
- name: baseline
type: probabilistic
probabilistic: {sampling_percentage: 5}status_code, latency, probabilistic, string_attribute, numeric_attribute, boolean_attribute.rate_limiting (spans per second), span_count, trace_state, ottl_condition.and (all sub-policies), composite (rate shares per sub-policy), drop (explicitly drop, e.g. health checks).num_traces × spans per trace × span size. Spans arriving after decision_wait are handled by the decision cache (if configured) or treated as a new trace.Every span of a trace must reach the same Collector instance. With several gateway replicas, add a load-balancing tier in front.
# tier 1: route spans by trace ID
exporters:
loadbalancing:
routing_key: traceID
protocol:
otlp:
tls: {insecure: true}
resolver:
k8s:
service: otel-sampler.otel # headless service
ports: [4317]When a gateway pod is added or removed, the hash ring changes and in-flight traces can be split. Keep the sampling tier stable and size it for peak, not average.
Sampled traces are a biased subset. Anything that counts must be computed before the sampler, or from metrics.
tail_sampling in the pipeline (section 37).TraceIDRatioBased is consistent because every service hashes the same trace ID.tracestate (ot=th:…) so later stages can adjust counts.parentbased_always_on in SDKs + tail sampling (errors, slow, 5–10 % baseline).A vendor-neutral process that receives, transforms and exports telemetry: the router between your apps and every backend.
Components are declared once, then wired into pipelines per signal under service.pipelines. A component that is declared but not used in a pipeline is not started.
This is the shape of otel-gateway in the labs: OTLP in, Kubernetes metadata added, traces to Tempo, span metrics to Prometheus.
receivers:
otlp:
protocols:
grpc: {endpoint: 0.0.0.0:4317}
http: {endpoint: 0.0.0.0:4318}
processors:
memory_limiter:
check_interval: 1s
limit_percentage: 80
spike_limit_percentage: 20
k8sattributes: {} # see next slides
batch: {}
connectors:
spanmetrics: {}
exporters:
otlp/tempo:
endpoint: tempo.tracing.svc:4317
tls: {insecure: true}
prometheusremotewrite:
endpoint: http://prometheus-operated.monitoring.svc:9090/api/v1/writeextensions:
health_check: {endpoint: 0.0.0.0:13133}
service:
extensions: [health_check]
pipelines:
traces:
receivers: [otlp]
processors: [memory_limiter, k8sattributes, batch]
exporters: [otlp/tempo, spanmetrics]
metrics/spanmetrics:
receivers: [spanmetrics]
processors: [memory_limiter, batch]
exporters: [prometheusremotewrite]
telemetry:
metrics:
readers:
- pull:
exporter:
prometheus: {host: 0.0.0.0, port: 8888}type/name (e.g. otlp/tempo) gives several instances of one component type. Validate before deploying: otelcol-contrib validate --config=config.yaml.
Order matters: processors run in the order listed in the pipeline.
| Processor | Purpose | Where in the pipeline |
|---|---|---|
memory_limiter | Refuses data when memory crosses the limit, so the Collector backs off instead of OOM-crashing | First |
k8sattributes | Adds k8s.namespace.name, k8s.pod.name, k8s.deployment.name, … by matching the sender's pod | Early, before anything that filters on them |
resource / attributes | Insert, upsert, delete or hash resource / span attributes | Middle |
transform | OTTL statements: rename, redact, set values conditionally | Middle |
filter | Drop spans, metrics or logs matching OTTL conditions (health checks, noisy debug logs) | Before expensive work |
tail_sampling | Keep or drop whole traces (section 34) | After enrichment, before batch |
batch | Groups data into larger requests; fewer, bigger exports | Last |
Newer Collector versions can also batch inside the exporter (sending_queue batching in exporterhelper); the batch processor is still widely used. Check the docs for the version you run.
Do metadata and hygiene once in the Collector instead of in every service.
processors:
k8sattributes:
auth_type: serviceAccount
# RBAC: get/list/watch pods,
# namespaces, replicasets
extract:
metadata: [k8s.namespace.name,
k8s.pod.name, k8s.deployment.name,
k8s.node.name]
pod_association:
- sources:
- from: resource_attribute
name: k8s.pod.ip
- sources:
- from: connection # sender IP
resource:
attributes:
- key: deployment.environment.name
value: lab
action: upsert transform:
error_mode: ignore
trace_statements:
- context: span
statements:
- delete_key(attributes,
"http.request.header.authorization")
- replace_pattern(attributes["url.query"],
"token=[^&]+", "token=REDACTED")
filter/health:
error_mode: ignore
traces:
span:
- 'attributes["url.path"] == "/healthz"'from: connection only works when the Collector sees the pod IP directly (agent DaemonSet, or a gateway without a proxy in front). Behind a load balancer, send k8s.pod.ip as a resource attribute from the agent.
Agents sit next to the workload; gateways are a central, scalable tier. Most clusters end up with both.
$(NODE_IP):4317 or a node-local service.filelog), kubelet stats (kubeletstats), host metrics.k8sattributes with from: connection works.otel-gateway in namespace otel, apps export to it directly.The Collector is a toolkit: a distribution is a binary with a chosen set of components.
| Distribution | Contents |
|---|---|
otelcol (core) | Small set: OTLP, debug, batch, memory_limiter |
otelcol-contrib | Almost every community component (tail_sampling, k8sattributes, spanmetrics, prometheusremotewrite) |
otelcol-k8s | Components curated for Kubernetes |
Custom (ocb builder) | Only what you list in a manifest: smaller binary, smaller attack surface |
| ADOT | AWS Distro for OpenTelemetry: AWS exporters and SigV4 auth (ch 12) |
| Grafana Alloy | Grafana's collector; Alloy 1.21 adds an OpenTelemetry Engine (public preview) |
# Helm (open-telemetry repo), contrib image, gateway mode
helm upgrade --install otel-gateway \
open-telemetry/opentelemetry-collector -n otel \
--set mode=deployment \
--set image.repository=otel/opentelemetry-collector-contrib \
-f gateway-values.yaml # 'config:' merged into chart defaults
# or with the OpenTelemetry Operator:
# kind: OpenTelemetryCollector (spec.mode, spec.config)
# Validate locally before you ship
docker run --rm -v $PWD/config.yaml:/c.yaml \
otel/opentelemetry-collector-contrib \
validate --config=/c.yamlA Collector that silently drops data is worse than none. Scrape its internal metrics (port 8888 above) and alert on refusals and failures.
| Metric | Meaning |
|---|---|
otelcol_receiver_accepted_spans_total | Spans received successfully |
otelcol_receiver_refused_spans_total | Rejected at the receiver (often memory_limiter back-pressure) |
otelcol_exporter_sent_spans_total | Delivered to the backend |
otelcol_exporter_send_failed_spans_total | Export failures after retries |
otelcol_exporter_queue_size / _capacity | Sending queue fill level |
# alert: the Collector is failing to export
sum by (exporter) (
rate(otelcol_exporter_send_failed_spans_total[5m])
) > 0
# queue more than 80 % full: backend too slow
max by (exporter) (
otelcol_exporter_queue_size
/ otelcol_exporter_queue_capacity
) > 0.8Same pattern for metrics and logs: _metric_points_ and _log_records_ variants. Exact names can change between Collector versions; check curl :8888/metrics on yours. The debug exporter (verbosity: detailed) is the quickest way to see what a pipeline really carries.
A trace store that only needs object storage, and a query language that selects spans by what they did.
Tempo stores traces in object storage as Parquet blocks and finds them by trace ID or by TraceQL. It indexes no labels up front.
tempo-cli.tempo-cli migrate config converts 2.x configs.grafana-community/helm-charts.A TraceQL query returns traces that contain matching spans. { … } selects a spanset; conditions use scoped attributes and intrinsics.
# all spans of the orders service
{ resource.service.name = "orders" }
# failed spans in payments
{ resource.service.name = "payments" && status = error }
# slow server spans on one route
{ span.http.route = "/orders" && kind = server && duration > 500ms }
# 5xx responses, any service
{ span.http.response.status_code >= 500 }
# whole-trace intrinsics
{ trace:rootService = "orders" && trace:duration > 1s }| Syntax | Means |
|---|---|
resource.x | Resource attribute (service.name, k8s.pod.name) |
span.x | Span attribute (http.route) |
name, status, kind, duration | Span intrinsics (span:name, span:status, … also accepted) |
status values | error, ok, unset |
kind values | server, client, producer, consumer, internal, unspecified |
trace:duration, trace:rootName, trace:rootService | Trace-level intrinsics |
Unscoped .x matches either scope but checks both; prefer span. or resource. — it reads one column instead of two.
Spansets can be combined by their position in the tree, and filtered with aggregates per trace.
# orders calls payments and the payments side fails
{ resource.service.name = "orders" }
>> { resource.service.name = "payments" && status = error }
# direct child only
{ name = "POST /orders" } > { kind = client }
# siblings: two operations under the same parent
{ name = "GET inventory" } ~ { name = "POST /charge" }
# both conditions, anywhere in the same trace
{ span.http.route = "/orders" } && { status = error }
# traces with more than 20 DB spans (N+1 hunt)
{ span.db.system.name != nil } | count() > 20
# traces whose orders spans average over 300 ms
{ resource.service.name = "orders" } | avg(duration) > 300ms
# show extra attributes in the result table
{ status = error } | select(span.http.route, resource.k8s.pod.name)| Operator | Meaning |
|---|---|
>> / << | Descendant / ancestor |
> / < | Child / parent |
~ | Sibling |
!>>, !>, !~ | Negated structural forms |
&&, || between {} | Both / either spanset in the trace |
| count(), avg(), min(), max(), sum() | Aggregate per spanset |
| select(...) | Return extra attributes |
TraceQL metrics turn a spanset into a time series at query time — useful for questions your metrics never anticipated.
# request rate per service (server spans)
{ kind = server } | rate() by (resource.service.name)
# error rate per route in orders
{ resource.service.name = "orders" && status = error }
| rate() by (span.http.route)
# p95 latency per route
{ resource.service.name = "orders" && kind = server }
| quantile_over_time(duration, .95) by (span.http.route)
# counts and histograms
{ name = "POST /charge" } | count_over_time()
{ name = "POST /charge" } | histogram_over_time(duration)
# what is different about slow spans?
{ resource.service.name = "payments" }
| compare({ duration > 500ms })All of these accept OTLP; the instrumentation in your apps does not change when you switch.
| Backend | Storage | Query | Notes |
|---|---|---|---|
| Grafana Tempo | Object storage (Parquet blocks) | TraceQL, TraceQL metrics | Cheap at scale; tight Grafana integration |
| Jaeger v2 | Cassandra, Elasticsearch/OpenSearch, Badger, memory | Jaeger UI, search by tags | v2 is built on the OpenTelemetry Collector |
| AWS X-Ray / CloudWatch | Managed | Trace map, Transaction Search | OTLP via ADOT or the CloudWatch agent (ch 12) |
| Commercial APMs | Managed | Vendor UIs | Same OTLP data; watch per-span pricing |
Keep the vendor decision in the Collector's exporters, never in application code.
Exemplars, trace IDs in logs and span metrics: one click from a spike on a graph to the request that caused it.
Correlation is not magic: each link needs a shared key in both signals and a data source setting in Grafana.
An exemplar is one example observation (value, timestamp, labels such as trace_id) stored next to a histogram bucket or counter.
// orders: observe latency with an exemplar (client_golang)
sc := trace.SpanContextFromContext(r.Context()) // inside otelhttp
h := reqDuration.WithLabelValues(r.Method, route)
if sc.IsSampled() {
h.(prometheus.ExemplarObserver).ObserveWithExemplar(
elapsed.Seconds(),
prometheus.Labels{"trace_id": sc.TraceID().String()},
)
} else {
h.Observe(elapsed.Seconds())
}
// exemplars are only exposed in OpenMetrics format
http.Handle("/metrics", promhttp.HandlerFor(reg,
promhttp.HandlerOpts{EnableOpenMetrics: true}))--enable-feature=exemplar-storage (kps: prometheus.prometheusSpec.enableFeatures: [exemplar-storage]).trace_id → Tempo data source.Logs carry trace_id (ch 6); Grafana uses it in both directions.
# Grafana data source provisioning (kps: grafana.additionalDataSources)
- name: Tempo
type: tempo
uid: tempo
url: http://tempo.tracing.svc:3200
jsonData:
tracesToLogsV2:
datasourceUid: loki
spanStartTimeShift: -5m
spanEndTimeShift: 5m
filterByTraceID: true
tags: [{key: service.name, value: service_name}]
serviceMap: {datasourceUid: prometheus}
nodeGraph: {enabled: true}- name: Loki
type: loki
uid: loki
url: http://loki.logging.svc:3100
jsonData:
derivedFields:
# trace_id inside the JSON log line → Tempo link
- name: TraceID
matcherRegex: '"trace_id":"(\w+)"'
url: '$${__value.raw}'
datasourceUid: tempo
# or, if Alloy stores trace_id as structured metadata:
# - name: TraceID
# matcherType: label
# matcherRegex: trace_id
# url: '$${__value.raw}'
# datasourceUid: tempo$$ escapes $ in provisioning files, which otherwise expand environment variables. The tags mapping renames a span resource attribute to the Loki label name used in the generated query.
Generate RED metrics from 100 % of spans before sampling, so dashboards stay correct while Tempo stores only a subset.
connectors:
spanmetrics:
histogram:
explicit:
buckets: [5ms, 25ms, 100ms, 250ms, 500ms, 1s, 2.5s]
dimensions:
- name: http.route
- name: http.response.status_code
servicegraph:
latency_histogram_buckets: [10ms, 100ms, 500ms, 1s]
forward/sample: {}
service:
pipelines:
traces: # before sampling
receivers: [otlp]
exporters: [spanmetrics, servicegraph, forward/sample]
traces/sampled:
receivers: [forward/sample]
processors: [tail_sampling, batch]
exporters: [otlp/tempo]
metrics/derived:
receivers: [spanmetrics, servicegraph]
exporters: [prometheusremotewrite]traces_span_metrics_calls_total and a duration histogram (unit ms by default) per service.name, span.name, span.kind, status.code + your dimensions.traces_service_graph_request_total, _failed_total and latency histograms per client/server pair: the service map.forward connector splits the pipeline: metrics see every span, Tempo only the sampled ones.prometheus.prometheusSpec.enableRemoteWriteReceiver: true.The workflow chapter 13 builds on, with the links from this section.
OrdersHighLatency fires/orders > 500 ms4bf92f…payments fraud-check 240 msEach step answers one question: is it real, where is it, which request, which span, why. If any link is missing, write down which one and fix the instrumentation after the incident.
Most "tracing does not work" tickets come from this list.
GET /orders/8812): every span unique, nothing groups.TraceIDRatioBased without ParentBased: half-traces.Error on server spans: error rate inflated by bad clients.tp.Shutdown(ctx): last spans lost on exit.memory_limiter: OOM-kill under load, all buffered data lost.:8888: export failures go unseen.debug exporter with verbosity: detailed in a test pipeline.otelcol-contrib validate in CI.{ trace:rootService != "orders" && resource.service.name = "payments" } finds orphaned payments roots.otelcol_exporter_send_failed_spans_total.Answer with the exact setting, operator or attribute.
404 set Error on a server span? On a client span?ParentBased?k8sattributes with from: connection fail?Practice: Lab 9 in Hands-on labs — Tempo in tracing, the gateway exporting to it with tail sampling and span metrics, exemplars and trace↔logs links in Grafana.
Dashboards that answer questions fast: RED for every service, USE for every resource, Grafana layouts that guide the eye, and dashboards kept as code.
Rate, errors and duration per service and route: the first screen you open when a user says "it is slow".
Rate, Errors, Duration (Tom Wilkie). They describe the work as the caller experiences it, so they line up with SLIs (the SRE deck).
5xx for servers; include timeouts.RED is for services (things that handle requests). Resources such as CPUs, disks and connection pools get USE (section 39). Google's four golden signals = RED + saturation.
All three from the two metrics in the shared conventions: http_requests_total and http_request_duration_seconds.
# Rate: requests per second per route
sum by (route) (rate(http_requests_total{job="$service"}[$__rate_interval]))
# Errors: ratio of 5xx (0–1, unit percentunit)
sum(rate(http_requests_total{job="$service", code=~"5.."}[$__rate_interval]))
/
sum(rate(http_requests_total{job="$service"}[$__rate_interval]))
# Errors per route, absolute
sum by (route) (rate(http_requests_total{job="$service", code=~"5.."}[$__rate_interval]))
# Duration: p50 / p95 / p99 per route (classic histogram)
histogram_quantile(0.95,
sum by (le, route) (rate(http_request_duration_seconds_bucket{job="$service"}[$__rate_interval])))
# Duration with native histograms (if enabled, ch 2): no le label needed
histogram_quantile(0.95,
sum by (route) (rate(http_request_duration_seconds{job="$service"}[$__rate_interval])))
# Share of requests faster than 300 ms (an SLI-style ratio; bucket must exist)
sum(rate(http_request_duration_seconds_bucket{job="$service", le="0.3"}[$__rate_interval]))
/
sum(rate(http_request_duration_seconds_count{job="$service"}[$__rate_interval]))If the error ratio has no traffic, 0/0 gives no data, not zero. In Grafana, show "No data" as 0 only when you are sure that zero traffic is healthy.
Latency is a distribution. Pick the view that shows the shape, not just one number.
histogram_quantile interpolates inside a bucket: the answer is only as good as your bucket bounds (ch 3).sum by (le) first, then take the quantile.rate(..._sum) / rate(..._count) = mean latency.sum by (le) (increase(..._bucket[$__rate_interval])), format Heatmap.Top row answers "is it healthy?", the middle "where?", the bottom "what changed and what does it depend on?".
A service is only as fast as its slowest dependency. Measure calls from the client side too.
otelhttp.NewTransport emits http.client.request.duration (ch 8).span_kind="SPAN_KIND_CLIENT".# orders → payments, from orders' CLIENT spans
# (span metrics connector, ch 9; duration in ms by default)
histogram_quantile(0.95, sum by (le) (
rate(traces_span_metrics_duration_milliseconds_bucket{
service_name="orders",
span_kind="SPAN_KIND_CLIENT"}[$__rate_interval])))
# service graph: failed requests between two services
sum by (client, server) (
rate(traces_service_graph_request_failed_total[$__rate_interval]))Label names produced by the span metrics connector (service_name, span_kind, span_name, status_code) are derived from the OTel attribute names; check them in Prometheus before hard-coding queries.
Utilization, saturation and errors for every resource: where requests wait when RED says "slow".
Brendan Gregg's method: for every resource, check utilization, saturation and errors. Resources include CPUs, memory, disks, NICs — and software resources such as thread pools and connection pools.
Workflow: when RED shows a problem, walk USE for every resource on the request path, starting with the one closest to the slow span (ch 9).
One row per resource. Queries use node_exporter metrics installed by kps (ch 5).
| Resource | Utilization | Saturation | Errors |
|---|---|---|---|
| CPU | 1 - avg by (instance) (rate(node_cpu_seconds_total{mode="idle"}[5m])) | rate(node_pressure_cpu_waiting_seconds_total[5m]); or node_load1 ÷ CPU count | — (rare; check dmesg/MCE) |
| Memory | 1 - node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes | rate(node_pressure_memory_waiting_seconds_total[5m]); rate(node_vmstat_pgmajfault[5m]) | OOM kills: increase(node_vmstat_oom_kill[1h]) |
| Disk | rate(node_disk_io_time_seconds_total[5m]) | rate(node_disk_io_time_weighted_seconds_total[5m]) (≈ avg queue length); rate(node_pressure_io_waiting_seconds_total[5m]) | filesystem node_filesystem_device_error |
| Network | bytes/s ÷ link speed: rate(node_network_receive_bytes_total[5m]) vs node_network_speed_bytes | rate(node_network_receive_drop_total[5m]), ..._transmit_drop_total | rate(node_network_receive_errs_total[5m]), ..._transmit_errs_total |
| Filesystem | 1 - node_filesystem_avail_bytes / node_filesystem_size_bytes | predict_linear(node_filesystem_avail_bytes[6h], 24*3600) < 0 | read-only remounts: node_filesystem_readonly |
PSI metrics need a kernel with PSI enabled (most modern distros) and the node_exporter pressure collector (on by default on Linux).
Kubernetes adds two limits a node view cannot show: CPU quota (throttling) and memory limit (OOM kill).
# CPU utilization vs request, per pod (cAdvisor + kube-state-metrics)
sum by (pod) (rate(container_cpu_usage_seconds_total{namespace="shop", container!=""}[$__rate_interval]))
/
sum by (pod) (kube_pod_container_resource_requests{namespace="shop", resource="cpu"})
# CPU saturation: share of CFS periods that were throttled
sum by (pod) (rate(container_cpu_cfs_throttled_periods_total{namespace="shop", container!=""}[$__rate_interval]))
/
sum by (pod) (rate(container_cpu_cfs_periods_total{namespace="shop", container!=""}[$__rate_interval]))
# Memory utilization vs limit (working set is what the OOM killer compares)
sum by (pod) (container_memory_working_set_bytes{namespace="shop", container!=""})
/
sum by (pod) (kube_pod_container_resource_limits{namespace="shop", resource="memory"})
# Errors: OOM kills and restarts
max by (pod, container) (kube_pod_container_status_last_terminated_reason{namespace="shop", reason="OOMKilled"})
increase(kube_pod_container_status_restarts_total{namespace="shop"}[1h])container!="" drops the pod-level cgroup series that cAdvisor also exports; without it, sums double-count.
The most common saturation point in a service is not the CPU — it is a pool or a queue inside the process.
sql.DBStats).go_* metrics).// Go: export database/sql pool stats (client_golang)
import "github.com/prometheus/client_golang/prometheus/collectors"
reg.MustRegister(collectors.NewDBStatsCollector(db, "orders"))
// go_sql_open_connections, go_sql_in_use_connections,
// go_sql_wait_count_total, go_sql_wait_duration_seconds_total ...
# saturation: average wait per request for a connection
rate(go_sql_wait_duration_seconds_total{db_name="orders"}[5m])
/
rate(go_sql_wait_count_total{db_name="orders"}[5m])Pool utilization near 100 % with growing wait time explains "high latency, low CPU" incidents — a classic USE finding that RED alone cannot explain.
Above pods: is the cluster itself out of room? These panels belong on a platform dashboard, not on each service.
# requested vs allocatable CPU, cluster-wide
sum(kube_pod_container_resource_requests{resource="cpu"})
/
sum(kube_node_status_allocatable{resource="cpu"})
# pods that cannot be scheduled (saturation of the cluster)
sum(kube_pod_status_phase{phase="Pending"})
# nodes not ready (errors)
count(kube_node_status_condition{condition="Ready", status="true"} == 0)Layouts, variables, units and panels that make a dashboard readable at 3 a.m.
A dashboard is a tool for a decision. Name the question it answers before adding the first panel.
| Dashboard | Audience | Question | Shape |
|---|---|---|---|
| Service overview (RED) | On-call, service team | Is my service healthy? Where is it hurting? | Stats on top, breakdowns below, links to drill-downs |
| Resource / pod (USE) | On-call, platform | Which resource is saturated? | One row per resource: U, S, E side by side |
| Platform / cluster | Platform team | Do we have capacity? Is the control plane OK? | Cluster totals, nodes, pending pods |
| Business / product | Product, management | Are orders flowing? | Few big numbers, long time ranges |
| Debug / exploration | Whoever is investigating | Anything | Use Explore, not a permanent dashboard |
Hierarchy beats volume: overview → service → pod, linked together. A 60-panel "everything" dashboard is slow to load and slow to read.
Variables become part of every query. Use query variables from label values, chained from broad to narrow.
# Variables (Dashboard settings → Variables)
datasource type: Data source (prometheus)
namespace label_values(kube_pod_info, namespace)
service label_values(
http_requests_total{namespace="$namespace"}, job)
route label_values(
http_requests_total{job="$service"}, route)
multi-value + "Include All"
# Using them in queries
sum by (route) (rate(http_requests_total{
job="$service", route=~"$route"}[$__rate_interval]))
# Built-ins
$__rate_interval # max(4×scrape, step + scrape)
$__interval # step from time range and width
$__range # whole selected range=~ in the matcher, not =.$__rate_interval for every rate()/increase() in panels: it never drops below 4 scrape intervals, so no gaps.$__rate_interval is correct.Each panel type answers a different kind of question.
| Panel | Use it for | Settings that matter |
|---|---|---|
| Time series | Trends and comparisons over time | Unit, min 0, legend as table with last/max, series overrides |
| Stat | One number with a threshold colour | Reduce (last, mean), thresholds, sparkline |
| Gauge / bar gauge | Value against a known max (utilization) | Min/max set explicitly |
| Heatmap | Latency distributions | Data format: heatmap buckets; le as bucket |
| Table | Top-N lists, current state per pod | Instant query, transformations, cell colouring |
| State timeline | Up/down or phases over time | Value mappings |
| Logs / Traces / Node graph | Loki lines, a Tempo trace, a service graph | Data source specific |
Always set the unit (reqps, percentunit, s, bytes). A latency of 0.25 without a unit is ambiguous; 250 ms is not.
Context turns a line into an answer: what is bad, what changed, and where to go next.
changes(kube_deployment_status_observed_generation{namespace="shop"}[2m]) > 0.Grafana 13 (Apr 2026) turned dynamic dashboards on by default: a new layout engine, editing experience and dashboard structure.
$service has no database.Every one of these has cost someone time in an incident.
rate().sum before rate, or avg of percentiles across pods.x_ratio_2?").job="orders" instead of variables.$__rate_interval, units and thresholds as defaults in the template.Provisioning, the JSON model, Git Sync and generators: dashboards reviewed and versioned like any other code.
They differ in who edits (UI or code) and in what is the source of truth.
| Approach | Source of truth | Edit in | Good for |
|---|---|---|---|
| File provisioning | JSON files on disk / ConfigMaps | Code (UI saves blocked by default) | Kubernetes installs, kps sidecar |
| Generators (Grafonnet, Foundation SDK) | Code that renders JSON | Code | Many similar dashboards from one template |
Terraform grafana provider | Terraform state + JSON | Code | Grafana Cloud / managed Grafana, folders and permissions too |
| Git Sync (GA in Grafana 13) | Git repository | UI → commit or pull request | Teams that want UI editing with Git review |
kube-prometheus-stack runs a sidecar next to Grafana that loads every ConfigMap labelled grafana_dashboard: "1".
# dashboards/shop-red.json → ConfigMap with the label
kubectl -n monitoring create configmap shop-red \
--from-file=shop-red.json=dashboards/shop-red.json \
--dry-run=client -o yaml \
| kubectl label --local -f - grafana_dashboard=1 -o yaml \
| kubectl apply -f -
# relevant kps values (defaults shown)
grafana:
sidecar:
dashboards:
enabled: true
label: grafana_dashboard
labelValue: "1"
datasources:
enabled: true
label: grafana_datasource
labelValue: "1"uid fixed in the JSON so links and bookmarks survive updates.grafana_datasource: "1", or via grafana.additionalDataSources.A classic (v1) dashboard is one JSON document: metadata, variables, and a list of panels with grid positions and queries.
{
"uid": "shop-red",
"title": "Shop / Service RED",
"tags": ["shop", "red"],
"time": {"from": "now-1h", "to": "now"},
"templating": {"list": [{
"name": "service", "type": "query",
"datasource": {"type": "prometheus", "uid": "prometheus"},
"query": "label_values(http_requests_total, job)",
"refresh": 2
}]},
"panels": [{
"type": "timeseries", "title": "Requests/s by route",
"gridPos": {"x": 0, "y": 0, "w": 12, "h": 8},
"datasource": {"type": "prometheus", "uid": "prometheus"},
"fieldConfig": {"defaults": {"unit": "reqps"}},
"targets": [{"refId": "A", "legendFormat": "{{route}}", "expr":
"sum by (route) (rate(http_requests_total{job=\"$service\"}[$__rate_interval]))"}]
}]
}uid is the stable identity; id is per-instance and should be omitted in files.datasource.uid: kps provisions its Prometheus data source with uid prometheus.refresh: 2 = refresh variable values on time range change.schemaVersion) on load; export from your Grafana version to see the current shape.elements (panels) from layout and is served as a Kubernetes-style resource; v1 JSON is still accepted for provisioning — verify for your version.When ten services need the same RED dashboard, write the template once.
grafana_dashboard with config_json = file(...) to push results.# Lab 10 style: one template, many services
def red_dashboard(service):
return {
"uid": f"red-{service}",
"title": f"Shop / {service} RED",
"panels": [
ts("Requests/s", rate_q(service), "reqps", 0),
ts("Error ratio", err_q(service), "percentunit", 8),
ts("p95 latency", p95_q(service), "s", 16),
],
}
for svc in ["orders", "payments"]:
write(f"dashboards/red-{svc}.json", red_dashboard(svc))Generate, review the diff in a pull request, then apply (ConfigMap, API, Terraform or Git Sync). Generated JSON is an artifact: do not hand-edit it.
Git Sync (GA in Grafana 13) connects folders to a Git repository. Saving a dashboard becomes a commit or a pull request.
Treat a dashboard like code: it has users, and a bad one costs time during incidents.
rate() uses $__rate_interval (or a window ≥ 4 × scrape).sum by (le) before histogram_quantile.uid, clear title, tags, description with runbook link.by (pod) on large namespaces in overview panels.Each one makes a healthy system look broken or a broken one look healthy.
avg(histogram_quantile(...)) across pods instead of quantile of summed buckets.rate(x[1m]) with a 30 s scrape interval: gaps and spikes.container!="" in cAdvisor sums: doubled usage.container_memory_usage_bytes (includes cache) instead of working set.uid in different ConfigMaps.FAIL_RATE, EXTRA_LATENCY_MS) and confirm every panel reacts.Answer with the exact query, panel setting or variable.
$service.container_cpu_cfs_throttled_periods_total measure?$__rate_interval and why use it?uid stay fixed?Practice: Lab 10 in Hands-on labs — generate RED and USE dashboards for the shop app as code and load them through the kps sidecar.
The AWS-native metrics store: how a CloudWatch metric is identified, published and queried, how alarms evaluate, and the curated views built on top.
Namespaces, dimensions, resolution, statistics, EMF, metric math, Metrics Insights and the 2026 OpenTelemetry + PromQL path.
If you know Prometheus (ch 2–4), map each concept before learning the AWS details. The differences are where people get surprised.
| Concept | Prometheus | CloudWatch |
|---|---|---|
| Series identity | Metric name + label set | Namespace + metric name + exact dimension set |
| How data arrives | Pull: server scrapes /metrics | Push: AWS services, PutMetricData, EMF log lines, CloudWatch agent, OTLP (2026) |
| Query language | PromQL | Metric math, SEARCH(), Metrics Insights (SQL); PromQL for OTel metrics (2026) |
| Pre-computed series | Recording rules | Metric math in alarms and dashboards (computed at read time); metric filters on logs |
| Alert evaluation | Alerting rule with for: | Alarm: period × evaluation periods, M out of N datapoints |
| Notification routing | Alertmanager (group, inhibit, silence) | Alarm actions → SNS / Lambda / EventBridge; composite alarms for suppression |
| Retention | Configured per server (often 15 d) + remote storage | 15 months, rolled up to coarser periods as data ages |
| Cost driver | Your servers and storage | Per custom metric, per API call, per alarm metric, per GB of logs |
Rule of thumb: CloudWatch is the default for AWS resource metrics (you get them for free and cannot scrape them); Prometheus is the default for Kubernetes and application metrics. Most teams run both (ch 12, part 4).
A metric is a time-ordered set of datapoints. Change any dimension name or value and you are writing to a different metric.
AWS/ are reserved for AWS services.{service} does not aggregate {service, route} for you.GetMetricData stat; use SEARCH() or Metrics Insights.Milliseconds, Count, Bytes, …) are metadata; CloudWatch does not convert between them.CloudWatch keeps metrics for 15 months, but not at the resolution you wrote them.
| Datapoint period | Available for |
|---|---|
| < 60 s (high resolution) | 3 hours |
| 60 s (1 minute) | 15 days |
| 300 s (5 minutes) | 63 days |
| 3600 s (1 hour) | 455 days (15 months) |
StorageResolution=1, down to 1 second; alarms can then use 10 s or 30 s periods (or multiples of 60 s).You choose the statistic when you read, not when you write — as long as CloudWatch has the raw values.
SampleCount, Sum, Average, Minimum, Maximum.pNN percentiles: p50, p90, p99, p99.9.TM(10%:90%), WM, IQM (inter-quartile mean).PR(:300) (share of values ≤ 300), TC, TS.p90, not P90).PutMetricData can send raw Values/Counts arrays or a pre-aggregated statistic set (SampleCount, Sum, Minimum, Maximum).pNN.p50 + p90/p99, never only Average (one slow call hides in the mean).Sum per period; rate = Sum / PERIOD(m1).Maximum or Average depending on burstiness.SampleCount of a latency metric = request count for free.Simple and synchronous. Fine for scripts and low volume; for applications prefer EMF or an agent so you do not block request paths on API calls.
# one datapoint, two dimensions (shorthand)
aws cloudwatch put-metric-data --namespace ObsLab \
--metric-name Latency --unit Milliseconds --value 87 \
--dimensions service=orders,route="POST /orders"
# several values in one call (raw values keep percentiles)
cat > md.json <<'EOF'
[{"MetricName": "Latency", "Unit": "Milliseconds",
"Dimensions": [{"Name": "service", "Value": "orders"},
{"Name": "route", "Value": "POST /orders"}],
"Values": [42, 87, 120, 950], "Counts": [10, 5, 2, 1],
"StorageResolution": 60}]
EOF
aws cloudwatch put-metric-data --namespace ObsLab \
--metric-data file://md.json
# read it back: what does CloudWatch know?
aws cloudwatch list-metrics --namespace ObsLab \
--metric-name LatencyValues + Counts = a compact histogram: value 42 seen 10 times.list-metrics a few minutes after the first write.cloudwatch:PutMetricData; restrict with condition key cloudwatch:namespace.order_id, user_id) in dimensions: every value is a new billed metric.Write structured JSON to a log; CloudWatch Logs extracts the metrics asynchronously. High-cardinality context stays in the log, not in dimensions.
{
"_aws": {
"Timestamp": 1794200000000,
"CloudWatchMetrics": [{
"Namespace": "ObsLab",
"Dimensions": [["service", "route"]],
"Metrics": [
{"Name": "Latency",
"Unit": "Milliseconds"},
{"Name": "Errors",
"Unit": "Count"},
{"Name": "Requests",
"Unit": "Count"}]
}]
},
"service": "orders",
"route": "POST /orders",
"Latency": 87.4, "Errors": 0,
"Requests": 1,
"level": "INFO",
"msg": "order created",
"order_id": "8c1f…",
"trace_id": "4bf92f35…"
}_aws.Timestamp in epoch milliseconds.Dimensions and Metrics must exist as a root-level key with a value.Dimensions is a list of dimension sets: [["service","route"],["service"]] publishes both aggregations.PutLogEvents with header x-amzn-logs-format: json/emf.order_id, trace_id) stay searchable in Logs Insights (ch 12).aws-embedded-metrics (Node, Python, Java, .NET), Powertools for AWS Lambda Metrics.list-metrics (Lab 11).Metric math combines up to hundreds of series at read time: the CloudWatch answer to simple PromQL arithmetic.
cat > q.json <<'EOF'
[{"Id": "req", "ReturnData": false,
"MetricStat": {"Metric": {"Namespace": "ObsLab",
"MetricName": "Requests", "Dimensions": [
{"Name": "service", "Value": "orders"},
{"Name": "route", "Value": "POST /orders"}]},
"Period": 60, "Stat": "Sum"}},
{"Id": "err", "ReturnData": false,
"MetricStat": {"Metric": {"Namespace": "ObsLab",
"MetricName": "Errors", "Dimensions": [
{"Name": "service", "Value": "orders"},
{"Name": "route", "Value": "POST /orders"}]},
"Period": 60, "Stat": "Sum"}},
{"Id": "error_pct", "Label": "error %",
"Expression": "100 * FILL(err, 0) / req"}]
EOF
aws cloudwatch get-metric-data --metric-data-queries file://q.json \
--start-time $(date -u -v-1H +%FT%TZ) --end-time $(date -u +%FT%TZ)
# Linux: date -u -d '-1 hour' +%FT%TZm1 / m2 * 100.FILL(m, 0) for gaps (no errors → no datapoints).IF(cond, a, b), RATE(m), DIFF(m), PERIOD(m).METRICS() = all series in the request; SUM(METRICS()).SEARCH('{ObsLab,service,route} MetricName="Errors"', 'Sum', 60) finds every matching series, like a regex selector.SEARCH(); use a Metrics Insights query instead (part 2).Metrics Insights queries the most recent metrics with a SQL dialect. It is how you ask "which route has the most errors?" across many series.
-- errors per route for the orders service
SELECT SUM(Errors)
FROM SCHEMA(ObsLab, route, service)
WHERE service = 'orders'
GROUP BY route
ORDER BY SUM() DESC
LIMIT 5
-- slowest Lambda functions by average duration
SELECT AVG(Duration)
FROM SCHEMA("AWS/Lambda", FunctionName)
GROUP BY FunctionName
ORDER BY AVG() DESC
LIMIT 10SCHEMA(ns, dim1, dim2) matches metrics with exactly those dimensions; plain FROM ObsLab matches any dimension set.GROUP BY + ORDER BY + LIMIT = PromQL topk(5, sum by (route)(…)).AVG, SUM, MIN, MAX, COUNT; no percentiles.GetMetricData (Expression = the SQL) and in alarms.Announced in public preview in April 2026 and then generally available (AWS announcement mid-2026). Check region availability and pricing on the day you use it.
ap-northeast-1 support before planning on it.The deck treats this as new: the labs use classic EMF metrics, which work in every Region. Read the "CloudWatch OpenTelemetry Metrics" and "PromQL querying" pages of the CloudWatch user guide for the current endpoint, limits and settings.
Prices change and differ by Region: learn the drivers, then read the pricing page. The free tier is small and does not cover high-resolution data.
GetMetricData (charged per metric requested), PutMetricData, ListMetrics.GetMetricData every 10 s are a classic surprise: check request counts in Cost Explorer.How an alarm evaluates, what missing data does, and the four alarm types: static, metric math, anomaly detection and composite.
An alarm looks at the most recent N periods and goes to ALARM when M of them breach. This is the CloudWatch version of for: (ch 4).
--period 60 --evaluation-periods 5 --datapoints-to-alarm 3 tolerates two good minutes inside a bad stretch, so the alarm does not flap. With M = N it behaves like a strict for: 5m.
EMF error metrics with value 0 are published, but metrics that only appear on failure (or a function that stops being invoked) leave gaps.
| Setting | A missing period counts as | Use for |
|---|---|---|
missing (default) | Ignored when enough other datapoints exist; all missing → INSUFFICIENT_DATA | Metrics that should always report (CPU) |
notBreaching | Good (within threshold) | Error counts that are only published when errors happen |
breaching | Bad (over threshold) | Heartbeats: "no data means dead" (e.g. a job that must report every 5 min) |
ignore | Not evaluated: the alarm keeps its current state | Sparse metrics where you want no state flip from gaps |
Percentile alarms on low traffic: --evaluate-low-sample-count-percentile ignore keeps the state when there are too few samples for a meaningful percentile; evaluate always evaluates.
The everyday alarm: one metric, one statistic, one threshold. The SNS topic is the bridge to email, chat or a pager.
TOPIC=$(aws sns create-topic --name obs-lab-alerts \
--query TopicArn --output text)
aws sns subscribe --topic-arn "$TOPIC" \
--protocol email --notification-endpoint you@example.com
aws cloudwatch put-metric-alarm \
--alarm-name obs-lab-orders-errors \
--alarm-description "orders: >=3 errors/min in 3 of 5 min. Runbook: <url>" \
--namespace ObsLab --metric-name Errors \
--dimensions Name=service,Value=orders "Name=route,Value=POST /orders" \
--statistic Sum --period 60 \
--evaluation-periods 5 --datapoints-to-alarm 3 \
--threshold 3 --comparison-operator GreaterThanOrEqualToThreshold \
--treat-missing-data notBreaching \
--alarm-actions "$TOPIC" --ok-actions "$TOPIC"
aws cloudwatch describe-alarms --alarm-names obs-lab-orders-errors \
--query 'MetricAlarms[0].[StateValue,StateReason]'obs-lab-orders-errors.list-metrics, not typed from memory: a typo means INSUFFICIENT_DATA forever (or OK forever with notBreaching).--ok-actions too, so on-call sees recovery.aws cloudwatch set-alarm-state --state-value ALARM --state-reason test fires the actions once.--extended-statistic p90 instead of --statistic.Counts scale with traffic; ratios do not. Error % is the CloudWatch way to write an availability-style alarm.
aws cloudwatch put-metric-alarm \
--alarm-name obs-lab-orders-error-pct \
--evaluation-periods 5 --datapoints-to-alarm 3 \
--threshold 5 --comparison-operator GreaterThanThreshold \
--treat-missing-data notBreaching \
--alarm-actions "$TOPIC" \
--metrics '[
{"Id":"req","ReturnData":false,"MetricStat":{"Metric":{
"Namespace":"ObsLab","MetricName":"Requests","Dimensions":[
{"Name":"service","Value":"orders"},
{"Name":"route","Value":"POST /orders"}]},
"Period":60,"Stat":"Sum"}},
{"Id":"err","ReturnData":false,"MetricStat":{"Metric":{
"Namespace":"ObsLab","MetricName":"Errors","Dimensions":[
{"Name":"service","Value":"orders"},
{"Name":"route","Value":"POST /orders"}]},
"Period":60,"Stat":"Sum"}},
{"Id":"pct","Label":"error %","ReturnData":true,
"Expression":"IF(req >= 10, 100 * FILL(err,0) / req, 0)"}]'ReturnData: true: the series the alarm evaluates.IF(req >= 10, …, 0): no paging on 1 error out of 2 requests at night.Expression (with Period) — one alarm covers every route.CloudWatch trains a model on the metric's history (trend, hourly and weekly seasonality) and alarms when values leave the band.
aws cloudwatch put-metric-alarm \
--alarm-name obs-lab-orders-latency-anomaly \
--evaluation-periods 3 --datapoints-to-alarm 3 \
--comparison-operator GreaterThanUpperThreshold \
--threshold-metric-id band \
--treat-missing-data notBreaching \
--metrics '[
{"Id":"m1","ReturnData":true,"MetricStat":{"Metric":{
"Namespace":"ObsLab","MetricName":"Latency","Dimensions":[
{"Name":"service","Value":"orders"},
{"Name":"route","Value":"POST /orders"}]},
"Period":60,"Stat":"p90"}},
{"Id":"band","Label":"expected p90",
"Expression":"ANOMALY_DETECTION_BAND(m1, 2)"}]'ANOMALY_DETECTION_BAND(m1, 2): width in standard deviations; larger = fewer alarms.GreaterThanUpperThreshold, LessThanLowerThreshold or LessThanLowerOrGreaterThanUpperThreshold.A composite alarm has a rule over other alarms' states. Child alarms can stay silent (no actions) while the composite pages once.
aws cloudwatch put-composite-alarm \
--alarm-name obs-lab-orders-page \
--alarm-rule 'ALARM("obs-lab-orders-errors") AND
ALARM("obs-lab-orders-latency-anomaly")' \
--alarm-actions "$TOPIC"
# rule language
# ALARM("x"), OK("x"), INSUFFICIENT_DATA("x")
# AND, OR, NOT, parentheses, TRUE, FALSE
# suppression (all three together):
# --actions-suppressor <alarm name or ARN>
# --actions-suppressor-wait-period 120
# --actions-suppressor-extension-period 300Typical design: per-symptom alarms without actions (they still show on dashboards), one composite per service that notifies, and a suppressor alarm for "dependency down" or maintenance so downstream services do not all page at once.
Actions run when the state changes. Every state change is also an EventBridge event, which is the most flexible hook.
# EventBridge rule pattern: every alarm
# named obs-lab-* that enters ALARM
{
"source": ["aws.cloudwatch"],
"detail-type":
["CloudWatch Alarm State Change"],
"detail": {
"state": {"value": ["ALARM"]},
"alarmName":
[{"prefix": "obs-lab-"}]
}
}
# targets: Lambda, Step Functions,
# SQS, an API destination (webhook)page, ticket), like Alertmanager receivers (ch 4).CloudWatch dashboards, Container Insights, Application Signals, Synthetics and the CloudWatch agent.
A dashboard is a JSON document of widgets. Keep it in Git and deploy it with put-dashboard or IaC, the same discipline as Grafana (ch 10).
{"widgets": [
{"type": "metric", "x": 0, "y": 0, "width": 12, "height": 6,
"properties": {"title": "orders latency",
"region": "ap-northeast-1",
"view": "timeSeries", "period": 60,
"metrics": [
["ObsLab", "Latency", "service", "orders", "route", "POST /orders",
{"stat": "p50"}],
["...", {"stat": "p90"}]]}},
{"type": "log", "x": 12, "y": 0, "width": 12, "height": 6,
"properties": {"title": "recent errors",
"region": "ap-northeast-1",
"view": "table",
"query": "SOURCE '/obs-lab/orders' | filter level = 'ERROR' | limit 20"}},
{"type": "alarm", "x": 0, "y": 6, "width": 24, "height": 3,
"properties": {"title": "alarms", "alarms": [
"arn:aws:cloudwatch:ap-northeast-1:<acct>:alarm:obs-lab-orders-page"
]}}
]}"..." repeats the previous metric's namespace, name and dimensions.service) from a dropdown.aws cloudwatch put-dashboard --dashboard-name obs-lab --dashboard-body file://dash.json.Curated metrics and logs for clusters, nodes, pods and containers, installed as an EKS add-on. CloudWatch's answer to kube-prometheus-stack dashboards (ch 5).
# EKS: install the observability add-on
aws eks create-addon \
--cluster-name <cluster> \
--addon-name \
amazon-cloudwatch-observability
# the node role (or the agent's Pod
# Identity / IRSA role) needs the
# managed policy
# CloudWatchAgentServerPolicy
# check
kubectl -n amazon-cloudwatch get pods
# cloudwatch-agent-* (DaemonSet)
# fluent-bit-* (DaemonSet)AWS's application performance monitoring (APM) layer. Instrumentation is OpenTelemetry-based (ADOT SDKs and the CloudWatch agent).
ApplicationSignals namespace.aws application-signals start-discovery (or the console button).amazon-cloudwatch-observability add-on can auto-instrument Java, Python, .NET and Node.js workloads via annotations.Black-box checks (ch 1) catch what in-process metrics cannot: DNS, TLS, CDN, login flows and the real user's browser.
SuccessPercent and Duration metrics in CloudWatchSynthetics; alarm on them.One agent for EC2, on-premises servers and containers. It publishes custom metrics, ships logs, and can receive OpenTelemetry and X-Ray data.
{
"agent": {"metrics_collection_interval": 60},
"metrics": {
"namespace": "ObsLab/Host",
"append_dimensions": {"InstanceId": "${aws:InstanceId}"},
"metrics_collected": {
"mem": {"measurement": ["mem_used_percent"]},
"disk": {"measurement": ["used_percent"],
"resources": ["/"]},
"procstat": [{"exe": "orders",
"measurement": ["cpu_usage", "memory_rss"]}],
"statsd": {"service_address": ":8125"}
}
},
"logs": {"logs_collected": {"files": {"collect_list": [{
"file_path": "/var/log/orders/*.log",
"log_group_name": "/obs-lab/orders",
"log_stream_name": "{instance_id}",
"retention_in_days": 7}]}}}
}procstat monitors individual processes; statsd and collectd accept app metrics.append_dimensions adds instance context; avoid adding high-cardinality dimensions.amazon-cloudwatch-agent-config-wizard; store config in SSM Parameter Store and roll out with Systems Manager.CloudWatchAgentServerPolicy on the instance role.Each one has caused either a silent outage or a surprise bill.
order_id or user_id as an EMF dimension → thousands of billed custom metrics.{service} and expecting it to sum {service, route} series.GetMetricData every few seconds.INSUFFICIENT_DATA (or OK with notBreaching) forever, nobody notices.treatMissingData: missing on an error count that only appears on failure.list-metrics after every new EMF field; copy dimensions from its output.INSUFFICIENT_DATA for critical alarms, or use describe-alarms --state-value INSUFFICIENT_DATA in a weekly review.set-alarm-state drill for each paging alarm.CW:MetricMonitorUsage, CW:Requests).Answer with the exact setting, field or expression.
Latency and Errors with dimensions service and route.TreatMissingData for a heartbeat metric, and which for an error count?SEARCH(), and what do you use instead?Practice: Lab 11 in Hands-on labs — a Lambda that emits EMF, a metric filter, static, anomaly and composite alarms, and a dashboard in ap-northeast-1.
CloudWatch Logs and Logs Insights, tracing on AWS after the X-Ray SDKs, and the managed open-source path: Amazon Managed Service for Prometheus, Amazon Managed Grafana and cross-account observability.
Log groups and classes, retention, metric and subscription filters, data protection and Live Tail.
Everything is regional. A log group is the unit of retention, encryption, class, access control and billing.
Name groups by owner and purpose (/obs-lab/orders, /aws/lambda/<fn>, /aws/eks/<cluster>/cluster) so IAM policies, retention rules and Logs Insights queries can target prefixes.
The class is chosen when the log group is created and cannot be changed later. Infrequent Access trades features for a lower ingestion price.
| Capability | Standard | Infrequent Access |
|---|---|---|
| Ingestion price | Baseline | About half of Standard per GB |
| Storage, Logs Insights scan | Same | Same |
| Logs Insights QL | Yes | Yes, except pattern, diff, unmask |
| OpenSearch PPL / SQL | Yes | Yes (since Mar 2026) |
| Data protection (masking) | Yes | Yes (since Mar 2026) |
| Metric filters, EMF, field indexes, facets | Yes | No |
| Subscription filters, Live Tail | Yes | No |
| Export to S3 | Yes | Yes |
GetLogEvents / FilterLogEvents | Yes | No (query with Logs Insights) |
aws logs create-log-group --log-group-name /obs-lab/orders-ia --log-group-class INFREQUENT_ACCESS.ESSENTIALS from Dec 2026). IA has gained features over time: check the docs table.New log groups keep data forever unless you set retention. That default is the single largest source of silent CloudWatch Logs cost.
# set retention (allowed values: 1, 3, 5, 7, 14, 30, 60, 90,
# 120, 150, 180, 365, 400, 545, 731, 1096, 1827, 2192,
# 2557, 2922, 3288, 3653 days)
aws logs put-retention-policy \
--log-group-name /obs-lab/orders --retention-in-days 7
# find groups that never expire, biggest first
aws logs describe-log-groups \
--query 'logGroups[?!retentionInDays].[logGroupName,storedBytes]' \
--output text | sort -k2 -n -r | head
# encrypt with a customer managed key (key policy must
# allow the logs.<region>.amazonaws.com principal)
aws logs associate-kms-key --log-group-name /obs-lab/orders \
--kms-key-id arn:aws:kms:ap-northeast-1:<acct>:key/<id>/aws/lambda/…, /aws/eks/…).storedBytes shows where the storage money goes./aws/lambda/<fn> on first invocation without retention — set LoggingConfig.LogGroup to a group you created, or create the default group first.A metric filter turns matching log events into a CloudWatch metric. It only sees events ingested after it was created.
# count ERROR events from JSON logs
aws logs put-metric-filter --log-group-name /obs-lab/orders \
--filter-name orders-error-logs \
--filter-pattern '{ $.level = "ERROR" }' \
--metric-transformations metricName=ErrorLogs,\
metricNamespace=ObsLab,metricValue=1,defaultValue=0
# sum a numeric field instead of counting events
# metricValue=$.duration_ms
# test a pattern against sample lines before creating it
aws logs test-metric-filter \
--filter-pattern '{ $.status >= 500 }' \
--log-event-messages '{"status":502,"level":"ERROR"}' \
'{"status":201,"level":"INFO"}'
# → matches[0].eventMessage shows only the 502 line| Style | Example |
|---|---|
| Terms | ERROR · "payment failed" · ?ERROR ?WARN (either) |
| JSON | { $.level = "ERROR" && $.route = "POST /orders" } |
| JSON numeric | { $.duration_ms > 500 } |
| Space-delimited | [ip, id, user, ts, req, status=5*, size] |
| Regex (subset) | %timeout|deadline% |
JSON and space-delimited patterns can add up to three dimensions from fields, but then defaultValue is not allowed, and high-cardinality fields multiply custom metrics. For metrics from structured app logs, EMF (ch 11) is usually the better tool.
Subscription filters deliver matching events in near real time. Export tasks copy a time range once. Subscription filters do not work on Infrequent Access groups; export tasks do.
put-account-policy --policy-type SUBSCRIPTION_FILTER_POLICY) covers all groups, with an exclusion list.# stream errors to Firehose
# → S3 / SIEM
FH=arn:aws:firehose:\
ap-northeast-1:<acct>:\
deliverystream/obs-lab-logs
aws logs put-subscription-filter \
--log-group-name /obs-lab/orders \
--filter-name to-firehose \
--filter-pattern \
'{ $.level = "ERROR" }' \
--destination-arn $FH \
--role-arn arn:aws:iam::<acct>:\
role/cwl-to-firehose
# one-off export to S3
# (times in epoch ms)
aws logs create-export-task \
--log-group-name /obs-lab/orders \
--from 1794000000000 \
--to 1794086400000 \
--destination obs-lab-archiveA data protection policy finds sensitive values with managed data identifiers and masks them in every read path. Seeing the raw value needs logs:Unmask.
cat > dp.json <<'EOF'
{"Name": "obs-lab-pii", "Version": "2021-06-01",
"Statement": [
{"Sid": "audit",
"DataIdentifier": [
"arn:aws:dataprotection::aws:data-identifier/EmailAddress"],
"Operation": {"Audit": {"FindingsDestination": {}}}},
{"Sid": "redact",
"DataIdentifier": [
"arn:aws:dataprotection::aws:data-identifier/EmailAddress"],
"Operation": {"Deidentify": {"MaskConfig": {}}}}]}
EOF
aws logs put-data-protection-policy \
--log-group-identifier /obs-lab/orders \
--policy-document file://dp.json
# the event is stored; reads show ****************
# a principal with logs:Unmask can run in Logs Insights:
# fields @timestamp, unmask(@message)tail -f for CloudWatch LogsInteractive streaming of new events from up to 10 log groups, with a filter pattern. Billed per minute of session; close it when done.
# CLI v2 streams to your terminal (Ctrl-C to stop)
aws logs start-live-tail \
--log-group-identifiers \
arn:aws:logs:ap-northeast-1:<acct>:log-group:/obs-lab/orders \
--log-event-filter-pattern '{ $.level = "ERROR" }'
# older way, still useful for scripting:
aws logs tail /obs-lab/orders --follow \
--filter-pattern '{ $.status >= 500 }' --format shortorder_id or trace_id.--log-group-identifiers takes ARNs (without the trailing :*).Logs Insights QL, OpenSearch PPL and SQL, facets and field indexes, and how scanning drives cost.
Commands are joined with | and run left to right. JSON fields are discovered automatically and addressed with dots (http.status).
| Command | Purpose |
|---|---|
fields | Choose or compute fields: fields @timestamp, route, duration_ms |
filter | Keep matching events: filter level = "ERROR" and duration_ms > 500 |
stats | Aggregate: stats count(*), pct(duration_ms, 95) by route, bin(5m) |
sort / limit | Order and cap the result (limit up to 10,000) |
parse | Extract fields from text with a glob or regex |
dedup | Keep the first event per value: dedup order_id |
pattern | Cluster messages into recurring patterns |
diff | Compare patterns with the previous equal-length period |
anomaly | Flag unusual patterns (pairs with pattern) |
@timestamp, @message, @logStream, @log, @ingestionTime.@requestId, @duration, @billedDuration, @maxMemoryUsed on REPORT lines.count, count_distinct, sum, avg, min, max, pct, earliest, latest.strcontains, concat, replace, toupper, regex like / =~.ispresent(field), coalesce(a, b), datefloor(@timestamp, 1h).#.All against /obs-lab/orders with the slog schema from ch 6 (level, msg, route, status, duration_ms, order_id, trace_id).
# 1. error rate per route over time
fields route, status
| stats sum(status >= 500) as errors, count(*) as total,
errors * 100 / total as error_pct by route, bin(5m)
# 2. latency distribution per route
filter ispresent(duration_ms)
| stats pct(duration_ms, 50) as p50, pct(duration_ms, 95) as p95,
pct(duration_ms, 99) as p99, count(*) as n by route
# 3. the 10 slowest requests, with trace IDs to open in traces
fields @timestamp, route, duration_ms, trace_id
| sort duration_ms desc | limit 10# 4. everything one request logged
filter order_id = "8c1f4c2e-…" or trace_id = "4bf92f35…"
| sort @timestamp asc
# 5. text logs: parse with a glob pattern
parse @message "* payment failed after *ms: *" as lvl, ms, reason
| stats count(*) by reason
# 6. top error messages, grouped into patterns
filter level = "ERROR"
| pattern msg
# 7. Lambda cold starts and memory headroom
filter @type = "REPORT"
| stats count(*) as invocations, max(@maxMemoryUsed) as maxMem,
sum(ispresent(@initDuration)) as coldStarts by bin(1h)Query 3 is the bridge to tracing: copy a trace_id into Transaction Search or Tempo (ch 9). That link only exists if your logger writes trace_id (ch 6).
Pick the language in the console or with --query-language CWLI | PPL | SQL. PPL feels like a pipeline; SQL is the familiar database syntax.
# OpenSearch PPL: errors by route
fields route, status, duration_ms
| where status >= 500
| stats count() as errors,
avg(duration_ms) as avg_ms by route
| sort - errors
| head 5
-- OpenSearch SQL: log group goes in FROM
SELECT route, COUNT(*) AS n,
AVG(duration_ms) AS avg_ms
FROM `/obs-lab/orders`
WHERE status >= 500
GROUP BY route ORDER BY n DESC LIMIT 5
-- several groups:
-- FROM `logGroups(logGroupIdentifier:
-- ['/obs-lab/orders','/obs-lab/payments'])`pattern, diff, anomaly; the only one with large-result pagination.eval, top, rare, fillnull, JSON functions, join and subqueries.trace_id), familiar syntax.--log-group-names.Logs Insights is asynchronous. Scripts start a query, poll its status, then read the results (also how runbooks and bots use it).
END=$(date +%s); START=$((END - 3600))
QID=$(aws logs start-query \
--log-group-names /obs-lab/orders \
--start-time $START --end-time $END \
--query-string 'filter level = "ERROR"
| stats count(*) as errors by route' \
--query queryId --output text)
# status: Scheduled → Running → Complete
aws logs get-query-results --query-id "$QID" \
--query '[status, results]'
# ["Complete", [[{"field":"route","value":"POST /orders"},
# {"field":"errors","value":"14"}]]]
# PPL: same call, different language
aws logs start-query --query-language PPL \
--log-group-names /obs-lab/orders \
--start-time $START --end-time $END \
--query-string 'fields route | where status >= 500
| stats count() by route'get-query-results again by ID).put-query-definition) so the team shares them.SOURCE command in the query can select groups by prefix or class instead of listing names.statistics.bytesScanned in the result is what you pay for.Logs Insights charges per GB scanned. The fastest, cheapest query is the one that reads the fewest bytes.
put-index-policy) on fields you filter by often (order_id, trace_id, status): queries skip events that cannot match.filterIndex to force an index-only lookup.# index the fields we search by
aws logs put-index-policy \
--log-group-identifier \
/obs-lab/orders \
--policy-document \
'{"Fields": ["order_id",
"trace_id", "route"]}'
# a query that uses the index
filterIndex order_id = "8c1f…"
| fields @timestamp, msgX-Ray concepts, the move to OpenTelemetry, trace ID formats, Transaction Search and Application Signals.
X-Ray is still the trace backend on AWS. What changed is how you instrument and ship data to it.
| X-Ray term | Meaning | OpenTelemetry equivalent (ch 8–9) |
|---|---|---|
| Trace | All segments sharing one trace ID | Trace |
| Segment | Work done by one service for a request | Server/consumer span (root span of a service) |
| Subsegment | A call or block inside a segment (DB, HTTP, AWS SDK) | Child span (client/internal) |
| Annotations | Indexed key-value pairs, usable in filter expressions | Span attributes chosen for indexing |
| Metadata | Non-indexed key-values for context | Other span attributes |
| Trace map / service map | Graph of services and edges with health | Service graph (Tempo, ch 9) |
| Sampling rules | Reservoir per second + fixed rate (default 1 req/s + 5%) | Samplers; xray remote sampler |
| Groups | Saved filter expression with its own metrics | TraceQL metrics query (ch 9) |
Filter expression examples: service("orders") AND fault = true, responsetime > 0.5, annotation.order_type = "express".
AWS made OpenTelemetry the primary instrumentation for X-Ray. Existing SDK-instrumented apps keep working; new work should start on OTel.
| Old | New |
|---|---|
| X-Ray SDK in the app | OpenTelemetry SDK or ADOT distribution (auto-instrumentation where available) |
| X-Ray daemon (UDP 2000) | CloudWatch agent or an OpenTelemetry Collector exporting OTLP to X-Ray |
X-Amzn-Trace-Id propagation | W3C traceparent (plus the xray propagator while mixed) |
| Annotations | Span attributes (index selected ones) |
| Lambda active tracing | Still available; add OTel (Application Signals) for in-function spans |
Mixed fleets break traces at the boundary where one side only understands the other format. Know both.
# X-Ray trace ID: version - epoch seconds (hex) - 96 random bits
1-5759e988-bd862e3fe1be46a994272793
# W3C trace ID (32 hex) — the same trace written for OTel:
5759e988bd862e3fe1be46a994272793
# HTTP headers
X-Amzn-Trace-Id: Root=1-5759e988-bd862e3fe1be46a994272793;
Parent=53995c3f42cd8ad8;Sampled=1
traceparent: 00-5759e988bd862e3fe1be46a994272793-53995c3f42cd8ad8-01X-Amzn-Trace-Id header is still what services add and forward.tracecontext,baggage,xray while some services still use X-Ray headers.trace_id) so Logs Insights, Loki and Tempo all match it (ch 6).1-.With Transaction Search, X-Ray stores spans in CloudWatch Logs (group aws/spans) and indexes a sample for trace summaries. Required for the OTLP traces endpoint.
# 1) allow X-Ray to write spans to CloudWatch Logs
A=<acct>; L=arn:aws:logs:ap-northeast-1:$A:log-group
cat > ts.json <<EOF
{"Version":"2012-10-17","Statement":[{
"Sid":"TransactionSearchXRayAccess","Effect":"Allow",
"Principal":{"Service":"xray.amazonaws.com"},
"Action":"logs:PutLogEvents",
"Resource":["$L:aws/spans:*",
"$L:/aws/application-signals/data:*"],
"Condition":{
"ArnLike":{"aws:SourceArn":"arn:aws:xray:ap-northeast-1:$A:*"},
"StringEquals":{"aws:SourceAccount":"$A"}}}]}
EOF
aws logs put-resource-policy --policy-name TransactionSearchXRay \
--policy-document file://ts.json
# 2) send segments to CloudWatch Logs instead of X-Ray storage
aws xray update-trace-segment-destination --destination CloudWatchLogs
aws xray get-trace-segment-destination
# {"Destination": "CloudWatchLogs", "Status": "ACTIVE"} (~10 min)
# 3) share of spans indexed as trace summaries (1% is the free tier)
aws xray update-indexing-rule --name Default \
--rule '{"Probabilistic": {"DesiredSamplingPercentage": 1}}'aws/spans, the Transaction Search console), not only sampled trace summaries.CloudWatch exposes OTLP/HTTP endpoints per Region. Requests are signed with SigV4, so a Collector needs the sigv4auth extension (or use the CloudWatch agent / ADOT Collector).
extensions:
sigv4auth/xray: {region: ap-northeast-1, service: xray}
sigv4auth/logs: {region: ap-northeast-1, service: logs}
exporters:
otlphttp/xray:
traces_endpoint:
https://xray.ap-northeast-1.amazonaws.com/v1/traces
auth: {authenticator: sigv4auth/xray}
otlphttp/logs:
logs_endpoint:
https://logs.ap-northeast-1.amazonaws.com/v1/logs
headers: {x-aws-log-group: /obs-lab/orders,
x-aws-log-stream: otel}
auth: {authenticator: sigv4auth/logs}
service:
extensions: [sigv4auth/xray, sigv4auth/logs]
pipelines: # receivers [otlp], processors [batch]
traces: {exporters: [otlphttp/xray], …}
logs: {exporters: [otlphttp/logs], …}Application Signals is the managed APM layer over the same OTel data. It shines where you can use auto-instrumentation.
aws application-signals start-discovery (once per account).POST /orders p99 < 300 ms, 99.5%).Compared with the open-source path in this deck: Application Signals ≈ span metrics + service graph + SLO tooling (ch 9, SRE deck), managed for you, priced per signal volume.
Amazon Managed Service for Prometheus, Amazon Managed Grafana, cross-account observability, and how to choose.
A Prometheus-compatible, multi-AZ store and query engine (based on Cortex). You bring the collector; AMP stores, evaluates rules and routes alerts.
remote_write with SigV4, the OTel Collector, Grafana Alloy, or an AWS managed scraper for EKS (agentless)./api/v1/query — Amazon Managed Grafana or any Grafana with SigV4.# create a workspace
WS=$(aws amp create-workspace \
--alias obs-lab \
--query workspaceId --output text)
URL=https://aps-workspaces.\
ap-northeast-1.amazonaws.com\
/workspaces/$WS
# Prometheus remote_write
remote_write:
- url: <URL>/api/v1/remote_write
sigv4:
region: ap-northeast-1
# rules (same YAML as ch 4)
aws amp create-rule-groups-namespace \
--workspace-id $WS --name shop \
--data fileb://rules/shop.yamlaws amp create-scraper), scrape config is base64 YAML, access through EKS access entries.Managed Grafana workspaces with AWS authentication and data sources pre-wired. The dashboards and skills from ch 10 transfer directly.
CloudWatch cross-account observability links source accounts to a sink in a monitoring account. Metrics, logs, traces and Application Signals data become visible there without copying.
# monitoring account
SINK=$(aws oam create-sink --name obs-sink \
--query Arn --output text)
aws oam put-sink-policy --sink-identifier $SINK \
--policy file://sink-policy.json # allow source accounts / OU
# each source account
aws oam create-link --label-template '$AccountName' \
--sink-identifier $SINK \
--resource-types AWS::CloudWatch::Metric \
AWS::Logs::LogGroup AWS::XRay::TraceSet it up with AWS Organizations (a StackSet or the console wizard) so new accounts link automatically. Sharing is per Region: create a sink and links in every Region you operate in.
There is no single right answer; there is a clear default per signal and per team. Decide explicitly and write it down (ch 13).
| Need | CloudWatch-native | Open source (self or AWS managed) |
|---|---|---|
| AWS service metrics (ALB, RDS, Lambda) | Built in; no choice needed | Pull via CloudWatch data source or exporters (adds API cost) |
| Kubernetes and app metrics | Container Insights, EMF, OTel metrics + PromQL (2026) | Prometheus + kps (ch 5), AMP for long term |
| Logs | CloudWatch Logs, Logs Insights, IA class | Loki (ch 7): cheaper at scale, label discipline required |
| Traces | X-Ray + Transaction Search, Application Signals | Tempo (ch 9) with TraceQL |
| Dashboards | CloudWatch dashboards | Grafana or AMG, can read CloudWatch too |
| Alerting | Alarms → SNS / EventBridge | Alertmanager (ch 4), AMP alert manager |
| Multi-cloud or on-prem | Possible with the agent; AWS-centric | Natural fit |
| Ops effort | Lowest | Highest self-managed; medium with AMP/AMG |
Common hybrid: OpenTelemetry everywhere (ch 8), Prometheus/Loki/Tempo for engineering debugging, CloudWatch alarms for AWS resources, and Grafana reading both. OTel keeps the switch cheap if you change your mind.
These show up on bills, in audits and in incidents where the trace stops at a service boundary.
X-Amzn-Trace-Id and traceparent without the xray propagator → broken traces.trace_id, so logs and traces cannot be joined.Answer with the exact command, field or setting.
status >= 500.5759e988bd862e3fe1be46a994272793 to an X-Ray trace ID.Practice: Lab 12 in Hands-on labs — Logs Insights in three languages, Live Tail and masking, an IA log group, and OpenTelemetry traces to Transaction Search (optional: AMP remote write from kind).
Use the stack under pressure: a repeatable debugging loop, alerts people trust, a bill you can explain, and monitoring for the monitoring itself.
From a page to a root cause in minutes: alert → RED dashboard → exemplar → trace → logs → change events.
Each step narrows the search space. Metrics tell you that and where, traces tell you which hop, logs tell you why.
The loop only works if the signals share keys: service.name / job, namespace, pod, and trace_id in logs (ch 6, ch 8, ch 9).
Under pressure people jump to the last outage's cause. A short written loop keeps the investigation honest.
route, code, pod, node, zone, version.Shop app (ch 0). At 14:05 the alert below fires. Nothing was deployed to orders.
# alert as received (Alertmanager → Slack)
[FIRING:1] ShopHighLatencyP95 severity=page
service: orders namespace: shop
summary: p95 latency of orders is 0.93s (threshold 0.5s)
runbook: https://runbooks.example.com/shop/latency
# rule behind it (PrometheusRule, ch 4)
- alert: ShopHighLatencyP95
expr: |
histogram_quantile(0.95,
sum by (le) (rate(
http_request_duration_seconds_bucket{job="orders"}[5m]
))) > 0.5
for: 5m
labels: {severity: page, service: orders}POST /orders).orders is CPU-throttled, (b) a dependency is slow, (c) traffic spike.Rate is flat, errors are flat, duration is up — and only on the route that calls payments.
# Rate: no traffic spike → hypothesis (c) rejected
sum by (route) (rate(http_requests_total{job="orders"}[5m]))
# Errors: unchanged
sum(rate(http_requests_total{job="orders",code=~"5.."}[5m]))
/ sum(rate(http_requests_total{job="orders"}[5m]))
# Duration by route: only POST /orders is slow
histogram_quantile(0.95, sum by (le, route) (
rate(http_request_duration_seconds_bucket{job="orders"}[5m])))
# payments, as seen by its own server metrics
histogram_quantile(0.95, sum by (le) (
rate(http_request_duration_seconds_bucket{job="payments"}[5m])))
# USE for orders: throttling? → hypothesis (a) rejected
sum by (pod) (rate(container_cpu_cfs_throttled_periods_total{
namespace="shop", container="orders"}[5m]))
/ sum by (pod) (rate(container_cpu_cfs_periods_total{
namespace="shop", container="orders"}[5m]))GET /orders/{id} p95 is normal: orders itself is healthy.POST /orders p95 jumped from ~0.12 s to ~0.93 s at 14:01.payments server-side p95 is ~0.85 s: the time is spent inside payments.orders is near 0.payments is slow. Next: one concrete slow request.An exemplar is a real request (with its trace_id) attached to a histogram bucket. Click it and you leave aggregates behind.
trace_id; the data source link opens the trace in Tempo (ch 9).# the trace as a waterfall (simplified)
orders POST /orders 942 ms
├─ orders SELECT orders (db) 6 ms
└─ orders POST /charge 921 ms client
└─ payments POST /charge 918 ms server
├─ payments validate card 2 ms
└─ (no child spans) ~910 ms ← time
inside the handlerWhat the trace adds: client ≈ server duration, so the network is not the problem; no child spans in the gap, so the time is inside the payments handler, not in a database or downstream call. Next: check k8s.pod.name and service.version on the spans — one pod or all?
One trace is an anecdote. TraceQL tells you whether it is every request, one pod, or one version.
# all slow payments server spans in the last 30 min
{ resource.service.name = "payments" && kind = server
&& duration > 500ms }
# which pods? (TraceQL metrics, ch 9)
{ resource.service.name = "payments" && kind = server }
| quantile_over_time(duration, .95) by (resource.k8s.pod.name)
# did errors start too?
{ resource.service.name = "payments" && status = error }
| rate()payments pod is slow: not a bad node or a single pod.payments.payments at 14:01 without a new version? → logs and change events.The trace_id in structured logs (ch 6) joins the trace to the log lines that explain it.
# logs of one slow request (Loki, LogQL)
{namespace="shop"} | json | trace_id="4bf92f3577b34da6a3ce929d0e0e4736"
# payments startup lines since 14:00 — config at boot
{namespace="shop", app="payments"} | json | msg=~"starting.*"
# what changed in Kubernetes?
kubectl -n shop rollout history deploy/payments
kubectl -n shop get deploy payments \
-o jsonpath='{.spec.template.spec.containers[0].env}'starting payments extra_latency_ms=800 fail_rate=0 at 14:00.EXTRA_LATENCY_MS (a test knob left on).kubectl -n shop rollout undo deploy/payments.POST /orders p95 back under 0.15 s on the same RED panel; alert resolves.Lab 13 replays this incident on your cluster, then adds errors (FAIL_RATE) to see how the picture changes.
Most incidents follow a change. Put changes on the same timeline as the symptoms.
kubectl rollout).POST /api/annotations) from CI/CD.kube_deployment_status_observed_generation from kube-state-metrics.loki.source.kubernetes_events).# annotation from a pipeline step
curl -s -X POST \
"$GRAFANA_URL/api/annotations" \
-H "Authorization: Bearer $TOKEN" \
-H 'Content-Type: application/json' \
-d '{"tags":["deploy","payments"],
"text":"payments rev 7"}'
# no "time" field → nowEvery page must be urgent, actionable and owned. Everything else is a ticket or a dashboard.
An alert is a request for a human to act now. If no action is possible or needed, it should not page.
team/service label to one rotation.summary, description with {{ $value }}, runbook_url.| Situation | Goes to |
|---|---|
| Users affected now | Page (severity: page) |
| Will hurt within days (disk 85%, cert expiry) | Ticket (severity: ticket) |
| Interesting, not actionable | Dashboard only |
| Cause behind a firing symptom | Inhibited, shown as context |
Prometheus and Alertmanager record their own alerting activity. Review it weekly.
# which alerts spent the most time firing (last 7 days)
# (samples ≈ minutes firing × evaluations per minute)
sort_desc(
count by (alertname) (
count_over_time(ALERTS{alertstate="firing"}[7d])))
# currently firing, by severity
count by (severity) (ALERTS{alertstate="firing"})
# notifications sent and failed, per integration
sum by (integration) (increase(alertmanager_notifications_total[7d]))
sum by (integration) (
increase(alertmanager_notifications_failed_total[7d]))
# alerts received by Alertmanager
sum(increase(alertmanager_alerts_received_total[7d]))for:, use keep_firing_for:, or alert on a longer window.group_by) or add an inhibition rule (ch 4).Linked from runbook_url, short enough to read at 3 a.m., with copy-paste queries.
# ShopHighLatencyP95 — runbook
## What it means
p95 latency of `orders` > 0.5 s for 5 min. Users wait at checkout.
## First 5 minutes
1. RED dashboard "Shop / Services", route panel.
2. Is `payments` slow too? (dependency panel)
3. Open an exemplar → trace in Tempo.
4. Logs for that trace_id in Loki.
5. Grafana annotations: any deploy/config change?
## Mitigations
- Recent change → `kubectl -n shop rollout undo deploy/<svc>`
- CPU throttling → raise limits / scale out
## Escalation
payments team: #payments-oncallrules/, runbooks/) and review them in the same PR.Put team (or service) on every alert; the routing tree stays small as alerts multiply.
route:
receiver: default-ticket
group_by: [alertname, namespace, service]
routes:
- matchers: ['severity="page"', 'team="payments"']
receiver: payments-pager
- matchers: ['severity="page"']
receiver: platform-pager
- matchers: ['severity="ticket"']
receiver: ticket-queue
inhibit_rules:
# a down target makes its latency alerts noise
- source_matchers: ['alertname="TargetDown"']
target_matchers: ['severity="page"']
equal: [namespace, service]team in the PrometheusRule (static label) or derive it from namespace ownership.amtool config routes test --config.file=alertmanager.yml severity=page team=payments.Every label, log line and span costs money. Decide what to keep on purpose.
Cost scales with different units per signal. Know the unit before you try to cut.
| Signal | Main cost unit | Typical blow-up | Primary control |
|---|---|---|---|
| Metrics (Prometheus, AMP, Mimir) | Active series × retention; samples ingested | Unbounded label (user_id, raw path, pod churn) | Label hygiene, relabel drop, recording rules |
| Logs (Loki, CloudWatch Logs) | GB ingested, GB stored, GB scanned by queries | Debug logs in prod, chatty health checks, stack traces per request | Levels, sampling, drop at the agent, retention tiers |
| Traces (Tempo, X-Ray) | Spans ingested and stored | 100% sampling of high-QPS routes | Head/tail sampling (ch 9), drop health-check spans |
| CloudWatch metrics | Custom metrics per month (each name + dimension set), API requests | High-cardinality dimensions in EMF | Fewer dimensions, EMF only for what you alert on |
| CloudWatch Logs Insights | GB scanned per query | Wide time ranges over all log groups | Narrow log groups and time, facets, saved queries |
Prices differ by Region and change; read the current pricing pages before estimating. The shape of the curve matters more than the number.
Find the top metrics first, then the label that explodes inside them.
# total active series in the head block
prometheus_tsdb_head_series
# top 10 metric names by series count (expensive: run rarely)
topk(10, count by (__name__) ({__name__=~".+"}))
# which label explodes inside one metric?
count(count by (route) (http_requests_total{job="orders"}))
count(count by (pod) (http_requests_total{job="orders"}))
# series per scrape target
topk(10, scrape_series_added)
topk(10, scrape_samples_post_metric_relabeling)# the same from the TSDB status API
kubectl -n monitoring port-forward svc/prometheus-operated 9090 &
curl -s localhost:9090/api/v1/status/tsdb \
| jq '.data.seriesCountByMetricName[:10]'
curl -s localhost:9090/api/v1/status/tsdb \
| jq '.data.labelValueCountByLabelName[:10]'
# offline, on a copy of the data directory
promtool tsdb analyze /prometheusA route label with thousands of values usually means the middleware records the raw path (/orders/8812) instead of the route template (/orders/{id}). Fix it in code (ch 2).
Cutting early is cheapest; cutting late is safest for teams you do not control. Use both.
info by default; debug behind a flag.metric_relabel_configs (drop, labeldrop) or metricRelabelings in a ServiceMonitor.sample_limit / label_limit per scrape job to stop surprises.filter and transform processors: drop health checks, delete attributes.loki.process stage.drop for noisy lines; tail sampling for traces.max_streams_per_user, ingestion_rate_mb (ch 7).Cost control is a social process backed by limits. Make usage visible before you enforce it.
namespace or team: count by (namespace) ({__name__=~".+"}) (run rarely, or record it).sum by (app) (bytes_over_time({namespace=~".+"}[1h])).sample_limit in Prometheus.promtool check metrics).info; debug needs an expiry.Who watches the watchers, and how to keep the whole setup reproducible.
If Prometheus stops evaluating rules or Alertmanager cannot send, every other alert is silently dead.
| Component | Signal to watch | Example |
|---|---|---|
| Prometheus scraping | Targets down, scrape errors, limits hit | up == 0, prometheus_target_scrapes_exceeded_sample_limit_total |
| Prometheus rules | Rule evaluation failures, slow groups | increase(prometheus_rule_evaluation_failures_total[5m]) > 0 |
| Prometheus → Alertmanager | Notifications dropped or erroring | prometheus_notifications_dropped_total, prometheus_notifications_errors_total |
| Alertmanager | Failed notifications, cluster size | alertmanager_notifications_failed_total, alertmanager_cluster_members |
| Config reloads | Bad config silently not applied | prometheus_config_last_reload_successful == 0, alertmanager_config_last_reload_successful == 0 |
| Loki | Rejected or discarded lines | loki_discarded_samples_total by reason |
| OTel Collector | Refused or failed telemetry, full queues | otelcol_receiver_refused_spans_total, otelcol_exporter_send_failed_spans_total, otelcol_exporter_queue_size |
kube-prometheus-stack already ships many of these rules (e.g. PrometheusRuleFailures, AlertmanagerFailedToSendAlerts, TargetDown). Check they route somewhere real.
An alert that always fires. If the external receiver stops hearing it, the whole chain is broken.
routes:
- matchers: ['alertname="Watchdog"']
receiver: heartbeat
group_wait: 0s
group_interval: 1m
repeat_interval: 1m # heartbeat every minute
receivers:
- name: heartbeat
webhook_configs:
- url: https://<heartbeat-service>/ping/<id>
A small PrometheusRule that covers the gaps the defaults do not know about: your own jobs and your own pipeline.
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
name: shop-meta
namespace: monitoring
labels: {release: kps} # picked up by kps
spec:
groups:
- name: shop-meta
rules:
- alert: ShopTargetMissing
expr: absent(up{job="orders"} == 1)
for: 5m
labels: {severity: page, team: platform}
annotations:
summary: no healthy orders target is scraped
- alert: CollectorExportFailing
expr: |
sum(rate(otelcol_exporter_send_failed_spans_total[5m])) > 0
for: 10m
labels: {severity: ticket, team: platform}absent(...) fires when the series is missing entirely — up == 0 cannot, because a target removed by service discovery produces no up series at all.:8888/metrics endpoint before writing the rule.Everything that defines what you see and what pages you lives in Git and is tested in CI.
obs-lab/
├── apps/ # instrumented services (ch 2, 6, 8)
├── deploy/
│ ├── kps-values.yaml # Prometheus, AM, Grafana
│ ├── loki-values.yaml # Loki + Alloy (ch 7)
│ ├── tempo-values.yaml # Tempo (ch 9)
│ └── otel-gateway.yaml # Collector config (ch 9)
├── rules/ # PrometheusRule + promtool tests
│ ├── shop-red.yaml
│ └── shop-red.test.yaml
├── alertmanager/ # routing tree + amtool tests
├── dashboards/ # JSON / schema v2 (ch 10)
└── runbooks/ # one .md per alertpromtool check rules and promtool test rules on every PR (ch 4).amtool check-config and amtool config routes test for routing.otelcol validate --config=… (or the distribution binary's validate).alloy fmt to check syntax.Use it to decide the next investment, not as a score.
| Stage | You have | Next step |
|---|---|---|
| 1 · Reactive | Host metrics, logs on nodes, users report outages | Central metrics and logs; up and error-rate alerts |
| 2 · Service metrics | RED per service, USE per node, dashboards per team | Recording rules, runbooks, routing by team |
| 3 · Correlated | Structured logs with trace_id, traces with sampling, exemplars | One-click alert → trace → logs; change annotations |
| 4 · SLO-driven | SLIs, error budgets, burn-rate paging (SRE deck) | Alert review, cost governance, meta-monitoring |
| 5 · Platform | Self-service instrumentation, golden-path defaults, per-team budgets | Everything as code; continuous verification (game days) |
Most teams get the biggest win from stage 2 → 3: shared keys across signals turn hours of guessing into minutes.
Each one has caused long outages or large bills somewhere.
runbook_url; alerts without an owner.Watchdog left routed to "null": nobody notices Alertmanager is down.Answer out loud with the exact query, label or setting.
team instead of by alertname?Watchdog alert protect against, and where must its receiver run?absent(up{job="orders"} == 1) rather than up == 0?Practice: Lab 13 in Hands-on labs — a game day on the obs cluster: inject latency and errors into payments, follow alert → dashboard → exemplar → trace → logs, wire up the Watchdog, and clean up a cardinality problem.
Everything in the deck on one cluster, plus a CloudWatch mirror. Build it after Lab 12; Lab 13 is its game day.
Same ideas, two stacks: RED metrics, structured logs and alarms on symptoms work the same in Prometheus and in CloudWatch.
Done means every box is ticked and proven with a query or a command, not a screenshot.
obs, shop app, loadgen (Lab 0).release: kps (Labs 2, 5).trace_id; Alloy → Loki (Labs 6, 7).otel-gateway → Tempo, with tail sampling (Labs 8, 9).obs-lab-orders with EMF, Logs Insights queries, alarm to SNS (Labs 11, 12).promtool test rules passes in CI; every alert has runbook_url and team.EXTRA_LATENCY_MS=800 pages within ~7 min; rollout undo resolves it.route uses templates).namespace, app, container (and similar); trace_id is not a label.amtool config routes test alertname=Watchdog → the heartbeat receiver.dashboards/ folder.Counters: rate first, then aggregate. Histograms: sum by (le) of the bucket rate, then histogram_quantile.
# request rate per route
sum by (route) (rate(http_requests_total{job="orders"}[5m]))
# error ratio (5xx / all)
sum(rate(http_requests_total{job="orders",code=~"5.."}[5m]))
/ sum(rate(http_requests_total{job="orders"}[5m]))
# p95 latency per route (classic histogram)
histogram_quantile(0.95, sum by (le, route) (
rate(http_request_duration_seconds_bucket{job="orders"}[5m])))# p95 with a native histogram (no _bucket, no le)
histogram_quantile(0.95, sum by (route) (
rate(http_request_duration_seconds{job="orders"}[5m])))
# average latency
sum(rate(http_request_duration_seconds_sum[5m]))
/ sum(rate(http_request_duration_seconds_count[5m]))
# add the node to per-pod CPU (many-to-one join)
sum by (pod) (rate(container_cpu_usage_seconds_total{
namespace="shop", container!=""}[5m]))
* on (pod) group_left (node)
max by (pod, node) (kube_pod_info{namespace="shop"})Selectors pick series, functions transform them, operators combine them.
| Construct | Meaning |
|---|---|
{job="orders", code=~"5.."} | Selector; =, !=, =~, !~ (regexes are fully anchored) |
[5m], offset 1d, @ end() | Range; shift back in time; pin the evaluation time |
rate / irate / increase | Per-second average over the range / from the last two samples / total increase over the range |
sum by (l) / sum without (l) | Keep only / drop the listed labels |
topk(5, …), quantile(0.9, …) | Top series; quantile across series (not over time) |
on (l) group_left (x) | Match on l; many left to one right; copy label x from the right |
and, or, unless | Set operations on label sets |
> bool 0.5 | Return 0/1 instead of filtering |
max_over_time(x[1h]) | Range function on a gauge |
max_over_time(rate(x[5m])[1h:1m]) | Subquery: 1 h of 1-min steps of an inner expression |
absent(up{job="orders"}) | 1 when no such series exists (for "missing" alerts) |
predict_linear(x[6h], 4*3600) | Linear forecast 4 h ahead (gauges) |
$__rate_interval | Grafana variable: a safe rate window for the scrape interval |
Recording rule names follow level:metric:operations. Every alert needs severity, an owner and a runbook_url.
groups:
- name: shop-red
interval: 30s
rules:
- record: job_route:http_requests:rate5m
expr: |
sum by (job, route) (
rate(http_requests_total[5m]))
- alert: ShopHighErrorRate
expr: |
sum(rate(http_requests_total{
job="orders",code=~"5.."}[5m]))
/ sum(rate(http_requests_total{
job="orders"}[5m])) > 0.05
for: 5m
keep_firing_for: 5m
labels: {severity: page, team: shop}
annotations:
summary: '5xx {{ $value | humanizePercentage }}'
runbook_url: https://runbooks.example.com/shop/errorsroute:
receiver: default
group_by: [alertname, namespace]
group_wait: 30s # defaults: 30s / 5m / 4h
group_interval: 5m
repeat_interval: 4h
routes:
- matchers: ['severity="page"']
receiver: pager
continue: false
inhibit_rules:
- source_matchers: ['severity="page"']
target_matchers: ['severity="ticket"']
equal: [alertname, namespace]
receivers:
- name: default
- name: pager
webhook_configs:
- url: https://pager.example.com/hookValidate in CI; test routing before you merge; silence with a comment and an expiry.
| Command | What it does |
|---|---|
promtool check config prometheus.yml | Validate the server config (and the rule files it references) |
promtool check rules rules.yaml | Validate rule syntax |
promtool test rules tests.yaml | Unit-test rules with input_series and expected alerts or values |
promtool query instant http://localhost:9090 'up' | Run a query from the shell |
promtool tsdb analyze <data-dir> | Cardinality report from a TSDB directory |
amtool check-config alertmanager.yml | Validate config and templates |
amtool config routes test --config.file=am.yml severity=page | Which receiver gets an alert with these labels? |
amtool config routes show --config.file=am.yml | Print the routing tree |
amtool alert query alertname=X --alertmanager.url=http://localhost:9093 | List firing alerts |
amtool silence add alertname=X --duration=2h --comment="maintenance" | Create a silence |
amtool silence query / amtool silence expire <id> | List / remove silences |
Select streams with labels (cheap), filter lines (cheap), then parse (costly). Metric queries wrap a log query in a range function.
# log queries
{namespace="shop", app="orders"} |= "error"
{namespace="shop"} != "healthz" |~ "timeout|refused"
{app="orders"} | json | status >= 500 and duration_ms > 300
{app="orders"} | json | trace_id="4bf92f3577b34da6a3ce929d0e0e4736"
{app="orders"} | logfmt | level="error"
{app="orders"} | json | line_format "{{.route}} {{.status}} {{.msg}}"
{app="orders"} | json | __error__="" # drop parse failures# metric queries
sum by (app) (rate({namespace="shop"} |= "error" [5m]))
sum by (app) (count_over_time({namespace="shop"}[1m]))
sum by (app) (bytes_over_time({namespace="shop"}[1h]))
topk(5, sum by (route) (count_over_time(
{app="orders"} | json | status >= 500 [15m])))
quantile_over_time(0.95,
{app="orders"} | json | unwrap duration_ms [5m]
) by (route)Labels in these examples follow the lab's Alloy setup (ch 7): namespace, app, container. Fields such as trace_id and status come from parsing the JSON line, never from labels.
A log query is a pipeline read left to right; every stage after the selector runs on each line.
| Part | Notes |
|---|---|
{label="v"} | Stream selector, required; =, !=, =~, !~ |
|= != |~ !~ | Line filters: contains / does not contain / regex / not regex |
| json | logfmt | Extract fields as labels for this query only |
| pattern "<a> <_> <b>" / | regexp "…" | Extract fields from unstructured lines |
| status >= 500 | Label filter after parsing (numbers, durations, bytes, strings) |
| line_format / | label_format | Rewrite the line with a template / rename or set labels |
| drop a, b / | keep a, b | Remove / keep only these labels |
| unwrap duration_ms | Use a numeric field as the sample value |
rate, count_over_time, bytes_over_time | Lines per second / lines in range / bytes in range |
sum_over_time, avg_over_time, quantile_over_time | Range functions over unwrapped values |
logcli query '{app="orders"}' --since=1h | Shell client; set LOKI_ADDR=http://localhost:3100 |
A query selects spansets with { … }, then pipes them through aggregates, select or metrics functions.
# by service, status, duration
{ resource.service.name = "payments" && status = error }
{ resource.service.name = "orders" && duration > 500ms }
{ span.http.route = "/orders"
&& span.http.response.status_code >= 500 }
# unscoped attribute (span or resource)
{ .deployment.environment.name = "lab" }
# structural: payments spans below orders spans
{ resource.service.name = "orders" }
>> { resource.service.name = "payments" }# aggregates per trace, then filter
{ resource.service.name = "orders" } | count() > 10
{ } | avg(duration) > 1s
# return extra attributes with the results
{ status = error }
| select(span.http.route, resource.k8s.pod.name)
# TraceQL metrics (time series from spans)
{ resource.service.name = "payments" } | rate()
{ kind = server }
| quantile_over_time(duration, .95) by (resource.service.name)
{ status = error } | count_over_time() by (span.http.route)Span intrinsics are bare (duration) or span:-scoped; trace intrinsics use trace:; attributes are resource., span. or unscoped ..
| Element | Meaning |
|---|---|
resource.x / span.x / .x | Resource attribute / span attribute / either |
duration, name, status, kind | Span intrinsics |
trace:duration, trace:rootName, trace:rootService | Trace-level intrinsics |
status = error | ok | unset | Span status values |
kind = server | client | producer | consumer | internal | Span kinds |
&&, || inside { } | Conditions on one span |
>> / > / ~ | Descendant / child / sibling spansets |
| count(), | avg(duration) | Aggregate per trace, then filter with a comparison |
| select(…) | Add attributes to the returned spans |
| rate(), | count_over_time(), | quantile_over_time() | TraceQL metrics (GA in Tempo 3) |
Define components once, wire them into pipelines. In processors, memory_limiter goes first and batch late.
receivers:
otlp:
protocols:
grpc: {endpoint: 0.0.0.0:4317}
http: {endpoint: 0.0.0.0:4318}
processors:
memory_limiter: {check_interval: 1s,
limit_percentage: 80, spike_limit_percentage: 25}
k8sattributes: {}
filter/health:
error_mode: ignore
traces:
span: ['attributes["url.path"] == "/healthz"']
tail_sampling:
decision_wait: 10s
policies:
- {name: errors, type: status_code,
status_code: {status_codes: [ERROR]}}
- {name: slow, type: latency,
latency: {threshold_ms: 500}}
- {name: rest, type: probabilistic,
probabilistic: {sampling_percentage: 10}}
batch: {}exporters:
otlp/tempo:
endpoint: tempo.tracing.svc:4317
tls: {insecure: true}
otlphttp/loki:
endpoint: http://loki.logging.svc:3100/otlp
debug: {verbosity: basic}
connectors:
spanmetrics: {}
service:
pipelines:
traces:
receivers: [otlp]
processors: [memory_limiter, k8sattributes,
filter/health, tail_sampling, batch]
exporters: [otlp/tempo, spanmetrics]
metrics:
receivers: [spanmetrics]
exporters: [debug]
logs:
receivers: [otlp]
processors: [memory_limiter, k8sattributes, batch]
exporters: [otlphttp/loki]
# check: otelcol-contrib validate --config=config.yamlk8sattributes needs RBAC to read pods; tail sampling needs every span of a trace on the same Collector (ch 9). Loki 3 accepts OTLP logs at /otlp. Send span metrics to Prometheus with an exporter of your choice (ch 9).
Logs Insights bills per GB scanned: pick few log groups and a short time range first.
# Logs Insights QL (JSON fields are discovered)
fields @timestamp, level, route, status, duration_ms
| filter level = "ERROR" and route = "/orders"
| sort @timestamp desc
| limit 50
filter status >= 500
| stats count(*) as errors by bin(5m)
stats pct(duration_ms, 95) as p95 by route
parse @message "status=* duration=*ms" as st, dur
| stats count(*) by st
pattern @message # cluster similar lines
filter trace_id = "4bf92f3577b34da6a3ce929d0e0e4736"
filter @type = "REPORT" # Lambda reports
| stats pct(@duration, 99) as p99msLG=/obs-lab/orders
aws logs tail $LG --since 15m --follow
aws logs tail $LG --since 1h \
--filter-pattern '{ $.level = "ERROR" }'
QID=$(aws logs start-query --log-group-name $LG \
--start-time $(($(date +%s) - 3600)) \
--end-time $(date +%s) \
--query-string 'stats count(*) by bin(5m)' \
--query queryId --output text)
aws logs get-query-results --query-id $QID
aws logs put-retention-policy --log-group-name $LG \
--retention-in-days 14
aws logs put-metric-filter --log-group-name $LG \
--filter-name orders-5xx \
--filter-pattern '{ $.status >= 500 }' \
--metric-transformations \
metricName=Orders5xx,metricNamespace=ObsLab,metricValue=1
aws cloudwatch describe-alarms --state-value ALARM
aws cloudwatch set-alarm-state --alarm-name <name> \
--state-value ALARM --state-reason test # test actionsLabs 0–10 and 13 run on a local kind cluster and are free. Labs 11–12 use a small AWS sandbox in ap-northeast-1.
obs (1 control plane + 2 workers).export AWS_REGION=ap-northeast-1, a budget alarm before Lab 11.obs-lab/ (local Git is enough); Go module obs-lab; images shop/orders:lab, shop/payments:lab loaded with kind load docker-image.kps in monitoring; every ServiceMonitor, PodMonitor and PrometheusRule carries label release: kps.brew install kind kubectl helm go jq hey \
prometheus alertmanager logcli awscli
kind version
kubectl version --client
helm version # v3.x or v4.x
go version # go1.25+
promtool --version # ships with prometheus
amtool --version # ships with alertmanager
logcli --version
aws --version # aws-cli/2.x
helm repo add prometheus-community \
https://prometheus-community.github.io/helm-charts
helm repo add grafana \
https://grafana.github.io/helm-charts
helm repo add grafana-community \
https://grafana-community.github.io/helm-charts
helm repo add open-telemetry \
https://open-telemetry.github.io/opentelemetry-helm-charts
helm repo updateGoal: a dependency whose error rate and latency you control with two environment variables. No telemetry yet: later labs add it.
mkdir -p obs-lab/apps/{orders,payments} obs-lab/deploy
cd obs-lab && go mod init obs-lab
cat > apps/payments/main.go <<'EOF'
package main
import (
"encoding/json"
"log/slog"
"math/rand/v2"
"net/http"
"os"
"strconv"
"time"
)
func envFloat(k string) float64 {
v, _ := strconv.ParseFloat(os.Getenv(k), 64)
return v
}
func main() {
h := slog.NewJSONHandler(os.Stdout, nil)
log := slog.New(h)
failRate := envFloat("FAIL_RATE") // 0..1
extra := envFloat("EXTRA_LATENCY_MS") // added delay
mux := http.NewServeMux()
mux.HandleFunc("GET /healthz",
func(http.ResponseWriter, *http.Request) {}) charge := func(w http.ResponseWriter, r *http.Request) {
var req struct {
OrderID string `json:"order_id"`
Amount float64 `json:"amount"`
}
err := json.NewDecoder(r.Body).Decode(&req)
if err != nil {
http.Error(w, "bad request", 400)
return
}
// base latency 20-80 ms plus the injected delay
ms := 20 + rand.Float64()*60 + extra
time.Sleep(time.Duration(ms * float64(time.Millisecond)))
if rand.Float64() < failRate {
log.Error("charge failed", "order_id", req.OrderID)
http.Error(w, "card declined", 502)
return
}
log.Info("charged", "order_id", req.OrderID,
"amount", req.Amount)
}
mux.HandleFunc("POST /charge", charge)
log.Info("payments listening", "addr", ":8080")
err := http.ListenAndServe(":8080", mux)
log.Error("server stopped", "err", err)
os.Exit(1)
}
EOFGoal: an API that keeps orders in memory and calls payments synchronously, so payment failures and latency surface on POST /orders.
cat > apps/orders/main.go <<'EOF'
package main
import (
"bytes"
"encoding/json"
"fmt"
"log/slog"
"net/http"
"os"
"sync"
"sync/atomic"
"time"
)
type Order struct {
ID string `json:"id"`
Item string `json:"item"`
Amount float64 `json:"amount"`
}
var (
log = slog.New(slog.NewJSONHandler(os.Stdout, nil))
client = &http.Client{Timeout: 2 * time.Second}
payURL = "http://payments:8080/charge"
seq atomic.Int64
mu sync.RWMutex
orders = map[string]Order{}
)func create(w http.ResponseWriter, r *http.Request) {
var o Order
if err := json.NewDecoder(r.Body).Decode(&o); err != nil {
http.Error(w, "bad request", 400)
return
}
o.ID = fmt.Sprintf("o-%d", seq.Add(1))
body, _ := json.Marshal(map[string]any{
"order_id": o.ID, "amount": o.Amount})
resp, err := client.Post(payURL, "application/json",
bytes.NewReader(body))
if err == nil {
resp.Body.Close()
if resp.StatusCode != http.StatusOK {
err = fmt.Errorf("payments: %s", resp.Status)
}
}
if err != nil {
log.Error("payment failed", "order_id", o.ID,
"err", err)
http.Error(w, "payment failed", 502)
return
}
mu.Lock()
orders[o.ID] = o
mu.Unlock()
w.WriteHeader(http.StatusCreated)
json.NewEncoder(w).Encode(o)
}Goal: finish orders, then run both services on your laptop before any container or cluster is involved.
func get(w http.ResponseWriter, r *http.Request) {
mu.RLock()
o, ok := orders[r.PathValue("id")]
mu.RUnlock()
if !ok {
http.NotFound(w, r)
return
}
json.NewEncoder(w).Encode(o)
}
func main() {
if v := os.Getenv("PAYMENTS_URL"); v != "" {
payURL = v
}
mux := http.NewServeMux()
mux.HandleFunc("POST /orders", create)
mux.HandleFunc("GET /orders/{id}", get)
mux.HandleFunc("GET /healthz",
func(http.ResponseWriter, *http.Request) {})
log.Info("orders listening", "addr", ":8080")
err := http.ListenAndServe(":8080", mux)
log.Error("server stopped", "err", err)
os.Exit(1)
}
EOFgofmt -l . ; go vet ./... # no output = good
# both listen on :8080: run payments in a container
docker run --rm -d --name pay -p 9001:8080 \
-v "$PWD":/src -w /src golang:1.27 \
go run ./apps/payments
# go run would leave the server running after kill
go build -o /tmp/orders ./apps/orders
PAYMENTS_URL=http://localhost:9001/charge \
/tmp/orders &
curl -s -X POST localhost:8080/orders \
-d '{"item":"pen","amount":3}'
# {"id":"o-1","item":"pen","amount":3}
curl -s localhost:8080/orders/o-1
# {"id":"o-1","item":"pen","amount":3}
# stop both
kill %1; docker rm -f payMethod patterns ("POST /orders") and r.PathValue need Go 1.22+; the images build with golang:1.27.
Goal: one Dockerfile for both services, a three-node cluster, and the images loaded into it (no registry needed).
cat > Dockerfile <<'EOF'
# syntax=docker/dockerfile:1
FROM golang:1.27 AS build
ARG APP
WORKDIR /src
COPY go.* ./
RUN go mod download
COPY apps ./apps
RUN CGO_ENABLED=0 go build -trimpath \
-o /out/app ./apps/${APP}
FROM gcr.io/distroless/static-debian12:nonroot
COPY --from=build /out/app /app
EXPOSE 8080
ENTRYPOINT ["/app"]
EOF
cat > deploy/kind.yaml <<'EOF'
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
name: obs
nodes:
- role: control-plane
- role: worker
- role: worker
EOFkind create cluster --config deploy/kind.yaml
kubectl get nodes
# obs-control-plane Ready control-plane
# obs-worker Ready <none>
# obs-worker2 Ready <none>
for a in orders payments; do
docker build --build-arg APP=$a -t shop/$a:lab .
kind load docker-image shop/$a:lab --name obs
done
# after any code change later in the deck:
# docker build --build-arg APP=orders \
# -t shop/orders:lab .
# kind load docker-image shop/orders:lab --name obs
# kubectl -n shop rollout restart deploy/ordersCOPY go.* ./ copies go.mod now and go.sum once Lab 2 adds dependencies; go mod download then caches them in their own layer.
Goal: Prometheus, Alertmanager, Grafana, kube-state-metrics and node-exporter in monitoring. Control-plane scrapes that kind cannot serve are switched off, so the lab starts without false alerts.
cat > deploy/kps-values.yaml <<'EOF'
# kind binds these components to 127.0.0.1:
# scraping them from Prometheus only fails
kubeControllerManager: {enabled: false}
kubeScheduler: {enabled: false}
kubeEtcd: {enabled: false}
kubeProxy: {enabled: false}
prometheus:
prometheusSpec:
retention: 7d # selectors stay default:
# objects need release: kps
EOF
# pin the chart; every later "helm upgrade kps"
# reuses this version (add the export to ~/.zshrc)
helm search repo kube-prometheus-stack # CHART VERSION
export KPS_VER=<that version>
helm install kps \
prometheus-community/kube-prometheus-stack \
-n monitoring --create-namespace \
--version $KPS_VER -f deploy/kps-values.yaml
kubectl -n monitoring get pods -w # ~2 minkubectl -n monitoring get svc
# kps-grafana 80/TCP
# kps-kube-prometheus-stack-alertmanager 9093/TCP
# kps-kube-prometheus-stack-prometheus 9090/TCP
# ...
# one terminal each (or append & to background)
NS="-n monitoring"
kubectl $NS port-forward \
svc/kps-kube-prometheus-stack-prometheus 9090
kubectl $NS port-forward \
svc/kps-kube-prometheus-stack-alertmanager 9093
kubectl $NS port-forward svc/kps-grafana 3000:80
# Grafana: user admin, password from the secret
kubectl $NS get secret kps-grafana \
-o jsonpath='{.data.admin-password}' \
| base64 -d; echoService names are derived from the release name. If a port-forward fails, use the names that kubectl -n monitoring get svc prints.
Goal: one Deployment and one Service per app, generated by a loop so both stay identical. The Service port is named http: ServiceMonitors (Lab 2) refer to it by name.
echo 'apiVersion: v1
kind: Namespace
metadata: {name: shop}' > deploy/shop.yaml
# only payments reads the two env vars
for a in orders payments; do
cat >> deploy/shop.yaml <<EOF
---
apiVersion: apps/v1
kind: Deployment
metadata: {name: $a, namespace: shop, labels: {app: $a}}
spec:
replicas: 2
selector: {matchLabels: {app: $a}}
template:
metadata: {labels: {app: $a}}
spec:
containers:
- name: $a
image: shop/$a:lab
imagePullPolicy: IfNotPresent
env:
- {name: FAIL_RATE, value: "0"}
- {name: EXTRA_LATENCY_MS, value: "0"}
ports: [{name: http, containerPort: 8080}]
readinessProbe:
httpGet: {path: /healthz, port: http}
resources:
requests: {cpu: 50m, memory: 32Mi}
limits: {memory: 64Mi}# (still inside the same heredoc and loop)
---
apiVersion: v1
kind: Service
metadata: {name: $a, namespace: shop, labels: {app: $a}}
spec:
selector: {app: $a}
ports: [{name: http, port: 8080, targetPort: http}]
EOF
done
kubectl apply -f deploy/shop.yaml
kubectl -n shop get deploy,svc
# deployment.apps/orders 2/2
# deployment.apps/payments 2/2
# service/orders ClusterIP ... 8080/TCP
# service/payments ClusterIP ... 8080/TCPGoal: steady traffic through orders → payments, so every later lab has data to look at.
cat >> deploy/shop.yaml <<'EOF'
---
apiVersion: apps/v1
kind: Deployment
metadata: {name: loadgen, namespace: shop}
spec:
replicas: 1
selector: {matchLabels: {app: loadgen}}
template:
metadata: {labels: {app: loadgen}}
spec:
containers:
- name: loadgen
image: curlimages/curl:8.11.1
command: ["/bin/sh", "-c"]
args:
- |
while true; do
curl -s -o /dev/null -X POST \
-d '{"item":"book","amount":42}' \
http://orders:8080/orders
sleep 0.2
done
EOFkubectl apply -f deploy/shop.yaml
kubectl -n shop rollout status deploy/orders
kubectl -n shop get pods
# loadgen-… 1/1 Running
# orders-… 1/1 Running (x2)
# payments-… 1/1 Running (x2)
kubectl -n shop logs deploy/payments --tail=2
# {"time":"…","level":"INFO","msg":"charged",
# "order_id":"o-…","amount":42}
# your own request through the Service
kubectl -n shop port-forward svc/orders 8080 &
curl -s -X POST localhost:8080/orders \
-d '{"item":"pen","amount":3}'
# {"id":"o-…","item":"pen","amount":3}curlimages/curl:8.11.1 is an example tag; any recent tag works. Each orders replica numbers its own orders, so IDs repeat across the two replicas: fine for a lab.
Everything later in the deck observes this setup. Make sure it is healthy before you add telemetry.
kubectl -n shop get pods: 2 orders, 2 payments, 1 loadgen Running.kubectl -n shop logs deploy/orders --tail=3: JSON lines from slog.Watchdog (always firing by design, ch 13) and possibly InfoInhibitor.shop.kubectl -n shop set env deploy/payments FAIL_RATE=0.3 → orders logs payment failed. Only logs show it: no metric exists yet (Lab 2).EXTRA_LATENCY_MS=2500 → the 2 s client timeout fires and orders returns 502. Reset both to 0.kubeEtcd: {enabled: false}, helm upgrade with the same values, and watch etcd targets go DOWN and alerts fire. Restore it.# keep the cluster: every later
# lab builds on it
# pause the load when not studying
kubectl -n shop scale \
deploy/loadgen --replicas=0
# full teardown (only at the
# very end of the deck)
# kind delete cluster --name obsGoal: answer "is it broken, and how badly?" for the shop app using only an outside probe and the logs Lab 0 writes. Note what you cannot answer.
kubectl -n shop port-forward svc/orders 8080 & # if not running
# black-box: status codes and latency of 50 requests
probe() {
for i in $(seq 50); do
curl -s -o /dev/null -w '%{http_code} %{time_total}\n' \
-X POST -d '{"item":"pen","amount":3}' \
localhost:8080/orders
done | awk '{n[$1]++; t+=$2; if ($2>m) m=$2}
END {for (c in n) print "code", c, n[c];
printf "avg %.3fs max %.3fs\n", t/NR, m}'
}
probe
# code 201 50
# avg 0.060s max 0.095s
kubectl -n shop set env deploy/payments FAIL_RATE=0.2 # 20%
kubectl -n shop rollout status deploy/payments
probe
# code 201 41
# code 502 9# logs: what do the services write?
kubectl -n shop logs deploy/orders --tail=3
# {"time":"…","level":"ERROR","msg":"payment failed",
# "order_id":"o-…","err":"payments: 502 Bad Gateway"}
# with -l, kubectl logs defaults to --tail=10 per pod!
L="-n shop -l app=payments --since=2m --tail=-1"
kubectl logs $L | jq -r .msg | sort | uniq -c
# 480 charged
# 121 charge failed
# error ratio from payments logs: 121 / 601 ≈ 20%
kubectl logs $L | jq -s 'map(select(.msg|test("^charge")))
| (map(select(.msg=="charge failed"))|length) / length'
# 0.2013…
# orders logs only failures: no total, no route,
# no duration → no error RATIO and no latency hereKeep a note of every question you could not answer from these signals. Lab 2 adds the metrics that answer them.
Goal: see what kube-prometheus-stack already collects about the shop pods, how many series exist, and write the shop's signal plan.
# Prometheus UI: http://localhost:9090 (port-forward from Lab 0)
# CPU used per pod (cores)
sum by (pod) (rate(container_cpu_usage_seconds_total{
namespace="shop", container!=""}[5m]))
# memory working set per pod (bytes) vs the 64Mi limit
sum by (pod) (container_memory_working_set_bytes{
namespace="shop", container!=""})
# restarts and availability
sum by (pod) (kube_pod_container_status_restarts_total{
namespace="shop"})
kube_deployment_status_replicas_available{namespace="shop"}
# app metrics? → "Empty query result" (none yet)
{job=~"orders|payments"}# series inventory from the TSDB status API
P=localhost:9090/api/v1/status/tsdb
curl -s $P | jq '.data.headStats.numSeries'
# e.g. 60000 (varies with versions and node count)
curl -s $P | jq -r '.data.seriesCountByMetricName[:5][]
| "\(.value)\t\(.name)"'
# top metrics by series count (bucket metrics lead)
# signal plan: copy the table from ch 1 and fill in
mkdir -p docs && cat > docs/signal-plan.md <<'EOF'
| Question | Signal | Query / source | Alert? |
|---|---|---|---|
| Are orders failing? | metric (R,E) | TODO in Lab 2 | page |
| Are orders slow? | metric (D) | TODO in Lab 2 | page |
| Is payments the cause? | metric + trace | … | no |
| Pods sized right? | USE (k8s) | queries above | ticket |
EOFcontainer!="" drops the pod-level cgroup series that cAdvisor also exports, which would otherwise double-count each pod.
The point of this lab: resource metrics and failure-only logs cannot give you RED. That gap is what app instrumentation fills.
probe with FAIL_RATE=0.2 shows ~20% 502s; the payments log ratio agrees.{job=~"orders|payments"} returns nothing — no app metrics yet.docs/signal-plan.md lists at least 6 questions, each with a signal and an alert decision.orders logs alone cannot give an error ratio.kubectl -n shop set env deploy/payments FAIL_RATE=0 EXTRA_LATENCY_MS=1500 → probe avg jumps to ~1.5 s, yet the CPU and memory queries barely move. Waiting is not utilization: a symptom with no USE cause.EXTRA_LATENCY_MS=2500 → every order is a 502 after 2 s (client timeout). Black-box sees it; USE still looks healthy.kubectl logs -n shop -l app=payments | wc -l without --tail=-1 → only 10 lines per pod. Your "error ratio" from logs was wrong without it.# reset the knobs
kubectl -n shop set env \
deploy/payments \
FAIL_RATE=0 EXTRA_LATENCY_MS=0
# stop the port-forward
kill %1
# keep docs/signal-plan.md:
# Lab 2 fills in the TODOsGoal: one shared package gives RED metrics for every route of both services. It lives under apps/ so the Lab 0 Dockerfile copies it.
mkdir -p apps/internal/metrics
cat > apps/internal/metrics/metrics.go <<'EOF'
package metrics
import (
"net/http"
"strconv"
"strings"
"time"
"github.com/prometheus/client_golang/prometheus"
"github.com/prometheus/client_golang/prometheus/promauto"
"github.com/prometheus/client_golang/prometheus/promhttp"
)
var requests = promauto.NewCounterVec(prometheus.CounterOpts{
Name: "http_requests_total",
Help: "HTTP requests by method, route template and code.",
}, []string{"method", "route", "code"})var duration = promauto.NewHistogramVec(prometheus.HistogramOpts{
Name: "http_request_duration_seconds",
Help: "HTTP request latency.",
Buckets: []float64{.01, .025, .05, .1, .25, .3, .5, 1, 2.5, 5},
// also expose a native histogram (protobuf scrape)
NativeHistogramBucketFactor: 1.1,
NativeHistogramMaxBucketNumber: 100,
NativeHistogramMinResetDuration: time.Hour,
}, []string{"method", "route"})
// recorder remembers the status code the handler wrote
type recorder struct {
http.ResponseWriter
code int
}
func (r *recorder) WriteHeader(c int) {
r.code = c
r.ResponseWriter.WriteHeader(c)
}
# (file continues on the next slide)The 0.3 s bucket is deliberate: it is the latency SLO threshold used in later chapters, so "share under 300 ms" is exact.
Goal: finish the package. r.Pattern (Go 1.23+) is set by the ServeMux, so route is the template, never the raw path.
// Middleware wraps the whole mux; the mux sets r.Pattern.
func Middleware(next http.Handler) http.Handler {
return http.HandlerFunc(func(w http.ResponseWriter,
r *http.Request) {
start := time.Now()
rec := &recorder{ResponseWriter: w, code: 200}
next.ServeHTTP(rec, r)
route := "unmatched" // 404 / 405
if _, p, ok := strings.Cut(r.Pattern, " "); ok {
route = p // "GET /orders/{id}" → "/orders/{id}"
}
requests.WithLabelValues(r.Method, route,
strconv.Itoa(rec.code)).Inc()
duration.WithLabelValues(r.Method, route).
Observe(time.Since(start).Seconds())
})
}
// Handler serves /metrics (OpenMetrics → exemplars, ch 9).
func Handler() http.Handler {
return promhttp.HandlerFor(prometheus.DefaultGatherer,
promhttp.HandlerOpts{EnableOpenMetrics: true})
}
EOFgo get github.com/prometheus/client_golang@v1.25.0
go mod tidy
go vet ./...
# what the middleware produces (tested):
# GET /orders/o-1 → route="/orders/{id}", code="200"
# GET /orders/o-2 → same series, value 2
# GET /nope → route="unmatched", code="404"
#
# http_requests_total{code="200",method="GET",
# route="/orders/{id}"} 2
# http_requests_total{code="404",method="GET",
# route="unmatched"} 1client_golang v1.25.0 was current when this was written; any newer v1 works. go.sum now exists, so the Dockerfile's COPY go.* ./ picks it up.
Goal: both services expose /metrics; kube-prometheus-stack scrapes them as job="orders" and job="payments" (the job label defaults to the Service name).
// apps/orders/main.go — add to imports:
// "obs-lab/apps/internal/metrics"
// "github.com/prometheus/client_golang/prometheus"
// "github.com/prometheus/client_golang/prometheus/promauto"
var ordersCreated = promauto.NewCounter(prometheus.CounterOpts{
Name: "orders_created_total",
Help: "Orders created successfully.",
})
// in create(), just before w.WriteHeader(http.StatusCreated):
ordersCreated.Inc()
// in main():
mux.Handle("GET /metrics", metrics.Handler())
err := http.ListenAndServe(":8080", metrics.Middleware(mux))
// apps/payments/main.go — same imports, then:
var chargeAmount = promauto.NewHistogram(prometheus.HistogramOpts{
Name: "payments_charge_amount",
Help: "Charged amount per successful charge.",
Buckets: prometheus.ExponentialBuckets(1, 2, 12),
})
// in charge(), after log.Info("charged", ...):
chargeAmount.Observe(req.Amount)
// in main(): the same two lines as ordersgofmt -l . ; go vet ./...
for a in orders payments; do
docker build --build-arg APP=$a -t shop/$a:lab .
kind load docker-image shop/$a:lab --name obs
done
kubectl -n shop rollout restart deploy/orders deploy/payments
cat > deploy/shop-monitor.yaml <<'EOF'
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
name: shop
namespace: shop
labels: {release: kps} # matched by kps
spec:
selector:
matchExpressions:
- {key: app, operator: In, values: [orders, payments]}
endpoints:
- port: http # Service port NAME
path: /metrics
interval: 15s
EOF
kubectl apply -f deploy/shop-monitor.yaml
# Prometheus → Status → Target health, after ~1 min:
# serviceMonitor/shop/shop/0 4/4 upGoal: write raw scrape_configs yourself (no Operator), scrape native and classic histograms side by side, and probe /healthz from outside.
cd local-prom # after writing prometheus.yml (right)
kubectl -n shop port-forward --address 0.0.0.0 svc/orders 8080 &
echo 'modules: {http_2xx: {prober: http, timeout: 5s}}' \
> blackbox.yml
docker network create obs-lab
docker run -d --name blackbox --network obs-lab \
--add-host=host.docker.internal:host-gateway \
-v "$PWD/blackbox.yml:/etc/blackbox_exporter/config.yml" \
prom/blackbox-exporter:v0.27.0
docker run -d --name prom --network obs-lab -p 9091:9090 \
--add-host=host.docker.internal:host-gateway \
-v "$PWD/prometheus.yml:/etc/prometheus/prometheus.yml" \
prom/prometheus:v3.15.1
# UI http://localhost:9091 (9090 = kps); after 5 min:
histogram_quantile(0.99, sum by (le) (rate( # classic
http_request_duration_seconds_bucket{job="orders-local"}[5m])))
histogram_quantile(0.99, sum(rate( # native
http_request_duration_seconds{job="orders-local"}[5m])))
histogram_fraction(0, 0.3, sum(rate( # share ≤ 0.3 s
http_request_duration_seconds{job="orders-local"}[5m])))mkdir -p local-prom && cat > local-prom/prometheus.yml <<'EOF'
global: {scrape_interval: 15s}
scrape_configs:
- job_name: orders-local
scrape_native_histograms: true # protobuf first
always_scrape_classic_histograms: true # keep _bucket
sample_limit: 2000
static_configs: [{targets: ['host.docker.internal:8080']}]
metric_relabel_configs:
- {source_labels: [__name__], regex: go_gc_.*, action: drop}
- job_name: blackbox-http
metrics_path: /probe
params: {module: [http_2xx]}
static_configs:
[{targets: ['http://host.docker.internal:8080/healthz']}]
relabel_configs:
- {source_labels: [__address__], target_label: __param_target}
- {source_labels: [__param_target], target_label: instance}
- {target_label: __address__, replacement: blackbox:9115}
EOF
docker run --rm -v "$PWD/local-prom:/c" --entrypoint promtool \
prom/prometheus:v3.15.1 check config /c/prometheus.ymlImage tags are examples; any Prometheus ≥ 3.8 supports scrape_native_histograms. Docker would create a missing bind-mount source as a directory, hence "write the file first".
From here on every lab queries these metrics for both services.
sum by (job, route, code) (rate(http_requests_total[5m])) shows both services at ~5 req/s.orders_created_total on localhost:8080/metrics grows with the loadgen.probe_success{job="blackbox-http"} is 1; native and classic p99 agree within a bucket.FAIL_RATE=0.2 on payments → sum(rate(http_requests_total{job="orders",code="502"}[5m])) / sum(rate(http_requests_total{job="orders"}[5m])) ≈ 0.2.release: kps label from the ServiceMonitor → after a minute the orders/payments targets vanish (gone, not DOWN). Restore it.route = r.URL.Path, rebuild, request /orders/o-1 … /orders/o-300 → count(http_requests_total{job="orders"}) grows by ~300; scrape_series_added spikes. Revert.always_scrape_classic_histograms, docker restart prom → no new _bucket samples.# local Prometheus only
docker rm -f prom blackbox
docker network rm obs-lab
kill %1 # port-forward
cd ..
# keep the metrics code and
# deploy/shop-monitor.yaml
kubectl -n shop set env \
deploy/payments FAIL_RATE=0Goal: answer rate, errors and duration questions for orders and payments from the command line. Needs Labs 0–2 (metrics scraped as job="orders" / job="payments").
kubectl -n monitoring port-forward svc/prometheus-operated 9090 &
# helper: instant query → "labels value"
q() {
curl -sG localhost:9090/api/v1/query \
--data-urlencode "query=$1" | jq -r '.data.result[]
| "\(.metric | del(.__name__) | tostring) \(.value[1])"'
}
# 0) both jobs scraped? (1 per pod)
q 'up{namespace="shop"}'
# 1) RATE: requests/s per job and route
q 'sum by (job, route) (
rate(http_requests_total{namespace="shop"}[5m]))'
# {"job":"orders","route":"/orders"} 4.9… (loadgen ≈ 5/s)
# 2) ERRORS: 5xx share per job (≈ 0 or empty for now)
q 'sum by (job) (rate(
http_requests_total{namespace="shop",code=~"5.."}[5m]))
/ sum by (job) (rate(
http_requests_total{namespace="shop"}[5m]))'# 3) DURATION: which buckets does the app expose?
q 'count by (le) (
http_request_duration_seconds_bucket{job="orders"})'
# 4) p99 per job and route
q 'histogram_quantile(0.99, sum by (job, route, le) (rate(
http_request_duration_seconds_bucket{namespace="shop"}[5m])))'
# 5) mean latency from _sum / _count
q 'sum by (job) (rate(
http_request_duration_seconds_sum{namespace="shop"}[5m]))
/ sum by (job) (rate(
http_request_duration_seconds_count{namespace="shop"}[5m]))'
# 6) business metric: orders created per minute
q 'sum(rate(orders_created_total[5m])) * 60'
# 7) week-over-week: empty, the cluster is younger
q 'sum(rate(http_requests_total{job="orders"}[5m] offset 1w))'Run each query in the Prometheus UI (Table tab) too and compare labels before and after each sum by. route values are route templates (/orders/{id}), not raw paths.
Copy node placement onto pod usage, count points in a range query, then use the payments knobs and watch the RED queries move.
# 8) JOIN: CPU cores per shop pod, plus its node
q 'sum by (namespace, pod) (rate(
container_cpu_usage_seconds_total{
namespace="shop", container!=""}[5m]))
* on (namespace, pod) group_left (node) kube_pod_info'
# {"namespace":"shop","node":"obs-worker","pod":"orders-…"} 0.003
# 9) memory per pod vs limit (empty if no limit is set)
q 'sum by (pod) (container_memory_working_set_bytes{
namespace="shop", container!=""})
/ sum by (pod) (kube_pod_container_resource_limits{
namespace="shop", resource="memory"})'
# 10) range query: points in the last 30 min at step 60s
S=$(( $(date +%s) - 1800 ))
curl -sG localhost:9090/api/v1/query_range \
--data-urlencode \
'query=sum(rate(http_requests_total{job="orders"}[5m]))' \
--data-urlencode "start=$S" \
--data-urlencode "end=$(date +%s)" \
--data-urlencode step=60 \
| jq '.data.result[0].values | length'
# 31# A) errors: 20% of charges fail
kubectl -n shop set env deploy/payments FAIL_RATE=0.2
kubectl -n shop rollout status deploy/payments
# wait 2–3 minutes, then
q 'sum by (job, code) (
rate(http_requests_total{namespace="shop"}[5m]))'
# payments 5xx ≈ 20% of its traffic;
# orders shows how the failure propagates upstream
# B) latency: +300 ms on every charge
kubectl -n shop set env deploy/payments \
FAIL_RATE=0 EXTRA_LATENCY_MS=300
q 'histogram_quantile(0.99, sum by (job, le) (rate(
http_request_duration_seconds_bucket{namespace="shop"}[5m])))'
# both p99 values rise by ≈ 0.3 s (orders waits for payments)
# C) presence checks
q 'absent(up{job="orders"})' # empty = good
q 'absent(up{job="does-not-exist"})' # {"job":"does-not-exist"} 1Two pods, one rollout restart: the correct query stays flat, the wrong one spikes.
# correct: rate per pod, then sum
GOOD='sum(rate(http_requests_total{job="orders"}[5m]))'
# wrong: sum raw counters, then rate (needs a subquery)
BAD='rate(sum(http_requests_total{job="orders"})[5m:15s])'
R=$(kubectl -n shop get deploy orders \
-o jsonpath='{.spec.replicas}')
kubectl -n shop scale deploy/orders --replicas=2
sleep 120
kubectl -n shop rollout restart deploy/orders
# watch both for ~5 minutes
for i in $(seq 10); do
echo "good=$(q "$GOOD" | awk '{print $2}')" \
"bad=$(q "$BAD" | awk '{print $2}')"
sleep 30
done
# good stays ≈ 5; bad jumps far above 5 after the
# restart: the summed counter dropped, rate() saw a reset# D) subquery: peak 5m rate in the last 30 minutes
q 'max_over_time(
sum(rate(http_requests_total{job="orders"}[5m]))[30m:1m])'
# resolution omitted → global evaluation_interval
q 'max_over_time(
sum(rate(http_requests_total{job="orders"}[5m]))[30m:])'
# reset everything
kubectl -n shop set env deploy/payments \
FAIL_RATE=0 EXTRA_LATENCY_MS=0
kubectl -n shop scale deploy/orders --replicas=$RAlso graph GOOD and BAD in the Graph tab over the last 15 minutes: the BAD spike is what a dashboard built that way shows after every deploy.
You should be able to explain every number you saw, including the wrong one.
orders /orders (the loadgen rate).FAIL_RATE=0.2, ≈ 20% of payments responses are 5xx within a few minutes.EXTRA_LATENCY_MS=300, p99 rises by ≈ 0.3 s; precision depends on the bucket containing it.node label.absent(up{job="orders"}) is empty while the app runs.le:** histogram_quantile(0.99, sum by (job) (rate(…_bucket[5m]))) → empty result (series without le are skipped). Explain why.rate(http_requests_total[15s]) with a 15s or 30s scrape → empty or flickering.rate(container_memory_working_set_bytes[5m]) returns numbers that mean nothing.* on (namespace) group_left kube_pod_info → "many-to-many matching not allowed".# knobs back to normal?
kubectl -n shop set env \
deploy/payments --list
# FAIL_RATE=0
# EXTRA_LATENCY_MS=0
kubectl -n shop get deploy
# keep everything for Lab 4:
# its recording rules use
# the same series
# stop the port-forward
# with Ctrl-CGoal: recording rules and alerts for the shop app in Git, proven by promtool before anything touches the cluster. Needs Labs 0–3.
mkdir -p rules && cat > rules/shop.rules.yml <<'EOF'
groups:
- name: shop-red
rules:
- record: job_route:http_requests:rate5m
expr: |
sum by (job, route) (
rate(http_requests_total{namespace="shop"}[5m]))
- record: job:http_requests:rate5m
expr: sum by (job) (job_route:http_requests:rate5m)
- record: job:http_request_errors:rate5m
expr: |
sum by (job) (rate(
http_requests_total{namespace="shop",code=~"5.."}[5m]))
- record: job:http_request_errors:ratio_rate5m
expr: job:http_request_errors:rate5m / job:http_requests:rate5m
- record: job:http_request_duration_seconds:p99_5m
expr: |
histogram_quantile(0.99, sum by (job, le) (rate(
http_request_duration_seconds_bucket{namespace="shop"}[5m])))
EOFcat >> rules/shop.rules.yml <<'EOF'
- name: shop-alerts
rules:
- alert: ShopHighErrorRate
expr: |
job:http_request_errors:ratio_rate5m > 0.05
and on (job) job:http_requests:rate5m > 0.5
for: 5m
labels: {severity: critical, team: shop}
annotations:
summary: "{{ $labels.job }}: {{ $value | humanizePercentage }} 5xx"
- alert: ShopHighLatencyP99
expr: job:http_request_duration_seconds:p99_5m > 0.5
for: 10m
labels: {severity: warning, team: shop}
annotations:
summary: "{{ $labels.job }} p99 {{ $value | humanizeDuration }}"
- alert: ShopJobAbsent
expr: absent(up{job="orders"})
for: 5m
labels: {severity: critical, team: shop, namespace: shop}
annotations: {summary: "No orders targets at all"}
EOF
promtool check rules rules/shop.rules.yml
# SUCCESS: 8 rules foundA 10% error ratio fires ShopHighErrorRate only after for: 5m (pending at 3m); a vanished target fires ShopJobAbsent.
cat > rules/shop.test.yml <<'EOF'
rule_files: [shop.rules.yml] # evaluation_interval defaults to 1m
tests:
- interval: 1m # 6 errors + 54 ok per minute = 10%, 1 req/s
input_series:
- series: 'http_requests_total{namespace="shop",job="orders",code="500"}'
values: '0+6x20'
- series: 'http_requests_total{namespace="shop",job="orders",code="200"}'
values: '0+54x20'
promql_expr_test:
- expr: job:http_request_errors:ratio_rate5m
eval_time: 10m
exp_samples:
- labels: 'job:http_request_errors:ratio_rate5m{job="orders"}'
value: 0.1
alert_rule_test:
- {eval_time: 3m, alertname: ShopHighErrorRate, exp_alerts: []}
- eval_time: 10m
alertname: ShopHighErrorRate
exp_alerts:
- exp_labels: {job: orders, severity: critical, team: shop}
exp_annotations: {summary: "orders: 10% 5xx"}
EOFcat >> rules/shop.test.yml <<'EOF'
- interval: 1m # target gone after 4 minutes
input_series:
- series: 'up{job="orders"}'
values: '1x4 _x15'
alert_rule_test:
- eval_time: 15m
alertname: ShopJobAbsent
exp_alerts:
- exp_labels:
job: orders
severity: critical
team: shop
namespace: shop
exp_annotations:
summary: No orders targets at all
EOF
cd rules && promtool test rules shop.test.yml && cd ..
# Unit Testing: shop.test.yml
# SUCCESS
# now make it fail on purpose: change value: 0.1 → 0.2,
# rerun, read the diff promtool prints, change it backWhy 15m in the second test: the last up sample is at 4m; with the 5m lookback, absent() becomes true just after 9m, then for: 5m → firing from ~14m.
Wrap the tested file in a PrometheusRule with label release: kps, and deploy a tiny service that prints every Alertmanager notification.
# generate the CR from the tested file (no hand copy)
mkdir -p deploy
{ cat <<'EOF'
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
name: shop-rules
namespace: shop
labels:
release: kps
spec:
EOF
sed 's/^/ /' rules/shop.rules.yml
} > deploy/shop-rules.yaml
kubectl apply -f deploy/shop-rules.yaml
kubectl apply -f deploy/alert-sink.yaml # written on the right
# loaded? (the config reload takes up to ~1 min)
kubectl -n monitoring port-forward svc/prometheus-operated 9090 &
sleep 60
curl -s localhost:9090/api/v1/rules | jq -r \
'.data.groups[] | select(.name|startswith("shop")) | .name'
# shop-red
# shop-alertscat > deploy/alert-sink.yaml <<'EOF'
apiVersion: apps/v1
kind: Deployment
metadata: {name: alert-sink, namespace: monitoring}
spec:
selector: {matchLabels: {app: alert-sink}}
template:
metadata: {labels: {app: alert-sink}}
spec:
containers:
- name: sink
image: python:3.12-alpine
command: ["python", "-u", "-c"]
args:
- |
import http.server as h, json
class H(h.BaseHTTPRequestHandler):
def do_POST(s):
b = json.loads(s.rfile.read(int(s.headers['Content-Length'])))
names = [a['labels']['alertname'] for a in b['alerts']]
print(s.path, b['status'], names)
s.send_response(200); s.end_headers()
h.HTTPServer(('', 8080), H).serve_forever()
---
apiVersion: v1
kind: Service
metadata: {name: alert-sink, namespace: monitoring}
spec: {selector: {app: alert-sink}, ports: [{port: 8080}]}
EOFCritical shop alerts go to /critical, other shop alerts to /warning, Watchdog is dropped, and a missing job inhibits the rest.
mkdir -p alertmanager && cat > alertmanager/alertmanager.yml <<'EOF'
route:
receiver: "null"
group_by: [alertname, job]
group_wait: 10s
group_interval: 1m
repeat_interval: 1h
routes:
- {receiver: "null", matchers: [alertname = "Watchdog"]}
- {receiver: shop-critical, matchers: [team="shop", severity="critical"]}
- {receiver: shop-warning, matchers: [team = "shop"]}
inhibit_rules:
- source_matchers: [alertname = "ShopJobAbsent"]
target_matchers: [alertname != "ShopJobAbsent"]
equal: [team]
receivers:
- name: "null"
- name: shop-critical
webhook_configs:
- url: http://alert-sink.monitoring.svc:8080/critical
- name: shop-warning
webhook_configs:
- url: http://alert-sink.monitoring.svc:8080/warning
EOFC=alertmanager/alertmanager.yml
amtool check-config $C
amtool config routes test --config.file=$C \
--verify.receivers=shop-critical team=shop severity=critical
amtool config routes test --config.file=$C \
team=shop severity=warning # shop-warning
# wrap the tested file as Helm values for kps
{ echo 'alertmanager:'; echo ' config:'
sed 's/^/ /' $C; } > deploy/am-values.yaml
# same chart version as Lab 0; keep the other values
helm upgrade kps prometheus-community/kube-prometheus-stack \
-n monitoring --reuse-values -f deploy/am-values.yaml \
--version $KPS_VER
kubectl -n monitoring port-forward svc/alertmanager-operated 9093 &
export AM=http://localhost:9093
# fire a real alert: 30% of charges fail
kubectl -n shop set env deploy/payments FAIL_RATE=0.3
kubectl -n monitoring logs -f deploy/alert-sink
# after ~6–7 minutes:
# /critical firing ['ShopHighErrorRate']Lists in Helm values replace the chart defaults instead of merging: this config replaces the chart's default routes, receivers and inhibit rules, which is why Watchdog and "null" are defined again here.
Follow one alert through every state, then break loading, muting and routing on purpose.
ShopHighErrorRate goes pending, then firing after 5 minutes.ALERTS{team="shop"} and read the alertstate label./critical firing ['ShopHighErrorRate'] once per group, not once per evaluation.FAIL_RATE=0: the sink prints resolved (webhooks send resolved by default).amtool alert query --alertmanager.url=$AM team=shop lists active shop alerts.kubectl -n shop label prometheusrule shop-rules release- → the shop groups disappear from /api/v1/rules, with no error anywhere. Put it back.amtool silence add --alertmanager.url=$AM --author=$USER --duration=15m --comment=lab job=payments → no notification while it holds.orders to 0 → after ~6 min ShopJobAbsent reaches /critical; other shop alerts show as inhibited in the UI. Scale back.severity = "critical" to "crit" and rerun the --verify.receivers test: it fails before deploy.kubectl -n shop set env \
deploy/payments FAIL_RATE=0
kubectl -n shop scale \
deploy/orders --replicas=2 # Lab 0 count
# keep shop-rules: ch 10 and
# Lab 13 use the recording rules
# remove only the sink
kubectl delete -f \
deploy/alert-sink.yaml
# stop port-forwards
kill %1 %2 2>/dev/nullGoal: see how the Operator selects objects and renders config, then prepare a target that exports 20,000 series.
# 1) what the Operator manages and what it selects
kubectl api-resources --api-group=monitoring.coreos.com
P=kps-kube-prometheus-stack-prometheus
kubectl -n monitoring get prometheus $P \
-o jsonpath='{.spec.podMonitorSelector}{"\n"}'
# {"matchLabels":{"release":"kps"}}
# 2) the config the Operator rendered (never edit it)
kubectl -n monitoring get secret prometheus-$P \
-o jsonpath='{.data.prometheus\.yaml\.gz}' \
| base64 -d | gunzip | grep job_name
# - job_name: serviceMonitor/shop/orders/0 ...
# 3) baseline (Prometheus port-forward on 9090)
curl -s localhost:9090/api/v1/query \
--data-urlencode 'query=prometheus_tsdb_head_series' \
| jq -r '.data.result[0].value[1]'
# note the number, e.g. 41000
# 4) write the bomb (right). It sets Content-Type
# explicitly: Prometheus 3 rejects scrapes without
# a valid one (unless fallback_scrape_protocol is set)# deploy/bomb.yaml — one label with 20,000 values
apiVersion: apps/v1
kind: Deployment
metadata: {name: bomb, namespace: shop, labels: {app: bomb}}
spec:
selector: {matchLabels: {app: bomb}}
template:
metadata: {labels: {app: bomb}}
spec:
containers:
- name: bomb
image: python:3.13-alpine
ports: [{name: metrics, containerPort: 9100}]
command: [python, -c]
args:
- |
import http.server
rows = [f'bomb_requests_total{{user_id="u{i}"}} 1'
for i in range(20000)]
body = ("\n".join(rows) + "\n").encode()
class H(http.server.BaseHTTPRequestHandler):
def do_GET(self):
self.send_response(200)
self.send_header("Content-Type",
"text/plain; version=0.0.4")
self.end_headers()
self.wfile.write(body)
http.server.HTTPServer(("", 9100), H).serve_forever()Goal: find the culprit with the cheap tools (TSDB status, scrape metrics), then compare a drop rule with a hard limit.
# PodMonitor (append to deploy/bomb.yaml)
cat >> deploy/bomb.yaml <<'EOF'
---
apiVersion: monitoring.coreos.com/v1
kind: PodMonitor
metadata: {name: bomb, namespace: shop, labels: {release: kps}}
spec:
selector: {matchLabels: {app: bomb}}
podMetricsEndpoints: [{port: metrics, interval: 30s}]
EOF
kubectl apply -f deploy/bomb.yaml
sleep 60 # repeat step 3: about +20,000
# top metrics and labels from the head block (cheap)
curl -s localhost:9090/api/v1/status/tsdb | jq '.data |
{top_metrics: .seriesCountByMetricName[:3],
top_labels: .labelValueCountByLabelName[:3]}'
# bomb_requests_total 20000 … / user_id 20000 …
# which target sends it (Prometheus UI)
topk(5, scrape_samples_post_metric_relabeling)
# {pod="bomb-…", namespace="shop", …} 20000# Control A — drop the metric after the scrape
kubectl -n shop patch podmonitor bomb --type=json -p '[
{"op":"add",
"path":"/spec/podMetricsEndpoints/0/metricRelabelings",
"value":[{"sourceLabels":["__name__"],
"regex":"bomb_.*","action":"drop"}]}]'
# after ~1 min:
# scrape_samples_scraped{pod=~"bomb-.*"} 20000
# scrape_samples_post_metric_relabeling{…} 0
# Control B — a hard limit instead of a drop rule
kubectl -n shop patch podmonitor bomb --type=json -p '[
{"op":"remove",
"path":"/spec/podMetricsEndpoints/0/metricRelabelings"},
{"op":"add","path":"/spec/sampleLimit","value":5000}]'
# after ~1 min:
# up{pod=~"bomb-.*"} 0
# increase(
# prometheus_target_scrapes_exceeded_sample_limit_total[5m])
# > 0; Target health: "sample limit exceeded" prometheus_tsdb_head_series falls only after the next head compaction (up to about 2–3 hours): dropped series stay in memory until then. scrape_samples_post_metric_relabeling shows the effect immediately.
Goal: produce the two classic container failures and find them with cAdvisor and kube-state-metrics queries.
# CPU: a busy loop with a 100m limit
kubectl -n shop create deploy burner \
--image=busybox:1.37 -- sh -c 'while :; do :; done'
kubectl -n shop set resources deploy/burner \
--requests=cpu=50m --limits=cpu=100m
# memory: allocate 64 MiB with a 32 MiB limit
kubectl -n shop run hog --image=python:3.13-alpine \
--restart=Always --overrides='{"spec":{"containers":[
{"name":"hog","image":"python:3.13-alpine",
"command":["python","-c",
"b=b\"x\"*(64*1024*1024)\nimport time; time.sleep(3600)"],
"resources":{"limits":{"memory":"32Mi"}}}]}}'
kubectl -n shop get pod hog -w
# OOMKilled → CrashLoopBackOff# throttling ratio (Prometheus UI, after ~5 min)
sum by (pod) (rate(container_cpu_cfs_throttled_periods_total{
pod=~"burner-.*"}[5m]))
/ sum by (pod) (rate(container_cpu_cfs_periods_total{
pod=~"burner-.*"}[5m]))
# → close to 1: throttled in almost every period
# CPU used: pinned at ~0.1 cores (the limit)
sum by (pod) (rate(container_cpu_usage_seconds_total{
pod=~"burner-.*", container!=""}[5m]))
# the OOMKill, as kube-state-metrics sees it
kube_pod_container_status_last_terminated_reason{pod="hog"}
# → {reason="OOMKilled", …} 1
increase(kube_pod_container_status_restarts_total{pod="hog"}[15m])
# bundled alerts that should fire in ~15–20 min (for: 15m):
# CPUThrottlingHigh (burner), KubePodCrashLooping (hog)Goal: a second Prometheus acts as "long-term storage"; kps sends it only the series you keep.
# a second Prometheus that accepts remote write
kubectl -n monitoring create deploy prom-receiver \
--image=quay.io/prometheus/prometheus:v3.5.3 \
--port=9090 -- /bin/prometheus \
--config.file=/etc/prometheus/prometheus.yml \
--storage.tsdb.path=/prometheus \
--web.enable-remote-write-receiver
kubectl -n monitoring expose deploy prom-receiver \
--port=9090
# kps sends only what we keep
cat > deploy/kps-remote-write.yaml <<'EOF'
prometheus:
prometheusSpec:
remoteWrite:
- name: receiver
url: http://prom-receiver.monitoring:9090/api/v1/write
writeRelabelConfigs:
- sourceLabels: [__name__]
regex: "up|http_requests_total"
action: keep
EOFhelm upgrade kps \
prometheus-community/kube-prometheus-stack \
-n monitoring --reuse-values --version $KPS_VER \
-f deploy/kps-remote-write.yaml
kubectl -n monitoring port-forward \
svc/prom-receiver 9091:9090 &
# on the receiver (localhost:9091):
count by (__name__) ({__name__=~".+", prometheus!=""})
# → up, http_requests_total (nothing else)
# on kps (localhost:9090): sending, and how far behind
rate(prometheus_remote_storage_samples_total[5m])
prometheus_remote_storage_highest_timestamp_in_seconds
- ignoring(remote_name, url) group_right
prometheus_remote_storage_queue_highest_sent_timestamp_seconds
# → a few seconds at mostUse the chart version you installed (helm list -n monitoring); --reuse-values keeps earlier values and adds these. The receiver has no volume: its data is lost when its pod restarts.
You found 20,000 series without an expensive query, compared two controls, and sent a curated subset off the cluster.
podMonitor/shop/bomb/0.bomb_requests_total and user_id at the top before Control A.burner near 1; hog shows OOMKilled and a growing restart count.prometheus and prometheus_replica.metricRelabelings: [{regex: user_id, action: labeldrop}] → 20,000 samples share one label set; rate(prometheus_target_scrapes_sample_duplicate_timestamp_total[5m]) > 0.release: kps from the bomb PodMonitor → its job disappears from the rendered config./api/v1/writ → 404, not retried: rate(prometheus_remote_storage_samples_failed_total[5m]) > 0, data dropped.writeRelabelConfigs → samples sent jump; on a paid backend, that is your bill.kubectl -n shop delete deploy bomb burner
kubectl -n shop delete podmonitor bomb
kubectl -n shop delete pod hog
kubectl -n monitoring delete \
deploy/prom-receiver svc/prom-receiver
# remove remote write again
cat > deploy/kps-no-rw.yaml <<'EOF'
prometheus:
prometheusSpec:
remoteWrite: []
EOF
helm upgrade kps \
prometheus-community/kube-prometheus-stack \
-n monitoring --reuse-values \
--version $KPS_VER \
-f deploy/kps-no-rw.yamlGoal: one package apps/internal/logx used by every service, so the shop log schema is enforced by code. It lives under apps/ because the Dockerfile copies only that directory.
mkdir -p apps/internal/logx
cat > apps/internal/logx/logx.go <<'EOF'
package logx
import (
"context"
"log/slog"
"os"
"strings"
"go.opentelemetry.io/otel/trace"
)
var Level slog.LevelVar // INFO unless LOG_LEVEL says otherwise
var sensitive = map[string]bool{"password": true, "token": true,
"authorization": true, "card_number": true, "cvv": true, "email": true}
func redact(_ []string, a slog.Attr) slog.Attr {
if sensitive[strings.ToLower(a.Key)] {
return slog.String(a.Key, "[REDACTED]")
}
return a
}
EOFcat >> apps/internal/logx/logx.go <<'EOF'
// New returns the service logger and makes it the default.
func New(service string) *slog.Logger {
_ = Level.UnmarshalText([]byte(os.Getenv("LOG_LEVEL")))
h := slog.NewJSONHandler(os.Stdout, &slog.HandlerOptions{
Level: &Level, ReplaceAttr: redact})
l := slog.New(traceHandler{h}).With("service", service)
slog.SetDefault(l)
return l
}
// traceHandler adds trace_id/span_id when ctx has a valid span
// (from Lab 8 on; until then the fields are simply absent).
type traceHandler struct{ slog.Handler }
func (h traceHandler) Handle(ctx context.Context, r slog.Record) error {
if sc := trace.SpanContextFromContext(ctx); sc.IsValid() {
r.AddAttrs(slog.String("trace_id", sc.TraceID().String()),
slog.String("span_id", sc.SpanID().String()))
}
return h.Handler.Handle(ctx, r)
}
EOFGoal: one wide request line per request with route, status and duration_ms; probes and scrapes only at DEBUG.
cat >> apps/internal/logx/logx.go <<'EOF'
func (h traceHandler) WithAttrs(a []slog.Attr) slog.Handler {
return traceHandler{h.Handler.WithAttrs(a)}
}
func (h traceHandler) WithGroup(n string) slog.Handler {
return traceHandler{h.Handler.WithGroup(n)}
}
EOF
cat > apps/internal/logx/middleware.go <<'EOF'
package logx
import (
"log/slog"
"net/http"
"time"
)
type recorder struct {
http.ResponseWriter
status int
}
func (r *recorder) WriteHeader(code int) {
r.status = code
r.ResponseWriter.WriteHeader(code)
}
EOF
go get go.opentelemetry.io/otel/trace@latest && go mod tidycat >> apps/internal/logx/middleware.go <<'EOF'
// Middleware must wrap the ServeMux directly (it reads r.Pattern).
func Middleware(log *slog.Logger, next http.Handler) http.Handler {
return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
start := time.Now()
rec := &recorder{ResponseWriter: w, status: http.StatusOK}
next.ServeHTTP(rec, r)
route := r.Pattern
if route == "" {
route = "unmatched"
}
lvl := slog.LevelInfo
switch {
case r.URL.Path == "/healthz" || r.URL.Path == "/metrics":
lvl = slog.LevelDebug
case rec.status >= 500:
lvl = slog.LevelError
case rec.status >= 400:
lvl = slog.LevelWarn
}
log.LogAttrs(r.Context(), lvl, "request",
slog.String("route", route),
slog.Int("status", rec.status),
slog.Int64("duration_ms",
time.Since(start).Milliseconds()))
})
}
EOF
go vet ./apps/internal/logxGoal: CI fails if a required key disappears or a sensitive key leaks.
cat > apps/internal/logx/logx_test.go <<'EOF'
package logx
import (
"bytes"
"encoding/json"
"log/slog"
"net/http"
"net/http/httptest"
"strings"
"testing"
)
func newTest(buf *bytes.Buffer) *slog.Logger {
h := slog.NewJSONHandler(buf,
&slog.HandlerOptions{ReplaceAttr: redact})
return slog.New(traceHandler{h}).With("service", "orders")
}
func TestRedaction(t *testing.T) {
var buf bytes.Buffer
newTest(&buf).Info("signup", "email", "a@example.com")
if strings.Contains(buf.String(), "a@example.com") {
t.Error("email leaked")
}
}
EOFcat >> apps/internal/logx/logx_test.go <<'EOF'
func TestRequestSchema(t *testing.T) {
var buf bytes.Buffer
mux := http.NewServeMux()
mux.HandleFunc("POST /orders",
func(w http.ResponseWriter, _ *http.Request) {
w.WriteHeader(http.StatusBadGateway)
})
Middleware(newTest(&buf), mux).ServeHTTP(
httptest.NewRecorder(),
httptest.NewRequest("POST", "/orders", nil))
var m map[string]any
if err := json.Unmarshal(buf.Bytes(), &m); err != nil {
t.Fatal(err)
}
for _, k := range []string{"time", "level", "msg",
"service", "route", "status", "duration_ms"} {
if _, ok := m[k]; !ok {
t.Errorf("missing key %q", k)
}
}
if m["route"] != "POST /orders" || m["level"] != "ERROR" {
t.Errorf("route=%v level=%v", m["route"], m["level"])
}
}
EOF
go test ./apps/internal/logx/ # okGoal: both services log the shared schema; business events carry order_id; every call passes the request context.
# apps/orders/main.go — edit these lines
import "obs-lab/apps/internal/logx" // add to imports
log = logx.New("orders") // was: slog.New(slog.NewJSONHandler(…))
// in create(): use the request context everywhere
log.ErrorContext(r.Context(), "payment failed",
"order_id", o.ID, "err", err)
// new, after the order is stored:
log.InfoContext(r.Context(), "order created",
"order_id", o.ID, "amount", o.Amount)
// in main(): the middleware wraps the mux DIRECTLY
err := http.ListenAndServe(":8080",
logx.Middleware(log, mux))
// a Lab 2 metrics wrapper stays outside logx.Middleware
# apps/payments/main.go — same pattern
log := logx.New("payments")
log.ErrorContext(r.Context(), "charge failed",
"order_id", req.OrderID)
log.InfoContext(r.Context(), "charged",
"order_id", req.OrderID, "amount", req.Amount)gofmt -l . ; go vet ./... # remove unused imports if reported
for a in orders payments; do
docker build --build-arg APP=$a -t shop/$a:lab .
kind load docker-image shop/$a:lab --name obs
kubectl -n shop rollout restart deploy/$a
done
kubectl -n shop rollout status deploy/orders
# request lines, typed fields
kubectl -n shop logs deploy/orders --since=1m \
| jq -c 'select(.msg=="request")
| {level, service, route, status, duration_ms}' \
| head -2
# {"level":"INFO","service":"orders",
# "route":"POST /orders","status":201,"duration_ms":64}
# business events have order_id
kubectl -n shop logs deploy/orders --since=1m \
| jq -c 'select(.msg=="order created")
| {order_id, amount}' | head -2
# top talkers by message
kubectl -n shop logs deploy/orders --since=5m \
| jq -r .msg | sort | uniq -c | sort -rnkubectl logs deploy/orders reads one pod of the Deployment; use -l app=orders --prefix for all replicas. trace_id stays absent until Lab 8 adds the OpenTelemetry SDK.
The logs are now ready for Loki (Lab 7) and for trace correlation (Lab 8).
go test ./apps/internal/logx/ passes.time, level, msg, service; request lines add route, status (a number), duration_ms.route is always a template (GET /orders/{id}), never /orders/o-123./healthz lines at INFO: probes are DEBUG now.FAIL_RATE=0.3 on payments: orders logs ERROR request lines with status 502 and payment failed lines with order_id.kubectl -n shop set env deploy/orders LOG_LEVEL=DEBUG → probe and scrape lines appear; compare --since=1m | wc -l before and after."customer_email", "a@example.com" in create → not redacted: key-based redaction only knows email.fmt.Println("starting") to main → jq stops with a parse error. Tolerant reading: jq -R 'fromjson? // {raw: .}'.http.TimeoutHandler (it clones the request) between logx.Middleware and the mux → route becomes unmatched.# keep logx: Labs 7–9 use it
# undo the break-it changes
kubectl -n shop set env \
deploy/orders LOG_LEVEL-
kubectl -n shop set env \
deploy/payments FAIL_RATE=0
# remove customer_email,
# fmt.Println and TimeoutHandler
# from the code, then rebuild,
# kind load and rollout restartGoal: one Loki replica with filesystem storage, TSDB v13 schema, 3-day retention and a ruler that talks to the kps Alertmanager. Caches, gateway and canary are off to save memory.
cat > deploy/loki-values.yaml <<'EOF'
deploymentMode: Monolithic # chart < 12.0.0: SingleBinary
loki:
auth_enabled: false # one tenant, named "fake"
commonConfig: {replication_factor: 1}
storage: {type: filesystem}
schemaConfig:
configs:
- {from: "2024-04-01", store: tsdb, schema: v13,
object_store: filesystem,
index: {prefix: index_, period: 24h}}
limits_config: {retention_period: 72h}
compactor:
retention_enabled: true
delete_request_store: filesystem
rulerConfig:
storage: {type: local, local: {directory: /rules}}
rule_path: /var/loki/rules-temp
alertmanager_url: http://alertmanager-operated.monitoring:9093
enable_api: true
singleBinary: {replicas: 1}
sidecar: {rules: {folder: /rules/fake}} # rules of tenant "fake"
gateway: {enabled: false}
chunksCache: {enabled: false}
resultsCache: {enabled: false}
lokiCanary: {enabled: false}
test: {enabled: false}
EOFhelm repo add grafana-community \
https://grafana-community.github.io/helm-charts
helm repo add grafana https://grafana.github.io/helm-charts
helm repo update
# key names move between chart majors: check yours
helm show values grafana-community/loki \
| grep -n -E '^deploymentMode|^singleBinary:|^sidecar:'
helm upgrade --install loki grafana-community/loki \
-n logging --create-namespace -f deploy/loki-values.yaml
kubectl -n logging get pods
# loki-0 2/2 Running (loki + rules sidecar)
kubectl -n logging port-forward svc/loki 3100 &
curl -s localhost:3100/ready
# ready (first ~15 s: "Ingester not ready: waiting …")
curl -s localhost:3100/loki/api/v1/status/buildinfo \
| jq -r .version
# 3.7.x
curl -s localhost:3100/loki/api/v1/labels
# {"status":"success"} (nothing ships logs yet)Goal: each Alloy pod tails only the pods on its own node, parses the shop JSON, labels level and keeps trace_id as structured metadata.
cat > deploy/config.alloy <<'EOF'
discovery.kubernetes "pods" {
role = "pod"
selectors {
role = "pod"
field = "spec.nodeName=" +
coalesce(sys.env("HOSTNAME"), constants.hostname)
}
}
discovery.relabel "pods" {
targets = discovery.kubernetes.pods.targets
rule {
source_labels = ["__meta_kubernetes_namespace"]
target_label = "namespace"
}
rule {
source_labels = ["__meta_kubernetes_pod_name"]
target_label = "pod"
}
rule {
source_labels = ["__meta_kubernetes_pod_container_name"]
target_label = "container"
}
rule {
source_labels = ["__meta_kubernetes_pod_label_app"]
target_label = "app"
}
}loki.source.kubernetes "pods" {
targets = discovery.relabel.pods.output
forward_to = [loki.process.shop.receiver]
}
loki.process "shop" {
stage.match {
selector = "{namespace=\"shop\"}"
stage.json {
expressions = {level = "level", trace_id = "trace_id", ts = "time"}
}
stage.labels {
values = {level = ""}
}
stage.structured_metadata {
values = {trace_id = ""}
}
stage.timestamp {
source = "ts"
format = "RFC3339Nano"
}
}
forward_to = [loki.write.default.receiver]
}
loki.write "default" {
endpoint {
url = "http://loki.logging:3100/loki/api/v1/push"
}
}
EOFGoal: logs flowing into Loki, Loki visible in Grafana, and payments failing 30 % of the time so there is something to find.
# config in its own ConfigMap: Helm never templates {{ }}
kubectl -n logging create configmap alloy-config \
--from-file=config.alloy=deploy/config.alloy
helm upgrade --install alloy grafana/alloy -n logging \
--set alloy.configMap.create=false \
--set alloy.configMap.name=alloy-config \
--set alloy.configMap.key=config.alloy
kubectl -n logging rollout status ds/alloy
# later edits: create … --dry-run=client -o yaml | kubectl apply -f -
# (the chart's config reloader picks them up)
# component graph and health: http://localhost:12345
kubectl -n logging port-forward ds/alloy 12345 &
cat > deploy/kps-loki.yaml <<'EOF'
grafana:
additionalDataSources:
- {name: Loki, type: loki, uid: loki,
url: "http://loki.logging.svc:3100"}
EOF
helm upgrade kps prometheus-community/kube-prometheus-stack \
-n monitoring --reuse-values --version $KPS_VER -f deploy/kps-loki.yaml
kubectl -n shop set env deploy/payments FAIL_RATE=0.3# terminal client (or download it from the Loki releases)
brew install logcli
export LOKI_ADDR=http://localhost:3100
logcli labels
# app
# container
# level
# namespace
# pod
# service_name (added by Loki, from app)
logcli series --analyze-labels '{namespace="shop"}'
# Total Streams: ~10 (pods × levels)
# Unique Labels: 6
# Label Name Unique Values Found In Streams
# pod 5 …
# level 3 …
logcli query --since=10m --limit=3 -o raw \
'{app="orders"} |= "payment failed"' | jq -c .
# {"time":"…","level":"ERROR","msg":"payment failed",
# "service":"orders","order_id":"o-412",…}Lab 9 replaces additionalDataSources with a list holding this Loki entry plus Tempo; keep uid: loki so links keep working.
Goal: run each query in Grafana → Explore → Loki (code mode) and explain its output before moving on.
# 1) failed payments, newest first
{namespace="shop", app="orders"} |= "payment failed"
# 2) request lines, typed filter, reshaped
{app="orders"} |= `"msg":"request"` | json
| status >= 500
| line_format "{{.route}} {{.status}} {{.duration_ms}}ms"
# POST /orders 502 71ms
# 3) level is already a label → the parsed one is renamed
{app="orders"} | json | keep level, level_extracted
# 4) find one order across both services
{namespace="shop"} |= "o-412" | json
| line_format "{{.service}} {{.msg}}"
# payments charge failed
# orders payment failed# 5) error lines per second per app
sum by (app) (
rate({namespace="shop", level="ERROR"}[5m]))
# 6) share of 5xx requests (logs-as-RED)
sum(count_over_time({app="orders"}
|= `"msg":"request"` | json | status >= 500 [5m]))
/
sum(count_over_time({app="orders"}
|= `"msg":"request"` [5m]))
# ≈ 0.3
# 7) p95 request duration per route, from logs
quantile_over_time(0.95, {app="orders"}
|= `"msg":"request"` | json | __error__=""
| unwrap duration_ms [5m]) by (route)
# Query inspector → Stats: compare "bytes processed"
# for query 6 with and without the |= line filterGoal: a LogQL alerting rule delivered by the rules sidecar, evaluated by Loki, and visible in the kps Alertmanager.
cat > deploy/loki-rules.yaml <<'EOF'
apiVersion: v1
kind: ConfigMap
metadata:
name: shop-log-rules
namespace: logging
labels: {loki_rule: "true"} # watched by the sidecar
data:
shop.yaml: |
groups:
- name: shop-logs
interval: 1m
rules:
- alert: OrdersPaymentErrors
expr: |
sum(count_over_time(
{namespace="shop", app="orders", level="ERROR"}
|= "payment failed" [5m])) > 10
for: 1m
labels: {severity: warning}
annotations:
summary: "orders: {{ $value }} payment failures in 5m"
- alert: OrdersLogsMissing
expr: absent_over_time({namespace="shop", app="orders"}[10m])
for: 5m
labels: {severity: warning}
EOF
kubectl apply -f deploy/loki-rules.yamlkubectl -n logging exec loki-0 -c loki -- ls /rules/fake
# shop.yaml
# loaded by the ruler (re-sync within ~1 min)
curl -s localhost:3100/loki/api/v1/rules | head -3
# shop.yaml: (namespace = file name)
# - name: shop-logs
# interval: 1m
# firing after ~2–3 min with FAIL_RATE=0.3
curl -s localhost:3100/prometheus/api/v1/alerts \
| jq -r '.data.alerts[] | "\(.labels.alertname) \(.state)"'
# OrdersPaymentErrors firing
# in Alertmanager (port-forward 9093 from Lab 0)
amtool --alertmanager.url=http://localhost:9093 \
alert query alertname=OrdersPaymentErrors
# troubleshooting
kubectl -n logging logs loki-0 -c loki-sc-rules | tail
kubectl -n logging logs loki-0 -c loki | grep -i rulerLabs 8–10 build on this: keep Loki and Alloy running.
logcli labels lists namespace, app, container, pod, level, service_name — no order_id, no trace_id.OrdersPaymentErrors fires in Alertmanager; with FAIL_RATE=0 it resolves within ~6 min.:12345) shows every component healthy.kube-system lines arrive too, unparsed (outside stage.match).order_id = "order_id" to stage.json and order_id = "" to stage.labels → logcli series --analyze-labels '{app="orders"}' shows hundreds of streams within minutes. Revert it.{app=~".*"} is rejected: a selector needs one matcher that cannot match empty.{app="orders"} | logfmt | status >= 500 → no results; | __error__!="" shows why.kubectl -n logging scale sts/loki --replicas=0 for 1 min → Alloy logs push errors and retries; scale back to 1 and look for lines from the gap.# keep Loki + Alloy for Labs 8–10
kubectl -n shop set env \
deploy/payments FAIL_RATE=0
# full removal (after Lab 10):
kubectl delete -f \
deploy/loki-rules.yaml
helm uninstall alloy -n logging
helm uninstall loki -n logging
kubectl -n logging delete pvc \
--all
kubectl delete ns logging
# and drop the Loki entry from
# grafana.additionalDataSourcesGoal: a Collector in otel that accepts OTLP and writes every span to its own log (debug exporter). Tempo replaces the debug exporter in Lab 9.
cat > deploy/otel-gateway.yaml <<'EOF'
mode: deployment
fullnameOverride: otel-gateway # Service otel-gateway.otel
replicaCount: 1
image:
repository: otel/opentelemetry-collector-contrib
# tag defaults to the chart's appVersion
config:
# merged into the chart defaults, which already define
# receivers.otlp (4317 gRPC, 4318 HTTP), memory_limiter,
# batch, exporters.debug and the health_check extension
exporters:
debug: {verbosity: detailed} # print every span
service:
pipelines:
traces:
receivers: [otlp] # lists replace: drops jaeger, zipkin
EOF
helm repo add open-telemetry \
https://open-telemetry.github.io/opentelemetry-helm-charts
helm repo update
helm upgrade --install otel-gateway \
open-telemetry/opentelemetry-collector \
-n otel --create-namespace -f deploy/otel-gateway.yaml
kubectl -n otel rollout status deploy/otel-gatewaykubectl -n otel get svc otel-gateway
# otel-gateway ClusterIP … …,4317/TCP,4318/TCP,…
kubectl -n otel logs deploy/otel-gateway | grep -i ready
# … Everything is ready. Begin running and processing data.
# smoke test: one span by hand over OTLP/HTTP + JSON
cat > deploy/span.json <<'EOF'
{"resourceSpans": [{
"resource": {"attributes": [{"key": "service.name",
"value": {"stringValue": "curl"}}]},
"scopeSpans": [{"spans": [{
"traceId": "5b8efff798038103d269b633813fc60c",
"spanId": "eee19b7ec3c1b174", "name": "smoke", "kind": 2,
"startTimeUnixNano": "1762560000000000000",
"endTimeUnixNano": "1762560000250000000"}]}]}]}
EOF
kubectl -n otel port-forward svc/otel-gateway 4318 &
curl -s localhost:4318/v1/traces \
-H 'Content-Type: application/json' -d @deploy/span.json
# {"partialSuccess":{}} (HTTP 200)
kubectl -n otel logs deploy/otel-gateway \
| grep -B4 -A1 'Name *: smoke'
# Trace ID : 5b8efff798038103d269b633813fc60c
# Name : smoke
# Kind : ServerGoal: a shared package (like logx in Lab 6), configured entirely from OTEL_* variables.
mkdir -p apps/internal/otelx && cat > apps/internal/otelx/setup.go <<'EOF'
package otelx
import (
"context"
"go.opentelemetry.io/otel"
"go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc"
"go.opentelemetry.io/otel/propagation"
"go.opentelemetry.io/otel/sdk/resource"
sdktrace "go.opentelemetry.io/otel/sdk/trace"
)
// Setup registers a TracerProvider and the W3C propagators.
func Setup(ctx context.Context) (func(context.Context) error, error) {
res, err := resource.New(ctx,
resource.WithFromEnv(), resource.WithTelemetrySDK())
if err != nil {
return nil, err
}
exp, err := otlptracegrpc.New(ctx) // OTEL_EXPORTER_OTLP_ENDPOINT
if err != nil {
return nil, err
}
tp := sdktrace.NewTracerProvider(
sdktrace.WithBatcher(exp), sdktrace.WithResource(res))
otel.SetTracerProvider(tp)
otel.SetTextMapPropagator(propagation.NewCompositeTextMapPropagator(
propagation.TraceContext{}, propagation.Baggage{}))
return tp.Shutdown, nil
}
EOFresource.WithFromEnv() reads OTEL_SERVICE_NAME and OTEL_RESOURCE_ATTRIBUTES (set in step 5).otlptracegrpc.New reads OTEL_EXPORTER_OTLP_ENDPOINT; the http:// scheme means plaintext gRPC.WithBatcher: spans are queued and sent in batches every 5 s, off the request path.ParentBased(AlwaysSample): every trace is kept for now; Lab 9 samples in the Collector instead.traceparent and baggage.Shutdown flushes the queue; step 4 calls it on SIGTERM.Goal: one wrapper outside the handler chain, one inside it, and a client that propagates context.
cat > apps/internal/otelx/http.go <<'EOF'
package otelx
import (
"net/http"
"strings"
"time"
"go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp"
"go.opentelemetry.io/otel/attribute"
"go.opentelemetry.io/otel/trace"
)
func Handler(h http.Handler, service string) http.Handler {
return otelhttp.NewHandler(h, service,
otelhttp.WithFilter(func(r *http.Request) bool {
return r.URL.Path != "/healthz" && r.URL.Path != "/metrics"
}))
}
func RouteSpan(next http.Handler) http.Handler {
return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
next.ServeHTTP(w, r)
if _, route, ok := strings.Cut(r.Pattern, " "); ok {
span := trace.SpanFromContext(r.Context())
span.SetName(r.Pattern)
span.SetAttributes(attribute.String("http.route", route))
}
})
}
var Client = &http.Client{Timeout: 2 * time.Second,
Transport: otelhttp.NewTransport(http.DefaultTransport)}
EOFHandler is the outermost wrapper; RouteSpan wraps the mux, innermost."orders" is only the fallback name.r.Pattern while routing; RouteSpan runs after it and renames the span POST /orders with http.route=/orders — low-cardinality names (ch 8, section 31).r.Pattern; doing it here keeps the result independent of the version you get.WithFilter returning false skips tracing for probes and scrapes.Client wraps the default transport: one CLIENT span per request, traceparent injected — but only for requests built with a context.Goal: otelhttp outside the Lab 2 metrics and Lab 6 logging middleware, so both see the span; the call to payments carries the context; spans are flushed on SIGTERM.
go get go.opentelemetry.io/otel go.opentelemetry.io/otel/sdk \
go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc \
go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp
go mod tidy
// apps/orders/main.go — imports: "context", "os/signal", "syscall",
// "obs-lab/apps/internal/otelx"
client = otelx.Client // in the var block, was &http.Client{…}
// create(): the call to payments must carry the request context
req, _ := http.NewRequestWithContext(r.Context(), "POST",
payURL, bytes.NewReader(body))
req.Header.Set("Content-Type", "application/json")
resp, err := client.Do(req) // was client.Post(…)
// main(): before building the mux
ctx, stop := signal.NotifyContext(context.Background(),
os.Interrupt, syscall.SIGTERM)
defer stop()
shutdown, err := otelx.Setup(ctx)
if err != nil {
log.Error("otel setup failed", "err", err)
os.Exit(1)
}// main(): replace the old ListenAndServe block
h := otelx.Handler(
metrics.Middleware(logx.Middleware(log, otelx.RouteSpan(mux))),
"orders")
srv := &http.Server{Addr: ":8080", Handler: h}
go func() {
<-ctx.Done() // SIGTERM from Kubernetes
srv.Shutdown(context.Background())
}()
log.Info("orders listening", "addr", ":8080")
err = srv.ListenAndServe() // http.ErrServerClosed on SIGTERM
log.Info("server stopped", "err", err)
c, cancel := context.WithTimeout(context.Background(),
5*time.Second)
defer cancel()
shutdown(c) // flush the last batch of spans
// apps/payments/main.go: the same main() changes, with
// "payments" as the name. No client change: it calls nobody.
// optional, in payments charge(): business detail on the span
trace.SpanFromContext(r.Context()).SetAttributes(
attribute.String("shop.order.id", req.OrderID))Order matters: otelx.Handler(metrics(logx(RouteSpan(mux)))). Lab 6 traceHandler and Lab 9 exemplars read the span from the request context, so they must run inside otelhttp.
Goal: the same trace ID on orders' SERVER and CLIENT spans and on payments' SERVER span, and in the request logs.
gofmt -l . ; go vet ./...
for a in orders payments; do
docker build --build-arg APP=$a -t shop/$a:lab .
kind load docker-image shop/$a:lab --name obs
# set env changes the pod template → new pods, new image
kubectl -n shop set env deploy/$a OTEL_SERVICE_NAME=$a \
OTEL_RESOURCE_ATTRIBUTES=deployment.environment.name=lab \
OTEL_EXPORTER_OTLP_ENDPOINT=http://otel-gateway.otel:4317
done
# after later code changes: rollout restart (Lab 0 convention)
kubectl -n shop rollout status deploy/orders
kubectl -n shop rollout status deploy/payments
# loadgen keeps calling orders; add some failures
kubectl -n shop set env deploy/payments FAIL_RATE=0.2
# request logs now carry the trace (Lab 6 traceHandler)
kubectl -n shop logs deploy/orders --since=1m \
| jq -c 'select(.msg=="request")
| {route, status, trace_id}' | head -2
# {"route":"POST /orders","status":201,
# "trace_id":"7c3f9d0e5b1a4c2f8e6d3b2a1f0e9a1e"}kubectl -n otel logs deploy/otel-gateway --since=1m \
| grep -E 'service.name|Trace ID|Parent ID|^ +ID|Name +:|Kind' \
| head -18
# one request, three spans (IDs shortened):
# -> service.name: Str(payments)
# Trace ID : 7c3f…9a1e
# Parent ID : 5d2b…01c4 ← orders' CLIENT span
# ID : 9e40…77aa
# Name : POST /charge
# Kind : Server
# -> service.name: Str(orders)
# Trace ID : 7c3f…9a1e
# Parent ID : 1a8f…e302 ← orders' SERVER span
# ID : 5d2b…01c4
# Name : POST (older otelhttp: HTTP POST)
# Kind : Client
# Trace ID : 7c3f…9a1e
# Parent ID : (root)
# ID : 1a8f…e302
# Name : POST /orders
# Kind : Server
# full detail of one trace: attributes, status, resource
kubectl -n otel logs deploy/otel-gateway --since=1m \
| grep -A40 'Trace ID *: 7c3f' | lessRun each break, find the symptom in the debug output or the app logs, explain it, then revert.
service.name, deployment.environment.name=lab, telemetry.sdk.language=go.GET /healthz or GET /metrics.kubectl -n shop rollout restart deploy/orders → no export errors in the old pods' last logs.SetTextMapPropagator in setup.go → payments spans get new trace IDs and an empty Parent ID.client.Post(…) → orders' CLIENT span becomes the root of a new trace; payments joins that one, not the order's.OTEL_EXPORTER_OTLP_ENDPOINT=http://otel-gateway.otel:4318 → export errors in the orders logs every few seconds; requests still return 201.kubectl -n otel scale deploy/otel-gateway --replicas=0 → the app keeps serving; spans are dropped once the SDK queue (2048) is full.# revert the breaks, then:
kubectl -n otel scale \
deploy/otel-gateway --replicas=1
kubectl -n shop set env \
deploy/payments FAIL_RATE=0
# keep otel-gateway and the SDK:
# Lab 9 adds Tempo, tail
# sampling and span metrics
# full removal (after Lab 9):
helm uninstall otel-gateway -n otel
kubectl delete ns otel
kubectl -n shop set env \
deploy/orders deploy/payments \
OTEL_EXPORTER_OTLP_ENDPOINT-Goal: traces from the shop app land in Tempo (monolithic, local storage), and Grafana can jump between metrics, traces and logs.
# 1) Tempo, monolithic mode (no Kafka needed)
helm repo add grafana-community \
https://grafana-community.github.io/helm-charts
helm repo update # check keys/appVersion:
helm show values grafana-community/tempo | less
cat > deploy/tempo-values.yaml <<'EOF'
tempo:
storage:
trace:
backend: local
receivers:
otlp:
protocols:
grpc: {endpoint: 0.0.0.0:4317}
http: {endpoint: 0.0.0.0:4318}
EOF
helm upgrade --install tempo grafana-community/tempo \
-n tracing --create-namespace -f deploy/tempo-values.yaml
kubectl -n tracing rollout status statefulset/tempo
kubectl -n tracing port-forward svc/tempo 3200 &
curl -s localhost:3200/ready # → ready
# 2) kps: exemplars, remote write receiver, data
# sources (file on the right), then:
helm upgrade kps prometheus-community/kube-prometheus-stack \
-n monitoring --reuse-values --version $KPS_VER -f deploy/kps-tracing.yaml# deploy/kps-tracing.yaml — the list below replaces
# Lab 7's; keep Lab 7's Loki uid/URL if they differ
prometheus:
prometheusSpec:
enableFeatures: [exemplar-storage]
enableRemoteWriteReceiver: true
grafana:
sidecar:
datasources:
exemplarTraceIdDestinations:
{datasourceUid: tempo, traceIdLabelName: trace_id}
additionalDataSources:
- name: Tempo
type: tempo
uid: tempo
url: http://tempo.tracing.svc:3200
jsonData:
tracesToLogsV2: {datasourceUid: loki,
filterByTraceID: true,
spanStartTimeShift: -5m, spanEndTimeShift: 5m}
serviceMap: {datasourceUid: prometheus}
- name: Loki
type: loki
uid: loki
url: http://loki.logging.svc:3100
jsonData:
derivedFields:
- {name: TraceID, datasourceUid: tempo,
matcherRegex: '"trace_id":"(\w+)"',
url: '$${__value.raw}'}Goal: RED metrics from 100 % of spans; Tempo keeps errors, slow traces and a 10 % baseline.
config: # deploy/gateway-tracing.yaml
connectors:
spanmetrics:
histogram: {explicit: {buckets:
[5ms, 25ms, 100ms, 250ms, 500ms, 1s, 2.5s]}}
dimensions: [{name: http.route}]
servicegraph: {}
forward/sample: {}
processors:
tail_sampling:
decision_wait: 10s
num_traces: 20000
policies:
- {name: errors, type: status_code,
status_code: {status_codes: [ERROR]}}
- {name: slow, type: latency,
latency: {threshold_ms: 500}}
- {name: baseline, type: probabilistic,
probabilistic: {sampling_percentage: 10}}
exporters:
otlp/tempo: {endpoint: "tempo.tracing.svc:4317",
tls: {insecure: true}}
prometheusremotewrite:
endpoint: http://prometheus-operated.monitoring.svc:9090/api/v1/write service:
pipelines:
traces:
receivers: [otlp]
processors: [memory_limiter]
exporters: [spanmetrics, servicegraph, forward/sample]
traces/sampled:
receivers: [forward/sample]
processors: [tail_sampling, batch]
exporters: [otlp/tempo]
metrics/derived:
receivers: [spanmetrics, servicegraph]
processors: [batch]
exporters: [prometheusremotewrite]
# apply (keeps image, mode and the rest of Lab 8's values)
helm upgrade otel-gateway open-telemetry/opentelemetry-collector \
-n otel --reuse-values -f deploy/gateway-tracing.yaml
kubectl -n otel logs -l app.kubernetes.io/instance=otel-gateway \
| grep -i -E 'error|Everything is ready'
# keep exactly ONE gateway replica: tail sampling needs all
# spans of a trace on the same instance (ch 9, section 34)
kubectl -n otel get deploy -l app.kubernetes.io/instance=otel-gatewayLists in Helm values replace, maps merge: the traces pipeline above replaces Lab 8's (which exported to debug). memory_limiter and batch come from the chart's default config.
Goal: latency samples carry the trace ID of a request, so Grafana shows clickable dots on the p95 graph.
// apps/orders/metrics.go — inside the otelhttp-wrapped handler
import (
"github.com/prometheus/client_golang/prometheus"
"github.com/prometheus/client_golang/prometheus/promhttp"
"go.opentelemetry.io/otel/trace"
)
func observe(r *http.Request, route string, d time.Duration) {
h := reqDuration.WithLabelValues(r.Method, route)
sc := trace.SpanContextFromContext(r.Context())
if sc.IsSampled() {
h.(prometheus.ExemplarObserver).ObserveWithExemplar(
d.Seconds(),
prometheus.Labels{"trace_id": sc.TraceID().String()})
return
}
h.Observe(d.Seconds())
}
// exemplars are only served in OpenMetrics format
mux.Handle("/metrics", promhttp.HandlerFor(reg,
promhttp.HandlerOpts{EnableOpenMetrics: true}))# rebuild and roll out orders
docker build --build-arg APP=orders -t shop/orders:lab . \
&& kind load docker-image shop/orders:lab --name obs \
&& kubectl -n shop rollout restart deploy/orders
# exemplars are exposed?
kubectl -n shop port-forward svc/orders 8080 &
curl -s -H 'Accept: application/openmetrics-text' \
localhost:8080/metrics | grep -m2 'trace_id'
# http_request_duration_seconds_bucket{...,le="0.1"} 812 \
# # {trace_id="4bf92f35..."} 0.043 1.762e+09
# inject errors and latency in payments
kubectl -n shop set env deploy/payments \
FAIL_RATE=0.2 EXTRA_LATENCY_MS=400
# Grafana: localhost:3000 (port-forward svc/kps-grafana 3000:80)
# Explore → Prometheus, Exemplars on:
histogram_quantile(0.95, sum by (le) (
rate(http_request_duration_seconds_bucket{job="orders"}[5m])))Use TraceQL, exemplars and the log link together. Each break shows one rule from chapter 9.
{ resource.service.name = "orders" } >> { resource.service.name = "payments" && status = error } returns traces.payments span.sum by (status_code) (rate(traces_span_metrics_calls_total{service_name="payments"}[5m])) shows ~20 % errors.tail_sampling into the first traces pipeline → span-metric error ratio jumps far above 20 %.1s with EXTRA_LATENCY_MS=2000 → traces arrive in pieces, root spans missing.EnableOpenMetrics → /metrics shows no exemplars and the dots vanish.# reset the fault knobs
kubectl -n shop set env deploy/payments \
FAIL_RATE=0 EXTRA_LATENCY_MS=0
# keep Tempo + gateway for
# Labs 10 and 13. Remove Tempo:
helm uninstall tempo -n tracing
kubectl delete namespace tracing
# then point the gateway back to debug:
# re-apply Lab 8's values
# (without gateway-tracing.yaml)Goal: dashboards are code. One Python script renders three dashboards from templates; nothing is built by hand in the UI.
# dashboards/gen.py — RED per service + one USE dashboard
import json, pathlib
DS = {"type": "prometheus", "uid": "prometheus"} # kps default uid
OUT = pathlib.Path(__file__).parent / "out"
OUT.mkdir(exist_ok=True)
RI = "[$__rate_interval]"
def ts(title, expr, unit, x, y, legend="{{route}}"):
return {"type": "timeseries", "title": title, "datasource": DS,
"gridPos": {"x": x, "y": y, "w": 8, "h": 8},
"fieldConfig": {"defaults": {"unit": unit}},
"targets": [{"refId": "A", "expr": expr,
"legendFormat": legend}]}
def stat(title, expr, unit, x, red_at):
steps = [{"color": "green", "value": None},
{"color": "red", "value": red_at}]
return {"type": "stat", "title": title, "datasource": DS,
"gridPos": {"x": x, "y": 8, "w": 8, "h": 4},
"fieldConfig": {"defaults": {"unit": unit, "thresholds":
{"mode": "absolute", "steps": steps}}},
"targets": [{"refId": "A", "expr": expr}]}def red(svc):
s = f'job="{svc}", route=~"$route"'
rate = f'sum by (route) (rate(http_requests_total{{{s}}}{RI}))'
err = (f'sum by (route) (rate(http_requests_total'
f'{{{s}, code=~"5.."}}{RI})) / {rate}')
p95 = ('histogram_quantile(0.95, sum by (le, route) (rate('
f'http_request_duration_seconds_bucket{{{s}}}{RI})))')
tot_err = (f'sum(rate(http_requests_total{{{s}, code=~"5.."}}{RI}))'
f' / sum(rate(http_requests_total{{{s}}}{RI}))')
return {
"uid": f"red-{svc}", "title": f"Shop / {svc} RED",
"tags": ["shop", "red"], "time": {"from": "now-1h", "to": "now"},
"templating": {"list": [{
"name": "route", "type": "query", "datasource": DS,
"query": f'label_values(http_requests_total{{job="{svc}"}}, route)',
"multi": True, "includeAll": True, "refresh": 2}]},
"panels": [ts("Requests/s", rate, "reqps", 0, 0),
ts("Error ratio", err, "percentunit", 8, 0),
ts("p95 latency", p95, "s", 16, 0),
stat("Error ratio (all routes)", tot_err,
"percentunit", 0, 0.05)]}
# use() is on the next slide
for name, d in [("red-orders", red("orders")),
("red-payments", red("payments")),
("use-shop", use())]:
(OUT / f"{name}.json").write_text(json.dumps(d, indent=1))The stat threshold (red at 5 %) should match the error-ratio alert from Lab 4.
Goal: a USE dashboard for the shop pods, and all three dashboards loaded from labelled ConfigMaps.
# dashboards/gen.py (continued; define before the loop)
def use():
ns = 'namespace="shop", container!=""'
p = "{{pod}}"
cpu = ('sum by (pod) (rate('
f'container_cpu_usage_seconds_total{{{ns}}}{RI}))')
thr = ('sum by (pod) (rate(container_cpu_cfs_throttled_periods_total'
f'{{{ns}}}{RI})) / sum by (pod) (rate('
f'container_cpu_cfs_periods_total{{{ns}}}{RI}))')
mem = f'sum by (pod) (container_memory_working_set_bytes{{{ns}}})'
rst = ('sum by (pod) (increase(kube_pod_container_status_'
'restarts_total{namespace="shop"}[1h]))')
return {"uid": "use-shop", "title": "Shop / USE (pods)",
"tags": ["shop", "use"],
"time": {"from": "now-1h", "to": "now"},
"panels": [
ts("CPU cores used (U)", cpu, "short", 0, 0, p),
ts("CPU throttled ratio (S)", thr, "percentunit", 8, 0, p),
ts("Memory working set (U)", mem, "bytes", 16, 0, p),
ts("Restarts in 1h (E)", rst, "short", 0, 8, p)]}# render and load
python3 dashboards/gen.py && ls dashboards/out
# red-orders.json red-payments.json use-shop.json
for f in dashboards/out/*.json; do
n=$(basename "$f" .json)
kubectl -n monitoring create configmap "dash-$n" \
--from-file="$n.json=$f" --dry-run=client -o yaml \
| kubectl label --local -f - grafana_dashboard=1 -o yaml \
| kubectl apply -f -
done
# check through the Grafana API
kubectl -n monitoring port-forward svc/kps-grafana 3000:80 &
PW=$(kubectl -n monitoring get secret kps-grafana \
-o jsonpath='{.data.admin-password}' | base64 -d)
curl -s -u "admin:$PW" 'localhost:3000/api/search?tag=shop' \
| jq -r '.[].title'
# Shop / orders RED
# Shop / payments RED
# Shop / USE (pods)
# commit gen.py and out/*.json; review diffs in PRsThe throttling panel stays empty while shop pods have no CPU limit: CFS throttling only happens under a quota. The break step below adds one.
Inject each fault and confirm the right panel moves. If a panel does not react, the panel is wrong.
kubectl -n shop set env deploy/payments FAIL_RATE=0.2 → orders "Error ratio" rises on /orders; the stat turns red above 5 %.EXTRA_LATENCY_MS=400 → orders p95 rises by roughly 0.4 s on /orders only.$route = one route: every RED panel follows the variable.orders/payments pod are drawn; restarts stay at 0.kubectl -n shop set resources deploy/orders --limits=cpu=50m → throttled ratio climbs, p95 rises, CPU flat at 0.05 cores.gen.py instead.kubectl -n monitoring label cm dash-use-shop grafana_dashboard- → the dashboard disappears.histogram_quantile(0.95, rate(...)) without sum by (le, route) → one line per pod and instance; compare.# reset faults and limits
kubectl -n shop set env deploy/payments \
FAIL_RATE=0 EXTRA_LATENCY_MS=0
# remove the CPU limit: re-apply the
# original manifest
kubectl apply -f deploy/shop.yaml
# delete ONLY the lab ConfigMaps by name.
# Never: delete cm -l grafana_dashboard=1
# — that also deletes kps's built-in
# dashboards.
kubectl -n monitoring delete configmap \
dash-red-orders dash-red-payments \
dash-use-shopGoal: CloudWatch metrics from structured logs, then alarms and a dashboard on them. Everything runs in ap-northeast-1.
# app.py — one EMF line per request, slog-style fields (ch 6)
import json, os, random, time, uuid
FAIL = float(os.environ.get("FAIL_RATE", "0"))
def handler(event, context):
t0, route = time.time(), event.get("route", "POST /orders")
fail = random.random() < FAIL
time.sleep(random.uniform(0.02, 0.12) + (0.4 if fail else 0))
ms = round((time.time() - t0) * 1000, 1)
status = 502 if fail else 201
print(json.dumps({"_aws": {
"Timestamp": int(time.time() * 1000),
"CloudWatchMetrics": [{"Namespace": "ObsLab",
"Dimensions": [["service", "route"]],
"Metrics": [{"Name": "Latency", "Unit": "Milliseconds"},
{"Name": "Errors", "Unit": "Count"},
{"Name": "Requests", "Unit": "Count"}]}]},
"service": "orders", "route": route,
"Latency": ms, "Errors": int(fail), "Requests": 1,
"level": "ERROR" if fail else "INFO",
"msg": "payment failed" if fail else "order created",
"order_id": str(uuid.uuid4()), "status": status,
"duration_ms": ms}))
return {"status": status}export AWS_REGION=ap-northeast-1
ACCT=$(aws sts get-caller-identity --query Account --output text)
# log group first, with retention (otherwise Lambda creates
# /aws/lambda/<fn> that never expires)
aws logs create-log-group --log-group-name /obs-lab/orders
aws logs put-retention-policy --log-group-name /obs-lab/orders \
--retention-in-days 7
# execution role: logs + X-Ray (active tracing, used in Lab 12)
aws iam create-role --role-name obs-lab-orders-role \
--assume-role-policy-document '{"Version":"2012-10-17",
"Statement":[{"Effect":"Allow","Action":"sts:AssumeRole",
"Principal":{"Service":"lambda.amazonaws.com"}}]}'
P=arn:aws:iam::aws:policy
aws iam attach-role-policy --role-name obs-lab-orders-role \
--policy-arn $P/service-role/AWSLambdaBasicExecutionRole
aws iam attach-role-policy --role-name obs-lab-orders-role \
--policy-arn $P/AWSXRayDaemonWriteAccessStep 2: ~10 minutes of traffic, then a metric filter and a static M-of-N alarm. LogFormat=Text keeps each print line intact, which EMF needs.
zip fn.zip app.py
aws lambda create-function --function-name obs-lab-orders \
--runtime python3.13 --handler app.handler \
--zip-file fileb://fn.zip --timeout 10 \
--role arn:aws:iam::$ACCT:role/obs-lab-orders-role \
--environment 'Variables={FAIL_RATE=0.05}' \
--logging-config LogFormat=Text,LogGroup=/obs-lab/orders \
--tracing-config Mode=Active
# "role cannot be assumed"? a new role needs a few seconds: retry
# traffic: second terminal, leave it running
while true; do
aws lambda invoke --function-name obs-lab-orders \
--cli-binary-format raw-in-base64-out \
--payload '{"route":"POST /orders"}' /dev/null >/dev/null
done
# after a few minutes
aws cloudwatch list-metrics --namespace ObsLab \
--query 'Metrics[].[MetricName,Dimensions[].Value]'# metric filter: count ERROR events
aws logs put-metric-filter --log-group-name /obs-lab/orders \
--filter-name orders-error-logs \
--filter-pattern '{ $.level = "ERROR" }' \
--metric-transformations metricName=ErrorLogs,\
metricNamespace=ObsLab,metricValue=1,defaultValue=0
TOPIC=$(aws sns create-topic --name obs-lab-alerts \
--query TopicArn --output text)
aws sns subscribe --topic-arn "$TOPIC" --protocol email \
--notification-endpoint you@example.com # confirm the mail
# static: >= 3 errors/min in 3 of 5 minutes (no actions)
aws cloudwatch put-metric-alarm \
--alarm-name obs-lab-orders-errors \
--namespace ObsLab --metric-name Errors \
--dimensions Name=service,Value=orders \
"Name=route,Value=POST /orders" \
--statistic Sum --period 60 --evaluation-periods 5 \
--datapoints-to-alarm 3 --threshold 3 \
--comparison-operator GreaterThanOrEqualToThreshold \
--treat-missing-data notBreachingStep 3: an anomaly band on p90 latency, a composite alarm that is the only one that notifies, and a SQL query over the metrics.
cat > anom.json <<'EOF'
[{"Id": "m1", "ReturnData": true, "MetricStat": {
"Metric": {"Namespace": "ObsLab", "MetricName": "Latency",
"Dimensions": [{"Name": "service", "Value": "orders"},
{"Name": "route", "Value": "POST /orders"}]},
"Period": 60, "Stat": "p90"}},
{"Id": "band", "Expression": "ANOMALY_DETECTION_BAND(m1, 2)"}]
EOF
aws cloudwatch put-metric-alarm \
--alarm-name obs-lab-orders-latency-anomaly \
--evaluation-periods 3 --datapoints-to-alarm 3 \
--comparison-operator GreaterThanUpperThreshold \
--threshold-metric-id band --treat-missing-data notBreaching \
--metrics file://anom.json
aws cloudwatch put-composite-alarm \
--alarm-name obs-lab-orders-page \
--alarm-rule 'ALARM("obs-lab-orders-errors") AND
ALARM("obs-lab-orders-latency-anomaly")' \
--alarm-actions "$TOPIC" --ok-actions "$TOPIC"# top routes by errors (Metrics Insights)
Q="SELECT SUM(Errors) FROM SCHEMA(ObsLab, route, service) \
GROUP BY route ORDER BY SUM() DESC LIMIT 5"
jq -n --arg q "$Q" \
'[{Id: "q1", Period: 300, Expression: $q}]' > mi.json
aws cloudwatch get-metric-data \
--metric-data-queries file://mi.json \
--start-time $(date -u -v-30M +%FT%TZ) \
--end-time $(date -u +%FT%TZ) \
--query 'MetricDataResults[].[Label,Values[0]]'
# Linux: date -u -d '-30 min' +%FT%TZ
# a second route = new custom metrics (the ch 11 cost lesson)
aws lambda invoke --function-name obs-lab-orders \
--cli-binary-format raw-in-base64-out \
--payload '{"route":"GET /orders/{id}"}' /dev/null
# run the query again: two rows
# the anomaly model needs history: with minutes of data the
# band is rough; in production give it days to weeksStep 4: latency percentiles, requests and errors, recent error logs and the composite alarm on one page, deployed from a file.
cat > dash.json <<EOF
{"widgets": [
{"type":"metric","x":0,"y":0,"width":12,"height":6,
"properties":{"title":"latency p50 / p90",
"region":"ap-northeast-1","view":"timeSeries","period":60,
"metrics":[["ObsLab","Latency","service","orders",
"route","POST /orders",{"stat":"p50"}],
["...",{"stat":"p90"}]]}},
{"type":"metric","x":12,"y":0,"width":12,"height":6,
"properties":{"title":"requests and errors / min",
"region":"ap-northeast-1","stat":"Sum","period":60,
"metrics":[["ObsLab","Requests","service","orders",
"route","POST /orders"],
[".","Errors",".",".",".","."]]}},
{"type":"log","x":0,"y":6,"width":16,"height":6,
"properties":{"title":"recent errors",
"region":"ap-northeast-1","view":"table",
"query":"SOURCE '/obs-lab/orders' | filter level = 'ERROR' | limit 20"}},
{"type":"alarm","x":16,"y":6,"width":8,"height":6,
"properties":{"title":"page","alarms":[
"arn:aws:cloudwatch:ap-northeast-1:$ACCT:alarm:obs-lab-orders-page"]}}
]}
EOFaws cloudwatch put-dashboard --dashboard-name obs-lab \
--dashboard-body file://dash.json
# {"DashboardValidationMessages": []} ← empty = valid
# read it back (what IaC would store)
aws cloudwatch get-dashboard --dashboard-name obs-lab \
--query DashboardBody --output text | jq .
# metrics array shorthand
# "." repeats the value at the same position
# in the row above
# "..." repeats everything up to the options object
# heredoc without quotes: $ACCT is expanded in dash.json
# console: CloudWatch → Dashboards → obs-lab
# - set the time range to 1 h
# - hover the p90 line, compare with the anomaly band
# on the alarm's own graphUse the knobs to drive each alarm through its states and watch which ones notify.
list-metrics shows Latency, Errors, Requests with service and route, plus ErrorLogs.FAIL_RATE=0.05.describe-alarms --alarm-name-prefix obs-lab → OK after a few minutes (the anomaly alarm can stay INSUFFICIENT_DATA until its model has data).set-alarm-state --alarm-name obs-lab-orders-page --state-value ALARM --state-reason drill → exactly one email.update-function-configuration --environment "Variables={FAIL_RATE=0.5}" → errors alarm, then the anomaly alarm (failed calls are slower), then one composite email.Value=order and default --treat-missing-data missing → INSUFFICIENT_DATA forever (with notBreaching it sits in OK and never pages); find it with describe-alarms --state-value INSUFFICIENT_DATA."route" from the JSON root → logs arrive, metrics stop.--treat-missing-data breaching → it alarms on no data.# stop the traffic loop, then
aws cloudwatch delete-alarms \
--alarm-names obs-lab-orders-page
aws cloudwatch delete-alarms \
--alarm-names obs-lab-orders-errors \
obs-lab-orders-latency-anomaly
aws cloudwatch delete-dashboards \
--dashboard-names obs-lab
aws logs delete-metric-filter \
--log-group-name /obs-lab/orders \
--filter-name orders-error-logs
aws sns delete-topic \
--topic-arn "$TOPIC"
# keep Lambda, role, log group for
# Lab 12 (removed there). Custom
# metrics cannot be deleted; they
# expire after 15 months.Goal: answer the same questions in Logs Insights QL, PPL and SQL against /obs-lab/orders. Restart the Lab 11 traffic loop for ~10 minutes first.
export AWS_REGION=ap-northeast-1
END=$(date +%s); START=$((END - 3600))
run() { # $1 = CWLI | PPL, $2 = query string
QID=$(aws logs start-query --query-language "$1" \
--log-group-names /obs-lab/orders \
--start-time $START --end-time $END \
--query-string "$2" --query queryId --output text)
until [ "$(aws logs get-query-results --query-id $QID \
--query status --output text)" = Complete ]; do
sleep 2; done
aws logs get-query-results --query-id $QID \
--query results --output text
}
# Logs Insights QL: latency percentiles per route
run CWLI 'filter ispresent(duration_ms)
| stats pct(duration_ms, 50) as p50,
pct(duration_ms, 95) as p95, count(*) as n by route'
# PPL: errors by route
run PPL 'fields route, status | where status >= 500
| stats count() as errors by route'# SQL: the log group is named in FROM, not in a flag
QID=$(aws logs start-query --query-language SQL \
--start-time $START --end-time $END \
--query-string "SELECT route, COUNT(*) AS n,
AVG(duration_ms) AS avg_ms
FROM \`/obs-lab/orders\`
GROUP BY route ORDER BY n DESC" \
--query queryId --output text)
sleep 5; aws logs get-query-results --query-id $QID
# recurring error messages (QL only)
run CWLI 'filter level = "ERROR" | pattern msg'
# Lambda platform lines: cold starts and memory
run CWLI 'filter @type = "REPORT"
| stats count(*) as inv, max(@maxMemoryUsed) as mem,
sum(ispresent(@initDuration)) as cold by bin(15m)'
# bytes scanned = what the last query cost
aws logs get-query-results --query-id $QID \
--query statisticsExpected: p50 ≈ 70 ms for POST /orders; p95 near 120 ms or 450+ ms (5% of calls fail slowly); PPL errors ≈ 5% of n; QL and SQL agree on n.
Step 2: stream errors as they happen, then prove that a data protection policy masks an email address in every read path.
ACCT=$(aws sts get-caller-identity \
--query Account --output text)
LG=arn:aws:logs:ap-northeast-1:$ACCT:\
log-group:/obs-lab/orders
# Live Tail: errors only (traffic must run)
aws logs start-live-tail \
--log-group-identifiers $LG \
--log-event-filter-pattern \
'{ $.level = "ERROR" }'
# Ctrl-C after a few events:
# billed per minute of session
# simpler, scriptable alternative
aws logs tail /obs-lab/orders --follow \
--filter-pattern '{ $.status >= 500 }' \
--format shortcat > dp.json <<'EOF'
{"Name": "obs-lab-pii", "Version": "2021-06-01",
"Statement": [
{"Sid": "audit", "DataIdentifier": [
"arn:aws:dataprotection::aws:data-identifier/EmailAddress"],
"Operation": {"Audit": {"FindingsDestination": {}}}},
{"Sid": "redact", "DataIdentifier": [
"arn:aws:dataprotection::aws:data-identifier/EmailAddress"],
"Operation": {"Deidentify": {"MaskConfig": {}}}}]}
EOF
aws logs put-data-protection-policy \
--log-group-identifier /obs-lab/orders \
--policy-document file://dp.json
# masking only applies to events ingested
# after the policy existsStep 3: write one bad log line by hand and read it back; then create an IA log group and hit its feature limits.
aws logs create-log-stream \
--log-group-name /obs-lab/orders \
--log-stream-name manual
M='{\"level\":\"INFO\",\"msg\":\"receipt sent\",'
M+='\"email\":\"alice@example.com\"}'
cat > ev.json <<EOF
[{"timestamp": $(date +%s)000, "message": "$M"}]
EOF
aws logs put-log-events \
--log-group-name /obs-lab/orders \
--log-stream-name manual \
--log-events file://ev.json
aws logs tail /obs-lab/orders \
--log-stream-names manual --since 5m
# … "email":"*****************"
# with logs:Unmask, in Logs Insights:
# filter msg = "receipt sent"
# | fields unmask(@message)# Infrequent Access: cheaper ingestion, fewer features
aws logs create-log-group \
--log-group-name /obs-lab/orders-ia \
--log-group-class INFREQUENT_ACCESS
aws logs describe-log-groups \
--log-group-name-prefix /obs-lab/orders-ia \
--query 'logGroups[0].logGroupClass'
# "INFREQUENT_ACCESS"
# not supported on this class → expect an error
aws logs put-metric-filter \
--log-group-name /obs-lab/orders-ia \
--filter-name x --filter-pattern ERROR \
--metric-transformations \
metricName=X,metricNamespace=ObsLab,metricValue=1
# supported: Logs Insights (QL, PPL, SQL)
# the class can never be changed: pick it at creationStep 4: store spans in CloudWatch Logs, then send OTLP spans from a local contrib Collector signed with SigV4. Spans take a few minutes to become searchable.
cat > ts.json <<EOF
{"Version":"2012-10-17","Statement":[{
"Sid":"TransactionSearchXRayAccess","Effect":"Allow",
"Principal":{"Service":"xray.amazonaws.com"},
"Action":"logs:PutLogEvents",
"Resource":[
"arn:aws:logs:ap-northeast-1:$ACCT:log-group:aws/spans:*",
"arn:aws:logs:ap-northeast-1:$ACCT:log-group:\
/aws/application-signals/data:*"],
"Condition":{"ArnLike":{"aws:SourceArn":
"arn:aws:xray:ap-northeast-1:$ACCT:*"},
"StringEquals":{"aws:SourceAccount":"$ACCT"}}}]}
EOF
aws logs put-resource-policy \
--policy-name TransactionSearchXRay \
--policy-document file://ts.json
aws xray update-trace-segment-destination \
--destination CloudWatchLogs
aws xray get-trace-segment-destination # wait: ACTIVE
# Lambda active tracing (Lab 11) also lands here
for i in 1 2 3; do aws lambda invoke \
--function-name obs-lab-orders /dev/null; done# col.yaml
extensions:
sigv4auth: {region: ap-northeast-1, service: xray}
receivers:
otlp: {protocols: {grpc: {endpoint: 0.0.0.0:4317}}}
exporters:
otlphttp:
traces_endpoint:
https://xray.ap-northeast-1.amazonaws.com/v1/traces
auth: {authenticator: sigv4auth}
service:
extensions: [sigv4auth]
pipelines:
traces: {receivers: [otlp], exporters: [otlphttp]}
# profile creds → container (needs xray:PutTraceSegments,
# e.g. AWSXrayWriteOnlyAccess)
eval "$(aws configure export-credentials --format env)"
docker run -d --name obs-col -p 4317:4317 \
-e AWS_ACCESS_KEY_ID -e AWS_SECRET_ACCESS_KEY \
-e AWS_SESSION_TOKEN \
-v $PWD/col.yaml:/etc/otelcol-contrib/config.yaml \
otel/opentelemetry-collector-contrib:<version>
# 20 synthetic traces, service.name = orders
docker run --rm ghcr.io/open-telemetry/\
opentelemetry-collector-contrib/telemetrygen:<version> \
traces --otlp-insecure --traces 20 --service orders \
--otlp-endpoint host.docker.internal:4317Step 5: spans are logs now, so Logs Insights works on them. Then (optional, small cost) send the kind cluster's shop metrics to an AMP workspace.
# spans are structured logs in aws/spans
run2() { aws logs start-query --log-group-names aws/spans \
--start-time $START --end-time $(date +%s) \
--query-string "$1" --query queryId --output text; }
QID=$(run2 'fields @timestamp, traceId, name, durationNano
| sort durationNano desc | limit 10')
sleep 5; aws logs get-query-results --query-id $QID
# console: Application Signals → Transaction Search
# AMP workspace + a remote-write-only lab user
WS=$(aws amp create-workspace --alias obs-lab \
--query workspaceId --output text)
aws iam create-user --user-name obs-lab-amp-writer
aws iam attach-user-policy \
--user-name obs-lab-amp-writer --policy-arn \
arn:aws:iam::aws:policy/AmazonPrometheusRemoteWriteAccess
aws iam create-access-key --user-name obs-lab-amp-writer
kubectl -n monitoring create secret generic amp-writer \
--from-literal=accessKey=<AccessKeyId> \
--from-literal=secretKey=<SecretAccessKey># amp-values.yaml for kps: keep only the shop jobs
prometheus:
prometheusSpec:
remoteWrite:
- url: <AMP_URL>/api/v1/remote_write
sigv4:
region: ap-northeast-1
accessKey: {name: amp-writer, key: accessKey}
secretKey: {name: amp-writer, key: secretKey}
writeRelabelConfigs:
- sourceLabels: [job]
regex: orders|payments
action: keep
# <AMP_URL> = https://aps-workspaces.ap-northeast-1
# .amazonaws.com/workspaces/<WS> (one string)
helm upgrade kps prometheus-community/kube-prometheus-stack \
-n monitoring --reuse-values --version $KPS_VER -f amp-values.yaml
# in Prometheus: samples flow, none fail
# rate(prometheus_remote_storage_samples_total[5m])
# rate(prometheus_remote_storage_samples_failed_total[5m])Static keys are a lab-only shortcut for a cluster outside AWS. On EKS use Pod Identity or IRSA for Prometheus, or an AMP managed scraper — no keys at all.
Each break produces a specific symptom: learn to recognise it.
ERROR lines.CloudWatchLogs / ACTIVE; orders spans appear in Transaction Search and aws/spans.up{job="orders"}.--log-group-names** → rejected: SQL names groups in FROM./obs-lab/orders-ia; queries work.XRay** → the Collector logs export errors from the traces endpoint (spans rejected).service: logs** in sigv4auth for traces → 403 signature errors (docker logs obs-col).keep** → remote-write samples jump: that is the bill.docker rm -f obs-col
aws xray update-trace-segment-destination \
--destination XRay
aws logs delete-resource-policy \
--policy-name TransactionSearchXRay
for g in aws/spans /obs-lab/orders-ia \
/obs-lab/orders; do
aws logs delete-log-group \
--log-group-name $g; done
aws lambda delete-function \
--function-name obs-lab-orders
R=obs-lab-orders-role
P=arn:aws:iam::aws:policy
for p in service-role/\
AWSLambdaBasicExecutionRole \
AWSXRayDaemonWriteAccess; do
aws iam detach-role-policy \
--role-name $R --policy-arn $P/$p
done
aws iam delete-role --role-name $R
# AMP: delete-workspace, drop
# remoteWrite from kps, delete the
# access key and the IAM userGoal: replay the ch 13 incident on the obs cluster. Needs Labs 2–10 running (kps, Loki + Alloy, otel-gateway, Tempo, RED dashboard).
# 0) everything up?
for ns in monitoring logging tracing otel shop; do
kubectl -n $ns get pods --no-headers | grep -v Running; done
# (no output from the loop = all Running)
# 1) port-forwards (one terminal each, or with &)
M="kubectl -n monitoring port-forward"
$M svc/prometheus-operated 9090 &
$M svc/alertmanager-operated 9093 &
$M svc/kps-grafana 3000:80 &
# 2) game-day latency rule (right): short "for" for a quick lab
kubectl apply -f rules/gameday.yaml
kubectl -n monitoring get prometheusrule shop-gameday
# 3) baseline p95 per route, before the change
promtool query instant http://localhost:9090 \
'histogram_quantile(0.95, sum by (le, route) (rate(
http_request_duration_seconds_bucket{job="orders"}[5m])))'
# 4) the "incident": a config-only change, same image
kubectl -n shop set env deploy/payments EXTRA_LATENCY_MS=800
kubectl -n shop rollout status deploy/payments
date # write down the time: this is your change event# rules/gameday.yaml
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
name: shop-gameday
namespace: monitoring
labels: {release: kps} # required by kps default selectors
spec:
groups:
- name: shop-gameday
rules:
- alert: ShopHighLatencyP95
expr: |
histogram_quantile(0.95, sum by (le) (rate(
http_request_duration_seconds_bucket{job="orders"}[5m]))) > 0.5
for: 2m
labels: {severity: critical, service: orders, team: shop}
annotations:
summary: 'orders p95 {{ $value | humanizeDuration }}'The 5-minute rate window plus for: 2m means the alert needs roughly 4–7 minutes to fire. Use the wait to predict which panels will move.
Follow the loop: alert → RED dashboard → exemplar → trace → logs → change. Write each hypothesis down before you run its query.
# 1) the alert
amtool --alertmanager.url=http://localhost:9093 \
alert query alertname=ShopHighLatencyP95
# also: Prometheus UI → Alerts (pending → firing)
# 2) RED dashboard (Lab 10), service=orders:
# rate flat? errors flat? which route is slow?
# then service=payments: is its server p95 high too?
# 3) rule out throttling of orders (USE)
promtool query instant http://localhost:9090 \
'sum by (pod) (rate(container_cpu_cfs_throttled_periods_total{
namespace="shop", container="orders"}[5m]))'
# 4) exemplar → trace: Grafana Explore, Prometheus,
# duration query, toggle "Exemplars", click a
# diamond in the slow band → open the trace
# No diamonds? is exemplar storage enabled?
kubectl -n monitoring get prometheus \
-o jsonpath='{.items[0].spec.enableFeatures}'# 5) TraceQL in Explore → Tempo: all pods or one?
{ resource.service.name = "payments" && kind = server
&& duration > 500ms }
{ resource.service.name = "payments" && kind = server }
| quantile_over_time(duration, .95)
by (resource.k8s.pod.name)
# 6) logs for the trace (Explore → Loki)
{namespace="shop"} | json | trace_id="<id from step 4>"
# 7) what changed?
kubectl -n shop rollout history deploy/payments
kubectl -n shop get deploy payments -o \
jsonpath='{.spec.template.spec.containers[0].env}'
# 8) mitigate, verify on the SAME panel (~5 min)
kubectl -n shop rollout undo deploy/payments
# 9) round 2: errors instead of latency
kubectl -n shop set env deploy/payments FAIL_RATE=0.2
# → Lab 4's ShopHighErrorRate (after ~6 min); Tempo: status = error
kubectl -n shop set env deploy/payments FAIL_RATE-If resource.k8s.pod.name is empty, the k8sattributes processor is not adding it in otel-gateway (ch 9): group by resource.service.name instead and fix it after the game day.
Make silence visible: the always-firing Watchdog must reach a receiver that would notice if it stopped.
# 1) Watchdog is always firing in kps — confirm
amtool --alertmanager.url=http://localhost:9093 \
alert query alertname=Watchdog
# 2) where does it go today? (kps default: "null")
amtool --alertmanager.url=http://localhost:9093 \
config routes test alertname=Watchdog
# 3) a stand-in heartbeat endpoint
kubectl -n monitoring create deployment heartbeat \
--image=mccutchen/go-httpbin --port=8080
kubectl -n monitoring expose deployment heartbeat \
--port=80 --target-port=8080
# 4) route Watchdog to it (config on the right)
helm upgrade kps \
prometheus-community/kube-prometheus-stack \
-n monitoring --reuse-values --version $KPS_VER \
-f deploy/am-values.yaml
# 5) check: routes and notification counter
amtool --alertmanager.url=http://localhost:9093 \
config routes test alertname=Watchdog # heartbeat# alertmanager/alertmanager.yml (Lab 4): Watchdog
# → heartbeat (go-httpbin stand-in) instead of "null"
route:
...
routes:
- receiver: heartbeat
matchers: [alertname = "Watchdog"]
repeat_interval: 1m # keep pinging
- {receiver: shop-critical, ...} # unchanged
- {receiver: shop-warning, ...} # unchanged
receivers:
- name: "null"
- name: heartbeat
webhook_configs:
- url: http://heartbeat.monitoring.svc/anything
# ... shop-critical and shop-warning unchanged
# test, then re-wrap exactly as in Lab 4
C=alertmanager/alertmanager.yml
amtool check-config $C
amtool config routes test --config.file=$C \
alertname=Watchdog # heartbeat
{ echo 'alertmanager:'; echo ' config:'
sed 's/^/ /' $C; } > deploy/am-values.yamlAlways change the one Alertmanager config file from Lab 4 and re-wrap it: a separate values file with its own alertmanager.config would replace Lab 4's routes and receivers (Helm replaces lists).
Catch a missing target with absent(), then find the biggest metrics and drop what nobody uses.
# 1) meta-alerts from ch 13 (shop-meta, release: kps)
kubectl apply -f rules/shop-meta.yaml
# 2) test: no orders pods → ShopTargetMissing after 5m
kubectl -n shop get deploy orders \
-o jsonpath='{.spec.replicas}'; echo # remember N
kubectl -n shop scale deploy/orders --replicas=0
amtool --alertmanager.url=http://localhost:9093 \
alert query alertname=ShopTargetMissing
kubectl -n shop scale deploy/orders --replicas=<N>
# 3) cardinality: biggest metric names
promtool query instant http://localhost:9090 \
'topk(10, count by (__name__) ({__name__=~".+"}))'
curl -s localhost:9090/api/v1/status/tsdb \
| jq '.data.seriesCountByMetricName[:5]'# 4) series per shop job, and go_gc_* among them
promtool query instant http://localhost:9090 \
'count by (job) ({job=~"orders|payments"})'
promtool query instant http://localhost:9090 \
'count({__name__=~"go_gc_.*", job=~"orders|payments"})'
# 5) drop them at scrape time (ServiceMonitor shop, Lab 2)
kubectl -n shop get servicemonitor
kubectl -n shop patch servicemonitor shop \
--type=json -p='[{"op":"add",
"path":"/spec/endpoints/0/metricRelabelings",
"value":[{"action":"drop",
"sourceLabels":["__name__"],
"regex":"go_gc_.*"}]}]'
# 6) after ~5 min the old series go stale
promtool query instant http://localhost:9090 \
'count({__name__=~"go_gc_.*", job=~"orders|payments"})'
# → empty resultWrite a five-line timeline: change, alert, diagnosis, mitigation, recovery — with the query that proved each step.
ShopHighLatencyP95: pending → firing → resolved, at times that match your date note.payments server span; log lines share its trace_id.rollout undo of the config-only revision restored p95.alertmanager_notifications_total{integration="webhook"} grows every minute.go_gc_* series left for the shop jobs.release: kps from shop-gameday → the rules vanish from Prometheus → Rules, silently.job="orders" to job="order" → the alert can never fire. Only a promtool test rules unit test (ch 4) catches this.kubectl -n monitoring scale deploy/heartbeat --replicas=0 → alertmanager_notifications_failed_total rises; kps fires AlertmanagerFailedToSendAlerts./orders/8812) → watch count(count by (route)(http_requests_total)) climb, then revert.# undo game-day changes
kubectl -n shop set env deploy/payments \
EXTRA_LATENCY_MS- FAIL_RATE-
kubectl delete -f rules/gameday.yaml
# keep shop-meta and the Watchdog route
# stop the port-forwards
pkill -f 'kubectl -n monitoring port-forward'
# end of the deck? remove everything:
kind delete cluster --name obsThe kind labs are free apart from laptop resources. The AWS labs (11–12) cost little but bill per GB, per metric and per alarm: clean up when you finish.
GetMetricData API calls.obs-lab-orders, log group /obs-lab/orders (and /aws/lambda/obs-lab-orders), alarms, metric filters, SNS topic obs-lab-alerts; any AMP workspace or Grafana workspace; delete the budget last.kind delete cluster --name obs; docker image prune.Short anchors; explain each in your own words.
| Term | Short meaning |
|---|---|
| Time series | Metric name + unique label set, with timestamped samples |
| Cardinality | Number of distinct series (or label values); drives memory and cost |
| Counter / gauge | Only-increasing value (resets on restart) / value that goes up and down |
| Histogram | Observations counted in buckets (le), plus _sum and _count |
| Native histogram | Histogram with dynamic exponential buckets in one series (stable since Prometheus 3.8) |
| Exemplar | Sample attached to a metric pointing to one trace (trace_id) |
| Recording rule | Precomputed query saved as a new series |
| Remote write | Protocol to stream samples to another store (Mimir, Thanos, AMP) |
| Term | Short meaning |
|---|---|
| RED | Rate, Errors, Duration — for request-driven services |
| USE | Utilization, Saturation, Errors — for resources |
| Golden signals | Latency, traffic, errors, saturation |
| Symptom vs cause | What users feel vs why it happens; page on symptoms |
for / keep_firing_for | Pending time before firing / hold time after the condition clears |
| Grouping | Alertmanager batching alerts with the same group_by labels into one notification |
| Inhibition | Suppressing target alerts while a matching source alert fires |
| Watchdog | Always-firing alert used as a dead man's switch |
Short anchors; explain each in your own words.
| Term | Short meaning |
|---|---|
| Structured log | Log line as key/value fields (JSON, logfmt) with a stable schema |
| Stream (Loki) | All lines with the same label set; the unit Loki indexes |
| Structured metadata | Per-line key/values stored with Loki entries, not indexed as labels |
| Span | One timed operation with attributes, status, events and links |
| Trace | Tree of spans sharing one trace ID |
| Context propagation | Passing trace context across process boundaries (W3C traceparent) |
| Resource | Attributes describing the telemetry source (service.name, k8s.pod.name) |
| Head / tail sampling | Decide at trace start / after the trace completes |
| OTLP | OpenTelemetry protocol; gRPC 4317, HTTP 4318 |
| Term | Short meaning |
|---|---|
| Collector | OpenTelemetry pipeline: receivers → processors → exporters |
| Alloy | Grafana's collector; replaces Promtail (EOL 2 Mar 2026) and Grafana Agent |
| Namespace / dimension (CloudWatch) | Container for metrics / name-value pair identifying a metric |
| EMF | Embedded Metric Format: JSON log lines CloudWatch turns into metrics |
| Composite alarm | Alarm whose state is a rule over other alarms |
| Logs Insights | CloudWatch log query engine (Logs Insights QL, PPL, SQL) |
| Application Signals | CloudWatch APM built on OpenTelemetry: services, SLOs, service map |
| ADOT | AWS Distro for OpenTelemetry |
| OAM | Observability Access Manager: sinks and links for cross-account observability |
Versions move every month; when a slide and the docs disagree, the docs win.
prometheus.io/docsprometheus.io/docs/alerting/latest/configuration/prometheus-operator.dev; Helm charts — github.com/prometheus-community/helm-chartsgrafana.com/docs/loki/latest/; Alloy — grafana.com/docs/alloy/latest/grafana.com/docs/tempo/latest/grafana.com/docs/grafana/latest/opentelemetry.io/docsopentelemetry.io/docs/specs/semconv/; W3C Trace Context — w3.org/TR/trace-context/sre.google/sre-book/monitoring-distributed-systems/brendangregg.com/usemethod.html; RED method — grafana.com/blog/2018/08/02/the-red-method-how-to-instrument-your-services/docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/; Logs and Logs Insights — …/latest/logs/aws.amazon.com/about-aws/whats-new/2026/06/amazon-cloudwatch-otel-metrics/docs.aws.amazon.com/xray/latest/devguide/xray-daemon-eos.htmldocs.aws.amazon.com/prometheus/