Runbooks

What you will be able to do

Turn each alert into a short list of steps that anyone on call can follow at 3 a.m. The steps say what the alert means and what to check first. They say how to protect customers, whom to call, and how to prove that the service is back.

Before you startWhat to know and to have readyAbout 50 minutes

What you need to know

• Severity levels and the error budget, from Service objectives. Owners, from Ownership and support.
• Alert rules in Grafana, from Quality and costs, and an alert in seven parts, from LLM/RAG regressions.
• Rollback with python -m ops rollout, from Controlled rollouts.

What you need on your computer

• Your project ticket-api at the end of Module 5, or a copy of the snapshot end-of-m05. python -m pytest prints 213 passed.
• Docker running PostgreSQL and Valkey (docker compose up -d db cache), for the tests.
• The saved telemetry in data/telemetry/, copied from the labs repository in Artifact lineage.

No account, no key and no money. The steps that read saved files need no Docker.

The note on the fridge

Parents who go out for the evening leave a note on the fridge for the babysitter. A good note does not say “call us if something is wrong”. It says:

• How you know: “If she is hot and does not want to play, she may have a fever.”
• What to check: “Take her temperature with the thermometer in the bathroom.”
• What to do first: “Give her water and the children’s medicine, 5 ml.”
• Whom to call: “Over 39 °C, call us. If we do not answer, call the doctor: this number.”
• How you know it is better: “Under 38 °C after an hour, and she plays again.”

The babysitter has never met this child with a fever. With the note, they do the right things in the right order, without panic.

The problem: an alert with no note

In Module 1 you met “last week”: prompt v3 went live for every ticket, and for two days nobody noticed. Since then, the service got alert rules. Run them on the practice replay of last week, and see what they would have said:

python -m ops alerts data/telemetry/release-full.jsonl.gz --after 1.5.0-767b56

Practice action 'ops rollout start 1.5.0-767b56 --reason the cheaper prompt, for every ticket' at +5.0 min
alert             severity  fired      resolved      after the action  owner
----------------  --------  ---------  ------------  ----------------  -----
draft-fast-burn   SEV-2     +10.7 min  +18.2 min     5.7 min           Amira
no-answer         SEV-2     +3.2 min   still firing  -                 Priya
cost-over-budget  SEV-3     +11.2 min  still firing  6.2 min           Kwame
triage-slow       SEV-3     +9.7 min   still firing  4.7 min           Amira
reviews-disagree  SEV-3     +10.8 min  still firing  5.8 min           Priya

Five alerts fired. That is good: the service now sees the problem. But imagine Amira’s phone at 3 a.m. with the message “draft-fast-burn: SEV-2”. She has to answer three questions alone, half awake:

• What does this alert mean, and is it real? no-answer had already fired before the change, for one ticket of prompt v2. Is this the same noise?
• What do I do first? Restart the workers? Raise the provider timeout? Both are wrong here. A restart changes nothing, and a longer timeout makes the bad prompt slower.
• When is it over? draft-fast-burn resolved at +18.2 minutes, but nobody rolled back. The alert stopped because its 5-minute share dipped under 10% for a moment. The bad answers did not stop: in the last 2 minutes, 31 tickets still had no valid draft.

An alert without instructions gives the person on call a problem, not a plan. Here the wrong first step is likely, and the false “resolved” is the most dangerous part.

The idea: one runbook per kind of failure

A runbook is the note on the fridge for a service: the written steps for one kind of failure. The name comes from the early computer rooms, where operators kept a book of the procedures to run. Each runbook has five parts, in the order that you use them:

1. Detection. What the alert means in plain words, and where to look: the dashboard panel or the command.
2. Triage. A few quick checks, about 5 minutes, that decide what kind of failure this is and how bad it is. Which release? Since when? How many customers?
3. Mitigation. What protects customers now, even before you know the cause. For a bad release, that is a rollback. Investigate later.
4. Escalation. Whom to call, and when: the owner, the backup, Grace for customers, and the provider. It also says when the severity goes up, for example “a safety ticket without an answer is SEV-1”.
5. Recovery checks. The evidence that the service is back, as boxes to tick. “The alert stopped” is not one of them, as you just saw.

The five parts of a runbook, in the order of use. When a recovery box stays empty, you go back to triage.

Text description of the diagram

An alert fires, for example draft-fast-burn, SEV-2, owned by Amira. Detection says what it means and where to look. Triage takes about 5 minutes and finds the release, the start time and the number of customers. Mitigation protects the customers first, for example with a rollback. Escalation says whom to call and when the severity rises. Recovery checks are boxes with evidence. When every box is ticked, the incident closes; when one is not, you go back to triage.

Mitigation comes before the cause on purpose. In the babysitter’s note, the water and the medicine come before anyone knows why the child is hot. A rollback is the same: it is safe, because the old release passed its gate, and it stops the harm while people find the cause.

A tiny example: when does the alert fire?

A runbook starts from the alert, so first read the alert’s rule exactly. draft-fast-burn says: more than 10% of finished tickets without a valid draft in the last 5 minutes, for at least 1 minute. Here are the real counts of the first 12 minutes of last week, minute by minute:

Run Python in your browser

Run the draft-fast-burn rule by hand

The box prints the share of bad tickets in the 5-minute window for each minute. It fires at minute 11, six minutes after prompt v3 went live at minute 5. Make the window shorter, or the threshold lower, and run it again.

Python code (you can change it)# "Last week", minute by minute: finished tickets, and those without a valid draft. # Prompt v3 went live at minute 5. finished = [113, 121, 119, 119, 121, 117, 120, 116, 118, 125, 120, 117] bad = [ 0, 0, 0, 1, 0, 13, 13, 11, 13, 15, 12, 12] # The rule of the alert draft-fast-burn: # more than 10% bad in the last 5 minutes, and that for at least 1 minute. above, window, hold = 0.10, 5, 1 since = None for minute in range(1, len(bad) + 1): first = max(0, minute - window) share = sum(bad[first:minute]) / sum(finished[first:minute]) if share > above: since = minute if since is None else since else: since = None firing = since is not None and minute - since >= hold print(f"minute {minute:2}: {share:6.1%} bad in the window", "FIRING" if firing else "") # Try: a window of 2 minutes, or "above = 0.05". When does it fire now?

minute  1:   0.0% bad in the window
minute  2:   0.0% bad in the window
minute  3:   0.0% bad in the window
minute  4:   0.2% bad in the window
minute  5:   0.2% bad in the window
minute  6:   2.3% bad in the window
minute  7:   4.5% bad in the window
minute  8:   6.4% bad in the window
minute  9:   8.4% bad in the window
minute 10:  10.9% bad in the window
minute 11:  10.7% bad in the window FIRING
minute 12:  10.6% bad in the window FIRING

From minute 6, about 11% of the tickets in each minute had no valid draft. The rule did not fire at once: the 5-minute window still held the good minutes of prompt v2. The share crossed 10% at minute 10 and fired one minute later. ops alerts says the same: +10.7 minutes.

Two things go into the runbook from this example:

• The delay is built in. The window and the wait trade speed for fewer false alarms. Customers got bad tickets for about 6 minutes before anyone knew. The runbook cannot shorten that; a canary can, as Module 5 showed.
• “Resolved” is a statement about the window, not about customers. With a steady 10.6% bad, the rule sits on its threshold. It can stop firing when one minute is a little better. So the runbook checks the tickets, not the alert.

Add the module’s files to your project

This module adds files for all three of its lessons. They are the runbooks and alert rules, the fallback and the customer status, and the post-incident review. Copy them now, once. Each lesson explains its own files.

1. In a terminal, go to your project and activate the virtual environment.

   macOS / Linux

   cd ~/projects/ticket-api
   source .venv/bin/activate

   Windows (PowerShell)

   cd ~\projects\ticket-api
   .venv\Scripts\Activate.ps1

2. Copy the module’s files from the snapshot end-of-m06. Fourteen files are new, and eleven change.

   macOS / Linux

   S=../nextia-learning-labs/C25/snapshots/end-of-m06
   mkdir -p operations/runbooks operations/reviews operations/releases
   for f in operations/alerts.toml operations/runbooks/README.md operations/runbooks/provider-outage.md \
            operations/runbooks/bad-release.md operations/runbooks/stale-index.md \
            operations/runbooks/model-version-changed.md operations/runbooks/quality-drop.md \
            operations/runbooks/tickets-refused.md operations/reviews/PIR-001-prompt-v3.md \
            operations/releases/1.5.0-4fafd1.toml operations/releases/1.5.0-767b56.toml \
            ops/cmd_alerts.py ops/cmd_incident.py ops/cmd_slo.py ticket_api/config.py \
            ticket_api/job_routes.py ticket_api/main.py ticket_api/worker.py scripts/scenario.py \
            tests/test_incidents.py tests/test_jobs.py tests/test_telemetry.py contract/openapi.json \
            .env.example README.md; do
     cp "$S/$f" "$f"
   done

   Windows (PowerShell)

   $S = "..\nextia-learning-labs\C25\snapshots\end-of-m06"
   New-Item -ItemType Directory -Force operations\runbooks, operations\reviews, operations\releases | Out-Null
   $files = 'operations\alerts.toml', 'operations\runbooks\README.md', 'operations\runbooks\provider-outage.md',
            'operations\runbooks\bad-release.md', 'operations\runbooks\stale-index.md',
            'operations\runbooks\model-version-changed.md', 'operations\runbooks\quality-drop.md',
            'operations\runbooks\tickets-refused.md', 'operations\reviews\PIR-001-prompt-v3.md',
            'operations\releases\1.5.0-4fafd1.toml', 'operations\releases\1.5.0-767b56.toml',
            'ops\cmd_alerts.py', 'ops\cmd_incident.py', 'ops\cmd_slo.py', 'ticket_api\config.py',
            'ticket_api\job_routes.py', 'ticket_api\main.py', 'ticket_api\worker.py', 'scripts\scenario.py',
            'tests\test_incidents.py', 'tests\test_jobs.py', 'tests\test_telemetry.py', 'contract\openapi.json',
            '.env.example', 'README.md'
   foreach ($f in $files) { Copy-Item "$S\$f" $f }

   The two release manifests are the course’s own. They match the code at the end of the course, so python -m ops release verify 1.5.0-4fafd1 now says that the release matches the files.

3. Add the new setting to your .env. It is also the default, so the line only makes it visible:

   TRIAGE_FALLBACK=person

4. Run the tests. This module has no new migration. Expect 218 passed, 5 more than before:

   python -m pytest

   218 passed, 1 deselected in 25.82s

Files	What they add	Lesson
operations/runbooks/ (7 files), operations/alerts.toml, ops/cmd_alerts.py	the runbooks; the alert rules with owners, severity and runbook; python -m ops alerts	this one
ticket_api/worker.py, config.py, job_routes.py, main.py, ops/cmd_incident.py, ops/cmd_slo.py	the fallback to a person, GET /v1/status, python -m ops incident	Provider and dependency outages
operations/reviews/PIR-001-prompt-v3.md, operations/releases/	the post-incident review of last week; the course’s two manifests	Post-incident learning
tests/, scripts/scenario.py, contract/openapi.json, .env.example, README.md	tests, a --scale option for shorter practice scenarios, the API contract, the commands	all three

Larkfield’s runbooks

operations/runbooks/README.md maps every alert to its runbook and its owners:

Runbook	Alerts	Owner
provider-outage.md	provider-circuit-open, drafts-late, triage-slow	Kwame (provider), Amira (service)
bad-release.md	no-answer, draft-fast-burn, cost-over-budget	Priya (prompt), Amira (release)
stale-index.md	old-sources	Tomás
model-version-changed.md	model-version-changed	Priya
quality-drop.md	reviews-disagree	Priya
tickets-refused.md	accept-fast-burn	Amira

Six runbooks cover ten alerts, because several alerts are signs of the same failure. A slow triage call can be a slow provider or a bad prompt. So triage-slow is in the provider runbook, and the bad-release runbook tells you to check the release first.

operations/alerts.toml is the other half. Each rule has the same parts as the alerts of Module 4, and it names its runbook:

[[alert]]
uid = "draft-fast-burn"
title = "Tickets without a valid draft (draft SLO burns fast)"
rule = "share of finished tickets whose outcome is not ok"
above = 0.10
window = 300
for = 60
severity = "SEV-2"
owner = "Amira"
runbook = "operations/runbooks/bad-release.md"

These are the rules of Grafana’s alert files (observability/alerts/), written so that python -m ops alerts can run them on saved telemetry, every 10 seconds of the file. So you can test a rule on any recorded run, without Docker.

Now read operations/runbooks/bad-release.md. Its triage and mitigation parts:

## Triage (5 minutes)

1. Which release made the bad answers? `python -m ops quality FILE --by release` or the
   release comparison dashboard. If only the canary has them, it is the release.
2. Since when? `python -m ops rollout history`: the last change and who made it.

## Mitigation

- Roll back first, investigate after: `python -m ops rollout rollback --reason "..."`.
  The worker takes it within 5 seconds. Rolling back is safe: the active release passed its gate.

Every step is a command or a panel, not an opinion. A step that says “check whether the release is fine” needs knowledge that the person on call may not have. A step that gives the command, and says what a bad result looks like, does not.

Walk the runbook through last week

Follow bad-release.md on the replay of last week, as Amira would have at 3 a.m. Only some steps need the saved files; you met the others in Module 5.

1. Detection. draft-fast-burn fired at +10.7 minutes, SEV-2. The draft SLO burns fast.

2. Triage, step 1: which release?

   python -m ops quality data/telemetry/release-full.jsonl.gz --by release

   release                         1.5.0-4fafd1       1.5.0-767b56
   ------------------------------  -----------------  -----------------
   tickets                         602                1,798
     ok                            601 (99.8%)        1,602 (89.1%)
     no_answer                     1 (0.2%)           186 (10.3%)
     dead_letter                   0 (0.0%)           10 (0.6%)
   sent to a person                162 (27.0% of ok)  475 (29.7% of ok)
   tokens in (median)              1,902              1,477
   tokens out (median)             260                627
   triage call p50 / p95 (s)       2.89 / 5.17        5.56 / 8.34
   time to a draft p95 (s)         5.2                7.9
   cost per 1,000 tickets (US$)    0.338              0.474
   agrees with the agent's review  288/307 (93.8%)    623/744 (83.7%)

   Release 1.5.0-767b56, with prompt v3, has 10.3% answers without text. The old release has one, the known case. So this is the release, not noise.

3. Triage, step 2: since when? In your live service, python -m ops rollout history shows the change, and who made it. In the replay, the first line of ops alerts above shows the same action at +5.0 minutes.

4. Mitigation. Roll back: python -m ops rollout rollback --reason "answers without text". In last week, nobody did. In the canary run of Module 5, the rollback came at +7.2 minutes.

5. Escalation. Any ticket that needed a person and got no answer makes it SEV-1. Last week had 10 of them, so Grace calls those customers.

6. Recovery checks. The canary run shows what the evidence looks like after a rollback:

   python -m ops canary data/telemetry/canary.jsonl.gz

   Its last line:

   After the last release change (+7.3 min: active 1.5.0-4fafd1, canary None): 1,533 tickets, 2 without text, US$0.341 per 1,000, triage p95 5.17 s, canary tickets 0

   Tick the boxes. No canary ticket came after the rollback, and the cost and time are back to v2’s values. The 2 answers without text are the known v2 case.

Alerts that cry wolf

The no-answer alert fired at +3.2 minutes of last week, before the change. Run the rules on the normal week, where nothing went wrong:

python -m ops alerts data/telemetry/normal-week.jsonl.gz

alert      severity  fired      resolved   after the action  owner
---------  --------  ---------  ---------  ----------------  -----
no-answer  SEV-2     +20.5 min  +25.5 min                 -  Priya
no-answer  SEV-2     +31.0 min  +38.2 min                 -  Priya

A SEV-2 alert fired twice in a healthy week. The cause is real: one of prompt v2’s 679 recordings, practice ticket T-65450, was cut off at the token limit. The practice runs draw it again and again. Each time, one ticket has no answer, and the rule “more than 0 in 5 minutes” fires.

Last week, as the rules on saved telemetry see it. The first alert was noise from the healthy prompt, so the real alerts arrived after people had learned to ignore it. A practice replay; the answers are real recordings.

Text description of the diagram

The x axis shows the 20 practice minutes. Prompt v3 went live at minute 5. The no-answer alert had fired at minute 3.2, because of prompt v2’s one cut-off recording. After the change, triage-slow fired at 9.7, and draft-fast-burn at 10.7 until 18.2. Reviews-disagree followed at 10.8, and cost-over-budget at 11.2. No rule fired in the first 4.7 minutes after the change, but the bad answers began at once.

This is how alerts get ignored. A person sees no-answer on a good day, checks, and finds nothing. The next time, they check less. When prompt v3 makes the same alert fire for real, it looks like the same noise. The post-incident review in this module lists it as a contributing condition.

A runbook protects against noise in two ways. Its triage step tells a known case from a new one: ops quality --by release showed one no-answer for v2 and 186 for v3. And it gives the known case an owner: Priya reviews T-65450, so that the alert can become trustworthy again. Do not just raise the threshold: then one safety ticket without an answer would no longer fire it.

Grafana’s own alerts, on the same run, fired a little later than the rules on saved telemetry. After the change, triage-slow took 4.95 minutes, draft-fast-burn 5.62 and cost-over-budget 8.79. Grafana reads metrics that the service sends every 15 seconds, and Prometheus computes the windows its own way. The order is the same, and the minutes are close. Write the runbook for the order, not the exact second.

When to write a runbook, and what it costs

Sam asks, “Why not just write one big document: ‘If something breaks, call Amira’?”

Grace answers: “Because Amira is asleep, or on holiday, and the person on call needs the first five minutes in writing.”

Write a runbook for every alert that pages a person, and keep the alert and the runbook in the same repository, as Larkfield does. Do not write one for a signal that needs no action: delete that alert, or make it a dashboard panel. An alert that never needs a step is noise by design.

What it costs: an hour to write each runbook, and its upkeep. A command that changed, or an owner who left, makes the runbook wrong without any warning. So test runbooks. Walk one through a recorded run, as you did above. Or run a tabletop exercise, as you will at the end of this module. A runbook that nobody has followed is a guess.

Common mistakes

Common mistake

ops alerts says that there is no such practice action

Symptom:

python -m ops alerts data/telemetry/outage.jsonl.gz --after "outage mode"

ops alerts: no practice action like 'outage mode' in the file

The exit code is 2.

Cause: --after looks for part of an action’s text, as python -m ops incident prints it, for example simulator mode: outage. “outage mode” is not part of that text.

Fix: run python -m ops incident FILE, read the action in the timeline, and copy part of it: --after "simulator mode: outage". Your own telemetry has no practice actions: run ops alerts without --after.

Common mistake

The alert resolved, so the incident is closed

Symptom: draft-fast-burn shows resolved at +18.2 minutes, and the person on call closes the incident.

Cause: the rule’s 5-minute share fell just under 10% for a moment. The release was never rolled back: 31 tickets in the last 2 minutes still had no valid draft.

Fix: close an incident only on the runbook’s recovery checks. Each one is a number on the tickets, after the mitigation, not the state of an alert.

Your turn: an alert and a runbook for slow answers

Larkfield has an objective for the answers on the website: 95% within 3 seconds (answer_time in operations/slos.toml). No alert watches it, and no runbook covers it.

1. Write the alert in operations/alerts.toml style: rule, threshold, window, for, severity, owner and runbook. Use the burn-rate idea of Service objectives for the threshold.
2. Write operations/runbooks/answers-slow.md with the five parts. Name at least one command or panel in each part.
3. Say which recovery check proves that customers are fine again.

Worked solution

The alert. The budget is 5% slow answers. A fast burn, at about 7 times the budget, is 35% slow in 5 minutes:

[[alert]]
uid = "answers-slow"
title = "Answers on the website are slow (answer-time SLO burns fast)"
rule = "share of answers served in more than 3 seconds"
above = 0.35
window = 300
for = 60
severity = "SEV-2"
owner = "Amira"
runbook = "operations/runbooks/answers-slow.md"

ops alerts has no code for this new uid yet. Test it in Grafana, or add a branch to value() in ops/cmd_alerts.py, as the other rules have.

The runbook, short:

• Detection: more than 35% of answers took over 3 seconds in 5 minutes. Panel: answer time on the overview dashboard.
• Triage: Is the provider slow? If triage-slow fires too, go to provider-outage.md. Is the cache empty? Most answers say cached: false after a restart or a reindex. Is the database busy? Check GET /ready and the pool’s numbers.
• Mitigation: if the provider is slow, change nothing in the service; the breaker protects it. If the cache was emptied, wait, or warm it with the top questions. Never raise the timeout to make the alert stop.
• Escalation: after 15 minutes, Kwame asks the provider. If answers fail, not only slow, Grace puts a note on the help page.
• Recovery checks: [ ] 95% of answers within 3 seconds in the last 15 minutes; [ ] triage-slow is not firing; [ ] the cache answers again (cached: true on repeated questions).

The check that proves customers are fine is the first one: the SLI itself, on the answers, after the mitigation.

See it end to endPost-incident review of a public outageCase study

The case study rebuilds a published outage from its reports. Then it asks which of Larkfield’s alerts would fire, and which runbook the person on call would open.

Knowledge check

Check your understanding

Name the five parts of a runbook, in order. Why does mitigation come before finding the cause?

Show the answer

Detection, triage, mitigation, escalation, recovery checks. Mitigation comes first because it protects customers now, and it is safe without knowing the cause. A rollback to a release that passed its gate stops the harm in seconds. Finding the cause can take hours, and every minute of last week gave about 13 more tickets without a valid draft.

Check your understanding

Predict: you lower the draft-fast-burn threshold from 10% to 5%, with the same 5-minute window. On last week's counts, does it fire sooner? What does it cost?

Show the answer

Yes. In the tiny example, the window share passes 5% at minute 8 (6.4%). So it fires at minute 9 instead of 11, about 2 minutes sooner. The cost is noise. A short bad spell on a healthy release now crosses 5% more easily, for example a few dead letters in a quiet minute. Every false alarm teaches people to trust the alert less. A canary is the better way to be faster: it limits how many customers are hit while the window fills.

Check your understanding

Practical check: on the saved outage run, when did the two outage alerts fire, measured from the start of the outage? Which runbook do they open?

Show the answer

python -m ops alerts data/telemetry/outage.jsonl.gz --after "simulator mode: outage" prints provider-circuit-open after 0.6 minutes and drafts-late after 3.1 minutes. Both are SEV-2, and both open operations/runbooks/provider-outage.md, owned by Kwame and Amira. The next lesson follows that runbook.

Recap

• A runbook is the written steps for one kind of failure, in five parts: detection, triage, mitigation, escalation and recovery checks. Each step is a command or a panel.
• Mitigation comes before the cause. A rollback to a release that passed its gate stops the harm while people look for the cause.
• An alert’s window and wait delay it: draft-fast-burn fired 5.7 minutes after prompt v3 went live. “Resolved” describes the window, not the customers, so close an incident only on the recovery checks.
• python -m ops alerts runs the alert rules on saved telemetry. An alert that fires on a healthy release, such as no-answer on prompt v2’s one cut-off answer, teaches people to ignore it.
• Your project has the module’s files, and python -m pytest prints 218 passed.

Next, you follow the provider runbook through an outage of the simulated provider, and you check what customers see while it lasts.

Next: Provider and dependency outages