Provider and dependency outages

What you will be able to do

Decide in advance what the service does when its AI provider fails. Keep the tickets, and say honestly that replies are late. Send what cannot wait to a person, and prove with numbers that it is over.

Before you startWhat to know and to have readyAbout 55 minutes

What you need to know

• Runbooks and python -m ops alerts, from Runbooks.
• The job queue, retries and dead letters, from Delivery and retries. The circuit breaker, from Failure containment.
• The draft SLO (a valid draft within 10 minutes), from Service objectives.

What you need on your computer

• Your project after the last lesson: python -m pytest prints 218 passed.
• Docker running PostgreSQL and Valkey, and the simulator, the API and the worker in three terminals, as in Run the live service. A fourth terminal for the commands.

The live steps take about 12 minutes. The saved outage run needs no Docker.

The post office on strike day

The sorting office in town stops for a day. A good post office does not lock its doors. The letterboxes stay open, and every letter is kept. A sign on the counter says: “Deliveries are late today. Your letter is safe. You do not need to send it again.” A letter marked urgent: medicine goes to a courier by hand. The next morning, the backlog goes out first, oldest first, and the manager checks that the letterboxes are empty before saying “we are back”.

Four plans are in that story. What still works: the letterboxes. What waits: the letters. What customers are told: the sign. How you know it is over: the empty letterboxes.

The problem: the service depends on someone else

Larkfield’s service cannot triage a ticket without the AI provider. The provider is a dependency: a service that yours needs, but that you do not run. The database and the cache are dependencies too, but Mei runs them. The provider is run by another company, and it fails on its own schedule.

In this lesson you read the outage run, a practice incident that we caused on purpose on the real stack. Customers sent 1 ticket a second. At +5.1 practice minutes, the simulated provider started to fail every call. It came back at +10.1 minutes.

Without a plan, a provider outage goes wrong in four ways:

• Retries make it worse. Each worker sends each ticket up to 5 times. 300 waiting tickets become 1,500 calls to a provider that cannot answer.
• Tickets are lost. After its last attempt, a job becomes a dead letter and waits for a person who does not know about it.
• Customers guess. The website still says “thank you”, and nothing arrives. Some customers send the ticket again, so the queue grows faster.
• Nobody knows when it is over. The provider is back, but is the backlog done? Are some tickets stuck?

The idea: plan the outage before it happens

You cannot prevent another company’s outage. You can decide, in advance and in code, how your service behaves during it. Four parts:

1. Fallback modes. A fallback is what the service does instead of the normal path. For Larkfield’s tickets: the ticket waits in the queue for the provider. A ticket that gets no valid answer goes to a person.
2. Queued work. Accept every ticket, keep it, and try again later without spending its attempts. The scaling course built this: the circuit breaker.
3. User status. Tell customers the truth, automatically: what works, what is late, and what they should do.
4. Recovery verification. “It is back” is a claim. The numbers on the tickets are the evidence.

Where a ticket goes when the provider fails. The ticket is saved first, so an outage delays it but does not lose it.

Text description of the diagram

The API saves every ticket first and answers 202 at once, so an outage never refuses a ticket. The worker asks the circuit breaker before each call. While the breaker is open, the job goes back to the queue without using an attempt. When the provider answers but the answer is not valid, the ticket goes to a person. When the oldest ticket has waited more than 120 seconds, the status endpoint tells the website to show the delayed message.

Which fallback?

An AI service has more fallback choices than a classic one. Kwame lists Larkfield’s:

Fallback	What it does	What it costs
Wait in the queue	the ticket is triaged when the provider is back	the delay; the queue must have room
Send to a person	an agent triages the ticket by hand	agents’ time; slower for the customer
A second provider or a smaller model	another model triages the ticket	a different behaviour that needs its own evaluation, release and contract
A fixed reply	“We received your ticket” with no triage	nothing useful for the help desk

Larkfield uses the first two. A second provider sounds safe, but it is a new release that no gate has tested. Switching to it in the middle of an outage repeats the mistake of last week.

A tiny example: how long can the outage last?

The draft SLO says: a valid draft within 10 minutes (600 seconds). How long can the provider be down before tickets break it? Take the outage run’s numbers:

Step	Value	From
Tickets arriving	1 a second	the scenario
The outage	300 s (+5.1 to +10.1)	the scenario
Tickets waiting at the end of the outage	about 300	1 × 300
Time from “provider back” to the oldest ticket’s draft	about 54 s	the breaker’s next test call, then the oldest ticket first
The oldest ticket’s wait	300 + 54 = 354 s	measured: 354.1 s
Room left under the SLO	600 − 354 = 246 s

The first ticket of the outage waits the longest: the whole outage, plus the time to start again. So in this setup, an outage of about 600 − 54 = 546 seconds, about 9 minutes, would break the draft SLO for the first tickets. Every minute longer adds 60 tickets that break it, at this rate. This is an estimate from one run: the restart time depends on the breaker and the number of workers.

The queue has a limit too. The practice runs set QUEUE_MAX=5000. At 1 ticket a second, that is 83 minutes of outage before the API must refuse tickets. At Larkfield’s real rate of 822 tickets a day, it is about 6 days.

Larkfield: what the service already does, and what this module adds

The scaling course already gave the service its queued work. The API saves each ticket and answers 202 before any model call. The worker’s circuit breaker opens after 5 failed calls in a row. While it is open, the worker makes no call: it gives each job back to the queue, without spending one of its 5 attempts. Every 30 seconds the breaker lets one test call through (half-open). If it works, the breaker closes.

This module adds two things.

Customer status. GET /v1/status needs no API key, so the website can call it. It reads the queue and returns Grace’s wording. From ticket_api/job_routes.py:

DELAYED_AFTER_SECONDS = 120  # the "drafts-late" alert's threshold

def status_message(stats: dict) -> ServiceStatus:
    """Grace's wording: say what works, what is late, and what the customer should do."""
    oldest = stats["oldest_queued_seconds"]
    pace = stats["succeeded_last_minute"] / 60
    wait = round(stats["queued"] / pace, 1) if pace > 0 else None
    if oldest <= DELAYED_AFTER_SECONDS:
        return ServiceStatus(status="ok", message="We answer new tickets as usual.", ...)
    minutes = "a few" if wait is None else str(max(1, round(wait / 60)))
    return ServiceStatus(
        status="delayed",
        message=(
            "Your ticket is saved, and nothing is lost. Our replies take longer than usual "
            f"(about {minutes} minutes). You do not need to send your ticket again. If something "
            "is unsafe, stop using the product and call us."
        ), ...

The message answers the customer’s three questions: is my ticket lost, how long, and what do I do. The last sentence is for safety tickets, which must not wait for a provider. The status uses the same 120 seconds as the drafts-late alert, so customers and the person on call see “late” at the same moment.

The fallback to a person. TRIAGE_FALLBACK=person changes what the worker does with a ticket that gets no valid answer. Before, the job failed: the ticket stayed in the default team, and nobody was told. Now the worker saves needs_human true and priority 1, so the ticket goes to the agents’ first queue. The job finishes with the outcome sent_to_person. This is an action from the post-incident review of the next lesson. During an outage it does not apply: the breaker gives the job back before any answer, so the ticket waits for the provider.

Make the provider fail on your own service

Use your four terminals. The simulator, the API and the worker run as in Run the live service. The steps send 1 ticket every 2 seconds, not 1 a second as the saved run. The simulated provider has a quota, as the real one had: 100,000 tokens a minute, about 45 tickets. The saved runs turned the quota off and used two workers. On Windows, type curl.exe instead of curl, and put each command on one line.

1. Ask for the status, as the website does:

   curl http://127.0.0.1:8000/v1/status

   {"status":"ok","message":"We answer new tickets as usual.","queued":0,"oldest_waiting_seconds":0.0,"estimated_wait_seconds":null}

2. Make the simulated provider fail every call. This is a practice outage, made on purpose:

   curl -X POST http://127.0.0.1:8300/admin/mode -H "Content-Type: application/json" -d '{"mode": "outage"}'

   {"mode":"outage"}

3. Send 90 practice tickets, 1 every 2 seconds. This takes 3 minutes; keep it running:

   python -m scripts.practice --tickets 90 --rate 0.5 --seed 7

   Sent 90 practice tickets in 178.0 s: 202: 90

   Every ticket was accepted (202), although no ticket could be triaged.

4. In the fourth terminal, read the queue and the status every 30 seconds:

   curl http://127.0.0.1:8000/v1/queue
   curl http://127.0.0.1:8000/v1/status

   After 90 seconds, the queue grows, and the status still says ok:

   {"queued":46,"ready":0,"running":0,"dead_letter":0,"failed":0,"succeeded_last_minute":0,"oldest_queued_seconds":90.0}
   {"status":"ok","message":"We answer new tickets as usual.","queued":46,"oldest_waiting_seconds":90.0,"estimated_wait_seconds":null}

   After 120 seconds, the oldest ticket passes 120 seconds, and the status changes:

   {"queued":61,"ready":1,"running":0,"dead_letter":0,"failed":0,"succeeded_last_minute":0,"oldest_queued_seconds":120.1}
   {"status":"delayed","message":"Your ticket is saved, and nothing is lost. Our replies take longer than usual (about a few minutes). You do not need to send your ticket again. If something is unsafe, stop using the product and call us.","queued":61,"oldest_waiting_seconds":120.1,"estimated_wait_seconds":null}

   Nothing finished in the last minute, so the service cannot estimate the wait, and it says “a few minutes”. It does not guess a number. "dead_letter":0: no ticket is lost.

5. When the practice script has finished, bring the provider back:

   curl -X POST http://127.0.0.1:8300/admin/mode -H "Content-Type: application/json" -d '{"mode": "normal"}'

6. Read the queue and the status every 20 seconds or so. After 40 seconds:

   {"queued":51,"ready":51,"running":4,"dead_letter":0,"failed":0,"succeeded_last_minute":35,"oldest_queued_seconds":218.4}
   {"status":"delayed","message":"Your ticket is saved, and nothing is lost. Our replies take longer than usual (about 1 minutes). You do not need to send your ticket again. If something is unsafe, stop using the product and call us.","queued":51,"oldest_waiting_seconds":218.4,"estimated_wait_seconds":87.4}

   Now the workers finish tickets again, so the status can estimate the wait: 51 tickets at 35 a minute, about 87 seconds. (“1 minutes” is a small grammar mistake in the message’s code.) After 140 seconds:

   {"queued":0,"ready":0,"running":0,"dead_letter":0,"failed":0,"succeeded_last_minute":16,"oldest_queued_seconds":0.0}
   {"status":"ok","message":"We answer new tickets as usual.","queued":0,"oldest_waiting_seconds":0.0,"estimated_wait_seconds":0.0}

7. A recovery needs new tickets to prove it. Send 60 more, as normal traffic:

   python -m scripts.practice --tickets 60 --rate 0.5 --seed 3

8. When the queue is empty, stop the API and the worker with Ctrl+C. The worker’s last line counts what it did:

   Done in 481.6 s: 896 taken, 170 succeeded, 60 waiting for the quota, 10 to retry later, 0 failed, 0 dead letters, 656 given back to the queue.

   170 tickets succeeded and 0 failed. “656 given back to the queue” is the open breaker at work: each one cost no attempt. “60 waiting for the quota” happened after the outage: the backlog hit the provider’s quota. Your counts can differ.

9. Read the incident from your own telemetry, one row per minute:

   python -m ops incident telemetry/ticket-api.jsonl telemetry/ticket-worker.jsonl --minute 1

   Timeline (what happened, in order):
   when      kind     event
   --------  -------  ------------------------------
   +0.5 min  breaker  circuit open
   +1.0 min  breaker  circuit half_open
   +1.0 min  breaker  circuit open
   +1.0 min  alert    provider-circuit-open fired
   +1.5 min  breaker  circuit half_open
   +1.5 min  breaker  circuit open
   +2.0 min  breaker  circuit half_open
   +2.0 min  breaker  circuit open
   +2.5 min  breaker  circuit half_open
   +2.5 min  breaker  circuit open
   +3.0 min  breaker  circuit half_open
   +3.0 min  breaker  circuit open
   +3.5 min  breaker  circuit half_open
   +3.5 min  alert    drafts-late fired
   +3.5 min  alert    provider-circuit-open resolved
   +3.5 min  breaker  circuit closed
   +5.7 min  alert    drafts-late resolved
   
   Per period (minutes):
   minutes  received  refused (5xx)  finished  no valid draft  oldest waiting (s)  time to draft p95 (s)
   -------  --------  -------------  --------  --------------  ------------------  ---------------------
   0-1            38              0        20               0                  35                   12.6
   1-2            30              0         0               0                  95                      -
   2-3            30              0         0               0                 155                      -
   3-4            12              0        31               0                 213                  200.5
   4-5             0              0        52               0                 161                  225.1
   5-6             8              0        12               0                   6                  178.6
   6-7            30              0        32               0                   0                    7.3
   7-8            22              0        23               0                   0                    5.2
   
   Recovery checks (the last full period against the first):
     [x] tickets are accepted: 0 refused
     [x] drafts are valid: 0 without a valid draft (first period 0)
     [x] nothing is waiting long: oldest waiting 0 s
     [x] drafts are on time: p95 7.3 s (first period 12.6 s)

   Your telemetry has no practice actions, so the timeline starts with the breaker. The minutes count from your first record.

Read the periods as a story. In minutes 1 to 3, tickets arrived, and nothing finished. From minute 3, the backlog was done in about 2.5 minutes. Minutes 6 and 7 hold the new tickets, finished in a few seconds each. Those last rows are the evidence: the service is back for new customers, not only for the backlog.

When we first tried these steps, we sent 1 ticket a second after the recovery. The last check stayed empty: [ ] drafts are on time: p95 105.2 s. The queue was empty and the status said ok, but the new tickets waited, because 1 a second is more than the quota allows. The recovery check caught what the status did not.

Read the outage run

The saved outage run is the same practice, longer and on the full stack with Grafana. ops incident puts everything on one timeline, then counts the tickets per period:

python -m ops incident data/telemetry/outage.jsonl.gz

Timeline (what happened, in order):
when       kind      event
---------  --------  ----------------------------------------------------------------
+0.0 min   release   release: active 1.5.0-4fafd1
+0.1 min   practice  scenario start
+0.1 min   practice  ops rollout start 1.5.0-4fafd1 --reason practice scenario outage
+0.1 min   practice  simulator mode: normal
+0.1 min   grafana   alert firing: cost-over-budget
+0.1 min   grafana   alert firing: reviews-disagree
+3.3 min   grafana   alert resolved: cost-over-budget
+5.1 min   practice  simulator mode: outage
+5.1 min   breaker   circuit open
+5.2 min   breaker   circuit closed
+5.3 min   breaker   circuit open
+5.7 min   alert     provider-circuit-open fired
+5.8 min   breaker   circuit half_open
+5.8 min   breaker   circuit open
+5.9 min   grafana   alert firing: provider-circuit-open
+6.3 min   breaker   circuit half_open
+6.3 min   breaker   circuit open
+6.8 min   breaker   circuit half_open
+6.8 min   breaker   circuit open
+7.3 min   breaker   circuit half_open
+7.3 min   breaker   circuit open
+7.8 min   breaker   circuit half_open
+7.8 min   breaker   circuit open
+8.2 min   alert     drafts-late fired
+8.3 min   breaker   circuit half_open
+8.3 min   breaker   circuit open
+8.4 min   grafana   alert firing: drafts-late
+8.8 min   breaker   circuit half_open
+8.8 min   breaker   circuit open
+9.3 min   breaker   circuit half_open
+9.3 min   breaker   circuit open
+9.8 min   breaker   circuit half_open
+9.8 min   breaker   circuit open
+10.1 min  practice  simulator mode: normal
+10.3 min  breaker   circuit half_open
+10.3 min  alert     provider-circuit-open resolved
+10.4 min  breaker   circuit closed
+10.6 min  grafana   alert resolved: provider-circuit-open
+11.5 min  alert     drafts-late resolved
+11.7 min  grafana   alert resolved: drafts-late
+22.1 min  practice  scenario end

Per period (minutes):
minutes  received  refused (5xx)  finished  no valid draft  oldest waiting (s)  time to draft p95 (s)
-------  --------  -------------  --------  --------------  ------------------  ---------------------
0-2           116              0       124               0                   3                   38.0
2-4           120              0       121               0                   2                    5.4
4-6           120              0        67               0                  55                    6.1
6-8           120              0         0               0                 175                      -
8-10          120              0         0               0                 295                      -
10-12         120              0       413               0                   2                  325.3
12-14         120              0       117               0                   5                    5.1
14-16         120              0       121               0                   4                    5.1
16-18         120              0       120               0                   4                    5.1
18-20         120              0       122               0                   2                    5.2
20-22           4              0         7               0                   0                    5.0
22-24           0              0         0               0                   0                      -

Recovery checks (the last full period against the first):
  [x] tickets are accepted: 0 refused
  [x] drafts are valid: 0 without a valid draft (first period 0)
  [x] nothing is waiting long: oldest waiting 0 s
  [x] drafts are on time: p95 5.0 s (first period 38.0 s)

Read it from the top. Rows of kind grafana are Grafana’s own alerts; rows of kind alert are the rules on saved telemetry. The two grafana alerts at +0.1 minutes come from the run before, which was still in Grafana’s windows; ignore them here.

• +5.1: the provider fails. The breaker opens at once. Each of the two workers has its own breaker and reports its own changes, so you see closed once more at +5.2, then open again.
• Every 0.5 minutes: half_open, then open. That is the breaker’s test call, which fails. Each test is one call, not a flood of retries.
• +5.7 and +8.2: the two alerts of the provider runbook. Detection took 0.6 and 3.1 minutes; Grafana’s took 0.85 and 3.28.
• +10.1: the provider is back. The next test call works, and the breaker closes at +10.4.
• The periods: no ticket was refused, at any time. From +6 to +10 minutes, nothing finished, and the oldest ticket’s wait grew to 295 seconds. Between +10 and +12 minutes, the two workers finished 413 tickets: the backlog plus the new ones.

The breaker gave jobs back 3,102 times and the worker retried 36 times, so no job used up its attempts: 0 dead letters. The draft SLI stayed at 100% for the run, because the oldest ticket waited 354 seconds, under the 600 of the SLO.

• Tickets waiting for a draft
• Oldest waiting ticket (seconds)

The outage run per half minute: the tickets that wait for a draft, and how long the oldest one has waited. Both rise in a straight line during the outage and fall within 2 minutes of the provider's return. A constructed practice outage on the real stack.

Text description of the chart

The x axis shows the 22 practice minutes. Before the outage, about 3 tickets wait at any moment, and the oldest has waited 2 to 3 seconds. From minute 5.1, both lines rise by about 30 every half minute: 30 more tickets wait, and the oldest is 30 seconds older. At minute 10, 296 tickets wait, and the oldest has waited 295 seconds. The provider is back at 10.1. At minute 11, 170 tickets still wait, but the oldest reached 355 seconds. By minute 12, both lines are back to 3 tickets and 2 seconds.

The two lines tell different stories. The number waiting falls as soon as the workers start again. The oldest wait keeps rising for one more minute, because the oldest ticket is not finished first in every case. The SLO is about the oldest wait, not the queue length.

Grafana shows the same shape. This panel has three series, and its colours may differ on your screen:

The outage run in Grafana: time to a draft (p50, p95) and the oldest waiting ticket. Grafana computes the percentiles over windows of a few minutes, so the peaks are wider and later than in the ops table. A constructed practice outage; your colours may differ.

Verify the recovery

The provider runbook’s recovery checks, with the evidence from the run:

The incident timeline and the recovery checklist of the outage run. Each box on the list is a number, not a feeling. A constructed practice outage on the real stack.

Text description of the diagram

The outage starts at +5.1 minutes, and the breaker opens at once. The rules on saved telemetry fire provider-circuit-open at +5.7 and drafts-late at +8.2; Grafana fires them at +5.9 and +8.4. The provider answers again at +10.1, the breaker closes at +10.4, and drafts-late resolves at +11.5. The checklist has six boxes. 0 tickets were refused out of 1,200. The oldest waiting ticket is back to 0 seconds, after a peak of 354. 0 tickets are without a valid draft, and drafts are on time, with a p95 of 5.0 seconds. There are 0 dead letters. The breaker is closed, and the status says ok.

Why so many boxes? Because each one can fail on its own:

• The breaker closed but the queue is still full: customers still wait.
• The queue drained but some jobs became dead letters: those customers never get a reply. Check dead_letter in GET /v1/queue, and retry each one (POST /v1/jobs/{id}/retry).
• Drafts are valid but late: the workers cannot keep up.
• The status says ok but drafts are invalid: the provider came back with another model, or an error.

The provider runbook (operations/runbooks/provider-outage.md) has these boxes. Tick each one with a number from ops incident or the API. Then the incident is over.

When to plan for an outage, and what it costs

Tomás asks, “The provider promises 99.9%. Why plan for something that hardly happens?”

Kwame answers: “99.9% still allows about 43 minutes a month. And their 99.9% does not count slow answers, bad answers or a changed model.”

Plan for it whenever a dependency is run by someone else and your users wait for it. A queue, a breaker and a status message cost little, and Larkfield had the first two already. Do not add a second provider just in case: it doubles the evaluation, the release and the contract work, and its answers differ. Add one only when the outage cost is higher than that work, and test it through the gate like any release.

What it costs:

• Agents’ time. During an outage, Grace’s team reads the queue by hand for safety words. With the fallback, every ticket without a valid answer goes to a person, about 10% of tickets with a bad prompt.
• An honest status. Customers see “late” more often than a page that always says “ok”. That is the point: it keeps them from sending tickets twice.
• Queue space. The queue must hold the whole outage. Size it from the arrival rate, as in the tiny example.

Common mistakes

Common mistake

The worker and the API do not start: TRIAGE_FALLBACK

Symptom: after you edit .env, the worker stops at once:

Settings error: TRIAGE_FALLBACK must be person or none

Cause: a typo, for example TRIAGE_FALLBACK=people. The service refuses an unknown value on purpose: a silent default could turn the fallback off.

Fix: write TRIAGE_FALLBACK=person, or remove the line (the default is person).

Common mistake

The provider is down, but the status still says ok

Symptom: the provider fails, and GET /v1/status keeps saying "status":"ok" for about 2 minutes.

Cause: this is by design. The status changes when the oldest ticket has waited 120 seconds, the same moment as the drafts-late alert. Before that, replies are not late yet.

Fix: nothing to fix. If Larkfield wants customers told sooner, it lowers DELAYED_AFTER_SECONDS and the alert’s threshold together, so they stay in step.

Common mistake

Raising the retries during an outage

Symptom: someone raises the worker’s attempts or the provider timeout “to get through”. The queue drains no faster, and the provider’s errors rise.

Cause: while the provider cannot answer, every extra attempt is another failed call. With the breaker open, no attempt is spent anyway.

Fix: set the values back. The runbook says it in its mitigation part: do nothing to the workers.

See it end to endPost-incident review of a public outageCase study

The case study follows a real dependency cascade: one cloud provider failed, and the services built on it failed too, AI products included.

Knowledge check

Check your understanding

During the outage run, no ticket became a dead letter, although the provider failed for 5 minutes and each job has 5 attempts. Why?

Show the answer

The circuit breaker opened after 5 failed calls in a row. While it was open, the worker made no call. It gave each job back to the queue without spending an attempt, 3,102 times in the run. Only 36 real retries happened, around the moments when the breaker opened. So no job used its 5 attempts, and every ticket was triaged after the provider came back.

Check your understanding

Predict: the same outage lasts 15 minutes instead of 5, at 1 ticket a second. What happens to the draft SLO, the queue and the status?

Show the answer

The oldest ticket waits about 15 × 60 + 54 ≈ 954 seconds, over the 600 of the SLO. So every ticket that waited more than 600 seconds is bad. If the drain takes about a minute as before, that is roughly the first 6 minutes of the outage: about 350 tickets. The draft SLI falls, and the draft-fast-burn alert may fire after the recovery. The queue holds about 900 tickets, well under its 5,000 limit, so no ticket is refused. The status says “delayed” from about 2 minutes into the outage until the backlog is done. It is an estimate: a longer backlog may take longer to drain.

Check your understanding

Practical check: after your own practice outage, which outputs prove that the service has recovered?

Show the answer

GET /v1/status says "status":"ok", and GET /v1/queue shows "queued" back to 0 or a few, oldest_queued_seconds under 120, and "dead_letter":0. python -m ops incident on your telemetry files shows the breaker closed, no refused ticket, no ticket without a valid draft, and a time to a draft back to a few seconds in the last period, with new tickets in it. Each box needs its own number.

Recap

• The AI provider is a dependency that you do not run. Plan its outage in advance: a fallback, queued work, customer status and recovery checks.
• The queue and the circuit breaker keep every ticket. While the breaker is open, jobs go back to the queue without spending an attempt: 0 dead letters in the outage run.
• GET /v1/status tells customers the truth, at the same 120 seconds as the drafts-late alert. TRIAGE_FALLBACK=person sends a ticket without a valid answer to a person.
• The oldest wait decides the draft SLO: 354 seconds in a 5-minute outage, so about 9 minutes of outage would break it at this rate (an estimate from one run).
• Recovery is a list of numbers from python -m ops incident and the API, with new tickets in the last period, not the state of an alert.

Next, you write down what last week taught, without blame, and you check that one of its actions works.

Next: Post-incident learning