Alerts that lead to action
- Lesson 24
- 40 min
- Module 7: Alerts, service levels and quality
Where we are
In the last lesson, we wrote four objectives for the workflow and read the replayed week against them. But a table that nobody reads at 18:00 on a Saturday changes nothing. In this lesson, we write alerts on the indicators we mean, with example runs and a first step, we remove a false alarm, and we check that each alert fires on its scenario and stays quiet on a healthy week. We start with the warning lights of a car.
Lesson 24 of 28 · Observability for AI Agents: Traces, Costs and Audit TrailsAlerts that lead to action
1. Why agents need their own observability
2. OpenTelemetry for agents
3. Instrument the workflow
4. Find agent failures in Grafana
5. Cost
6. Audit trails
7. Alerts, service levels and quality
8. Tools: Langfuse, others and the run inspector
What you will be able to do
Turn the workflow’s indicators into alerts that a person can act on. Each one names its indicator, waits for enough runs, has one owner, links to example runs and says what to do first. Then check each alert on a week with problems and on a healthy week.
Before you startWhat to know and to have ready
What you need to know
- The four objectives and
obs slo, from Service levels for agents. - The five failure patterns and
obs find, from Agent failure patterns. - Optional: runbooks and alert noise for one service, from Runbooks in the MLOps and LLMOps course.
What you need on your computer
- Your project with this module’s files: the snapshot
end-of-m07, wherepython -m pytest -qends with112 passed, 1 skipped, 1 deselected. - Optional: Docker, to see the same rules in Grafana. Every step works without it.
Tested on macOS (Apple silicon) with Python 3.12.13. The Windows and Linux commands are written, not run. The Grafana times come from the course team’s runs of the replayed week at 1,000 times its real speed.
Two warning lights and a smoke alarm
Section titled “Two warning lights and a smoke alarm”A car has two kinds of warning light. The red oil light has one meaning: the engine has no oil pressure. The manual says what to do first: stop the engine now. Every driver acts on it. The amber “check engine” light can mean a hundred things, from a loose fuel cap to a broken sensor. The driver cannot act on it alone: a mechanic must read a code first.
Now think of a smoke alarm above the toaster. It goes off at every breakfast. After a week, someone takes out the battery. When a real fire starts, the alarm says nothing.
A useful alarm has one meaning, says what to do first, and does not go off for nothing.
The problem: a false alarm on Monday, and a real warning on Wednesday
Section titled “The problem: a false alarm on Monday, and a real warning on Wednesday”The course team ran the replayed week through Grafana with the first version of the alert rules. On Monday at 09:50 in practice time, the alert “Too many tickets handed to a person” fired. Nothing was wrong: it was the healthy part of the week, release 2.0.0 with no incident.
Two days later, on Wednesday at 13:36, another alert fired on the saved runs: “A write timed out and nobody checked whether it happened”. Its example run, R-1014-024, created a return label. The order system had only one label: the first call had timed out before it wrote anything. So the alert looked wrong too.
It was not. Twenty hours later, on Thursday morning, the workflow refunded two customers twice. They were R-1014-051 and R-1014-040, the double refund of Module 1. The Wednesday alert had shown the cause in its first run. A team that had learned on Monday that alerts cry wolf would not have looked.
The idea: an alert is an indicator, a rule and a message
Section titled “The idea: an alert is an indicator, a rule and a message”An alert is a rule on an indicator that tells a person when the rule becomes true. At Larkfield, each alert has five parts:
- The indicator you mean. Count the thing that must not happen (“a write timed out and no reconcile checked it”), or the indicator of an objective. Not a proxy, such as “errors in the logs”.
- A window and a minimum of runs. The rule looks at the last 45 closed runs. A rate needs enough runs before it means anything.
- A severity and one owner. SEV-1 is “customers may lose money now”; SEV-3 is “look at it during the working day”. One person is called first, as in the MLOps course’s ownership lesson.
- Example runs. The alert names the runs behind it, so the first click opens a trace.
- A first step. One command or check, from the runbook. The MLOps course’s Runbooks teaches the rest.
Here is one rule from grafana/alerts/rules.json, the file that obs alerts reads:
{ "uid": "unreconciled-writes", "title": "A write timed out and nobody checked whether it happened", "indicator": "unreconciled_writes", "op": ">", "threshold": 0, "window": 45, "severity": "SEV-1", "owner": "Amira", "first_step": "Run `python -m obs find duplicate_write` (or the TraceQL query `duplicate write` in grafana/traceql.md), then ask the system that owns the write (payments for a refund, orders for a label or a reshipment) about each operation ID before anyone sends the change again.", "example_query": "{ span.larkfield.write.outcome = \"timeout\" }", "promql": "sum(increase(larkfield_writes_total{larkfield_write_outcome=\"timeout\"}[90s])) - sum(increase(larkfield_reconciles_total[90s]))"}The same rule exists twice. rules.json has it for the saved runs, with its window counted in runs. agent-alerts.yaml has it for Grafana, as the PromQL query in the last line. The course team writes both files from one list, so the two never say different things.
A tiny worked example: why Monday fired
Section titled “A tiny worked example: why Monday fired”The handoff rule says “more than 45% of the last 45 closed runs were handed to a person”. At the start of a week, there are not 45 runs yet. Without a minimum, the rule divides by whatever it has. Here are the first five runs of Monday, the same in both weeks:
| After closed run | Handed to a person | Handoff rate | Over 45%? |
|---|---|---|---|
| 1 | 0 of 1 | 0.0% | no |
| 2 | 1 of 2 | 50.0% | yes: the alert fires |
| 3 | 2 of 3 | 66.7% | yes |
| 4 | 2 of 4 | 50.0% | yes |
| 5 | 2 of 5 | 40.0% | no: it stops |
Two runs closed, one went to a person, and the rate is 50%. That is not a rate; it is one ticket. The fix is a minimum of runs: the rule is not evaluated until its window holds 20 runs. With 20 runs, one ticket moves the rate by 5 points, not by 50.
- the replayed week
- the healthy week
- threshold and run 20
Text description of the chart
Two lines over the 316 closed runs of each week. Both start with a spike to 67% after the third run. From run 20 on, the healthy week stays between about 16% and 42%. The replayed week climbs over 45% from run 282 and peaks at 53%, when the local model hands most of its tickets to a person.
The minimum removes the spike of the first five runs, which is noise, and keeps Saturday’s rise, which is real.
Larkfield’s five alerts
Section titled “Larkfield’s five alerts”| Alert | Rule over the last 45 closed runs | Severity, owner | Why this rule |
|---|---|---|---|
unreconciled-writes |
write timeouts with no reconcile > 0 | SEV-1, Amira | A write that may have happened twice can cost a customer money. One is enough. |
reply-failures |
failed replies > 0 | SEV-2, Grace | A customer who got no reply waits. One is enough. |
loop-stops |
runs stopped by a loop limit ≥ 3 | SEV-3, Amira | One stop is the limit doing its job. Three in a day is a pattern. |
cost-per-resolved |
US$ per resolved ticket > 0.002 | SEV-3, Kwame | The budget objective of the last lesson. |
handoff-rate |
handed to a person > 45% | SEV-3, Grace | The handoff objective of the last lesson. |
The first three count things that must not happen; one is already a reason to act. The last two are rates against an objective, and only they need the minimum of 20 runs. So in the rules file, only these two have min_runs: 20; the rule above has none.
Replay the week
Section titled “Replay the week”-
Run the five rules on the replayed week:
Terminal window python -m obs alerts --data data/telemetry/weekunreconciled-writes (unreconciled_writes > 0 over the last 45 runs): FIRED 1 time(s)from Wed 14 Oct 13:36 (run 129, R-1014-024) until Thu 15 Oct 18:12 (run 202); peak 3example runs: R-1014-024 (python -m obs trace RUN_ID)first step: Run `python -m obs find duplicate_write` (or the TraceQL query `duplicate write` in grafana/traceql.md), then ask the system that owns the write (payments for a refund, orders for a label or a reshipment) about each operation ID before anyone sends the change again.reply-failures (reply_failures > 0 over the last 45 runs): FIRED 1 time(s)from Thu 15 Oct 12:32 (run 173, R-1015-017) until Fri 16 Oct 12:36 (run 229); peak 9example runs: R-1015-017 (python -m obs trace RUN_ID)first step: Open the example runs (`python -m obs trace RUN_ID`): did each ticket go to a person, or close as resolved? `python -m obs find no_action` lists the tickets closed without a reply. Send those replies when the reply service works again.loop-stops (loop_stops >= 3 over the last 45 runs): FIRED 1 time(s)from Sat 17 Oct 10:40 (run 267, R-1017-007) still firing at the end; peak 11example runs: R-1017-007, R-1017-006, R-1017-005 (python -m obs trace RUN_ID)first step: Open three stopped runs (`python -m obs find loop`): which model, which step repeats? Compare with the last release.cost-per-resolved (cost_per_resolved > 0.002 over the last 45 runs): FIRED 1 time(s)from Wed 14 Oct 10:39 (run 120, R-1014-016) still firing at the end; peak 0.0064example runs: R-1014-009, R-1014-016, R-1014-011 (python -m obs trace RUN_ID)first step: Split the cost by model and slice (`python -m obs cost --by model`, the Cost panels): which change raised it? The budget (US$0.002 per resolved ticket) is an assumption: Omar's.handoff-rate (handoff_rate > 0.45 over the last 45 runs): FIRED 1 time(s)from Sat 17 Oct 17:43 (run 282, R-1017-022) until Sun 18 Oct 13:18 (run 305); peak 0.5333example runs: R-1017-022, R-1017-021, R-1017-020 (python -m obs trace RUN_ID)first step: Split the handed-over runs by model (`python -m obs cost --by model --outcome handed_to_person`) and compare with the day before: a model or a release with many more handoffs is the first suspect. Open its example runs. -
Run them on the healthy week:
python -m obs alerts --data data/telemetry/healthy-week. All five lines end withquiet. -
Now remove the minimum and run the healthy week again. The option
--min-runs 1evaluates every rule from the first run on:Terminal window python -m obs alerts --data data/telemetry/healthy-week --min-runs 1Every rule is evaluated from closed run 1 on (the rules file's min_runs is not used).unreconciled-writes (unreconciled_writes > 0 over the last 45 runs): quietreply-failures (reply_failures > 0 over the last 45 runs): quietloop-stops (loop_stops >= 3 over the last 45 runs): quietcost-per-resolved (cost_per_resolved > 0.002 over the last 45 runs): quiethandoff-rate (handoff_rate > 0.45 over the last 45 runs): FIRED 1 time(s)from Mon 12 Oct 03:12 (run 2, R-1012-002) until Mon 12 Oct 08:43 (run 5); peak 0.6667the window held only 2 run(s) when it firedexample runs: R-1012-002 (python -m obs trace RUN_ID)first step: Split the handed-over runs by model (`python -m obs cost --by model --outcome handed_to_person`) and compare with the day before: a model or a release with many more handoffs is the first suspect. Open its example runs.This is the Monday false alarm, on a week with no incident. The line
the window held only 2 run(s)says why. -
Run the module’s tests of the rules:
python -m pytest -q tests/test_stage_m07.py tests/test_saved_week.py. The last line says14 passed. One test checks that each rule fires on the week and stays quiet on the healthy week. Another checks the minimum. Without it, the handoff rule fires on Monday of the healthy week. With it, the healthy week is quiet, and Saturday still fires at run 282.
Text description of the diagram
Each alert starts soon after its cause. The cost alert fires 1 hour 39 minutes after release 2.1.0, and the reply alert two minutes into the outage. The loop and handoff alerts fire during the local model’s trial. Grafana’s circles come a few practice hours later. The healthy week stays quiet on all five.
What each alert caught
Section titled “What each alert caught”- Cost per resolved ticket, Wednesday 10:39: Omar’s switch to chat-strong for hard tickets. Its three example runs are the three chat-strong runs that Module 5 found behind the first crossing.
- Unreconciled writes, Wednesday 13:36: release 2.1.0’s naive retry, on a return label. The next section reads it.
- Reply failures, Thursday 12:32: the reply service was down from 12:30 to 15:30, a practice incident we caused on purpose. Nine replies failed in one window.
- Loop stops, Saturday 10:40: the weekend trial of the free local model gemma3:4b. Thirteen of its 16 runs were stopped by a loop limit: 10 for too many tool errors and 3 for a repeated call.
- Handoff rate, Saturday 17:43: the same trial. Every one of gemma3:4b’s 8 runs on Saturday went to a person.
The warning on Wednesday
Section titled “The warning on Wednesday”Here is the alert’s example run:
python -m obs trace R-1014-024 --data data/telemetry/weekRun R-1014-024 | ticket T-90604 | chat-small (agent) | release 2.1.0 | language en2 model call(s), 1 read(s), 2 write call(s), 4571 tokens in, 157 out, US$0.00054…Trace 2 0bf792cf609032796a723248493cf3cd Wed 14 Oct 13:36:16 10.0 s 0.0 s |========================| 10.00 s approve support-resolution approved by usr-sam (agent, limit 100) 0.0 s |========================| 10.00 s execute_tool create_return_label ERROR timeout LK-645334 LADDER-3 key 2ed3b2a93ab0f494 10.0 s | =| 0.00 s execute_tool create_return_label done LK-645334 LADDER-3 key 2ed3b2a93ab0f494-retry1
…Read the two keys. The first call timed out after 10 seconds. The retry went out at once, with a new idempotency key: 2ed3b2a93ab0f494-retry1. A new key tells the order system “this is a different request”. Here, the first call had not written anything, so the customer got one label. With a refund, the same code pays twice whenever the first call times out after the write. That is exactly what happened on Thursday morning, during the slow-payments incident.
This is a real surprise of the course’s data. The alert was right about its indicator: a write timed out and nobody checked it. It was “wrong” only about the harm. The trace cannot tell whether a timed-out write happened, as Module 4 showed; the system that owns the write can. So the first step does not say “find the double refund”. It says: ask that system about each operation ID, before anyone sends the change again.
Text description of the diagram
On Wednesday at 13:36, the count of unchecked write timeouts became 1, and the rule fired. The message names a severity, an owner, an example run and a first step. The example run shows the retry with a new key. The runbook step checks each operation in the system that owns it and stops that retry. Then new runs bring the indicator back to 0.
If Amira had followed it on Wednesday afternoon, she would have found the new key and turned the naive retry off. Release 2.1.1 did that on Friday at 11:30, a day after the two double refunds.
The same rules in Grafana
Section titled “The same rules in Grafana”In Grafana, each rule is a PromQL query on the workflow’s counters. Grafana checks it every 10 seconds over a 90-second window, about one practice day at 1,000 times the real speed. The course team’s run fired all five. Handoff rate fired on Monday at 09:50 and on Saturday at 18:14, and cost on Wednesday at 16:41. Unreconciled writes fired on Wednesday at 19:37, reply failures on Thursday at 16:44 and loop stops on Saturday at 15:23. A 10-second check is 2.8 practice hours, so Grafana is later than obs. Write the runbook for the order of the alerts, not the minute.
Grafana’s window is a time, not a count of runs. So the number of runs in it changes with the traffic: few at night, few on Monday morning. The guard in agent-alerts.yaml puts the minimum into the query of the two rate rules:
(sum(increase(larkfield_runs_total{larkfield_outcome="handed_to_person"}[90s])) / sum(increase(larkfield_runs_total[90s]))) and sum(increase(larkfield_runs_total[90s])) >= 20and … >= 20 keeps the result only when the window holds 20 closed runs or more. With fewer, the query returns nothing, and the rule’s noDataState: "OK" keeps the alert quiet. Each rule also carries its example runs as an annotation, a TraceQL query to paste into Explore:
example_runs: "Explore, Tempo, TraceQL: { span.larkfield.outcome = \"handed_to_person\" }"The course team ran both weeks in Grafana again with these rules, plus one copy of the handoff rule without the guard, to compare. On the replayed week, the copy without the guard fired on Monday at 09:45 and on Saturday; the guarded rule fired only on Saturday, at 20:51. The other four rules fired as before: cost per resolved ticket on Wednesday at 17:32, the unreconciled write on Wednesday at 18:56, reply failures on Thursday at 17:14 and loop stops on Saturday at 15:16. The course team ran the healthy week in Grafana three times. In the run that had the copy without the guard, the copy fired on Monday at 08:22: the same false alarm, with no incident at all. Four rules stayed quiet every time. The guarded handoff rule stayed quiet once, and fired once on Saturday at about 22:10 in the other two runs. So the guard removes the Monday alarm, but not every alarm: on a weekend evening, a window of 90 seconds holds just over 20 runs, and the share of handoffs still jumps. A larger minimum or a longer window would quiet it, and would also make the alert slower. obs alerts counts its window in runs, not in time, and stays quiet on the healthy week. Grafana’s alert is the one that pages Grace, so measure it on a healthy week before you trust it. The times are practice times, from a replay at 1,000 times speed; Grafana checks a rule every 10 seconds, which is about 2.8 practice hours, so your times can differ by a few hours. The saved runs and the module’s test show the same result.
One more detail makes these rules work at all. Prometheus’s increase() needs two samples of a series to see a change. A counter that starts at 1, at the first unchecked write, has no earlier sample, so the first event is invisible. That is the MLOps course’s alert that cannot fire. The workflow avoids it: telemetry.zero_series() adds 0 to every series that an alert reads, when the process starts.
When to alert, and what it costs
Section titled “When to alert, and what it costs”Sam asks: “Why not alert on everything, to be safe?”
Amira answers: “Because every alert wakes a person. If three of them are noise, the fourth one is ignored, like the Wednesday one almost was.”
Alert on what a person must act on soon: a count of things that must not happen, and an objective that breaks. Do not alert on what nobody can act on: put it on a dashboard. Each alert costs an owner’s attention, a runbook and tests. The minimum of runs has a cost too: on a quiet night, a rate rule sees nothing. That is why the counts of harm have no minimum in Grafana.
A rule over a window tells you that something is wrong now. It does not say how fast an objective’s error budget is being spent. For that, the MLOps course alerts on the burn rate, the speed at which the budget goes, in Service objectives. Use burn-rate alerts for objectives with a large budget, and simple rules for the things that must not happen at all.
Common mistake
The alert went quiet, but nothing was fixed
Symptom: unreconciled-writes fires on Wednesday at 13:36 and stops on Thursday at 18:12. Someone closes the incident because “the alert resolved”.
Cause: the rule counts the last 45 closed runs. On Thursday at 18:12, the runs with unchecked writes left the window. The two customers who were refunded twice on Thursday morning were still refunded twice.
Diagnose: look at what the alert counts, not at its state. python -m obs find duplicate_write --data data/telemetry/week still lists R-1014-024, R-1014-040 and R-1014-051.
Recover: close an incident on evidence from the system of record. Check each operation ID, handle each extra refund, and remove the cause (release 2.1.1). The MLOps course calls these the recovery checks of a runbook.
Your turn: two objectives and their alerts
Section titled “Your turn: two objectives and their alerts”Write two new rules and replay both weeks with them. Use the indicators that obs/alerts.py computes: resolved_rate, handoff_rate, cost_per_resolved, unreconciled_writes, reply_failures, loop_stops, flagged_runs and approval_wait_p95_h. The operators are >, >= and <.
-
Create the folder
workif it does not exist. Create the filework/my-rules.jsonwith this shape:{"rules": [{"uid": "...", "title": "...", "indicator": "...", "op": "<", "threshold": 0.5, "window": 45, "min_runs": 20,"severity": "SEV-3", "owner": "...", "first_step": "..."}]} -
Write one rule for an objective, for example “at least half of the tickets end resolved”. Write one rule for a count of harm, for example “three or more flagged runs”. Leave
min_runsout of the count rule: one harm is enough. -
Replay them:
python -m obs alerts --data data/telemetry/week --rules work/my-rules.json, then the same withdata/telemetry/healthy-week. -
For each rule, write down when it fires, which scenario it caught, and whether the healthy week stays quiet.
Our rules and their results are in the first knowledge check.
Knowledge check
Section titled “Knowledge check”Check your understanding
Your turn: which two rules did you write, when do they fire on the week, and are they quiet on the healthy week?
Show the answer
Ours: resolved-share (resolved_rate < 0.5, SEV-3, Grace) and flagged-runs (flagged_runs >= 3, SEV-2, Amira), over 45 runs; the rate rule from run 20 on, the count rule from the first run. On the replayed week, resolved-share fires from Saturday 17:43 (run 282) to Sunday 12:38, lowest 0.4222: the local model’s trial. It fires a second time from Sunday 12:55 to 13:11, because it flaps around 50%. A real rule needs a margin before it resolves. flagged-runs fires from Thursday 09:17 (run 157) to Friday 10:25, peak 6. Its example runs are the double refunds: R-1014-040, R-1014-051 and R-1014-024. It fires again from Sunday 11:15, with a hidden tool error and the loops. Both are quiet on the healthy week. flagged-runs overlaps the existing alerts: decide whether it adds anything before you keep it.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
Predict: the handoff rule without a minimum fired on Monday of the healthy week. Why, and why do the three count rules not need the minimum in Grafana?
Show the answer
After two closed runs, one was handed to a person: 1 ÷ 2 = 50%, over 45%. After three, 2 ÷ 3 = 66.7%. That is one or two tickets, not a rate. With a minimum of 20 runs, one ticket moves the rate by at most 5 points. The count rules ask a different question: did something that must not happen happen? One unchecked write is already a reason to act. A minimum would hide it at night, when Grafana’s 90-second window holds few runs.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
The unreconciled-writes alert fired on Wednesday at 13:36 on a run that did not pay twice. Was it a false alarm? What should Amira have done?
Show the answer
No. Its indicator was true: a write timed out and nobody checked whether it happened. R-1014-024’s first create_return_label call timed out, and the retry went out with a new key, …-retry1. The trace cannot say whether the first call wrote anything; here it had not. Amira should have followed the first step: ask the system that owns the write about each operation ID. The trace also showed the cause, release 2.1.0’s retry with a new key. Turning it off on Wednesday would have prevented Thursday’s two double refunds (R-1014-051 and R-1014-040). Release 2.1.1 did it only on Friday at 11:30.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Recap
- An alert is an indicator, a rule and a message. It has the indicator you mean, a window, a severity, one owner, example runs and a first step.
- A rate needs enough runs. Without a minimum, the handoff rule fired on Monday of the healthy week after two runs. With 20 runs, the healthy week is quiet and Saturday’s real rise still fires. Counts of harm need no minimum.
- All five alerts fire on their scenario in the replayed week and stay quiet on the healthy week, on saved runs (
obs alerts) and in the tests. In Grafana they fire a few practice hours later, and the guarded handoff rule fired once on the healthy week’s Saturday evening in two of three runs: a window in time is not a window in runs. - The Wednesday 13:36 alert looked wrong, and it was the warning. Its example run showed the retry with a new key, 20 hours before the first double refund.
- An alert that stops is not a fix: check the system of record.
You can now hear about the failures that traces and counters can see. But some failures leave no mark at all: a polite, confident reply that is wrong, like the German tickets that the agent resolved. No counter goes up. Next, you sample live runs for a person’s review and attach the scores to the traces.
Next: Quality from live runs