Skip to content
Français

Quality from live runs

  • Lesson 25
  • 35 min
  • Module 7: Alerts, service levels and quality

Where we are

In the last lesson, our alerts learned to catch the failures that traces and counters can see. Some failures leave no mark in a trace: a polite, confident reply that is wrong. In this lesson, we sample live runs for a person's review, attach the scores to the traces, and compare slices such as the language of the ticket. We start in a restaurant kitchen.

What you will be able to do

Measure whether the agent’s outcomes are right, from live runs, without a review of every run. Pick a fair sample, put each score on the run’s trace, and read the scores by slice. Then you can see a failure that no trace, counter or alert shows.

Before you startWhat to know and to have readyAbout 35 minutes

What you need to know

What you need on your computer

  • Your project at the snapshot end-of-m07 (112 passed, 1 skipped, 1 deselected), the virtual environment active.

Tested on macOS (Apple silicon) with Python 3.12.13. The Windows and Linux commands are written, not run. The week’s reviews are simulated: the agents course’s labels stand in for Priya’s judgement.

A busy kitchen sends out 300 plates a night. The chef cannot taste them all. So she tastes about one plate in five, chosen without looking at which cook made it. She writes her note on the order ticket of that plate: table 12, the risotto, too salty. Now the note points at one cook, one dish and one time.

There is a catch. A dish that only two tables order each night is tasted perhaps once a week. If that dish is always wrong, the chef may never find out. So for a new or rare dish, she tastes every plate.

The problem: a wrong answer that leaves no mark

Section titled “The problem: a wrong answer that leaves no mark”

Here is one run of the replayed week, a German ticket on Sunday:

Terminal window
python -m obs trace R-1018-011 --data data/telemetry/week
Run R-1018-011 | ticket T-90305 | chat-small (agent) | release 2.1.1 | language de
2 model call(s), 1 read(s), 2 write call(s), 4591 tokens in, 230 out, US$0.00057
Trace 1 bde727680ccdef70dc993305b94b2706 Sun 18 Oct 12:25:03 3.6 s
0.0 s |========================| 3.62 s invoke_agent support-resolver resolve rule W5
0.0 s |======== | 1.14 s chat chat-small 2172 in / 21 out US$0.00023 tool_calls
1.1 s | = | 0.00 s execute_tool get_order ok LK-642445
1.1 s | =================| 2.48 s chat chat-small 2419 in / 209 out US$0.00035 tool_calls
3.6 s | =| 0.00 s approval.request needs agent
... waited 10 min 57 s for a person (this trace has a link to 95b49ebebaf9f57a in trace bde72768...)
Trace 2 025c9cc6bf598b20f635df2170ec74ae Sun 18 Oct 12:36:03 0.0 s
0.0 s |========================| 0.01 s approve support-resolution approved by usr-sam (agent, limit 100)
0.0 s |=== | 0.00 s execute_tool reship_item done LK-642445 COMPOST x 1 key f323726a8e7dbc2e
0.0 s | ============ | 0.01 s execute_tool create_return_label done LK-642445 COMPOST key ef179176fffd19df
0.0 s | ==========| 0.00 s execute_tool send_reply sent
Outcome: resolved

Every span is green. No tool failed, no write timed out, the cost is small, and Sam approved it in 11 minutes. No flag lights up and no alert counts it. Now read the ticket and what Larkfield’s rules expect:

Terminal window
python -m resolver show T-90305 --expected
T-90305 | customer C-52445 | slice needs_person
Attachments: IMG_3381.jpg
Hallo, meine Bestellung LK-642445 ist beschädigt angekommen. Der Deckel des Komposters ist gebrochen. Foto im Anhang.
Expected: hand_to_person (rule H5) []
Why: German.

Rule H5 sends German tickets to a person. The agent reshipped the compost bin instead, and Sam approved it. The run is wrong, and nothing in its telemetry says so. Only a person who knows the rules can tell, by reading the run and the ticket.

The idea: sample, score on the trace, read by slice

Section titled “The idea: sample, score on the trace, read by slice”

Three ideas solve this, and each one has a name.

A sampled review is a person’s judgement of a sample of runs: “was the outcome right?” At Larkfield, Priya reviews about one closed run in five. She uses the same rules as the agents course’s labels.

The sampler decides which runs she sees. It must be fair: it must not pick the runs that look interesting, or the ones from the morning. Larkfield’s sampler turns the run ID into a hash, a number from 0 to 99. It picks the run when the number is under 20. A hash gives the same answer every time and in every process, so any service can decide alone. It is the same idea as sampling traces by their trace ID in The Collector.

A score on the trace is the review written as a telemetry event, gen_ai.evaluation.result, with the trace ID and span ID of the run span. Its name and attributes come from the GenAI conventions, which are still in development, like every gen_ai name in this course. Because the score carries the trace ID, it points at one run, as the chef’s note points at one plate. Any tool that reads the trace can show the score next to it.

A slice is a group of runs that share one property: the language of the ticket, the model, the release. A failure that hides in the average often shows in one slice.

A tiny worked example: the sampler by hand

Section titled “A tiny worked example: the sampler by hand”

Here are six run IDs from Monday morning, with their hash numbers:

Terminal window
python -c "from obs.alerts import sampled; import hashlib; [print(r, int(hashlib.sha256(r.encode()).hexdigest(), 16) % 100, sampled(r)) for r in ['R-1012-001', 'R-1012-002', 'R-1012-003', 'R-1012-004', 'R-1012-005', 'R-1012-006']]"
R-1012-001 50 False
R-1012-002 48 False
R-1012-003 50 False
R-1012-004 67 False
R-1012-005 2 True
R-1012-006 22 False

Only R-1012-005 has a number under 20, so it is the only one picked. In the first lesson of this module, it was the one reviewed run of the ten, and it passed. R-1012-006, at 22, is just outside. R-1012-001 and R-1012-006 were both wrong by the labels, and neither was picked. Over 316 runs, about one in five is picked; in six runs, it can be one or none.

A loop from left to right. A run closes; its run span has a trace ID. The sampler asks: is the hash of the run ID in the first 20 of 100, or is the run in a small slice? No, for 4 runs in 5: not reviewed. Yes: Priya reads the run and the ticket and asks whether the outcome is right. Her score goes on the trace as a gen_ai.evaluation.result event, pass or fail. The scores are read per slice with obs reviews --by language. Then someone decides: a fix, or a new case for the evaluation set. A dotted arrow from the decision goes back to the first box: the next release.
The sampled-review loop: from a live run to a score on its trace, to a view by slice, to a decision.
Text description of the diagram

The sampler picks about one closed run in five by the hash of its run ID, and every run of a small slice. Priya reviews each one against Larkfield’s rules. Her score becomes an evaluation event with the run span’s trace ID, so it stays attached to that run. The scores are read by slice. A failing slice leads to a fix and to new cases for the evaluation set, and the next release goes through the same loop.

  1. Read the week’s 60 reviews by the language of the ticket:

    Terminal window
    python -m obs reviews --data data/telemetry/week
    group runs reviewed share pass fail pass rate
    de 6 1 16.7% 0 1 0.0%
    en 302 58 19.2% 52 6 89.7%
    fr 8 1 12.5% 1 0 100.0%
    total 316 60 19.0% 53 7 88.3%
    Fewer than 5 reviews in: de, fr. A pass rate of 1 or 2 reviews says almost nothing.
  2. Read them by model: python -m obs reviews --by model --data data/telemetry/week.

    group runs reviewed share pass fail pass rate
    chat-small 260 48 18.5% 45 3 93.8%
    chat-strong 40 8 20.0% 7 1 87.5%
    gemma3:4b 16 4 25.0% 1 3 25.0%
    total 316 60 19.0% 53 7 88.3%
    Fewer than 5 reviews in: gemma3:4b. A pass rate of 1 or 2 reviews says almost nothing.
  3. List the failed reviews and their trace IDs: python -m obs reviews --failed --data data/telemetry/week.

    run ticket language model outcome reviewed trace
    R-1013-032 T-90501 en chat-small resolved Wed 14:09 94678f574e2b63b9b1f853342ffa7bb2
    R-1014-040 T-90104 en chat-strong resolved Fri 09:17 f0433c4403e2767e1d69ece18a947526
    R-1015-002 T-90207 en chat-small resolved Fri 09:28 96b3c9ad4158dee283f796d028e31bbb
    R-1017-007 T-64872 en gemma3:4b handed_to_person Sun 10:40 7aedb15baced5db31b4c6324b3b76044
    R-1017-010 T-90605 en gemma3:4b handed_to_person Sun 11:05 7ad8163fbd3bd553a71127f1e2e689c5
    R-1018-007 T-90103 en gemma3:4b resolved Mon 10:24 46e00e2b1dcdc3558d8011de1affa068
    R-1018-011 T-90305 de chat-small resolved Mon 12:36 bde727680ccdef70dc993305b94b2706
    7 failed review(s) of 60
  4. Read the score of the German run as the event it is: python -m obs events R-1018-011 --name gen_ai.evaluation.result --data data/telemetry/week.

    Mon 19 Oct 12:36:03 gen_ai.evaluation.result trace bde72768 {"gen_ai.evaluation.name": "sampled-review", "gen_ai.evaluation.score.value": 0.0, "gen_ai.evaluation.score.label": "fail", "run.id": "R-1018-011", "ticket.id": "T-90305", "reviewer": "usr-priya"}
    1 event(s)

Three things matter here. The score’s trace ID, bde72768…, is the first trace of R-1018-011: the score is on that run, not on the ticket. The score arrived on Monday 19 October, one day after the run closed: reviews are delayed, as in the MLOps course’s Delayed labels. And the double refund, R-1014-040, failed its review too.

The model split also tells a story. The local model’s trial failed 3 of its 4 reviews. That is the same weekend that the loop and handoff alerts saw, now from the side of quality.

A sample estimates the whole. Here is the sample next to the truth, which a real team never has: the course’s labels for all 316 runs of each week.

Simulated reviews (the course's labels stand in for Priya) and the course's labels for every run; replayed decisions, constructed traffic.
Text description of the chart

Four bars. The replayed week: the sample says 88.3% (53 of 60), all runs 86.4% (273 of 316). The healthy week: the sample says 91.7% (55 of 60), all runs 85.1% (269 of 316).

The samples are close to the truth, within 2 to 7 points. But they rank the two weeks the wrong way round. The replayed week was slightly better by the labels, because chat-strong got the hard tickets right. Sixty reviews cannot see a difference of one point; one review more or less moves the rate by 1.7 points. Use a weekly sample to watch for large changes, and do not compare two weeks that are close. How to put an interval on a sample is in Uncertainty and slices in the evaluation course.

Camille, who answers the French tickets, reads the first table and worries: “Non-English tickets fail.” Put the two languages together, and the labels say 9 of 14 were right: 64%. That looks like a problem for Camille’s queue.

Split them, and the picture changes:

The course's labels for all 316 runs, which a real team does not have; replayed decisions, constructed traffic. The German slice is one practice ticket, T-90305, drawn 6 times.
Text description of the chart

Four bars. English: 264 of 302 runs right, 87.4%. French: 8 of 8, 100%. German: 1 of 6, 16.7%. Non-English together: 9 of 14, 64.3%. The low non-English bar comes from German alone.

French tickets were right 8 of 8 times. The slice that fails is German: 1 of 6, and the one right run was the local model’s, which handed the ticket over. A slice “non-English” hides which language fails, and it would have sent Camille to fix a queue that works. Slice by each language.

Two more facts change what to do about it. First, the German slice is one practice ticket, T-90305, drawn six times: one ticket, not a population. Second, the healthy week fails it too, 0 of 6. So it is not a release that broke German. It is a standing gap: chat-small does not follow rule H5. No alert can catch it, because nothing changed.

The 20% sampler saw one German run a week. One review says 0% or 100%, nothing in between. The fix is the chef’s: review every run of a small slice. obs reviews --queue lists the runs that the sampler picks and that nobody has reviewed yet:

Terminal window
python -m obs reviews --queue --every de,fr --data data/telemetry/week
run ticket closed language model outcome
R-1012-036 T-90702 Mon 16:39 fr chat-small resolved
R-1012-047 T-90305 Tue 08:54 de chat-small resolved
…
R-1018-013 T-90305 Sun 12:55 de gemma3:4b handed_to_person
12 run(s) to review: 20% of runs by the hash of the run ID, and every run in de, fr; 60 run(s) already reviewed.

Twelve more reviews a week, 20% more of Priya’s time, and every German and French run is seen. With all six German runs reviewed, the German line would say 1 of 6, a clear failure. This is called stratified sampling: a share of the large slices, and all of the small ones.

Now be the reviewer. You run the German ticket, approve it, review it, and read the score from your own telemetry.

  1. Delete the folder work. In PowerShell: Remove-Item -Recurse work.

  2. Run the German ticket: python -m resolver run T-90305. The proposal is a reshipment and a return label, with a reply draft in German. Its last line is Nothing has changed yet. Next: python -m resolver approve T-90305-agent-chat-small-1 --as agent (or: reject).

  3. Approve it, as Sam did: python -m resolver approve T-90305-agent-chat-small-1 --as agent. The last line says Reply sent. Ticket: resolved.

  4. Check it against the rules: python -m resolver show T-90305 --expected says Expected: hand_to_person (rule H5). So the outcome is wrong.

  5. Write the review:

    Terminal window
    python -m resolver review T-90305-agent-chat-small-1 fail --by usr-priya
    Review of T-90305-agent-chat-small-1 (resolved): fail, by usr-priya. Attached to trace …, span ….

    Your trace and span IDs differ from ours: they are new for every run.

  6. Read your reviews: python -m obs reviews.

    group runs reviewed share pass fail pass rate
    de 1 1 100.0% 0 1 0.0%
    total 1 1 100.0% 0 1 0.0%
    Fewer than 5 reviews in: de. A pass rate of 1 or 2 reviews says almost nothing.
  7. Check that the score is on the run’s trace: python -m obs reviews --failed. The trace column shows the same trace ID as the line Trace 1 of python -m obs trace T-90305-agent-chat-small-1.

  8. Run one more ticket, python -m resolver run T-80008, and look at the queue with every run picked: python -m obs reviews --queue --share 1. It lists T-80008’s run, the one that nobody has reviewed. Review it: python -m resolver review T-80008-agent-chat-small-1 pass.

resolver review refuses a run that is not closed yet, and a run with no trace. A second review of the same run replaces the first in obs reviews; the event of the first one stays in the telemetry.

With a Collector, the course’s configuration sends the score to Loki in Grafana, like the workflow’s other events. It keeps its trace ID, so a log line leads to the trace. In Module 8, the course team’s 60 sampled reviews become Langfuse scores on the same traces.

Hand judges and datasets to the evaluation course

Section titled “Hand judges and datasets to the evaluation course”

A person’s review is slow and costly, but it is the reference. Two tools make it go further, and the evaluation course teaches both:

This course’s part is the plumbing: a fair sample, each score on the right trace, and a view by slice.

When to sample for review, and what it costs

Section titled “When to sample for review, and what it costs”

Sam asks: “Why not ask the customers? They know if the answer was right.”

Priya answers: “Some do, and we read it. But most customers say nothing, and an angry one is not a sample. A review follows our rules, on runs chosen fairly.”

Use sampled review for the one indicator that telemetry cannot compute: whether the outcome was right. Its costs are real. Priya’s time grows with the sample, and the results arrive a day late. The reviewer must read the ticket, whose text is not in the telemetry, on purpose: the review opens it in the help desk. Keep free-text comments out of the event too, because a note can carry a customer’s name.

Do not use reviews for what telemetry counts for free, such as handoffs or cost. And do not review only flagged runs: they say nothing about the rest.

Common mistake

The review lands on the wrong run

Symptom: a tool joins reviews to runs by ticket ID. Priya fails T-90305, and the dashboard shows the failure on a run that handed the ticket to a person, which was right.

Cause: a ticket can have many runs. T-90305 had six in the week. A join by ticket picks one of them, here the last: R-1018-013, the local model’s run that handed the ticket over. python -m obs trace T-90305 --data data/telemetry/week shows the same choice: a ticket opens its latest run.

Diagnose: compare the score’s trace ID with the run’s traces. In obs reviews --failed, the trace column must equal one of the run’s traces in obs trace RUN_ID.

Recover: attach each score to the run span’s trace, as resolver review and the week’s reviews do, and join by trace ID. Keep the run ID in the event too, for tools that cannot read traces.

The healthy week met all four objectives in the first lesson. Check its quality by slice:

  1. Run python -m obs reviews --data data/telemetry/healthy-week, then --by model.
  2. Run python -m obs reviews --queue --every de,fr --data data/telemetry/healthy-week. How many reviews does the stratified sample add?
  3. Write two sentences for Grace: is the healthy week’s quality good, and what should Priya change?

Our answer is in the first knowledge check.

Check your understanding

Your turn: what do the healthy week's reviews say, and what should Priya change?

Show the answer

The sample passes 55 of 60 reviews, 91.7%, above the 85% objective. English is 54 of 58 and French 1 of 1, but German is 0 of 1 again. There is only one model, so --by model shows the same 55 of 60. The stratified queue adds 12 runs, all German and French. By the course’s labels, the healthy week was right on 0 of its 6 German runs. So the week meets its objective, but the German ticket fails in both weeks: a standing gap, not an incident. Priya should review every German and French run, and add T-90305 to the evaluation set.

Check your understanding

Predict: the samples say the healthy week (91.7%) was better than the replayed week (88.3%). The labels of all runs say the opposite (85.1% against 86.4%). How can both be true?

Show the answer

Each week’s sample is 60 runs picked by the hash of the run ID. One review more or less moves a pass rate by 1.7 points. So two samples of 60 often differ by a few points for no reason. The true difference between the weeks is 1.3 points: too small for 60 reviews to see. The replayed week was slightly better by the labels because chat-strong got the hard tickets right. Use a weekly sample to watch for large changes and for slices that fail badly, not to rank two weeks that are close. For an interval on a sample, see the evaluation course.

Check your understanding

Completion check for this module: show that each alert fires on its scenario and stays quiet on a healthy week. Give the commands, the times, and the test.

Show the answer

python -m obs alerts --data data/telemetry/week fires all five rules, each soon after the cause the course created. cost-per-resolved fires at Wednesday 10:39, 1 hour 39 minutes after the switch to chat-strong. unreconciled-writes fires at Wednesday 13:36, on R-1014-024, the first retry with a new key. reply-failures fires at Thursday 12:32, two minutes into the reply outage. loop-stops (Saturday 10:40) and handoff-rate (Saturday 17:43) fire during the local model’s trial. python -m obs alerts --data data/telemetry/healthy-week prints quiet for all five. The tests say the same: test_every_alert_fires_on_the_week and test_the_healthy_week_has_no_alert in tests/test_saved_week.py, and test_a_ratio_alert_waits_for_enough_runs in tests/test_stage_m07.py, which also shows the false alarm that the minimum of 20 runs removes. A good answer adds that Grafana fires the same alerts a few practice hours later, and that a quiet alert is not a recovery check.

Recap

  • Some failures leave no mark in a trace. R-1018-011 resolved a German ticket that rule H5 sends to a person, and every span was green.
  • A sampled review is a person’s judgement of a fair sample. Larkfield’s sampler picks a run when the hash of its run ID is under 20 of 100: the same answer everywhere.
  • A score is a gen_ai.evaluation.result event with the run span’s trace ID. It points at one run, not at a ticket, and it arrives a day late.
  • Read scores by slice, by each language. French was right 8 of 8; German 1 of 6, one ticket that fails in both weeks. Review small slices in full: 12 more reviews a week.
  • A sample of 60 cannot rank two close weeks. Judges and evaluation sets belong to the evaluation course; this course keeps the sample fair and the score on the right trace.

In this module, you gave the workflow objectives, alerts that lead to action, and a measure of quality from live runs. But all of it lives in Grafana and in the course’s own obs. Priya would like the scores next to each model call, in a tool made for language models. Next, in Module 8, you send the same traces to Langfuse.

Next: Send the same traces to Langfuse