Delayed labels

What you will be able to do

Measure whether the assistant’s decisions are right, although the true answers arrive hours or days later. Decide which quick signals to watch until then, and how many tickets people must check by hand to trust the result.

Before you startWhat to know and to have readyAbout 50 minutes

What you need to know

• Data drift, performance decay, PSI and the drift run, from Data and concept drift.
• The quality SLO and the safety SLO, from Service objectives.
• Confidence intervals, from Uncertainty and slices. Random review samples, from Production feedback.

What you need on your computer

• Your project after the last lesson: python -m pytest prints 207 passed.
• Docker running PostgreSQL and Valkey, and four terminals with the virtual environment active.
• TELEMETRY_FILE=telemetry/{service}.jsonl in your .env, from Logs, metrics and traces.

Waiting for the lab result

You feel ill, and the doctor takes a blood sample. The lab result comes in three days. That result is the truth: it says what you have.

The doctor does not wait three days and do nothing. A nurse checks your temperature and pulse every few hours. These quick signs do not give the diagnosis. But if your temperature jumps, the doctor acts at once, without the lab.

The lab also does not test all your blood. A small sample is enough, if it is taken well. A bigger sample gives a surer answer, and it costs more.

The help desk’s assistant lives with the same three facts. The truth comes late, the quick signs come early, and the truth comes from a sample.

The problem: a quality alert that stayed quiet

Larkfield’s quality SLO is “90% of reviewed tickets agree with the agent’s review”. In Grafana, Priya wrote an alert for it, reviews-disagree. It fires when more than 10% of the reviews of the last 30 minutes disagree with the assistant.

In the drift run of the last lesson, agreement fell to 85.3% in the policy-silent block. That is 14.7% disagreement, well over 10%. The alert did not fire.

Why? The run lasted 27.5 practice minutes. So a 30-minute window always held every review since the start. It mixed the good reviews of the first 2,000 tickets with the bad ones of the last 1,000. The course’s authors replayed that window over the saved run, review by review:

1445 reviews, 114 disagree (7.9% over the whole run)
30-minute window (after the first 50 reviews): highest disagreement 7.9% at minute 26.7

The good old reviews hid the bad new ones. The window was too long for the change, so the alert could not fire.

There is a second problem. When the quality falls, the reviews that would show it do not exist yet. In the drift run they came 120 practice seconds later. At Larkfield, they come hours or days later. Until then, nothing measures the quality at all.

The idea: labels, delayed labels and proxy signals

A label is the true answer for one input. For the assistant, it is the team that was right and whether a person had to handle the ticket. At Larkfield, labels come from the agents. When an agent finishes a ticket, the help desk’s tool records what was right: a review.

A delayed label is a label that arrives long after the decision. It is the lab result. The quality SLO and the safety SLO can only use delayed labels.

A proxy signal is a quick signal that often moves with quality, but is not quality. It is the temperature. An answer without text, a jump in tokens or in the share sent to a person: you see them seconds after the ticket. They tell you where to look. They cannot tell you that the valid decisions are right.

The feedback timeline puts them in order:

The feedback timeline of one ticket. Quick signals come in seconds; the truth comes with the review, hours or days later. Between them is a gap that only the quick signals cover.

Text description of the diagram

A ticket arrives at 0 seconds, and the triage is done about 3 seconds later. Four groups of signals are known at once: the input, the decision, the failures that you can see, and the cost and time. Then comes a gap: the decisions are made but not checked. The agent’s review arrives at the end, 120 seconds later in the practice runs and hours to days later in real life. Only the review gives the agreement and the safety check. A note says how many reviews a useful check needs: 139 for a margin of about 5 points. That is about 1 in 6 of Larkfield’s 822 tickets a day.

Reviews in the project

Larkfield’s help desk tool sends each review to the service: POST /v1/tickets/{id}/review. The new file ticket_api/reviews.py saves it in the table ticket_reviews (migration 018). It saves the assistant’s decision at that moment next to it, with the prompt version and the release ID. Then it logs a ticket.reviewed event with the delay since the ticket arrived.

So the review and the decision travel together. A later release cannot change what the old decision was.

For practice, scripts/send_reviews.py plays the agents. The practice tickets carry Grace’s labels. For the tickets that arrived at least --delay seconds ago, it sends the label as a review, for a random --share of them. Teams that are not one of the five help-desk teams are never reviewed, as in real life.

Send tickets, then their reviews

1. Start from a clean practice state, so that your outputs look like the ones below. Your database and your files hold the tickets of earlier lessons. Empty the database and load the small data again. Then move the old telemetry and practice files to a folder old:

   macOS / Linux

   python -m scripts.load --reset
   mkdir -p old && mv telemetry practice old/

   Windows (PowerShell)

   python -m scripts.load --reset
   New-Item -ItemType Directory -Force old | Out-Null; Move-Item telemetry, practice old\

   If the move says that a folder does not exist, there was nothing to move: go on.

2. Start the three processes, one terminal for each, as in Run the live service:

   python -m simulator
   fastapi dev
   python -m ticket_api.worker

3. In a fourth terminal, send 20 practice tickets:

   python -m scripts.practice --tickets 20 --rate 2

   Sent 20 practice tickets in 9.5 s: 202: 20
   Saved the ticket IDs in practice/sent.jsonl. The worker does their AI work; watch it with
     python -m scripts.queue_watch

4. Look at the first of the 20 tickets. The last 20 lines of practice/sent.jsonl are this run’s tickets.

   macOS / Linux

   tail -n 20 practice/sent.jsonl | head -n 1

   Windows (PowerShell)

   Get-Content practice\sent.jsonl -Tail 20 | Select-Object -First 1

   {"ticket_id": "T-500001", "case_id": "T-65129", "sent": 1791706790.798}

   Your ticket number can differ. Use yours in the next step.

5. Act as an agent: send a review for that ticket. On Windows, type curl.exe instead of curl, and put the command on one line without the \.

   curl -X POST http://127.0.0.1:8000/v1/tickets/T-500001/review -H "Content-Type: application/json" \
     -d '{"team": "delivery", "needs_human": false, "reviewed_by": "agent-07"}'

   {"ticket_id":"T-500001","team":"delivery","needs_human":false,"assistant_team":"delivery","assistant_needs_human":false,"agrees":true,"prompt_version":"assistant-v2","release_id":null,"delay_seconds":34.7}

   The answer compares the review with the assistant’s decision: agrees is true. If the assistant chose another team for your ticket, agrees is false. release_id is null if you run without RELEASE_ID.

6. Wait until the tickets are at least 2 minutes old. Then let the practice agents review half of them:

   python -m scripts.send_reviews --delay 120 --share 0.5

   Reviews sent: 11; tickets not reviewed: 9.

   The script chooses the reviewed tickets at random, with a fixed seed. A second review of the same ticket replaces the first one.

7. Look at the saved reviews:

   docker compose exec db psql -U tickets -c "SELECT ticket_id, team, needs_human, assistant_team, assistant_needs_human, prompt_version FROM ticket_reviews ORDER BY ticket_id"

    ticket_id |   team   | needs_human | assistant_team | assistant_needs_human | prompt_version
   -----------+----------+-------------+----------------+-----------------------+----------------
    T-500001  | delivery | f           | delivery       | f                     | assistant-v2
    T-500003  | warranty | t           | warranty       | t                     | assistant-v2
    T-500004  | warranty | f           | warranty       | f                     | assistant-v2
    T-500006  | warranty | t           | warranty       | t                     | assistant-v2
    T-500007  | payment  | f           | payment        | f                     | assistant-v2
    T-500008  | delivery | f           | delivery       | f                     | assistant-v2
    T-500010  | account  | f           | account        | f                     | assistant-v2
    T-500014  | warranty | f           | warranty       | f                     | assistant-v2
    T-500015  | warranty | f           | warranty       | f                     | assistant-v2
    T-500018  | account  | f           | delivery       | f                     | assistant-v2
    T-500019  | payment  | f           | payment        | f                     | assistant-v2
   (11 rows)

   Ten rows agree. In T-500018, the agent says account and the assistant said delivery: one disagreement.

8. Stop the API and the worker with Ctrl+C, so that they write their last telemetry. Then read your own reviews with ops labels:

   python -m ops labels telemetry/ticket-api.jsonl telemetry/ticket-worker.jsonl --windows 10

   20 tickets, 11 reviewed (55%); review delay median 131 s, p95 135 s
   window     tickets  reviewed  agrees (95% interval)   no answer  to a person  French
   ---------  -------  --------  ----------------------  ---------  -----------  ------
        1-10       10         7  100.0% [64.6%, 100.0%]       0.0%        40.0%    0.0%
       11-20       10         4  75.0% [30.1%, 95.4%]         0.0%        10.0%   20.0%
   Reviews arrive later: the last windows have fewer. For a margin of about 5 points on an agreement near 90%, a window needs 139 reviews.

Look at the intervals. Seven of seven reviews agree, and the interval still goes down to 64.6%. With 4 reviews, it goes from 30.1% to 95.4%. Eleven reviews say almost nothing about the quality. The next section shows why.

A tiny example: how sure is 17 of 20?

Agents reviewed 20 tickets, and 17 agree with the assistant: 85%. Is the assistant below the 90% SLO?

You cannot tell from 20. The Wilson interval gives the range of agreement values that fit the reviews, with 95% confidence. It works well for small counts and for shares near 0% or 100%, where the simple formula fails. Run it for 20, 100 and 1,000 reviews with the same 85%:

Run Python in your browser

An interval for the agreement, and the reviews it needs

The box prints the three intervals and 139. Then try wilson(0, 139) and reviews_needed(0.9, 0.02).

Python code (you can change it)import math def wilson(good, n, z=1.96): """A 95% interval for a share measured on n items (the Wilson interval).""" p = good / n centre = (p + z * z / (2 * n)) / (1 + z * z / n) half = z * math.sqrt(p * (1 - p) / n + z * z / (4 * n * n)) / (1 + z * z / n) return max(0.0, centre - half), min(1.0, centre + half) def reviews_needed(share, margin, z=1.96): """Reviews needed so that a share near `share` has about this margin.""" return math.ceil(z * z * share * (1 - share) / (margin * margin)) for good, n in [(17, 20), (85, 100), (850, 1000)]: lo, hi = wilson(good, n) print(f"{good} of {n}: {good / n:.1%}, 95% interval [{lo:.1%}, {hi:.1%}]") print("Reviews for a margin of 5 points near 90%:", reviews_needed(0.9, 0.05)) # Try: wilson(0, 139) for a safety check with no miss, or reviews_needed(0.9, 0.02).

17 of 20: 85.0%, 95% interval [64.0%, 94.8%]
85 of 100: 85.0%, 95% interval [76.7%, 90.7%]
850 of 1000: 85.0%, 95% interval [82.7%, 87.1%]
Reviews for a margin of 5 points near 90%: 139

With 20 reviews, the truth could be 64% or 94.8%: the SLO could be broken or met. With 100, the interval still reaches 90.7%. Only with 1,000 is the whole interval below 90%.

Now turn the question around: how many reviews do you need for a margin of ±5 points? The project’s formula: n = 1.96² × p × (1 − p) ÷ margin². With p = 0.9 and a margin of 0.05: 3.8416 × 0.9 × 0.1 ÷ 0.0025 = 138.3, so 139 reviews. A smaller margin costs much more, because the margin is squared:

Halving the margin from 10 to 5 points needs four times the reviews: 35, then 139. A margin of 2 points needs 865.

Text description of the chart

Four columns. A margin of plus or minus 10 points needs 35 reviews. Plus or minus 5 points needs 139, the column in a second colour, the course’s choice. Plus or minus 3 points needs 385. Plus or minus 2 points needs 865.

How many tickets should people check?

Grace’s question is practical: “How many tickets must my agents review, and when?”

Larkfield gets 822 tickets a day. For a daily check with a margin of about 5 points, the agents must review 139 tickets a day: about 1 in 6. A weekly check needs the same 139 from 5,754 tickets: fewer than 1 in 40. But you learn about a drop up to a week late. That is the trade: more reviews buy an earlier, surer answer.

Three rules make the sample useful:

• Random, not chosen. Review tickets picked at random, not only the tickets that customers complain about. Complaints are biased, as Production feedback showed.
• Also by slice. If French tickets are 5% of the total, 139 random reviews hold about 7 French ones. To know about French tickets, review more of them on purpose. Drift tells you which slice to sample, as in the last lesson.
• Safety is not a sample. The safety SLO has no budget: one ticket that needed a person and reached nobody is a SEV-1. With 139 reviews and no miss, the Wilson interval of the miss rate still goes up to 2.7%. A sample cannot prove “none”. So safety needs a rule on every ticket, for example a person for every ticket without a valid answer. Module 6 adds that fallback.

Here is the drift run again, per block of 1,000 tickets, with one more row: the last 754 tickets of the normal week. Its reviews came 300 practice seconds later, so at the end of the run only 42 had arrived: python -m ops labels data/telemetry/normal-week.jsonl.gz --windows 1000 shows it in its last row.

Each row is a block of tickets with its agreement and 95% Wilson interval. The policy-silent block is below the SLO for sure. The last block of the normal week looks good, but its 42 reviews cannot show it.

Text description of the chart

Four rows on an axis from 75% to 100%, with a line at 90%, the quality SLO. The normal block of the drift run is at 95.1%, from 92.8% to 96.7%. The French block is at 95.8%, from 93.6% to 97.2%. The policy-silent block is at 85.3%, from 81.8% to 88.2%: the whole interval is left of the line. The last 754 tickets of the normal week are at 95.2%. But only 42 reviews had arrived, so the interval goes from 84.2% to 98.7% and crosses the line.

Proxy signals: when they help, and when they cannot

Which quick signal would have warned you before the reviews? Compare two practice incidents. “Last week” is the prompt v3 change (a practice replay of the incident; the answers are real recordings):

python -m ops labels data/telemetry/release-full.jsonl.gz --by prompt

2,400 tickets, 1,051 reviewed (44%); review delay median 120 s, p95 121 s
prompt        tickets  reviewed  agrees (95% interval)  no answer  to a person  French
------------  -------  --------  ---------------------  ---------  -----------  ------
assistant-v2      602       307  93.8% [90.5%, 96.0%]        0.2%        27.0%    4.5%
assistant-v3     1798       744  83.7% [80.9%, 86.2%]       10.3%        29.7%    6.5%
Reviews arrive later: the last windows have fewer. For a margin of about 5 points on an agreement near 90%, a window needs 139 reviews.

Signal	“Last week”: v2 → v3	Drift run: normal → policy-silent
Answers without text (known at once)	0.2% → 10.3%	0.1% → 0.3%
Sent to a person (known at once)	27.0% → 29.7%	25.1% → 24.8%
French messages (known at once)	4.5% → 6.5%	5.4% → 3.4%
Agreement with the reviews (known later)	93.8% → 83.7%	95.1% → 85.3%

In “last week”, a proxy signal jumped at once: one answer in ten had no text. The no-answer alert of Quality and costs sees that in about a minute, long before any review. In the policy-silent block, no proxy moved. The decisions were valid and wrong, and only the reviews could show it.

So proxies catch failures that show in the output: no text, errors, slow answers, a cost jump. Reviews catch decisions that look fine and are wrong. You need both, and a dashboard with only proxies is green during the second kind.

When to use delayed labels, and what they cost

Sam asks, “Why not let the assistant check itself, or ask a second model?”

Priya answers: “A model judge is another assistant with its own mistakes. It can help to sort tickets for review. But the SLO is about what our agents say is right, so the label must come from them.”

Use delayed labels for every quality and safety number. Write the delay next to the SLO, so that people know how late it is. Count windows in reviews, not in minutes, when the review rate changes.

Use proxy signals for fast alerts on failures that show in the output. Never report a proxy as quality.

What it costs: agent time. At 139 reviews a day, and a minute or two per review, that is two to five hours of work a day. The review tool must also send each review to the service, so that it is joined with the decision. And quality is always known late: at Larkfield’s speed, a day late for a daily check.

Common mistakes

Common mistake

send_reviews sends nothing

Symptom: python -m scripts.send_reviews --delay 120 --share 0.5 prints:

Reviews sent: 0; tickets not reviewed: 0.

Cause: no ticket is old enough. The script reviews only tickets that arrived at least --delay seconds ago, and it has already handled the older ones (practice/reviewed.jsonl). It also prints this when practice/sent.jsonl does not exist yet.

Fix: wait 2 minutes and run it again, or add --follow to keep it running until Ctrl+C. Send practice tickets first if you have none.

Common mistake

A window in ops labels shows '-' for agreement

Symptom: the last row of ops labels has 0 in “reviewed” and - in “agrees”.

Cause: the reviews of those tickets have not arrived yet. This is the gap of the feedback timeline, not a bug.

Fix: read the quality only from windows with enough reviews: 139 for a margin of about 5 points. Read the proxy columns for the newest window, and wait for its reviews.

See it end to endMonitor a model on public data that changesCase study

The case study’s label, “closed within a week”, is complete 7 days after a month ends. The labels that arrive first are the requests that closed fast, so the early accuracy is too high: in September it says 88.0%, and all the labels say 86.1%.

Knowledge check

Check your understanding

Why did the 30-minute reviews-disagree alert stay quiet in the drift run, although agreement fell to 85.3% in the last block? Name the cause and one change that fixes it.

Show the answer

The run was 27.5 practice minutes long, so a 30-minute window always held all the reviews since the start. The good reviews of the first 2,000 tickets diluted the bad ones: at most 7.9% disagreed, under the 10% threshold. A fix: count the window in reviews, for example the last 139 reviews, so that old reviews leave the window as new ones arrive. A shorter time window also helps, as long as it holds enough reviews.

Check your understanding

Predict: Grace's team can review 35 tickets a day. What margin does a daily agreement near 90% get? Is that enough to say that the quality SLO is broken at 85%?

Show the answer

35 reviews give a margin of about ±10 points (the bar chart: 35 reviews for 10 points). With 30 of 35 reviews in agreement (85.7%), the Wilson interval goes from 70.6% to 93.7%. It includes 90%, so 35 reviews cannot show that the SLO is broken. Combine several days, which makes the answer later, or review more tickets.

Check your understanding

Practical check: show that a review in your project is saved next to the assistant's decision, and read the agreement with its interval. Which outputs prove it?

Show the answer

curl -X POST …/v1/tickets/T-500001/review returns JSON with assistant_team, agrees and delay_seconds. SELECT … FROM ticket_reviews shows the agent’s team and the assistant’s team in the same row, with the prompt version. python -m ops labels telemetry/ticket-api.jsonl telemetry/ticket-worker.jsonl --windows 10 prints the reviewed count and the agreement with a 95% interval for each window.

Recap

• A label is the true answer; at Larkfield it is an agent’s review. A delayed label arrives hours or days after the decision. The quality and safety SLOs can only use delayed labels.
• A proxy signal is quick and cheap: answers without text, errors, cost, the share sent to a person. It says where to look. It caught “last week” (10.3% without text), and it missed the policy-silent block, where only the reviews fell.
• The feedback timeline shows the gap between the decision and its review. Only quick signals cover that gap.
• A Wilson interval says how sure an agreement is. A margin of about 5 points near 90% needs 139 reviews, about 1 in 6 of Larkfield’s 822 tickets a day. Safety needs a rule on every ticket, not a sample.
• A quality alert over a time window can blend old good reviews with new bad ones. Count the window in reviews.

Next, the failures that only a language-model service has: answers that quote old documents, and a model that the provider changes without telling you.

Next: LLM/RAG regressions