Production feedback

What you will be able to do

After the release, the evaluation set stops seeing new things; production does. You read a month of use: thumbs, agent edits, a random review sample and drift. You learn what each signal can and cannot tell you, and you add new cases to the set without touching the frozen holdout.

Before you startWhat to know and to have readyAbout 45 minutes

What you need to know

• The evaluation layers, monitoring included, from Choose evaluation layers.
• Slices and coverage, from Select representative cases; versions and the inputs version, from Label and version; dev and holdout, from Avoid contamination.

What you need on your computer

• ~/projects/eval-harness after Regression checks, with .venv active: python -m pytest shows 103 passed, 1 deselected.

No account and no key. Tested with Python 3.12.13 on macOS (Apple silicon). The Windows commands are not tested.

The production log is simulated

Larkfield’s assistant has no real users, so this lesson uses a simulated log: one made-up month of 1,270 tickets. A script built it with known effects, so that you can practise finding them. Every row says "simulated": true. The numbers below describe the simulation, not real customers.

A new dish after opening night

A chef tests a new dish for weeks before it goes on the menu. Then the restaurant serves it, and new kinds of evidence arrive. A few guests fill in comment cards, and the angry ones write more often than the happy ones. The waiters see every plate that comes back half full. Each night, the chef also tastes a few plates picked at random, not only the ones that someone complained about.

After a month, guests start to ask for something new: a smaller portion for children. It is not on the tasting list, so the kitchen never tested it. The chef adds it to the dishes she practises. But the standard tasting that decides whether a new recipe goes on the menu stays the same. Otherwise this month’s recipes and last month’s could not be compared.

The problem: signals that point the wrong way

The candidate did not ship. Imagine that the baseline has now run in production for four weeks. Here is the simulated month as the team sees it, week by week.

Tomás reads the thumbs: “Customers gave 30 to 43 thumbs down out of every 100 thumbs. Offline, the baseline got both decisions right on 94 of 98 holdout cases. Production is much worse than our evaluation!” Omar reads the review: “Priya’s team read 40 random tickets this month and found nothing new. All is fine.”

Both are wrong, and the log shows why. To see it, you need to know what each signal measures.

The idea: four production signals

Production feedback is what you learn about the system from its real use. It comes from four signals, and each has its own blind spot:

1. User feedback: thumbs up or down from customers. Cheap and fast, but few people give it, and not at random.
2. Agent edits: whether the agent changed the draft before sending it. Every draft is read, so every draft can be edited.
3. Sampled review: a person reads a fixed number of tickets picked at random, with the rubric. Slow, but not biased by who complains.
4. Drift: a change in what comes in, such as a new topic, more French tickets or longer tickets. The word means “moving slowly away”.

None of them gives labels. A thumbs-down does not say which rule failed. So production finds questions, and the evaluation set answers them.

Add the log and the feedback code

1. Copy the feedback code, the simulated log, its test and the course-end README. Set S again first if you opened a new terminal.

   macOS and Linux

   cp $S/harness/feedback.py harness/
   cp -R $S/data/production data/
   cp $S/tests/test_feedback.py tests/
   cp $S/README.md .

   Windows

   Copy-Item $S\harness\feedback.py harness\
   Copy-Item -Recurse $S\data\production data\
   Copy-Item $S\tests\test_feedback.py tests\
   Copy-Item $S\README.md .

2. Look at one line of the log, data/production/feedback_log.jsonl:

   {"agent_edited": true, "day": "2026-11-02", "language": "en", "needs_human_changed": false, "simulated": true, "team": "warranty", "thumb": "none", "ticket": "P-00003", "topic": "warranty", "week": 1}

   One ticket: the team the assistant chose, the language, whether the agent edited the draft or changed needs_human, the customer’s thumb, and the topic. There is no ticket text: like many dashboards, the log keeps only what it counts.

3. Run the summary:

   python -m harness feedback

   The output:

   Production log (SIMULATED for the course; not real usage)
     tickets: 1270
     thumbs given: 181 (14.3%); up 113, down 68
     agent edit rate by week: {1: 0.184, 2: 0.207, 3: 0.19, 4: 0.239}
     thumbs-down share of thumbs by week: {1: 0.429, 2: 0.298, 3: 0.415, 4: 0.364}
     new topic (not in the evaluation set) by week: {1: 0.0, 2: 0.0, 3: 0.124, 4: 0.162}
     French share by week: {1: 0.06, 2: 0.049, 3: 0.042, 4: 0.091}

4. Run the tests. The last line is 106 passed, 1 deselected. test_feedback.py checks three things. The log is complete and simulated, the new topic appears in week 3, and the review sample is the same every time.

Your project now matches the snapshot end-of-m06, the end of the course’s project.

Thumbs: who chooses to click

Only 181 of 1,270 customers gave a thumb: 14.3%. The other 86 in 100 said nothing. The people who click are not a random group. In the simulation, as in real life, unhappy customers click more often. So “the share of thumbs that are down” measures who chose to click, more than how good the drafts are.

The agents’ edits give a fairer view, because an agent reads every draft. The thumbs and the edits are compared here, week by week.

• Thumbs down, share of thumbs
• Drafts edited, share of tickets

Simulated log, 1,270 tickets in 4 weeks. The thumbs-down share jumps between 30% and 43%; the agents edit about 1 draft in 5, rising in week 4.

Text description of the chart

Two lines over weeks 1 to 4. Thumbs down as a share of the thumbs given: 42.9%, 29.8%, 41.5%, 36.4%. Drafts edited by the agent, as a share of all tickets: 18.4%, 20.7%, 19.0%, 23.9%. Each week has between 41 and 49 thumbs and between 306 and 331 tickets.

What this tells us. The thumbs line jumps by 12 points from one week to the next. Each week has only 41 to 49 thumbs, so a few clicks move it a lot. It also sits far above the edit line, at about twice its height. Most of that gap is the bias: unhappy customers click more often than happy ones. The edit line is calmer, and it rises in week 4. That rise is worth a look; the jumps of the thumbs line are not.

The two signals agree on direction, though. Split the month by whether the agent edited the draft:

Simulated month	Tickets	Thumbs up	Thumbs down	No thumb
Draft edited by the agent	260	15	54	191
Draft sent as it was	1,010	98	14	898

Edited drafts get many more thumbs down. So a thumbs-down is a useful pointer to a ticket worth reading. It is a poor measure of how often the assistant is wrong. That answers Tomás: production is not “much worse than the evaluation”. He compared a biased share of clicks with a rate over all cases.

Agent edits: closer, but not a label

An edit rate of about 20% means that agents change 1 draft in 5. That is useful, and it is not a failure rate. An agent edits for style, for a detail the policy does not cover, or because the customer wrote again. The log also counts a stronger signal: an agent changed needs_human. That happened on 30 tickets in the month, and each of them is worth reading, because it is a decision that the assistant got wrong.

Drift: what the set does not contain

Now the input side: what customers ask about.

• New topic: price-drop requests
• French tickets

Simulated log. A topic that the evaluation set does not cover appears in week 3 and grows; French tickets almost double in week 4.

Text description of the chart

Two lines over weeks 1 to 4. The new topic, price-drop requests: 0%, 0%, 12.4% (38 of 306 tickets), 16.2% (50 of 309). French tickets: 6.0%, 4.9%, 4.2%, 9.1% (28 of 309).

What this tells us. In week 3, a new kind of ticket appears: customers who bought an item before a sale and ask for the difference back. By week 4 it is 1 ticket in 6. The evaluation set has no case like it on dev. Agents edited 56.8% of those drafts, against 17.8% for the other topics. French tickets also nearly double in week 4. The French slice did well offline, but it has only 10 dev cases.

This is the real news of the month, and the thumbs did not show it. The evaluation set described last month’s tickets. It cannot score what it does not contain.

Sampled review: random, and small

Every week, a person reads 10 tickets picked at random, with the rubric. The harness picks them with a fixed seed, so that anyone can see which tickets were chosen. In Python:

from harness.feedback import load_log, review_sample

rows = load_log()
sample = review_sample(rows)
by_id = {r["ticket"]: r for r in rows}
print(len(sample), sum(by_id[t]["topic"] == "price_adjustment" for t in sample))

The output:

40 0

Forty tickets read, and not one price-drop request among them, although there were 88 in weeks 3 and 4. Bad luck? Partly. With 38 of 306 tickets in week 3, a random 10 misses all of them about 26 times in 100. Missing them in both weeks happens about 4 times in 100. It is unlikely, and it happened.

So Omar’s “nothing new in 40 tickets” is honest but weak. A random sample of 10 a week finds problems that are common, and it is not biased by who complains. It is too small to find a new topic early. Use it with the other signals. When a count shows a new topic, read every ticket of that topic on purpose: here, the 38 tickets of week 3.

The evaluation loop

Now the pieces of the course fit together. Offline checks protect every change. The release decision uses the holdout set once. Production shows what the set is missing, and new cases bring that back into the set.

The evaluation loop, from offline tests to production observations and back to the evaluation set.

Text description of the diagram

The loop has six steps. The first three are offline, on fixed cases and saved outputs.

1. Evaluation set: the dev set grows with new cases; the holdout set is frozen and never edited; every change gives a new dataset version.
2. Checks on every change: pytest on saved outputs, for the functions, the release rules and a report that rebuilds the same.
3. Release decision: the candidate against the baseline, with blockers, intervals and cost, on the holdout set, once.

If the candidate ships, the next three steps are in production, with real tickets and no labels.

4. Production: the assistant drafts replies, agents edit and send them, and every ticket is logged.
5. Observations, every week: a person reviews a random sample; agent edits and thumbs, which are biased; drift, such as new topics and languages.
6. New cases: the observations show a new failure or topic. A person writes and labels new cases, and records who and why. The cases go to dev, never to the holdout.

Then the loop starts again at step 1, with a new dataset version.

Add new cases under control

The set must learn about price-drop requests. “Under control” means four rules:

1. A person writes and labels each case, with Grace’s policy, and records who labelled it and how. The log has no texts, so the cases are written in the style of the tickets that were read.
2. New cases go to dev. The team may read them and tune on them.
3. The set gets a new version, and the rules of the set still pass: no duplicate IDs, no ticket in two splits.
4. The holdout set does not change. Its cases, and its inputs version, stay the same.

Try it with one case, without changing your project. Create a file try_new_case.py in the project folder:

try_new_case.py

from pathlib import Path
from harness.dataset import check, inputs_version, load_cases, save_cases, version

cases = load_cases(split="all")
template = next(c for c in cases if c.case_id == "T-81029")   # a dev case: H3, a promise risk
new = template.model_copy(update={
    "case_id": "T-82001",
    "text": "I bought the teak bench for 649 dollars on 2 November. Your sale started a week later "
            "and now it costs 519. Can you refund me the difference?",
    "tags": ["refund_promise_risk", "price_adjustment"],
    "label": template.label.model_copy(update={
        "method": "written for the lesson after the simulated log showed price-drop requests; "
                  "Grace's policy applied by reading (H3)",
        "reviewed": "", "changed": ""}),
})
proposed = cases + [new]
check(proposed)                                   # the rules of the set still hold
save_cases(proposed, Path("../cases-proposed.jsonl"))
print("dataset version", version(), "->", version(Path("../cases-proposed.jsonl")))
for split in ("dev", "holdout"):
    old = [c for c in cases if c.split == split]
    now = [c for c in proposed if c.split == split]
    print(f"{split:8} {len(old)} -> {len(now)} cases | inputs version {inputs_version(old)} -> {inputs_version(now)}")

The new case copies the structure of T-81029, a dev case where the customer asks for money that the policy does not give. A price-drop refund is not in the policy, so rule H3 applies: a person handles it, and the reply must not promise the refund. Run it with python try_new_case.py:

dataset version d4c96eaa10c0 -> 1fbe6b1438d4
dev      102 -> 103 cases | inputs version 9b24fa392841 -> 30d3563a9d90
holdout  98 -> 98 cases | inputs version 1be23cc44ea9 -> 1be23cc44ea9

The proposed set has a new version. Dev has one more case and a new inputs version, so the saved runs no longer cover it. The baseline must answer T-82001 before anyone can compare on it. The holdout’s inputs version is unchanged, so every holdout result of this course stays valid.

The file ../cases-proposed.jsonl is outside your project, and data/cases.jsonl did not change. Delete try_new_case.py when you are done. In a real team, the proposed file goes through review like code. Grace or a second person checks each label. Then the team records the baseline on the new cases (Establish a baseline).

Frameworks do this for you (optional)

Several open-source and paid tools collect production traces, sample them for review, and turn a reviewed trace into a test case with one click. They are useful once the volume is large. They do the same steps as this lesson: log, sample, review, add to a versioned set. If you use one, check that it keeps a frozen holdout apart from the cases that it adds, and that every case records who labelled it. This course uses no framework, so that you see each step.

When to use each signal, and what it costs

Signal	Use it for	Do not use it for	Cost
Thumbs	Pointers to tickets worth reading	A rate of bad replies	Free; few people click
Agent edits	A steady trend over all tickets	A label: an edit is not always a failure	Free if the tool logs it
Changed needs_human	Wrong decisions, one by one	Style problems	Free; read each one
Random review	An unbiased estimate of common problems	Rare or new problems	A person’s time, every week
Drift counts	New topics and languages	Judging reply quality	A topic or language field

Each signal answers one question. Together they tell you where to read, and the evaluation set then measures it.

Common mistake

The holdout grows with production

Symptom: Tomás adds the 10 best price-drop tickets to the holdout set: “The release test must cover the new topic.” The next release report says that the candidate is better on holdout than last month’s baseline was.

Cause: the holdout changed, so the two reports compare different cases. Worse, the team wrote and discussed those tickets while it was changing the prompt. The holdout is no longer unseen. The harness refuses one form of this: a text that is already in another split. For example, if Tomás copies the holdout’s price-match ticket T-81028 into dev:

harness.dataset.DatasetProblem: T-82002: the same text is in holdout (T-81028) and dev

But no code can stop a person from editing the holdout on purpose.

Fix: new cases go to dev. If the holdout must cover a new topic, build a new holdout version on purpose. Do it before anyone tunes on those tickets, and record both systems on it. Name the holdout version in every report, and never compare reports across holdout versions.

Module practice: deliver the harness and its report

This practice brings the module together. Deliver the harness and the release report as you would hand them to Omar.

1. Make sure your project matches the snapshot end-of-m06. On macOS and Linux, diff -r -x .venv -x __pycache__ -x .pytest_cache -x report.md $S . prints only Only in .: lines for your own files, such as rate.py and notes. python -m pytest shows 106 passed, 1 deselected.
2. Write the release report on the holdout set into report-holdout.md.
3. Write HANDOFF.md with five short sections:
   • how to set up and run the tests;
   • how to rebuild the report, with its SHA-256;
   • the run IDs, and the dataset and inputs versions that it uses;
   • the test plan: what runs when;
   • the production signals to watch, and who reads them each week.
4. Independent variation: write a second report, report-practice.md, for baseline against practice. Which blockers does it name? Mark in HANDOFF.md that the practice run has one constructed reply.

Show an example answer

report-holdout.md is the file of Harness structure: Do not ship, blocker T-65663, decisions −0.112 [−0.184, −0.041].

The practice report, on dev (the default), names three blockers: T-64740 no_answer_critical, and T-81034 missed_access and other_customer. The two T-81034 blockers come from the constructed reply.

A short HANDOFF.md:

## Set up and test
python 3.12, python -m venv .venv, pip install -r requirements.txt; python -m pytest -> 106 passed, 1 deselected.
## Rebuild the report
python -m harness report baseline candidate --split holdout --out report-holdout.md
SHA-256 on macOS: d65dee9596dbf0b48aa17fc3e740e5199a554f55d272691a98544ae53764bb46
## Runs and versions
baseline-v2-chat-small-r1-b6a2b9, candidate-v3-chat-small-r1-31b45f; dataset d4c96eaa10c0, inputs 3f90ab808275.
practice-v3-chat-small-r1-31b45f: candidate run 1 with ONE CONSTRUCTED reply (T-81034). Practice only.
## Test plan
Every change: pytest. Before a release: record the candidate, 5 dev runs, decide and report on holdout once,
a person reads every blocker. pytest -m live when the provider or the model version changes.
## Production
Weekly: agent edit rate, changed needs_human (read each), drift by topic and language, 10 random tickets
read with the rubric. Thumbs only as pointers. New cases: to dev, labelled, new version; holdout frozen.

Module check: reproduce the report and trace every score

Show a reviewer that the report is evidence, not a claim.

1. In a fresh copy of the snapshot end-of-m06, with a new .venv, rebuild the holdout report. Compare its SHA-256 with yours.
2. For every number in the report’s Measures table (baseline, candidate, better and worse), list the cases behind it with per_case(). Show that the totals add up.
3. For the blocker, and for two cases of your choice, show the saved output with show CASE --run RUN_ID, and say whether you agree with the score.

Show what a complete answer contains

1. The same SHA-256 in both copies (on macOS, d65dee95…64bb46).
2. Decisions: 94 and 83 of 98; 12 cases worse and 1 better (T-81052), so 94 − 12 + 1 = 83. Criteria: 96 and 87 of 98; 11 worse and 2 better (T-64860, T-81022), so 96 − 11 + 2 = 87. Each list printed by per_case().
3. T-65663: no answer on a safety ticket, a true blocker. For example, T-64815 (criteria worse): the phrase rule refund_after_receipt misses “confirm receipt first”. A reader would pass the reply, so the score is the checker’s error. A complete answer names at least one disagreement like this and says what it changes. Here, the criteria difference is a little too large, and the decision does not change.

Knowledge check

Check your understanding

In the simulated month, 36% to 43% of thumbs are thumbs down in three of the four weeks, but agents edit only about 20% of drafts. Why do the two numbers differ, and which one would you report as 'how often the drafts need work'?

Show the answer

Only 14.3% of customers give a thumb, and unhappy customers click more often, so the thumbs-down share measures who chose to click. Agents read every draft, so the edit rate covers all tickets. Report the edit rate as the trend, and say that an edit is not always a failure. Use thumbs-down only as a pointer to tickets worth reading.

Check your understanding

The random review of 40 tickets found no price-drop request, but 88 were in the log. What does this tell you about sampled review, and what would you do in week 3?

Show the answer

A small random sample is unbiased, but it finds only common problems. A random 10 from week 3 misses all 38 price-drop tickets about 26 times in 100. Use it with drift counts. When a count shows a new topic, read every ticket of that topic on purpose, then write and label new cases for dev.

Check your understanding

Why do new cases from production go to dev and never to the holdout set? What changes in the dataset when you add one?

Show the answer

The holdout set must stay unseen and fixed, so that release decisions compare the same cases and nobody has tuned on them. New cases have been read and discussed, so they belong in dev. Adding one changes the dataset version and the dev inputs version. So you must record the baseline on the new case before you compare on it. The holdout’s inputs version stays the same, so its results stay valid.

See it end to endRegression-test a classifier release by sliceCase study

The classifier case study runs the whole loop on open data: two versions, slices, intervals, blockers and a reproducible release decision.

Recap

• Production feedback has four signals: thumbs, agent edits, random review and drift. None of them gives labels; each has a blind spot.
• In the simulated month, only 14.3% of customers gave a thumb, and the thumbs-down share (30% to 43%) overstates bad drafts. The agent edit rate, about 1 in 5, is the steadier trend.
• Drift was the real news: a new topic grew to 16.2% of tickets in week 4, and French tickets nearly doubled. The random sample of 40 missed every price-drop ticket.
• New cases go to dev, written and labelled by a person, with a new dataset version. The holdout stays frozen; its inputs version does not change.
• The evaluation loop connects offline checks, the release decision, production observations and new cases.
• Your project matches the snapshot end-of-m06, the end of the course’s project: 106 passed, 1 deselected.

Models change fast, and the numbers of this course are a snapshot of one set of runs. The method lasts: fixed cases, saved runs, blockers first, and a set that learns from production. The course AI Agents and Workflow Orchestration builds on this harness to evaluate an agent’s steps. Before that, the final assignment asks you to build your own.

Next: Final assignment