Label and version

What you will be able to do

Write down what a correct answer must contain, not one perfect reply, and record who decided it and why. Then give the whole set a version, so that every score names the exact data that it came from.

Before you startWhat to know and to have readyAbout 50 minutes

What you need to know

• Slices and the coverage table, from Select representative cases.
• Grace’s policy, with the teams and the rules H1 to H5 that make needs_human true. It is in data/policy.md of your project.

What you need on your computer

• The project eval-harness with python -m harness coverage from the last lesson, and its virtual environment active. python -m pytest -q ends with 8 passed.
• If your project is broken, copy the snapshot C11/snapshots/end-of-m01 as in Reset the project, and do the steps of the coverage table again (about five minutes).

No account and no key.

A cake order

A customer orders a birthday cake. They do not bring a photo and say “make exactly this cake”. They write what matters on the order form: chocolate, no nuts, “Happy birthday, Sam” on top, ready by 3 p.m. on Saturday. Any cake that meets every line is right. A beautiful cake with nuts in it is wrong, because a guest has an allergy.

The form also says who took the order and when. If the customer calls to change it, the baker does not rub out the old line. The baker writes the change, the reason and the date, and the order gets a new version. On Saturday, everybody knows which version of the order the cake must match.

The problem: one golden reply

The easiest way to label a test case is to write the perfect answer and compare the system’s answer with it. A perfect answer of this kind is often called a golden answer. Amira tries it on ticket T-64707, a customer charged twice for a bathroom cabinet. She writes a golden reply. The baseline’s real reply follows it, and then a wrong reply that she wrote to test the idea. Her golden reply and the wrong reply are constructed for this lesson; the middle reply is the real recorded one.

Reply	Text	Golden words that it contains
Golden (constructed)	“I’m sorry about the double charge on order LK-561208. Once we confirm it, we refund the extra charge within 3 business days.”	19 of 19
Baseline (real)	“I’m sorry you were charged twice. Our payment team will review the charges, and a confirmed double charge is refunded within 3 business days after confirmation. You can share the screenshots to help with the review.”	9 of 19 (0.47)
Wrong (constructed)	“I’m sorry about the double charge on order LK-561208. We have refunded the extra charge, and it will reach your card within 3 business days.”	16 of 19 (0.84)

Neither reply matches the golden text exactly, so an exact-match test fails both. A word-overlap test is worse: it prefers the wrong reply. That reply says “We have refunded”, but the assistant cannot refund anything. Rule 8 of the policy says that only a person on the team does that. The baseline’s reply is correct, and it uses other words. Many different replies are right. And a reply can look very close to the golden one and still break the most important rule.

The idea: label what must be true

So the labels describe what must be true of a correct answer, like the order form. A label is the answer that a good system gives for a case, written down before you score anything. Each of Larkfield’s cases has three kinds of labels:

• Decisions, which code can compare exactly: the team, and whether a person must handle the ticket (needs_human), with the rule of the policy that says so.
• Criteria for the reply: what it must contain (for example photo, person_will_contact, stop_using, reply_in_french) and what it must_not contain (claims_action, asks_secret, other_customer_details, promises_outcome). Module 3 turns these codes into checks.
• Provenance: who labelled the case, when, how, and what changed later. Provenance is the history of where something came from.

A golden reply still has a use, as a golden example: a good answer that shows a reader what the criteria mean. It is not the target. The diagram takes one real case apart.

One case and its labels: two decisions, rules for the reply, and the record of who decided and what changed.

Text description of the diagram

The case T-64944 is in the dev split and the slice “difficult” (it has typos). The customer writes: “hi sorry to bother u but the rattan patio set that came is the wrong one. i ordered the dark brown and they sent the light one. how do i send it back?? thank u”. The case has four groups of labels. Decisions, which code scores: the team is returns, and needs_human is false, because no rule applies. Rules for the reply: it must ask for a photo; it must not claim an action or ask for a secret. Who, when and how: the course author labelled it on 2026-10-09 by applying Grace’s policy. The team comes from the deep learning course’s label, read again. Review after the runs: the next step changed from return_steps to photo. The reason is policy rule 3: a wrong item on arrival needs a photo.

Read the case with the project. Run this command in the VS Code terminal:

python -m harness show T-64944 --run baseline

The output:

T-64944 | split dev | slice difficult | source c06-valid
Text: hi sorry to bother u but the rattan patio set that came is the wrong one. i ordered the dark brown and they sent the light one. how do i send it back?? thank u
Expected: team returns | needs_human false | next step photo
Reply must: photo | must not: claims_action, asks_secret
Labelled by: course author, 2026-10-09; Grace's policy (teams, H1-H5) applied by reading; team: the deep learning course's label (two models agreed), read again by the course author; next_step: return_steps -> photo on review (policy rule 3: a wrong or damaged item on arrival needs a photo within 14 days, then a replacement or a refund)

baseline-v2-chat-small-r1-b6a2b9: {"team": "returns", "needs_human": false, "reason": "They want to return the wrong item, so this is a returns request."}
Reply: I’m sorry you received the wrong colour. Please send us a photo of the item within 14 days of delivery; return shipping is covered for a wrong item. The returns team can guide you through sending it back.

The baseline’s reply passes this case: right team, right decision, a photo, no claimed action, no request for a secret. Its words are its own.

Who labelled, and how

Provenance only helps if it is honest, so here it is in full. The course author wrote the labels by applying Grace’s written policy, and scripts checked them. The course author read each ticket and decided the team, the rule that makes needs_human true, the next step and the reply criteria. For the 140 reused tickets, the team came from the deep learning course, where two models agreed on it. The course author read each one again. It is one person’s reading; another reader could decide some boundary cases differently. The course author also wrote the reference ratings of Module 4, scores for 30 replies, with Grace’s rubric.

Grace and Priya can discuss the labels, and you will disagree with some of them. That is expected: a label is one reading of a written policy. Write your disagreement down with a reason. That is what a reviewer does.

When a system disagrees with a label

Labels change. A label can be wrong, the policy can be unclear, or a reader can see something new. What matters is that a change follows a procedure, and that the procedure is written down. This was the procedure for Larkfield’s set. Find step 6 in the diagram.

The life of the evaluation set. The reviewer changes labels only with a reason, every change makes a new version, and nothing crosses the lines between the splits.

Text description of the diagram

The diagram has three bands. The first band builds the set in four steps. Collect the tickets. Label each case with rules for the reply and a record of who labelled it. Split the cases: shuffle each slice and deal the cases in turn. Give the file its first version, a hash. The second band uses and reviews the set. Run the systems and save every output with the version. Then a reviewer, drawn in a dashed box, reads each case where a system disagrees with a label. The reviewer changes a label only with a written reason. This makes version 2, with a new file hash but the same inputs hash, so the saved outputs stay valid. The third band shows the split boundaries, with thick lines between dev, holdout and contaminated: no ticket crosses a line. Dev is open: you read it, tune on it and score it as often as you need. A dashed arrow goes from dev back to the run step. Holdout is frozen and scored once, for the release decision. Contaminated holds the tickets that the prompt was built on, and it is used only to measure the gap.

After the first runs of the baseline and the candidate, the course author read some cases again. These were the cases where a system disagreed with a label or failed a reply criterion: 63 of the 200 dev and holdout cases. The result:

What happened	Cases	Why
Next step changed: return steps → photo	10	Policy rule 3: a wrong or damaged item on arrival needs a photo first.
Next step changed on safety tickets: warranty → none	16	On a safety ticket the next step is to stop using the product and wait for a person. Applied to every rule-H1 case, as one decision.
Next step changed: no promise to check	5	A person handles these tickets anyway (a legal threat, instructions in the ticket, a lost order).
Next step changed: the trimmer was tested	1	A tested trimmer is not unused, so the return steps do not apply.
needs_human stayed false, although a system said true	21	No rule of the policy applies (for example paying in instalments, a locked account).
Team stayed, although a system chose another	2	T-65388 stays returns; T-65615 stays account.

So 32 labels changed, all of them the next step. No team label and no needs_human label changed. Each changed case says what changed and why, in its provenance, as you saw for T-64944.

Now look at the weakness of this procedure. The review read only the cases where a system disagreed. When the label and both systems were wrong together, nobody looked. And each change moved a label toward what a system had said. A review like this can only make the systems look better. A stronger procedure has four parts:

1. Two reviewers label the same cases, each without seeing the other’s labels. Module 4 measures how far two reviewers agree, in Calibrate reviewers.
2. The owner of the policy settles each disagreement. For Larkfield, that is Grace.
3. The reason is written into the case, as label.changed does.
4. A decision is applied to every case it covers, not only to the ones a system failed. The safety decision above was applied in this way, to all rule-H1 cases.

What it costs: a second reviewer doubles the labelling time, and settling disagreements needs the owner’s time. For a set of 200 cases, that is a few days of work. Do it before a release decision depends on the labels, not after.

Give the set a version

The review changed 32 labels. A score from before the review and a score from after it are measured against different answers. So every score must name the exact data that it came from. A name like “the April set” is not enough, because someone can edit the file and keep the name.

The answer is a content hash: a short code computed from every byte of the file. SHA-256 is a common method. Change one character, and the hash changes completely. Here are the first 12 characters of three hashes. The texts differ by one word, then by one capital letter:

Text	First 12 characters of its SHA-256
T-64944 next_step: return_steps	718710ff1de3
T-64944 next_step: photo	0778f2b8d724
T-64944 next_step: Photo	29dfe1f7908c

The project uses the first 12 characters of the file’s SHA-256 as the dataset version. python -m harness cases prints it on its first line: {ds.version}. Before the review, it was {ds.version_before_review}.

That raises a question. The baseline was run before the review. Is its saved run still valid? The answer depends on what the system saw. The system saw the case IDs, the texts and the attachments, never the labels. So the project has a second hash, the inputs version: a hash of the IDs, texts and attachments only. The review did not change it, so the saved outputs are still the outputs for these exact tickets.

Two versions of one set. The labels changed, so the file hash changed; the texts did not, so the inputs hash and the saved outputs stay valid.

Text description of the diagram

Two columns. The left column is the set before the review, on which the baseline ran. Its file hash, over every byte, is 263bbd4f180d. Its inputs hash, over the case IDs, texts and attachments only, is 3f90ab808275. The right column is the set after the review: 32 next-step labels changed, and no text changed. Its file hash is d4c96eaa10c0, and its inputs hash is still 3f90ab808275. The file hashes are different. The inputs hashes are the same.

Add the inputs version to the project

1. Open harness/dataset.py. The new function uses json, which the file already imports. Check that the imports start like this:

   harness/dataset.py

   import hashlib
   import json
   from collections import Counter
   from pathlib import Path

2. Find the function version. Below it, leave two blank lines and add this function:

   harness/dataset.py

   def inputs_version(cases: list[Case]) -> str:
       """A version of what the system sees (case IDs, texts and attachments), without the labels.
   
       A label can change after review; the saved outputs stay valid as long as this version is the same."""
       lines = sorted(json.dumps([c.case_id, c.text, c.attachments], ensure_ascii=False) for c in cases)
       return hashlib.sha256("\n".join(lines).encode("utf-8")).hexdigest()[:12]

   The function writes one line per case with only the three inputs, and sorts the lines. Then the order of the file does not matter. Last, it hashes the lines.

3. Save the file. Print the inputs version of all 271 cases:

   python -c "from harness.dataset import inputs_version, load_cases; print(inputs_version(load_cases(split='all')))"

   The output:

   3f90ab808275

4. Open outputs/runs.json. Find dataset_version and inputs_version in the baseline’s run. They are 263bbd4f180d and 3f90ab808275. The file version is older, but the inputs version matches step 3, so the run is valid for today’s set.

Guided practice: change one label

You now change one label on purpose and watch both versions. Make a backup first, because the next lessons need the original file.

1. Copy the evaluation set to a backup file:

   macOS and Linux

   cp data/cases.jsonl data/cases.backup.jsonl

   Windows

   Copy-Item data\cases.jsonl data\cases.backup.jsonl

2. Open data/cases.jsonl in VS Code. Search (Ctrl+F, or Cmd+F on macOS) for T-64944. On that line, change "next_step": "photo" to "next_step": "return_steps": the label before the review. Save the file.

3. Run python -m harness cases. The first line now shows a new version:

   271 cases, dataset version d7ae5a8497ad

4. Run the command of the last section’s step 3 again. The inputs version is still 3f90ab808275: you changed a label, not an input.

5. Put the original file back:

   macOS and Linux

   mv data/cases.backup.jsonl data/cases.jsonl

   Windows

   Move-Item -Force data\cases.backup.jsonl data\cases.jsonl

6. Run python -m harness cases once more. The version must be {ds.version} again.

What if you change a ticket’s text instead?

Amira tried it: she removed one question mark from the text of T-64944. The dataset version became fd626b625f82, and the inputs version became 07c4e9eae983. A changed input means that the saved outputs no longer match the cases, and every system must run again on the new set.

Your turn: you are the reviewer

Three labels changed in the review. Read each case with python -m harness show <ID> --run baseline, and read the policy in data/policy.md. For each one, write “agree” or “disagree” and one sentence of reason.

1. T-64740: a parasol pole cracked and the canopy dropped toward a child. Next step: warranty_claim → none.
2. T-81024: an order four weeks late and a legal threat. Next step: investigate → none.
3. T-65003: a hedge trimmer that the customer tested and now wants to return. Next step: return_steps → none, and needs_human stays false.

Show one reviewer’s notes

These are notes, not the answer. Another careful reviewer can disagree.

1. Agree. Rule H1 applies, so a person handles it. The reply must say “stop using it” and that a person will contact the customer. The warranty route is that person’s decision, so the reply should not promise it.
2. Agree. Rule H2 applies (a lawyer). A person handles the ticket, so the reply must not promise to investigate. It only says that a person will contact the customer.
3. Disagree, or at least ask Grace. The provenance itself says that “whether it can be returned is a person’s call”. If a person must decide, maybe rule H3 (outside the policy) applies, and needs_human should be true. The baseline said true. The policy is not clear here; write the question down for Grace, and do not change the label alone.

Common mistake

The dataset version is not d4c96eaa10c0

Symptom: python -m harness cases prints another version on its first line, for example 271 cases, dataset version d7ae5a8497ad. Nothing else looks wrong, and the tests still pass.

Cause: data/cases.jsonl changed. Perhaps the backup was not put back after the practice, or an editor changed the file when you saved it, for example the line endings. A score from this file is not comparable with the scores in this course.

Fix:

1. Check the file against the checksums that come with it. On macOS and Linux, run shasum -a 256 -c SHA256SUMS in the data folder. cases.jsonl: OK means the file is the original. On Windows, run Get-FileHash data\cases.jsonl and compare the hash with the first line of data/SHA256SUMS. Windows prints it in capital letters; the letters are the same.
2. If the file is not the original, put the backup back (step 5 of the practice). If there is no backup, copy data/cases.jsonl from the labs snapshot C11/snapshots/end-of-m01.
3. Run python -m harness cases. The version must be {ds.version}.

Check your understanding

Tomás proposes to score replies by how many words they share with one golden reply. Use ticket T-64707 to explain the problem, and say what the labels contain instead.

Show the answer

On T-64707, the baseline’s correct reply shares only 9 of the golden reply’s 19 words (0.47). A constructed wrong reply, which claims “We have refunded” although the assistant cannot refund anything, shares 16 (0.84). So word overlap prefers the wrong reply. The labels instead say what must be true. They give the decisions (team, needs_human with its rule) and what the reply must contain: here, the double-charge timing. They also say what it must not contain: a claimed action or a request for a secret.

Check your understanding

The review changed 32 labels after the baseline was run. Why is the baseline's saved run still valid, and when would it not be?

Show the answer

The system only saw the case IDs, the texts and the attachments. The review changed labels, not inputs, so the inputs version stayed 3f90ab808275, and the saved outputs are still the outputs for these exact tickets. The scores change, because they are measured against new labels, but the outputs do not. If a text or an attachment changed, the inputs version would change, and the systems would have to run again.

Check your understanding

The review read only the cases where a system disagreed with a label. Name one problem with that procedure, and one change that fixes it.

Show the answer

Nobody read the cases where the label and both systems were wrong in the same way. And every change moved a label toward what a system said. So the review can only make the systems look better. One fix is a second reviewer who labels the same cases without seeing the first labels. Another is that Grace, who owns the policy, settles each disagreement. A third is to apply each decision to every case that it covers, as the safety decision was.

Recap

• A golden reply is a poor target: many replies are right, and a wrong reply can share most of its words. Use it as a golden example for readers, not as the test.
• Each case has labels: decisions (team, needs_human and its rule), criteria for the reply (must, must_not), and provenance (who, when, how, what changed and why).
• The course author wrote the labels by applying Grace’s written policy, and scripts checked them. It is one person’s reading; in Module 4, you are the second rater of some replies.
• The review read 63 cases where a system disagreed and changed 32 next-step labels. Reviewing only where a system disagrees can only flatter the systems; two independent reviewers and an owner who settles disagreements are stronger.
• The dataset version is the first 12 characters of the file’s SHA-256: {ds.version}. The inputs version ({ds.inputs_version}) covers only what the system sees, so saved outputs stay valid after a label change.
• Your project now has inputs_version(). The tests still pass: 8 passed.

Next, you keep the three parts of the set apart, and you measure what happens when a test case was seen before.

Next: Avoid contamination