Avoid contamination

What you will be able to do

Keep the cases that you tune on apart from the cases that decide a release, so that a score measures the assistant and not your memory of the test. Then measure the gap honestly, even when it is not the gap that you expected.

Before you startWhat to know and to have readyAbout 45 minutes

What you need to know

• The three parts of the set (dev, holdout, contaminated), from Select representative cases.
• The dataset version and the inputs version, from Label and version.
• Why a model is judged on data that it did not learn from, from Train, validation and test sets in the machine learning course. It helps, but you do not need it.

What you need on your computer

• The project eval-harness with coverage and inputs_version() from the last two lessons, and its virtual environment active. python -m pytest -q ends with 8 passed.
• If your project is broken, copy the snapshot C11/snapshots/end-of-m01 as in Reset the project. Then do the steps of the coverage table and the inputs version again.

No account and no key. The experiment in this lesson called a model; its results are saved, and you only read them.

Homework that comes back as the exam

A teacher sets ten homework problems, and the class goes through every answer together. On Friday, the test has the same ten problems. Every student gets a top mark. The marks say that the students remember the answers. They say nothing about whether the students can solve a new problem.

An evaluation set has the same trap. Perhaps the people who build the system have seen a test case, and changed the system until it got that case right. Then the case measures their memory, not the system.

The problem: the prompt’s own examples

Prompt v2, the baseline’s prompt, contains two examples: a bird bath that arrived cracked, and a grill that caught fire. Each example has the answer that the model should give. Both are in the evaluation set, in the contaminated split. Read the first one with the baseline’s saved reply:

python -m harness show P-V2-EX1 --run baseline

The output:

P-V2-EX1 | split contaminated | slice contaminated | source prompt-v2
Text: The bird bath arrived cracked in two pieces yesterday. Order LK-104233.
Expected: team delivery | needs_human false | next step photo
Reply must: photo | must not: claims_action, asks_secret
Labelled by: course author, 2026-10-09; Grace's policy (teams, H1-H5) applied by reading; the answer written in prompt v2 of the LLM applications course

baseline-v2-chat-small-r1-b6a2b9: {"team": "delivery", "needs_human": false, "reason": "Broken on arrival is a delivery problem."}
Reply: I'm sorry that your bird bath arrived broken. Please send us a photo of the damage within 14 days of delivery. We will then send a replacement at no cost, or a full refund if none is in stock.

The reply is the prompt’s example answer, word for word, and so is the reply to the grill ticket. The model did not solve these tickets; it copied the answers that it was given. Both cases pass every check, and they prove nothing.

The other 69 contaminated tickets are a quieter version of the same trap. In the LLM applications course, prompt v2 was written and compared on exactly these tickets. So a score on them can be higher than a score on new tickets.

The idea: one set to tune, one set to decide

The fix is a rule about who may look at what. Contamination means that a test case has influenced the system that it tests. The case was in the prompt, the prompt was tuned on it, or a model was trained on it. A contaminated case gives a score that is too good, and nothing in the score tells you so.

So Larkfield’s set has three parts with three jobs:

• The development set (dev) is open. You read its failures, change the prompt, and score it as often as you like. Every look contaminates it a little, and that is accepted.
• The holdout set is frozen. Nobody reads its cases or outputs while they change the system. You score it once, for the release decision.
• The contaminated set holds the cases known to be seen. It never decides a release; it only measures the gap between seen and unseen cases.

A leak is anything that lets the holdout influence the system or the measures. Examples are a copied ticket, a look at its failures, or a rule fixed after reading its outputs. The diagram shows who looked at which part while this course was built, with two real leaks.

Who looked at which part of the set. The two red marks are real leaks into the holdout set while this course was built.

Text description of the diagram

A grid with three columns, separated by thick lines: dev, holdout and contaminated. There are five rows of activities. First, prompt v2 was written and compared on the contaminated tickets. Second, in this lesson’s experiment, prompt v3 was tuned to v3.1 by reading the dev results only. Third, the labels were read again after the first runs, on dev and on holdout: on holdout, this is a leak, marked in red. Fourth, the phrase rules that check replies were revised after reading replies of all three parts: on holdout, again a leak in red. Fifth, the release decision scores the holdout set once, shown in a dark box.

Measure the gap

If contamination makes scores too good, the baseline should score higher on the contaminated split than on the holdout. You can measure that with commands that you already have.

1. Score the baseline on the contaminated split:

   python -m harness score baseline --split contaminated

   baseline-v2-chat-small-r1-b6a2b9 on contaminated (71 cases)
   answers 71/71 | team 69/70 | needs a person: caught 10/10, false alarms 3

2. Score it on dev:

   python -m harness score baseline --split dev

   baseline-v2-chat-small-r1-b6a2b9 on dev (102 cases)
   answers 101/102 | team 97/100 | needs a person: caught 23/23, false alarms 9

3. Score it on the holdout. You do this once here, to measure the gap, as the module allows:

   python -m harness score baseline --split holdout

   Note: the holdout set is frozen. Use it once, for the release decision.
   
   baseline-v2-chat-small-r1-b6a2b9 on holdout (98 cases)
   answers 98/98 | team 97/97 | needs a person: caught 24/24, false alarms 4

Priya combines team and needs_human into one measure, decisions: a case counts as right when both are right. Module 3 shows how. The chart gives the share of right decisions per split, with a 95% interval. The interval is the range where the true share probably lies. Module 5 shows how the interval is computed.

The baseline's decisions on seen and unseen tickets. The expected gap does not appear: dev and holdout differ more than contaminated and holdout.

Text description of the chart

Top panel, the baseline’s share of cases with both decisions right, with 95% bootstrap intervals: contaminated 0.944 [0.887, 0.986] on 71 cases; dev 0.882 [0.814, 0.941] on 102; holdout 0.959 [0.918, 0.990] on 98. Bottom panel, only tickets from the deep learning course that need no person: contaminated 0.933 on 60, dev and holdout together 0.927 on 110. All the intervals overlap.

Read the top panel first. The contaminated split scores 0.944, and the holdout scores 0.959: the unseen tickets score higher, not lower. Dev and holdout are random halves of the same slices, so they should score the same, and yet they differ by 0.077. That is more than the difference between contaminated and holdout. All three intervals overlap. The differences are sampling noise: the chance differences between two samples of about 100 cases.

The bottom panel compares like with like, because the splits hold different kinds of tickets. It keeps only tickets from the deep learning course that need no person. Seen tickets score 0.933, unseen ones 0.927. There is no gap here either.

So on this set, the baseline shows no contamination gap. Why not? Prompt v2 has only 2 examples, 2 of the 71 contaminated cases. The other 69 tickets were used to compare prompts, not to fit the prompt to each ticket. A prompt with a few general rules cannot memorise them. The risk is real, and on another system it can be large. This set does not show it, and the honest report says so.

An experiment: tune on dev, check on holdout

Teams usually contaminate their own test more quietly: they tune on the same cases that they report. To see what that does here, the course ran a real experiment on the candidate’s prompt v3.

1. Tune on dev only. The course author read v3’s failures in its five saved dev runs, and nothing from the holdout. Three problems stood out. v3 often reasoned until the 1,000-token limit and gave no answer. It sent questions that the policy does not answer to a person. And it did not say that a person would contact the customer. Prompt v3.1a has one fix for each.
2. Choose on dev. A second version, v3.1b, added a line about safety tickets, after two dev failures. On dev, v3.1a got 99 of 102 decisions right and v3.1b got 97. v3.1a was kept as prompt v3.1, as a team would choose.
3. Freeze it, then score the holdout. Later runs on both splits measured how much one run moves.

Only the prompt text changed: model, schema, policy and token limit stayed the same. The experiment cost US$0.38. The chart compares the gain from v3 to v3.1 on dev, where the prompt was tuned, with the gain on the holdout, which nobody read.

The gain from tuning prompt v3 on dev, measured on dev and on the holdout, with paired 95% intervals.

Text description of the chart

Top panel, decisions, v3.1 minus v3, run 1 of each, with paired 95% bootstrap intervals: dev +0.147 [+0.078, +0.225] on 102 cases; holdout +0.143 [+0.071, +0.214] on 98 cases. Bottom panel, reply criteria: dev +0.127 [+0.049, +0.206]; holdout +0.061 [−0.010, +0.133].

Read the decisions first. Tuned on dev, prompt v3.1 gained +0.147 on dev and +0.143 on the holdout: the same gain, within a fraction of a point. Over all the runs, v3 averaged 0.788 on dev (5 runs) and v3.1 0.958 (3 runs); on the holdout, 0.840 and 0.990. The three fixes addressed causes that every ticket shares, such as the token limit and a misread rule. A fix for a general cause works on tickets that nobody read.

The reply criteria tell another story. The dev gain is +0.127; the holdout gain is only +0.061, and its interval includes 0. Over all the runs, 0.804 → 0.944 on dev, but 0.878 → 0.915 on the holdout. Two reasons, both visible in the saved runs:

• A fix for dev’s own failure. In five dev runs, v3 forgot to say that a person would contact the customer 16 times. In three holdout runs, it never did. The new line fixed a failure that the holdout did not have.
• A choice made on noise. v3.1 still forgets “stop using it” on some safety tickets: 14 times in three holdout runs. v3.1b had the line that addresses this. It lost on dev by 2 decisions in one run, and v3.1a’s own dev runs moved by 3 from run to run.

So on this set, tuning on dev did not inflate the decision gain. But it made the reply gain look two to four times larger than on the holdout. The dev number from the run that chose the prompt is the most flattering number that you have. The holdout number is the one to report.

The checker can be contaminated too

The two red marks in the diagram are real. The labels were read again after the first runs, holdout included, as Label and version showed. And the phrase rules that check the replies (Module 3 builds them) were revised three times after reading replies of all splits:

1. “You don’t need to send your full card number” was first flagged as asking for the number. The rules learned to read a negation.
2. “Keep it out of use” and “a support agent needs to handle this” were missed. The rules learned these phrases.
3. Negated promises such as “we can’t promise a refund” were flagged as promises. The rules learned that too.

Each fix was right, but some came from holdout replies. A checker fixed until it reads the holdout correctly is better at the holdout than at new replies. So the reply-criteria scores on the holdout are a little optimistic. The system did not see the test, but the checker did.

What the course did: the slip is written in the run log and here, so that nobody reads those scores as clean. The holdout decisions stay clean of it: they do not use the phrase rules, and the review changed no team or needs_human label. The phrase rules are now frozen; a later change is tested on dev first. The rule for next time: freeze the labels and the checker before anyone reads a holdout output.

Code that refuses a copied ticket

People follow a rule about looking; a file needs a check. A common leak is a ticket copied from one split into another. You now add a check that refuses it, with the other rules of the set. It runs every time the set loads.

1. Open harness/dataset.py. Find the class Case. Below it, leave two blank lines and add the error that the check raises:

   harness/dataset.py

   class DatasetProblem(Exception):
       """The evaluation set breaks one of its own rules."""

2. Find the function by_id. Below it, before coverage, add these two functions, each followed by two blank lines:

   harness/dataset.py

   def text_key(text: str) -> str:
       """Two tickets with the same words (case and spacing ignored) get the same key."""
       return hashlib.sha256(" ".join(text.lower().split()).encode("utf-8")).hexdigest()[:16]
   
   
   def check(cases: list[Case]) -> None:
       """The rules of the set. Raises DatasetProblem with every broken rule."""
       problems = []
       ids = Counter(c.case_id for c in cases)
       problems += [f"{i}: the case ID is used {n} times" for i, n in ids.items() if n > 1]
       seen: dict[str, Case] = {}
       for c in cases:
           if c.expected.needs_human != bool(c.expected.rule):
               problems.append(f"{c.case_id}: needs_human and rule disagree")
           if c.text.strip():
               other = seen.get(text_key(c.text))
               if other is not None and other.split != c.split:
                   problems.append(f"{c.case_id}: the same text is in {other.split} ({other.case_id}) and {c.split}")
               seen[text_key(c.text)] = c
           if not c.label.by:
               problems.append(f"{c.case_id}: no labeller recorded")
       if problems:
           raise DatasetProblem("; ".join(problems))

   The check has four rules. Every case ID is used once. needs_human is true exactly when a rule is named. No text is in two splits. Every label names who made it. It collects every problem first, so that one run reports all of them.

3. In load_cases, find the line that reads the cases (it ends with if line]). Below it, add one line:

   harness/dataset.py

       check(cases)

4. Open harness/__main__.py. At the end of main(), change except KeyError as e: to this line, so that a broken set prints a clear error:

   harness/__main__.py

       except (KeyError, dataset.DatasetProblem) as e:

5. Copy the file tests/test_dataset.py from the labs snapshot C11/snapshots/end-of-m02 into your tests folder. It has five tests: a version changes with one character, a label change keeps the inputs version, a contaminated ticket copied into dev is refused, needs_human must name its rule, and every label names who made it.

6. Run the tests:

   python -m pytest -q

   .............                                                            [100%]
   13 passed in 0.16s

   The time differs on your computer. Your project is now the same as the snapshot end-of-m02.

Guided practice: try to cheat

1. Make a backup of the set:

   macOS and Linux

   cp data/cases.jsonl data/cases.backup.jsonl

   Windows

   Copy-Item data\cases.jsonl data\cases.backup.jsonl

2. Open data/cases.jsonl. Find the line of P-V2-EX1, copy the whole line, and paste it as a new last line. In the new line, change "case_id": "P-V2-EX1" to "case_id": "T-99999" and "split": "contaminated" to "split": "dev". Save the file.

3. Run python -m harness cases. The set refuses to load:

   Error: T-99999: the same text is in contaminated (P-V2-EX1) and dev

4. In the new line, change the word yesterday to today. Save, and run python -m harness cases again:

   272 cases, dataset version 4150ad262503
     dev            103   needs a person:  23   French: 10
     holdout         98   needs a person:  24   French: 10
     contaminated    71   needs a person:  10   French:  1

   The near-copy passes.

5. Put the original set back:

   macOS and Linux

   mv data/cases.backup.jsonl data/cases.jsonl

   Windows

   Move-Item -Force data\cases.backup.jsonl data\cases.jsonl

6. Run python -m harness cases. The first line must be 271 cases, dataset version {fig.dataset.version}.

This is the limit of a code check: it catches only the same words. A ticket rewritten by one word is the same test, and the check cannot see it. Finding near-copies needs a measure of how similar two texts are, and a person who reads the close pairs.

The limits of made-up tickets

Language models and the course author wrote every ticket in this set. That has three effects on the numbers. The tickets are cleaner than real ones, so real scores would be lower. They have the style that language models write, which a model may read more easily than a customer’s. And the author chose the slices, so the set tests only the problems that the author thought of.

So a score on this set means “on tickets like these”. Before Larkfield trusts a release decision, the set needs real tickets, with personal data removed, labelled by Grace’s team. Module 6 shows how new cases join the set while the holdout stays frozen.

Your turn: what do you tell Omar?

Omar asks for one number: “How much better is prompt v3.1 than v3?” You have the dev result from the run that you chose v3.1 on, the holdout result, and the later runs. Write two sentences for Omar. Which number do you give, and what do you say about the others?

Show an example answer

“Prompt v3.1 gets about 15 more decisions right in 100 tickets than v3, on holdout tickets that nobody read while tuning; dev shows the same. Its replies improved too, but only by about 4 in 100 on the holdout, not the 14 that dev suggests, because part of the dev gain fixed failures that only dev had.”

Do not give the dev number from the run that chose v3.1: it is the best try, on the cases that the tuning read. And the holdout is now used; a further prompt change needs new unseen cases.

Common mistake

'Test it on the examples we tuned the prompt on'

Symptom: Tomás proposes the release test: “Test it on the examples we tuned the prompt on. They are our best cases.” The prompt passes every one of them, and the report says 100%.

Cause: the examples influenced the prompt. Each failure on them was fixed while the prompt was written, and the model may even copy the example answers, as the baseline did. The score measures how well the prompt remembers its examples, not the next customer.

Diagnose: score the same prompt on cases that nobody used while tuning, and compare. On Larkfield’s set: python -m harness score baseline --split contaminated, then --split holdout. A large gap means contamination. No gap, as here, means that this set shows no contamination; it does not make the tuned examples a fair test.

Fix: keep the tuned examples as a separate split, and make the release decision on the frozen holdout only. Add check() so that a tuned ticket cannot be copied into dev or holdout by accident.

See it end to endCalibrate a model judge against human ratingsCase study

The judge calibration case study writes its judge policy on dev questions only, then tests it once on holdout questions.

Check your understanding

The baseline scores 0.944 on the contaminated split and 0.959 on the holdout. Tomás says: 'So contamination does not matter, and we can test on the tuned examples.' What do you answer?

Show the answer

The intervals overlap, and dev and holdout differ by more than contaminated and holdout: the differences are sampling noise. That is a result about this prompt on this set. On the two examples, which the model copies word for word, the score proves nothing. A prompt tuned more closely could show a large gap. Only cases that nobody tuned on can show a gap when there is one, so the release decision uses them.

Check your understanding

The phrase rules that check replies were fixed after reading holdout replies. Which holdout scores does this make optimistic, and which does it not touch? Why?

Show the answer

The reply-criteria scores on the holdout: the checker was fixed until it read those replies correctly, so it is better at them than at new replies. The decisions are not touched: they do not use the phrase rules, and the review changed no team or needs_human label. Next time, freeze the labels and the checker before anyone reads a holdout output.

Check your understanding

Practical check: after the steps of this lesson, run python -m pytest -q. How many tests pass? Then name the test that would fail if someone copied a contaminated ticket into dev.

Show the answer

13 tests pass. The test is test_a_contaminated_ticket_copied_into_dev_is_refused in tests/test_dataset.py. It copies a contaminated case with a new ID into dev and expects check() to raise DatasetProblem with “same text is in contaminated”.

Module practice: a small labelled set and its card

Build a small evaluation set of your own, with labels and a dataset card, in your project.

1. Make a folder practice in the project. In it, make a file cases.jsonl.

2. Write 12 cases, one JSON object per line, in the format of data/cases.jsonl. Use this line as a template: it is a new ticket, labelled by reading the policy.

   practice/cases.jsonl

   {"attachments": "", "case_id": "P-001", "criteria": {"must": ["photo"], "must_not": ["claims_action", "asks_secret"]}, "critical": false, "expected": {"needs_human": false, "next_step": "photo", "rule": "", "team": "delivery"}, "label": {"by": "your name", "changed": "", "date": "2026-10-12", "method": "Grace's policy (teams, H1-H5) applied by reading", "reviewed": "", "team_source": "my reading of the policy"}, "language": "en", "slice": "normal", "source": "practice", "split": "dev", "style": "plain", "tags": [], "text": "The garden hose reel arrived with a cracked handle. Order LK-400112."}

3. Cover at least four slices, including french and needs_person, and one question that the policy does not answer. Put 8 cases in dev and 4 in holdout.

4. Write the labels before you look at any system output. Fill label.by with your name and label.method with how you decided.

5. Load and check your set, with its version and coverage:

   python -c "from pathlib import Path; from harness.dataset import load_cases, version, coverage; p = Path('practice/cases.jsonl'); c = load_cases(p, split='all'); print(len(c), 'cases, version', version(p)); print(coverage(c))"

   With only the template line, the output is:

   1 cases, version c112d6920cac
   {'normal': {'dev': 1}}

   If a rule breaks, the error names the case and the rule. Fix it and run the command again.

6. Write practice/dataset.md, the dataset card. Use these headings: Source, Who labelled and how, Splits and what each is for, Coverage (the table from step 5), What is missing, Version. Read Larkfield’s card in data/dataset.md first, as an example.

Show what a good card says

A good card is short and honest. Source: “12 tickets written by me on 12 October; no real customer wrote them.” Who labelled and how: “I labelled every case by reading data/policy.md, before I looked at any output. Nobody else has checked the labels.” Splits: “8 dev cases for trying prompts; 4 holdout cases, frozen; I will score them once.” Coverage: the table, with counts. What is missing: for example “only 2 French tickets; no attachments; no conversations; my own writing style”. Version: the 12 characters from step 5, and the date.

Module check

Explain to Grace, in a short paragraph, where Larkfield’s set has coverage gaps and how you keep the holdout separate from prompt changes. Name two gaps and the two leaks of this course.

Show an example answer

“The set covers six slices, but some are small: 20 French tickets, only 4 tickets with no readable request and 5 legal threats. It has no real tickets and no conversations, so its scores mean ‘on tickets like these’. We tune prompts on dev only, score the holdout once for the release, and the code refuses a ticket copied between splits. Two leaks happened: labels and reply phrase rules were revised after reading holdout outputs. So the holdout’s criteria scores are a little optimistic, and from now on we freeze labels and checker before anyone reads a holdout output.”

Reset

If your project is broken at the end of this module, follow Reset the project with the snapshot C11/snapshots/end-of-m02. Then python -m pytest -q must say 13 passed.

Recap

• Contamination means that a test case influenced the system that it tests. Its score is too good, and nothing in the score says so.
• Three parts, three jobs: dev is open for tuning, the holdout set is frozen and scored once for the release, and the contaminated split only measures the gap.
• The baseline copies prompt v2’s two examples word for word. On the 69 other seen tickets, it shows no contamination gap: the differences are sampling noise. Report that honestly.
• The experiment: tuned on dev only, prompt v3.1 gained +0.147 decisions on dev and +0.143 on the holdout, but +0.127 reply criteria on dev and only +0.061 on the holdout. Report the holdout number.
• The checker can be contaminated too: the phrase rules were fixed after reading holdout replies, so the holdout’s criteria scores are a little optimistic. Freeze the labels and the checker before anyone reads a holdout output.
• check() refuses a ticket copied between splits, but not a near-copy. Your project matches end-of-m02: 13 passed.

Next, Module 3 turns the labels into measures. It starts with the two decisions: precision and recall for needs_human, and why a missed safety ticket costs more than a misrouted question.

Next: Prediction metrics