Harness structure

What you will be able to do

Put the harness in order: load the cases, run the system, score each case, and write a report. Each stage saves files with a run ID, so that anyone can rebuild the report from saved outputs and trace every number to its cases.

Before you startWhat to know and to have readyAbout 45 minutes

What you need to know

• Run IDs and the baseline, from Establish a baseline.
• Paired differences with intervals, from Uncertainty and slices, and the release decision, from Trade-offs and regressions.

What you need on your computer

• ~/projects/eval-harness at the end of Module 5, with .venv active: python -m pytest shows 101 passed. If it does not, copy the snapshot end-of-m05 as Use a snapshot explains.
• The labs repository in ~/projects/nextia-learning-labs, next to your project, up to date (git -C ../nextia-learning-labs pull).

No account and no key: every step reads saved files. Tested with Python 3.12.13 on macOS (Apple silicon). The Windows commands are not tested.

An exam that someone can check

A school runs a written exam in four separate steps. First, it prints the question paper and gives it a number. Next, the students write their answers, and the school collects and keeps every answer sheet. Then the markers mark each answer against a marking scheme, one mark per question. Last, the office adds the marks and prints a results sheet.

Now a student says: “My grade is wrong.” The school can check it, because every step left a record. It finds the answer sheet by the student’s number, and the mark for each question. It can also mark the sheet again without asking the student to sit the exam a second time. If the markers had written only a final grade on a list, nobody could check anything.

The problem: a number nobody can check

Tomás sends a short message to the team: “Candidate decisions: 81 of 102 on dev, difference −0.088 [−0.167, −0.020]. Worse, but close.”

Priya wants to see the 21 cases behind the number, so she asks: “Which run?” The candidate has five recorded runs on the development cases (Controlled experiments). She compares two of them with the baseline:

python -m harness compare baseline candidate-v3-chat-small-r2-64de63
python -m harness compare baseline candidate-v3-chat-small-r5-504de7

The output:

candidate-v3-chat-small-r2-64de63 (candidate) against baseline-v2-chat-small-r1-b6a2b9 (baseline), dev, 102 cases
decisions  baseline 90 | candidate 81 | difference -0.088 [-0.167, -0.020] (n=102) | better 3, worse 12
criteria   baseline 100 | candidate 80 | difference -0.196 [-0.284, -0.118] (n=102) | better 1, worse 21
candidate-v3-chat-small-r5-504de7 (candidate) against baseline-v2-chat-small-r1-b6a2b9 (baseline), dev, 102 cases
decisions  baseline 90 | candidate 81 | difference -0.088 [-0.167, -0.020] (n=102) | better 3, worse 12
criteria   baseline 100 | candidate 84 | difference -0.157 [-0.235, -0.088] (n=102) | better 1, worse 17

Run 2 and run 5 give exactly Tomás’s numbers. But they are not the same results: the two runs disagree on 8 of the 102 cases. So the message does not say which tickets failed, and nobody can check it. Tomás could ask the model again, but it would answer differently a third time.

The number needs a record of where it came from: the run, the cases and the saved outputs.

The idea: four stages, with files between them

An evaluation harness is the code that runs a system on the evaluation set and turns the results into a decision. The word comes from the straps that hold a horse to a cart: the harness holds every part in place, so the parts work together.

A good harness has four separate stages, like the exam:

1. Load: read the cases of one split, check the set’s rules, and note the dataset version.
2. Run: give every case to the system and save each output, under a run ID.
3. Score: give each case a result, such as 1 or 0, then add the results.
4. Report: write the decision and the measures, with the cases behind each number.

Each stage reads files that the stage before it wrote. That is the whole trick. Because the outputs are saved, you can score and report again at any time without running the system. And because each file carries a run ID, a number in the report can name its run and its cases.

The four stages of Larkfield's harness, and what each one hands to the next.

Text description of the diagram

Stage 1, load: harness/dataset.py reads data/cases.jsonl, checks the set’s rules and computes the dataset version, d4c96eaa10c0. It gives the cases of one split.

Stage 2, run: the assistant answers every case. In this course, this happened once, with the real model, and the outputs were saved; run.py reads them. It gives outputs/<run ID>.jsonl and outputs/runs.json.

Stage 3, score: the scorers and compare.py. The function per_case() gives 1 or 0 for every case, then the harness adds totals and intervals.

Stage 4, report: release.py and report.py check the blockers first, then the measures, the slices, the cost and the time. It gives report.md, a decision that names its cases.

Each stage reads files that the stage before it wrote. Score and report never call a model, so you can run them again at any time and get the same result.

In your harness, stage 2 has already happened. The course ran the baseline and the candidate with the real model, with the settings of Establish a baseline and Controlled experiments. The outputs are in outputs/. When Amira records a new candidate, only stage 2 is new: it writes one more file and one more line in runs.json. The other three stages do not change.

A tiny example: five cases, two runs

Here is the score stage on five made-up cases. A result of 1 means that both decisions are right on that case.

Case	Baseline	Candidate	Change
A	1	1
B	1	0	worse
C	0	1	better
D	1	0	worse
E	1	1
Total	4 of 5	3 of 5	1 better, 2 worse

The candidate’s total is 3 of 5, against 4 of 5. The paired difference is (3 − 4) ÷ 5 = −0.2. Notice what the table keeps. The total alone says “one fewer”. The table also says that the candidate fixed C and broke B and D. A reader who doubts the −0.2 can open cases B, C and D and check them.

The report of Larkfield’s harness is this same table, for 98 holdout cases, with intervals.

Add the report to the harness

1. In the project folder, with .venv active, set the variable for the snapshot:

   macOS and Linux

   S=../nextia-learning-labs/C11/snapshots/end-of-m06

   Windows

   $S = "..\nextia-learning-labs\C11\snapshots\end-of-m06"

   If you open a new terminal, set it again.

2. Copy the report code:

   macOS and Linux

   cp $S/harness/report.py harness/

   Windows

   Copy-Item $S\harness\report.py harness\

3. Predict, then run:

   python -m harness report baseline candidate --split holdout --out report.md

   The output ends with:

   python -m harness: error: argument command: invalid choice: 'report' (choose from cases, show, coverage, runs, score, retrieval, judge, pairwise, agree, public, compare, repeats, decide)

   The code is there, but the command line does not know the command yet.

4. Copy the new command line, and run the command again:

   macOS and Linux

   cp $S/harness/__main__.py harness/
   python -m harness report baseline candidate --split holdout --out report.md

   Windows

   Copy-Item $S\harness\__main__.py harness\
   python -m harness report baseline candidate --split holdout --out report.md

   The output:

   Note: the holdout set is frozen. Use it once, for the release decision.
   
   Wrote report.md

   The new __main__.py also has the command feedback, for the third lesson of this module.

5. Open report.md in VS Code. Press Ctrl+Shift+V (macOS: Cmd+Shift+V) to see it as a formatted page.

Read the report

The report opens with the facts that a reader needs to check it:

# Release report: candidate-v3-chat-small-r1-31b45f against baseline-v2-chat-small-r1-b6a2b9

- Split: holdout (98 cases); dataset version d4c96eaa10c0 (runs recorded on 263bbd4f180d; inputs version 3f90ab808275)
- Baseline: baseline, prompt v2, chat-small (gpt-6-luna-2026-09-22)
- Candidate: candidate, prompt v3, chat-small (gpt-6-luna-2026-09-22)

Both run IDs, the split and the number of cases. Then two dataset versions: the set is d4c96eaa10c0 now, but the runs were recorded on 263bbd4f180d, before the label review of Label and version. Is that a problem? No: the inputs version is the same, 3f90ab808275. The review changed labels, not the texts that the system saw, so the saved outputs are still valid.

Next comes the decision, in the order of Trade-offs and regressions: blockers first, then the measures. Each blocker is one row with its case:

## Blockers (any one fails the release)

| Case | Blocker | Evidence | Found by |
|---|---|---|---|
| T-65663 | No valid answer on a critical case. |  | code |

The measures follow, and each row names the cases that moved:

| Measure | Baseline | Candidate | Candidate - baseline | Cases better / worse |
|---|---|---|---|---|
| decisions | 94/98 | 83/98 | -0.112 [-0.184, -0.041] (n=98) | T-81052 / T-64736, T-64842, T-64859, T-65289, T-65452, T-65522 and 6 more |
| criteria | 96/98 | 87/98 | -0.092 [-0.163, -0.020] (n=98) | T-64860, T-81022 / T-64736, T-64815, T-64859, T-64893, T-65022, T-65032 and 5 more |

The rest of the file has the needs-a-person counts, the criteria by slice with intervals, and the recorded cost and time. Every number in it comes from the saved files. Nothing in it comes from a person’s memory or from a new model call.

Reproduce the report

A report that you cannot rebuild is a claim, not evidence. So build it a second time, into another file, and compare the two files.

macOS and Linux

python -m harness report baseline candidate --split holdout --out report-2.md
shasum -a 256 report.md report-2.md

Windows

python -m harness report baseline candidate --split holdout --out report-2.md
Get-FileHash report.md, report-2.md

The output (macOS):

d65dee9596dbf0b48aa17fc3e740e5199a554f55d272691a98544ae53764bb46  report.md
d65dee9596dbf0b48aa17fc3e740e5199a554f55d272691a98544ae53764bb46  report-2.md

The two fingerprints are the same, so the two files are identical, byte for byte. On Windows, the value is different from this one, because Windows ends each line with two characters instead of one. But your two files still share one value. Three choices make this work. The report reads only saved files. The bootstrap uses a fixed seed (2026), so the intervals are the same each time. And the cases are always in the same order. Delete report-2.md when you are done.

A test keeps it that way. Copy it, and run the tests:

macOS and Linux

cp $S/tests/test_report.py tests/
python -m pytest

Windows

Copy-Item $S\tests\test_report.py tests\
python -m pytest

The last line is 103 passed. test_report.py builds the report twice and checks that the two are equal. It also checks that the report says **Do not ship.** and names T-65663 as the blocker.

Trace a score to its case

The report lists six cases and then “and 6 more”. To check the whole number, you need every case. The score stage keeps them: per_case() gives each case its 1 or 0. Start Python in the project folder with python, and type:

from harness.dataset import load_cases
from harness.run import find_run, load_outputs
from harness.compare import per_case

cases = load_cases(split="holdout")
b = per_case(cases, load_outputs(find_run("baseline")), "decisions")
c = per_case(cases, load_outputs(find_run("candidate")), "decisions")
print(sum(b.values()), sum(c.values()))
for case_id in sorted(b):
    if b[case_id] != c[case_id]:
        print(case_id, b[case_id], "->", c[case_id])

The output:

94.0 83.0
T-64736 1.0 -> 0.0
T-64842 1.0 -> 0.0
T-64859 1.0 -> 0.0
T-65289 1.0 -> 0.0
T-65452 1.0 -> 0.0
T-65522 1.0 -> 0.0
T-65525 1.0 -> 0.0
T-65663 1.0 -> 0.0
T-65793 1.0 -> 0.0
T-65794 1.0 -> 0.0
T-65809 1.0 -> 0.0
T-81052 0.0 -> 1.0
T-81059 1.0 -> 0.0

Twelve cases worse and one better: 94 − 12 + 1 = 83, the candidate’s total in the report. Now follow one of them, T-65663, all the way down. Type exit() to leave Python, then:

python -m harness show T-65663 --run candidate

The last lines of the output:

candidate-v3-chat-small-r1-31b45f: no valid answer (finish_reason: length)
Reply:
  FAIL  must     reply  (no reply)

The case is a safety ticket (needs_human true (H1)): a frying pan with a hot, cracked handle. The candidate gave no answer, so no person would be told. The saved line in outputs/candidate-v3-chat-small-r1-31b45f.jsonl shows why: "answer": null, "finish_reason": "length" and "reasoning_tokens": 1000. The model reasoned until its limit and wrote nothing.

One number of the release report, traced to its case and its saved output.

Text description of the diagram

1. The report, section Measures: decisions 94 of 98 against 83 of 98. The difference is −0.112, with the interval −0.184 to −0.041. The worse cases: T-64736, T-64842, T-64859, T-65289, T-65452, T-65522 and 6 more.
2. Per case, with per_case() on the 98 holdout cases: 13 cases differ, 12 worse and 1 better (T-81052). T-65663: baseline 1, candidate 0.
3. The case, with show T-65663 --run candidate: expected team warranty and needs_human true (rule H1). The candidate has no valid answer (finish_reason: length), and the reply check fails: no reply.
4. The saved output, the line of T-65663 in the run’s file: answer null, finish_reason length, 1,000 reasoning tokens. runs.json gives the settings: prompt v3, chat-small, model version gpt-6-luna-2026-09-22.

That is the completion check of this module, done once. You followed one number in the report to its case, its saved output and its settings.

Check the run IDs themselves

A run ID is only useful if it is honest. make_run_id() builds it from the settings in runs.json, so you can rebuild every ID and compare. In Python:

from harness.run import load_runs, make_run_id

for r in load_runs().values():
    rebuilt = make_run_id(r.system, r.prompt, r.prompt_sha256, r.model, r.dataset_version, r.repeat, r.started)
    print(r.run_id, rebuilt == r.run_id)

The output:

baseline-v2-chat-small-r1-b6a2b9 True
baseline-v2-chat-small-r2-ef711b True
baseline-v2-chat-small-r3-feb481 True
baseline-v2-chat-small-r4-e2030c True
baseline-v2-chat-small-r5-3a8295 True
candidate-v3-chat-small-r1-31b45f True
candidate-v3-chat-small-r2-64de63 True
candidate-v3-chat-small-r3-548f7f True
candidate-v3-chat-small-r4-92e51b True
candidate-v3-chat-small-r5-504de7 True
strong-v2-chat-strong-r1-160302 True
practice-v3-chat-small-r1-31b45f False

Eleven IDs match their settings. One does not: the practice run. The course built it by hand from the candidate’s run 1, for the release practice of Module 5. One reply in it is constructed. It kept the candidate’s hash, so the check finds it. Its note in runs.json says so. This is what a stable run ID buys you: a run that was not made the normal way shows up.

When to structure a harness like this, and what it costs

Use separate stages with saved files whenever more than one person reads the results, or a result decides a release. Saved outputs let you score again for free when a scorer changes. They also let a reviewer check a number without a key, and let you compare runs months later, after the model has changed.

The costs are small but real. The saved outputs take disk space: here, 12 runs in about 2.3 MB. Every new scorer must work from saved files, not from a live call. And the run IDs only help if nobody edits an output file by hand. To correct a saved output, make a new run with its own ID and a note. The practice run did only half of this: it has its own name and a note, but it kept the candidate’s hash. That is why the check above finds it.

A very small, one-off check does not need all this. If you test three prompts once by reading ten replies, a notebook is enough. Move to a harness when the check repeats.

Common mistake

The release report on the wrong split

Symptom: Tomás runs python -m harness report baseline candidate --out report.md, without --split, and sends the file to Omar as the release report. Omar reads that the decisions difference is −0.059 [−0.127, 0.010]. “The interval includes zero. So maybe it is not worse?”

Cause: without --split, the harness uses the development cases. The report says so in its second line:

- Split: dev (102 cases); dataset version d4c96eaa10c0 (runs recorded on 263bbd4f180d; inputs version 3f90ab808275)

Dev is the set that the team reads and tunes on. Its blocker is another case, T-64740, and its interval is wider than the holdout’s −0.184 to −0.041. The release decision belongs to the holdout set, used once.

Fix: write the release report with --split holdout, and put the split in the file name, for example report-holdout.md. Before you send a report, read its first three lines: run IDs, split, versions.

Guided practice: trace the criteria row

The report’s criteria row says 96 of 98 against 87 of 98, with 2 better and 11 worse. Check it.

1. Start Python, and run the code of Trace a score to its case again, with one change: "criteria" instead of "decisions" in both per_case() calls.
2. Count the lines. Does 96 − worse + better give 87?
3. Pick T-64815 and run python -m harness show T-64815 --run candidate. Which rule fails?
4. Read the reply. Would a reader fail it too?

Show the answers

Thirteen lines: 11 worse (T-64736, T-64815, T-64859, T-64893, T-65022, T-65032, T-65289, T-65452, T-65663, T-65793, T-65809) and 2 better (T-64860, T-81022). 96 − 11 + 2 = 87.

T-64815 fails refund_after_receipt. The reply says: “The returns team needs to confirm receipt first; if the shears were unused and returned within 30 days of delivery, the return qualifies for a full refund.” A reader would probably accept it: the refund comes after receipt. The phrase rule looks for words such as “once we receive” or “receipt of the”, and it does not match “confirm receipt first”. So this failure is the checker’s error, not the candidate’s. Tracing finds these too: Read every failure before you count it.

Your turn: is the report reproducible on your computer?

Prove to Omar that the release report does not depend on your copy of the project or on your memory.

1. Copy the snapshot end-of-m06 to a new folder, make a fresh .venv in it, and install the requirements, as Use a snapshot explains.
2. Run the holdout report there, and compare its SHA-256 with your report.md.
3. Write two sentences for Omar: what you compared, and what it shows.

Show an example answer

On macOS, the fresh copy gives d65dee9596dbf0b48aa17fc3e740e5199a554f55d272691a98544ae53764bb46, the same as report.md in the project.

“I rebuilt the holdout report in a fresh copy of the project, with a new environment, from the saved outputs only. The file is identical, byte for byte, so every number in it comes from the saved runs, not from my copy of the project.”

Knowledge check

Check your understanding

Name the four stages of the harness, and say what each one writes for the next.

Show the answer

Load reads the cases of one split, checks the rules and notes the dataset version. Run gives every case to the system and saves each output under a run ID, in outputs/<run ID>.jsonl and runs.json. Score gives each case a result, 1 or 0, and adds them, with intervals. Report writes the decision and the measures, with the cases behind each number, in report.md. Because each stage reads saved files, you can score and report again without running the system.

Check your understanding

Two candidate runs both get 81 of 102 on dev, with the same difference from the baseline. Are they the same result? What do you need to tell them apart?

Show the answer

No. Runs 2 and 5 have the same totals but disagree on 8 cases, so different tickets failed. You need the run ID with every number, and the per-case results, which the score stage keeps. With the run ID, show CASE --run RUN_ID opens the exact saved output behind each failure.

Check your understanding

The report says 'dataset version d4c96eaa10c0 (runs recorded on 263bbd4f180d; inputs version 3f90ab808275)'. Can you trust the scores? Explain.

Show the answer

Yes. The two dataset versions differ because labels changed after the runs were recorded. The inputs version covers only what the system saw: case IDs, texts and attachments. It is the same, so the saved outputs answer the same tickets. The scores use the new labels, which is what the review was for. If the inputs version had changed, the outputs would no longer match the cases, and you would need new runs.

Recap

• An evaluation harness has four stages: load, run, score, report. Each one reads files that the stage before it wrote.
• The run stage saves every output under a run ID. Score and report never call a model, so they give the same result every time.
• python -m harness report baseline candidate --split holdout --out report.md writes the release report. Its first lines name the runs, the split and the versions.
• You can reproduce the report from saved outputs, byte for byte, and trace each number to its cases with per_case() and show.
• A number without its run ID cannot be checked: two candidate runs give the same 81 of 102 on different cases.
• Your project has report.py, the new command line and test_report.py: 103 passed.

Next, the release rules become tests that run on every change.

Next: Regression checks