Controlled experiments

What you will be able to do

Make a comparison fair. Change one thing, keep everything else the same, write every setting down, and run the model more than once. Then a difference in the results means something.

Before you startWhat to know and to have readyAbout 45 minutes

What you need to know

• Baselines, candidates and run IDs, from Establish a baseline.
• The dataset version and the inputs version, from Label and version.
• The candidate’s run and the model judges, from Model-based judging.

What you need on your computer

• ~/projects/eval-harness at the end of Module 4, with .venv active: python -m pytest shows 34 passed. If it does not, copy the snapshot end-of-m04 as Use a snapshot explains.
• The labs repository in ~/projects/nextia-learning-labs, with the newest files.

No account and no key. This lesson adds one package, numpy, for the computations of the module. Tested with Python 3.12.13 on macOS (Apple silicon). The Windows commands are not tested.

Two cakes and one new flour

You want to know if a new flour makes a better cake. So you bake two cakes and taste them. The new-flour cake is better. But you also used a different oven, more sugar and a shorter baking time. Which change made the difference? You cannot say.

So you try again. Same recipe, same oven, same time; only the flour changes. You write the recipe down, so that you can bake it again. And you bake each cake twice, because your oven is not the same every time. If the two old-flour cakes already differ a lot, a small difference between the flours means little.

The problem: a difference with many causes

Tomás has the candidate’s numbers and wants to compare them. “Last month, the old assistant chose the right team on 69 of 70 tickets. The candidate gets 88 of 100 here. So the candidate is worse by 10 points.” But the two numbers come from different tickets, on a different day. Establish a baseline showed that the same assistant gets 69 of 70 on one set and 97 of 100 on another. The tickets alone move the number.

There is a second problem, and it is less visible. The model does not give the same answer twice. Priya sent the same 102 dev tickets to the candidate five times. In run 1, the candidate gave no answer to T-64740, a safety ticket: a parasol fell toward a child. Nobody would hear about it. In run 2, the same ticket went to a person, as it must. If Priya had made only run 2, she would report: “no blocker on dev”.

The idea: a controlled experiment

A controlled experiment changes one thing and keeps everything else fixed. The thing you change is the variable under test: here, the prompt. Everything you keep fixed is a control: the cases, the model and its version, the output schema, the output limit. Then a difference in the results comes from the change, or from chance. Nothing else is left.

That leaves chance. A language model is stochastic: it picks each word with some randomness, so two runs on the same input can differ. A run of a stochastic system is one sample of its behaviour, like one cake from your oven. So you repeat the run, and you look at the spread between the repeats before you trust a difference.

A controlled experiment has three rules:

1. Hold the inputs fixed. Both systems see the same cases, proved by the same inputs version.
2. Version every setting. The run records the prompt and its hash, the model and its version, the dataset version. Change one, and the run ID changes.
3. Repeat stochastic runs. Run each system several times on the same cases, and report the spread, not the best run.

A tiny example: two runs, one change

Here are the settings of the baseline’s run 1 and the candidate’s run 1, side by side. Every setting comes from the run’s record in outputs/runs.json, except the output schema and limit, which come from the recording log.

The baseline's and the candidate's run 1. Only the prompt differs, so a difference in results comes from the prompt or from chance.

Text description of the diagram

The diagram has a row for each setting, with the baseline on the left and the candidate on the right. The run IDs differ, because they name the system and the prompt. The cases seen are the same: 271 cases with inputs version 3f90ab808275. The model is the same: chat-small, version gpt-6-luna-2026-09-22. The output is the same: a strict schema and a limit of 1,000 tokens. Only the prompt differs: v2, with the hash 728add8f0b0d, against v3, with the hash 1f6de460aaf2.

Note the inputs version. The labels changed after the first runs (32 next steps, on review), so the dataset version changed. The inputs version covers only what the system saw: the case IDs, the texts and the attachments. It did not change, so the saved outputs are still valid. Label and version explains the two versions.

You can check all of this with code, instead of trusting the diagram. That is the next step.

Add the repeats to the project

The project gets the eight repeated runs, the comparison code and a new command. The comparison code needs numpy, a Python library for fast computation on lists of numbers. You install it with the new requirements.txt.

1. Get the newest labs files, and go to the project:

   cd ~/projects
   git -C nextia-learning-labs pull
   cd eval-harness

2. Copy the new requirements, the saved runs, the comparison code and the command line from the snapshot end-of-m05.

   macOS and Linux

   cp ../nextia-learning-labs/C11/snapshots/end-of-m05/requirements.txt .
   cp ../nextia-learning-labs/C11/snapshots/end-of-m05/outputs/* outputs/
   cp ../nextia-learning-labs/C11/snapshots/end-of-m05/harness/compare.py harness/
   cp ../nextia-learning-labs/C11/snapshots/end-of-m05/harness/__main__.py harness/

   Windows

   Copy-Item ..\nextia-learning-labs\C11\snapshots\end-of-m05\requirements.txt .
   Copy-Item ..\nextia-learning-labs\C11\snapshots\end-of-m05\outputs\* outputs\
   Copy-Item ..\nextia-learning-labs\C11\snapshots\end-of-m05\harness\compare.py harness\
   Copy-Item ..\nextia-learning-labs\C11\snapshots\end-of-m05\harness\__main__.py harness\

3. Install numpy. The .venv must be active. One of the last lines starts with Successfully installed numpy:

   python -m pip install -r requirements.txt

4. Run the tests. The last line is 34 passed. The new code gets its tests in the next lesson.

   python -m pytest

5. List the runs. Each system now has five runs: run 1 on all 271 cases, runs 2 to 5 on the 102 dev cases. The practice run is for Trade-offs and regressions: it has one constructed reply, written by the course author.

   python -m harness runs

   baseline-v2-chat-small-r1-b6a2b9            271 cases  contaminated, dev, holdout gpt-6-luna-2026-09-22  started 2026-10-09T14:56:12-04:00
   baseline-v2-chat-small-r2-ef711b            102 cases  dev                        gpt-6-luna-2026-09-22  started 2026-10-09T15:10:14-04:00
   baseline-v2-chat-small-r3-feb481            102 cases  dev                        gpt-6-luna-2026-09-22  started 2026-10-09T15:17:06-04:00
   baseline-v2-chat-small-r4-e2030c            102 cases  dev                        gpt-6-luna-2026-09-22  started 2026-10-09T15:23:21-04:00
   baseline-v2-chat-small-r5-3a8295            102 cases  dev                        gpt-6-luna-2026-09-22  started 2026-10-09T15:28:29-04:00
   candidate-v3-chat-small-r1-31b45f           271 cases  contaminated, dev, holdout gpt-6-luna-2026-09-22  started 2026-10-09T15:01:11-04:00
   candidate-v3-chat-small-r2-64de63           102 cases  dev                        gpt-6-luna-2026-09-22  started 2026-10-09T15:12:11-04:00
   candidate-v3-chat-small-r3-548f7f           102 cases  dev                        gpt-6-luna-2026-09-22  started 2026-10-09T15:20:06-04:00
   candidate-v3-chat-small-r4-92e51b           102 cases  dev                        gpt-6-luna-2026-09-22  started 2026-10-09T15:25:18-04:00
   candidate-v3-chat-small-r5-504de7           102 cases  dev                        gpt-6-luna-2026-09-22  started 2026-10-09T15:31:06-04:00
   strong-v2-chat-strong-r1-160302             102 cases  dev                        gpt-6.1-sol-2026-09-29  started 2026-10-09T15:34:30-04:00
   practice-v3-chat-small-r1-31b45f            271 cases  contaminated, dev, holdout gpt-6-luna-2026-09-22  started 2026-10-09T15:01:11-04:00

Look at the start times. Priya ran the baseline and the candidate in turns, a few minutes apart, on the same afternoon. If the provider changed something that afternoon, both systems met it. That is also a control.

Check the controls with code

Start Python in the project folder (python), and compare the records of the two run 1s field by field:

>>> from harness.run import Run, load_runs, load_outputs
>>> runs = load_runs()
>>> b = runs["baseline-v2-chat-small-r1-b6a2b9"]
>>> c = runs["candidate-v3-chat-small-r1-31b45f"]
>>> [k for k in Run.model_fields if getattr(b, k) != getattr(c, k)]
['run_id', 'system', 'prompt', 'prompt_sha256', 'started', 'finished', 'note']
>>> b.model_version == c.model_version, b.inputs_version == c.inputs_version
(True, True)

The fields that differ are the names, the times, the note and the prompt. The model version and the inputs are the same. That is a one-change comparison, checked by code.

The repeats saw only the dev cases, so their inputs version is different: it covers 102 cases, not 271. Check that it is the inputs version of the same 102 dev cases:

>>> from harness.dataset import load_cases, inputs_version
>>> dev = load_cases(split="dev")
>>> inputs_version(dev)
'9b24fa392841'
>>> runs["candidate-v3-chat-small-r3-548f7f"].inputs_version
'9b24fa392841'

Five runs of each system

Now look at the spread. The repeats command scores every run of one system on the 102 dev cases:

python -m harness repeats baseline
python -m harness repeats candidate

baseline x 5 runs, decisions: totals [90.0, 96.0, 94.0, 92.0, 95.0] out of 102 | min 90, max 96 | 16 cases changed between runs
baseline x 5 runs, criteria: totals [100.0, 97.0, 98.0, 98.0, 97.0] out of 102 | min 97, max 100 | 7 cases changed between runs
candidate x 5 runs, decisions: totals [84.0, 81.0, 76.0, 80.0, 81.0] out of 102 | min 76, max 84 | 24 cases changed between runs
candidate x 5 runs, criteria: totals [85.0, 80.0, 79.0, 82.0, 84.0] out of 102 | min 79, max 85 | 40 cases changed between runs

“Decisions” counts the cases where both decisions are right: the team, if the case has one, and needs_human. “Criteria” counts the cases where every reply rule passed. The chart shows the decisions of each run as a dot.

Five recorded runs of each system on the same 102 dev cases. Each system moves by several cases from run to run; the two ranges do not overlap.

Text description of the chart

Two rows of five dots. The baseline: 90, 96, 94, 92, 95 of 102, from 90 to 96. The candidate: 84, 81, 76, 80, 81 of 102, from 76 to 84. The baseline’s worst run is still better than the candidate’s best run.

Read it in three steps.

1. One system moves by itself. The baseline got both decisions right on 90 cases in run 1 and on 96 in run 2. Nothing changed between those runs. A difference of 6 cases is normal for this system.
2. The ranges do not overlap. The baseline’s worst run (90) beats the candidate’s best run (84). So “the candidate is worse on decisions” is not an accident of one run.
3. Different cases move. 16 cases changed result between the baseline’s runs, 24 between the candidate’s. A case that changes is a case to read.

Here is one that changes. T-81058 asks: “What is the address of your warehouse? I would like to drop off my return in person.” No rule says that a person must answer it. Ask what the baseline said in each run:

>>> base = sorted((r for r in runs.values() if r.system == "baseline"), key=lambda r: r.repeat)
>>> [load_outputs(r)["T-81058"].answer["needs_human"] for r in base]
[True, False, False, False, False]

The same prompt, the same model, the same ticket: a false alarm in run 1, and the right answer in the four others.

The silent failure, run by run

The candidate’s biggest problem is not a wrong decision. It is no answer: the model reasons until it reaches the limit of 1,000 tokens and returns nothing. Count it in each run, and list the critical cases among those cut off:

>>> critical = {x.case_id for x in dev if x.critical}
>>> cand = sorted((r for r in runs.values() if r.system == "candidate"), key=lambda r: r.repeat)
>>> for r in cand:
...     outs = load_outputs(r)
...     cut = [x.case_id for x in dev if outs[x.case_id].finish_reason == "length"]
...     print(r.repeat, len(cut), [i for i in cut if i in critical])
...
1 11 ['T-64740']
2 16 []
3 19 ['T-81036']
4 14 []
5 14 []

Five recorded runs on the 102 dev cases. The candidate is cut off on 11 to 19 cases per run, and a different critical ticket goes silent in different runs.

Text description of the diagram

The top part counts the answers cut off at the 1,000-token limit in each of the five runs. The baseline: 1, 0, 0, 0, 0. The candidate: 11, 16, 19, 14, 14.

The bottom part follows two critical tickets in the candidate’s runs. The baseline sent both to a person in all five runs. T-64740 is a safety ticket: a parasol fell toward a child. The candidate gave no answer in run 1 and sent it to a person in runs 2 to 5. T-81036 is an access ticket: a parcel for number 16 was left with the customer, who asks who it is for. The candidate sent it to a person in runs 1, 2, 4 and 5, and gave no answer in run 3.

Across the five runs, 34 different dev cases were cut off at least once, and only 3 in every run. The failure moves from case to case. So one run can miss it: runs 2, 4 and 5 show no silent critical case at all. Two runs of five show one. A release decision based on run 4 alone would miss the problem completely.

This is why the release rules look at every case of every run, not only at an average. Trade-offs and regressions turns these silent cases into blockers.

When to repeat, and what it costs

Repeat a run when the system is stochastic and the decision matters. For a language model, that is almost always. Where a model lets you set its randomness to the lowest value, its answers still are not fully repeatable, so the rule stays.

How many repeats? Look at the spread of one system first. If the difference between the systems is much larger than the spread, as here, three to five runs are enough. If the difference is about the size of the spread, more runs of a weak result will not make it strong. The next lesson shows how to put a number on that.

Repeats cost money and time. One dev run of the candidate cost about US$0.05 and a few minutes. That is cheap here, because the set is small and the model is small. With a large model, 10,000 cases and five repeats, it is a budget decision. When you must save, repeat the critical cases more often than the rest: they decide the release.

Deterministic code needs no repeats. The scorers in harness/scorers/ give the same result every time on the same saved output, so one score is enough. That is one reason to save every output: you score it again for free.

Common mistake

Reporting the best run

Symptom: Tomás runs the candidate five times and reports run 1: “84 of 102, only 6 behind the baseline’s 90.” The baseline is also from run 1. In run 3, the candidate gets 76, 14 behind the same baseline run.

Cause: Tomás compares the best of five runs of one system with one run of the other. The best run is high partly by chance, so the comparison favours the system that got more tries. The same happens when you rerun a test “until it passes”.

Fix: decide the number of runs before you look at the results. Run both systems the same number of times on the same cases. Report every run: the minimum, the maximum and the blockers of each run. In the harness: repeats baseline and repeats candidate, side by side.

Guided practice: the same check for the baseline

1. In the same Python session, run the loop of The silent failure, run by run for the baseline: replace cand with base.
2. Before you run it, predict: how many answers does the baseline lose to the limit in each run?
3. Run it. Compare with the five-runs diagram.

Show the expected output

1 1 []
2 0 []
3 0 []
4 0 []
5 0 []

The baseline lost one answer in run 1 (T-65450, a collection question, not critical) and none in the other runs. The silent failure belongs to the new prompt, not to the model alone. Prompt v3 asks for complete, specific answers, and the model reasons longer to write them.

Your turn: which changes are silent?

The candidate has 24 dev cases whose decisions changed between runs. Find how many of them were cut off in at least one run.

1. Import repeat_spread from harness.compare.
2. Load the outputs of the five candidate runs into a list: outs = [load_outputs(r) for r in cand].
3. Call repeat_spread(dev, outs, "decisions") and take its "unstable_cases".
4. Keep the cases where any run has finish_reason == "length".

Show an example answer

>>> from harness.compare import repeat_spread
>>> outs = [load_outputs(r) for r in cand]
>>> unstable = repeat_spread(dev, outs, "decisions")["unstable_cases"]
>>> silent = [i for i in unstable if any(o[i].finish_reason == "length" for o in outs)]
>>> len(unstable), len(silent)
(24, 22)

22 of the 24 changing cases are cut off in at least one run. So the candidate’s instability is mostly one failure, no answer, and not a mix of different mistakes. That points to one fix to test: a higher output limit, or a prompt that asks for less. Each fix is a new candidate, with its own controlled runs.

The numbers in this module come from recordings of each system. Models change fast; the way to compare them lasts.

Check your understanding

Omar wants to compare prompt v3 on the model chat-strong with prompt v2 on chat-small, in one run each. What is wrong with that experiment, and how would you change it?

Show the answer

Two things change at once, the prompt and the model, so a difference cannot be put on either one. Change one thing per comparison: for example, prompt v3 on chat-small against prompt v2 on chat-small, then the model as a separate step. And one run each is not enough for a stochastic model. Run each version several times on the same cases, with the same inputs version, and report the spread.

Check your understanding

The candidate's run 4 shows no critical case without an answer on dev. Is that evidence that the silent failure is fixed? Explain with the repeats.

Show the answer

No. The same prompt and model left T-64740 without an answer in run 1, and T-81036 in run 3. Runs 2, 4 and 5 show none. The failure moves from case to case, and 34 different dev cases were cut off at least once. One clean run is one sample. To trust a fix, repeat the runs, find no silent critical case in any of them, then check the holdout.

Recap

• A controlled experiment changes one thing and holds the rest fixed. Here only the prompt changed: the same 271 inputs, model version and limit, checked by code from the run records.
• A language model is stochastic: five runs of the same baseline gave 90 to 96 of 102. Report the spread, not the best run.
• The candidate’s range (76 to 84) does not overlap the baseline’s, so its loss on decisions is not one bad run.
• The candidate is cut off on 11, 16, 19, 14 and 14 dev cases. A different critical ticket goes silent in different runs, and three runs of five show none.
• Repeats cost money and time. Repeat the stochastic system, not the deterministic scorers, and repeat the critical cases first.

Next: how sure can you be about one difference on 98 or 102 cases? Confidence intervals, paired comparisons and slices.

Next: Uncertainty and slices