Uncertainty and slices

What you will be able to do

Say how sure you are. A score on 98 cases would be a little different on 98 other cases. You will put an interval around every number, compare two systems case by case, and read slices without letting small groups fool you.

Before you startWhat to know and to have readyAbout 55 minutes

What you need to know

• Controlled runs and repeats, from Controlled experiments.
• Slices, from Generated-output metrics.
• Samples, confidence intervals and the bootstrap, from Samples and populations and Estimate uncertainty in Essential Mathematics and Statistics for AI. This lesson uses them; it does not teach them again.

What you need on your computer

• ~/projects/eval-harness after Controlled experiments, with .venv active and numpy installed: python -m harness repeats baseline works.

No account and no key. Tested with Python 3.12.13 on macOS (Apple silicon). The Windows commands are not tested.

Two restaurants

Two restaurants are on a map app. The first has 5 stars, from 4 reviews. The second has 4.6 stars, from 2,000 reviews. Which is better? Most people trust the 4.6. Four happy friends of the owner can give 5 stars; 2,000 strangers are harder to fool.

The number alone does not tell you that. You also need to know how many reviews are behind it. With 4 reviews, one more bad review would move the score a lot. With 2,000, one review hardly moves it.

The problem: the same system, two different numbers

The baseline is one system: prompt v2, chat-small, one recorded run. On the 102 dev cases it gets both decisions right on 0.882 of them. On the 98 holdout cases, the same run gets 0.959. Nothing changed but the cases, and the number moved by 0.077. The dev and holdout cases come from the same sources, split at random. Avoid contamination shows that this gap is sampling noise.

Now compare the candidate with the baseline. On dev, the candidate is worse by 0.059. On holdout, it is worse by 0.112. Is the candidate a little worse, or much worse? Could it even be no worse at all, with bad luck on dev? A difference of 0.059 means nothing until you know how much a number moves by chance on 102 cases.

The idea: a confidence interval

A confidence interval is a range around a measured value that shows how far the value could move by chance. A 95% interval comes from a method that covers the true value 95 times in 100, over many new samples. Wide means unsure; narrow means sure. It is the “number of reviews” of an evaluation result, turned into a range you can read.

The size of the sample decides most of the width. Here is the baseline’s share of correct decisions on the first 25, 50, 100 and 200 cases of the set. Each has its 95% interval from the harness:

Cases	Both decisions right	95% interval	Width
25	0.84	0.68 to 0.96	0.28
50	0.82	0.72 to 0.92	0.20
100	0.89	0.83 to 0.95	0.12
200	0.92	0.880 to 0.955	0.075

Eight times more cases give an interval almost four times narrower. Most of that comes from the size: the width shrinks with the square root of the number of cases, as Estimate uncertainty explains. The rest comes from the score itself, which is closer to 1 on 200 cases, where results spread less. So the first cases you add help a lot, and later ones help less.

The bootstrap: make many samples from one

You have one evaluation set, not many. The bootstrap makes new samples from it. Pick cases from your set at random, with replacement, until you have as many as the set. Some cases come up twice, some not at all. Compute the score on this resample. Repeat this 2,000 times. The middle 95% of the 2,000 scores is the bootstrap interval. The name comes from “pulling yourself up by your own bootstraps”: the data measures its own uncertainty.

Here is the whole method on five cases. Case C is wrong, the others are right, so the score is 0.8. The resamples below are the first four that the harness itself draws.

The bootstrap on five cases, with the first four resamples of the harness (numpy, seed 2026). A resample can leave out the one failure, so it can score better than the real cases.

Text description of the diagram

Five cases, A to E, with the values 1, 1, 0, 1 and 1: the mean is 0.8. Each resample picks five cases at random, with replacement. Resample 1 picks E, A, A, D and B: the mean is 1.0, and C is left out. Resample 2 picks C, A, B, D and B: the mean is 0.8, and E is left out. Resample 3 picks E, D, D, E and D: the mean is 1.0, and A, B and C are left out. Resample 4 picks A, E, D, A and B: the mean is 1.0, and C is left out. After 2,000 resamples, the middle 95% of the means runs from 0.4 to 1.0.

The means of the harness's 2,000 resamples of the five cases. The lowest 2.5% end at 0.4 and the highest reach 1.0, so the 95% interval is 0.4 to 1.0.

Text description of the chart

Five bars, one per possible mean. 0.2: 10 resamples. 0.4: 101. 0.6: 411. 0.8: 825. 1.0: 653. Most resamples land near the real mean of 0.8, but many reach 1.0 and some fall to 0.4.

The interval runs from 0.4 to 1.0. Five cases cannot tell a system that is right 4 times in 10 from one that is always right. That is the restaurant with 4 reviews. The harness does the same in harness/compare.py, with bootstrap_mean(). In Python:

>>> from harness.compare import bootstrap_mean
>>> print(bootstrap_mean([1, 1, 0, 1, 1]))
0.800 [0.400, 1.000] (n=5)
>>> print(bootstrap_mean([1, 1, 0, 1, 1] * 20))
0.800 [0.720, 0.880] (n=100)

The same share, 0.8, from 100 cases: the interval is now 0.72 to 0.88. The harness fixes the seed (2026), so it gives the same interval every time, and anyone can check a report.

Explore: resample the real results

The explorer below uses the real per-case results of both recorded runs on the 200 dev and holdout cases. Pick a sample size, draw resamples, and watch the interval form.

Explore

Resample the harness's real results: how wide is the interval?

Sample

first 25

Real cases

0.840 (21 of 25)

Resamples

2000

Middle 95%

0.680 to 0.960

Harness, 2,000

0.680 to 0.960

You reached 2000 resamples, as many as the harness draws. Pick another sample size to compare the width.

• Both decisions right
• Wrong (or no answer)

Your intervals: the baseline

Cases	Value	Middle 95%	Width
25	0.840	0.680 to 0.960	0.280
50	0.820	0.700 to 0.920	0.220
100	0.890	0.820 to 0.950	0.130
200	0.920	0.880 to 0.955	0.075

The baseline (prompt v2) and the candidate (prompt v3), chat-small, recorded run 1, on the first 25 to 200 of the 200 dev and holdout cases. Each resample draws the same number of cases with replacement. The table keeps the interval of each size.

Text description of the explorer

The controls choose the sample size: the first 25, 50, 100 or 200 cases. They also choose what to resample: the baseline’s results, or the difference candidate minus baseline on each case. Buttons draw 1 resample, 100 resamples, or up to 2,000.

A grid shows one square per case. For the baseline, blue squares have both decisions right and orange squares do not. For the difference, blue means the candidate is better on that case, orange worse, grey the same. After a resample, the number in a square shows how often the resample picked it. An empty square is a case that it left out.

A histogram collects the value of each resample. A dashed line marks the value on the real cases, and a bracket marks the middle 95%. A table keeps the interval for each size. With 2,000 resamples, the explorer’s intervals are close to the harness’s. The baseline: 25 cases 0.68 to 0.96; 50 cases 0.72 to 0.92; 100 cases 0.83 to 0.95; 200 cases 0.88 to 0.955. The difference: 100 cases −0.130 to +0.010, which contains 0; 200 cases −0.135 to −0.030, which does not.

Three things to see:

1. The width falls with the size. From 25 to 200 cases, the interval narrows from about 0.28 to about 0.075.
2. The difference needs more cases than a single score. On the first 100 cases, the interval of the difference still touches 0. On 200, it does not.
3. The middle stays in place, the edges move. Draw 100 resamples twice: the edges change a little each time. 2,000 resamples make them steady. The harness uses 2,000.

Compare on the same cases: the paired difference

Both systems answered the same cases. So compare them case by case: for each case, take the candidate’s result minus the baseline’s. A case that both get right gives 0. A case that only the candidate gets right gives +1, and one that only the baseline gets right gives −1. The mean of these is the difference, and the bootstrap of these per-case values is a paired interval. It is narrower than two separate intervals, because the easy and hard cases are the same for both systems. Compare alternatives in the mathematics course shows why.

A tiny example with six cases:

>>> from harness.compare import paired_difference, wins_losses
>>> base = {"A": 1, "B": 1, "C": 1, "D": 0, "E": 1, "F": 1}
>>> cand = {"A": 1, "B": 0, "C": 1, "D": 1, "E": 1, "F": 0}
>>> wins_losses(base, cand)
{'better': ['D'], 'worse': ['B', 'F'], 'same': ['A', 'C', 'E']}
>>> print(paired_difference(base, cand))
-0.167 [-0.667, 0.333] (n=6)

Three cases say nothing: both systems agree on them. The evidence is in the cases where they differ: one better, two worse. So the difference is (1 − 2) ÷ 6 = −0.167, and with six cases its interval is very wide. wins_losses() names the cases behind the number, so you can read them.

Compare the candidate with the baseline

1. Copy the tests of the comparison code from the snapshot end-of-m05:

   macOS and Linux

   cp ../nextia-learning-labs/C11/snapshots/end-of-m05/tests/test_compare.py tests/

   Windows

   Copy-Item ..\nextia-learning-labs\C11\snapshots\end-of-m05\tests\test_compare.py tests\

2. Open tests/test_compare.py and read its three tests. The same seed gives the same interval. More cases give a narrower interval. The paired difference names its cases. Then run the tests. The last line is 37 passed:

   python -m pytest

3. Compare the two systems on dev:

   python -m harness compare baseline candidate

   candidate-v3-chat-small-r1-31b45f (candidate) against baseline-v2-chat-small-r1-b6a2b9 (baseline), dev, 102 cases
   decisions  baseline 90 | candidate 84 | difference -0.059 [-0.127, 0.010] (n=102) | better 4, worse 10
   criteria   baseline 100 | candidate 85 | difference -0.147 [-0.225, -0.078] (n=102) | better 1, worse 16

4. Compare them on the holdout set. This is the set for the release decision, so the harness reminds you that it is frozen:

   python -m harness compare baseline candidate --split holdout

   Note: the holdout set is frozen. Use it once, for the release decision.
   
   candidate-v3-chat-small-r1-31b45f (candidate) against baseline-v2-chat-small-r1-b6a2b9 (baseline), holdout, 98 cases
   decisions  baseline 94 | candidate 83 | difference -0.112 [-0.184, -0.041] (n=98) | better 1, worse 12
   criteria   baseline 96 | candidate 87 | difference -0.092 [-0.163, -0.020] (n=98) | better 2, worse 11

Here are the decisions on a chart: each system alone, then the paired difference on dev, on holdout, and on both.

Both decisions right, with 95% bootstrap intervals (2,000 resamples). Top: each system alone on the holdout set. Bottom: the paired difference, candidate − baseline, with a line at no difference. Larkfield's limit is −0.03.

Text description of the chart

The top panel shows each system on the 98 holdout cases. The baseline: 0.959 (0.918 to 0.99). The candidate: 0.847 (0.776 to 0.918). The two intervals do not overlap.

The bottom panel shows the paired difference, candidate minus baseline. A dashed line marks 0, no difference. Larkfield’s limit is −0.03. Dev, 102 cases: −0.059 [−0.127, +0.010]; the interval crosses 0. Holdout, 98 cases: −0.112 [−0.184, −0.041]; the whole interval is below −0.03. Dev and holdout together, 200 cases: −0.085 [−0.135, −0.030]; the whole interval is below 0, and its upper end is at the limit.

Read the dev line first. The difference is −0.059, but the interval runs from −0.127 to +0.010: it includes 0. On dev alone, you cannot say that the candidate is worse on decisions. You can say that it is not better by more than 0.01.

On holdout, the interval runs from −0.184 to −0.041. It is entirely below 0, and entirely below the limit of −0.03 that Grace and Omar set in Module 1. Together, 200 cases give −0.085, from −0.135 to −0.030. Two separate sets point the same way.

Then read the cases. “better 1, worse 12” on holdout means that the whole difference rests on 13 cases. They are the cases to open with show. One of the 12 is T-65663, a safety ticket with no answer. An interval tells you how sure you are about the size; it does not tell you which case matters most. The next lesson handles that.

Slices, with caution

A slice is a group of cases of one kind: French tickets, tickets that need a person, questions the policy does not answer. Generated-output metrics showed that a slice can hide a failure. Now the same slices get intervals. score prints one per slice:

python -m harness score candidate --split holdout --by slice

Note: the holdout set is frozen. Use it once, for the release decision.

candidate-v3-chat-small-r1-31b45f on holdout (98 cases)
valid answers 90/98 | team 89/97 | needs a person: caught 23/24, false alarms 7 | reply criteria passed 87/98
  boundary      criteria 0.714 [0.500, 0.929] (n=14)
  difficult     criteria 0.862 [0.724, 0.966] (n=29)
  french        criteria 1.000 [1.000, 1.000] (n=10)
  needs_person  criteria 0.952 [0.857, 1.000] (n=21)
  normal        criteria 0.909 [0.727, 1.000] (n=11)
  unanswerable  criteria 0.923 [0.769, 1.000] (n=13)

The overall line covers 98 cases; each slice has 10 to 29. With 14 boundary cases, the interval runs from 0.50 to 0.93. With 11 normal cases, one case moves the score by 0.09. The chart shows the paired difference of the decisions per slice, on dev and holdout together (200 cases).

Candidate − baseline, both decisions right, per slice, on the 200 dev and holdout cases, with paired 95% bootstrap intervals. Most intervals are wide, and several cross 0.

Text description of the chart

One row per slice, with a dashed line at 0. normal (23 cases): −0.043 [−0.174, +0.087]. boundary (28 cases): −0.107 [−0.214, 0.000]. difficult (59 cases): −0.119 [−0.220, −0.034]. French (20 cases): 0.000 [0.000, 0.000]. needs a person (43 cases): −0.047 [−0.116, 0.000]. unanswerable (27 cases): −0.148 [−0.370, +0.074]. Only the difficult slice has an interval entirely below 0. The unanswerable slice has the largest drop and the widest interval. The French interval is a single point at 0.

Three cautions when you read slices:

1. A small slice has a wide interval. The unanswerable slice drops the most, by 0.148. But its interval runs from −0.370 to +0.074. Read the cases; do not announce the number.
2. Many slices, some surprises by chance. With six slices, one of them can look bad or good by luck alone. A slice that you found by searching after you saw the results is a question for the next evaluation, not a finding.
3. The bootstrap cannot see a failure that never happened. The candidate gets all 20 French cases right. So every resample is also 20 of 20, and the interval is 1.000 to 1.000. That does not mean that it never fails on French. It means that 20 cases did not show a failure.

Read the slices as a list of cases to open, not as a ranking. The difficult and unanswerable slices hold 15 of the candidate’s 22 lost decisions on dev and holdout. Eight of those 15 are answers cut off at the limit.

When to use intervals, and what they cost

Put an interval on every number that a decision depends on, and the number of cases next to it. That is the cheapest protection against overreading a small set. The bootstrap costs nothing but a second of computer time on saved outputs, and it works for any measure, even when no formula exists.

Use the paired difference whenever both systems answered the same cases, which a controlled experiment guarantees. Do not use it to compare runs on different cases, such as last month’s numbers.

An interval has limits. It covers sampling noise, not a biased set: if the set lacks a kind of ticket, no interval warns you. It covers the noise of one run, not the run-to-run noise of Controlled experiments. And it says how big the difference may be, not whether that size matters. Grace’s limit of −0.03 decides that.

Common mistake

Trusting a perfect slice

Symptom: Tomás reads the slice table: “French: 1.000, interval 1.000 to 1.000. The candidate is perfect on French. Let us at least ship it for French tickets.”

Cause: a slice of 20 cases with no failure. The bootstrap only resamples what it has, so it cannot make a failure that it never saw: the interval collapses to one point. A system that fails on 1 French ticket in 10 would still show 20 of 20 about 1 time in 8.

Fix: report the slice with its size: “20 of 20 French cases, no failure seen”. A common rule of thumb: with no failure in n cases, the true failure rate can still be up to about 3 ÷ n. Here that is 15%. To ship for French tickets alone, add French cases until the interval is narrow enough for the decision.

Guided practice: the slices of the comparison

1. Run python -m harness compare baseline candidate --split holdout --by slice.
2. Before you read the output, predict: in which slice does the candidate lose the most decisions on holdout?
3. Find the slice with the largest drop and its number of cases. Is that drop more than one or two cases?

Show the expected output and an answer

Note: the holdout set is frozen. Use it once, for the release decision.

candidate-v3-chat-small-r1-31b45f (candidate) against baseline-v2-chat-small-r1-b6a2b9 (baseline), holdout, 98 cases
decisions  baseline 94 | candidate 83 | difference -0.112 [-0.184, -0.041] (n=98) | better 1, worse 12
    boundary      baseline 0.93  candidate 0.86  (n=14)
    difficult     baseline 1.00  candidate 0.83  (n=29)
    french        baseline 1.00  candidate 1.00  (n=10)
    needs_person  baseline 1.00  candidate 0.95  (n=21)
    normal        baseline 1.00  candidate 0.91  (n=11)
    unanswerable  baseline 0.77  candidate 0.54  (n=13)
criteria   baseline 96 | candidate 87 | difference -0.092 [-0.163, -0.020] (n=98) | better 2, worse 11
    boundary      baseline 1.00  candidate 0.71  (n=14)
    difficult     baseline 0.97  candidate 0.86  (n=29)
    french        baseline 1.00  candidate 1.00  (n=10)
    needs_person  baseline 0.95  candidate 0.95  (n=21)
    normal        baseline 1.00  candidate 0.91  (n=11)
    unanswerable  baseline 1.00  candidate 0.92  (n=13)

On decisions, the unanswerable slice drops the most: 0.77 to 0.54, on 13 cases, so 3 cases. The difficult slice drops less, 1.00 to 0.83, but on 29 cases: 5 cases. In cases, the difficult slice loses more. Always turn a slice’s share back into cases before you compare slices.

Your turn: the reply rules, on 200 cases

The release rule in Module 1 uses the decisions. Check the reply rules too, on dev and holdout together.

1. In Python, load the dev and holdout cases: [c for c in load_cases(split="all") if c.split != "contaminated"].
2. Use per_case(cases, outputs, "criteria") for both runs, then paired_difference() and wins_losses().
3. Write one sentence for Omar with the difference, its interval, the number of cases, and how many cases got better and worse.

Show an example answer

>>> from harness.dataset import load_cases
>>> from harness.run import find_run, load_outputs
>>> from harness.compare import per_case, paired_difference, wins_losses
>>> cases = [c for c in load_cases(split="all") if c.split != "contaminated"]
>>> b = per_case(cases, load_outputs(find_run("baseline")), "criteria")
>>> c = per_case(cases, load_outputs(find_run("candidate")), "criteria")
>>> print(paired_difference(b, c))
-0.120 [-0.170, -0.070] (n=200)
>>> wl = wins_losses(b, c)
>>> len(wl["better"]), len(wl["worse"])
(3, 27)

“On 200 cases, the candidate passes every reply rule on 0.120 fewer cases than the baseline (95% interval: 0.070 to 0.170 fewer). It is better on 3 cases and worse on 27.” The phrase rules have errors of their own, such as the missed “cesser”. So read the 27 cases before you quote them.

See it end to endCalibrate a model judge against human ratingsCase study

The judge calibration case study puts a bootstrap interval on every agreement number, and splits it by category.

See it end to endEvaluate summaries against expert ratingsCase study

The summaries case study resamples whole articles, not single summaries, and shows how the wrong unit makes an interval too narrow.

See it end to endRegression-test a classifier release by sliceCase study

The classifier case study compares two versions slice by slice with paired intervals, and finds the one slice that gets worse.

Check your understanding

On dev, the decisions difference is −0.059 with the interval [−0.127, 0.010]. Tomás says: 'The interval includes 0, so the candidate is as good as the baseline.' What is wrong with that?

Show the answer

An interval that includes 0 means that the data cannot rule out “no difference”; it does not show that there is none. The same interval allows a loss of 0.127. And the holdout set, with its own cases, gives −0.112 [−0.184, −0.041], entirely below 0; both sets together give −0.085 [−0.135, −0.030]. Report the size and the interval, and use more cases when the interval is too wide for the decision.

Check your understanding

Why does the harness compute the difference case by case (paired), and not as the candidate's score minus the baseline's score with two separate intervals?

Show the answer

Both systems answered the same cases, so hard cases are hard for both. The paired difference removes that shared part and keeps only the cases where the systems differ, so its interval is narrower and fair. Pairing also names the cases behind the difference: on holdout, 1 better and 12 worse, which you can read one by one.

Check your understanding

A slice of 14 boundary cases drops from 1.00 to 0.71 on the reply rules. What do you report, and what do you do next?

Show the answer

Report it with its size and interval: 10 of 14 against 14 of 14. The candidate’s interval is about 0.50 to 0.93. Then read the 4 failing cases, because 14 cases are too few for a firm number. In this run, 2 of the 4 are answers cut off at the limit. Treat a slice as a list of cases to read, not as a verdict.

Recap

• Give every number its sample size and a 95% confidence interval. On 25 cases the baseline’s interval is about 0.28 wide; on 200, about 0.075.
• The bootstrap resamples your cases with replacement, 2,000 times; the middle 95% is the interval. Five cases with one failure give 0.4 to 1.0.
• The paired difference compares the systems case by case. Dev: −0.059 [−0.127, 0.010]; holdout: −0.112 [−0.184, −0.041]; both: −0.085 [−0.135, −0.030].
• Slices are small: read them as cases to open, not as verdicts. A perfect slice of 20 still allows failures that 20 cases did not show.

Next: the candidate is worse with good evidence. But Tomás points to a judge that prefers it 126 to 48. How do quality, cost, time and one silent safety ticket make one decision?

Next: Trade-offs and regressions