Compare alternatives

What you will be able to do

Decide what a small difference between two models supports: how large it is, whether chance can explain it, whether it matters, and what caused it.

NotebookOptional: check the numbers in a notebookColab · Kaggle · GitHub

The notebook is optional: this lesson works with a calculator. It has the results of models A and B on the same 300 tickets. You flip the coins of this page’s explorer 10,000 times and calculate the exact p-value and the interval for the difference. Then you do the module practice with model C, with checks. It uses only Python’s standard library, so it needs no install.

Open in ColabOpen in KaggleView on GitHub

Colab and Kaggle open your own copy. They need a free Google or Kaggle account. Stuck? See the solution notebook.

Before you startWhat to know and to have readyAbout 50 minutes

What you need to know

• How to calculate a 95% confidence interval for an accuracy, from Estimate uncertainty: Worked example 2. The intervals of model A (87.8% to 94.2%) and model B (88.9% to 95.1%) are from Estimate uncertainty: Guided practice.
• Coin flips and the binomial distribution, from Distributions in practice: Binomial. The p-value in this lesson counts coin flips.
• What makes a test set fair, from Samples and populations: Worked example. The module practice starts with this check.

What you need to have ready (the notebook needs none of it)

• A calculator with a square root key. A phone calculator is enough.
• Your c03-notes.md file, with the heading ## Sampling and statistical evidence from the “Before you start” box of Samples and populations. If you cannot find the file, make a new empty file with that name: this lesson does not use earlier answers.
• About 15 minutes at the end for the module practice and the module check.

The notebook was tested in Jupyter with Python 3.14.6 (standard library only).

Where you meet this in AI

• A new model version. A team tests a new model on the same test set as the old one. The new one is 1 point better. Is it time to switch?
• A prompt change. A new prompt for an assistant gives a good answer on 47 of 50 test questions; the old one on 45. Is the new prompt better, or was it chance?
• An A/B test. Half of the users get version A and half get version B, for two weeks. Version B has 3% more clicks. Did B cause that?
• A before-and-after report. “Since we started the new model, complaints fell by 20%.” What else changed at the same time?

Tomás’s claim

Model B is correct on 276 of the 300 test tickets (92.0%), model A on 273 (91.0%). The tickets are the same for both models. “B is better, let us use it,” says Tomás.

Priya separates three questions. Mixing them up is the most common mistake in comparisons:

1. Observed effect: how large is the difference that we measured?
2. Uncertainty: could chance alone give a difference this large?
3. Cause: what made the difference, and will it also happen in daily work?

Observed effect: how large is the difference?

An effect size is the size of a difference, in a unit that people understand. One difference can be written in several units:

Unit	Model A	Model B	Difference
Accuracy	91.0%	92.0%	+1.0 percentage point
Correct tickets of 300	273	276	3 tickets
Wrong tickets of 300	27	24	3 fewer errors, or 11% fewer errors (3 ÷ 27)
Wrong-team tickets per week, at 1,200 tickets	about 108	about 96	about 12 fewer per week

All four rows describe the same result. “11% fewer errors” sounds bigger than “1 point”, but it is the same 3 tickets. A good report gives the absolute numbers first: 1 point, 3 of 300 tickets.

Uncertainty: look at the tickets where the models disagree

Both models sorted the same 300 tickets. Priya puts the results in a table with one row and one column per model:

	B correct	B wrong	Total
A correct	261	12	273
A wrong	15	12	27
Total	276	24	300

On 261 tickets, both models are right. On 12, both are wrong. Those 273 tickets say nothing about which model is better.

The evidence is in the 27 tickets where the models disagree. B is right and A wrong on 15. A is right and B wrong on 12. The difference, 15 − 12 = 3, is the 3 tickets from the effect table.

Now imagine that the two models are equally good. Then each of the 27 disagreements could go to A or to B with the same chance, like a coin. Would a coin give 15 to 12, or something more uneven? Try it.

Explore

If A and B were equally good: flip a coin for each disagreement

Shuffles

300

Latest difference

+3

Shuffles at least as extreme as the real result

220 of 300 (73.3%)

Exact p-value

0.70 (3 or more)

73% of shuffles give a difference of 3 or more, in either direction, by chance alone. A difference like the real one is common without any real difference.

• Model A right, Model B wrong
• Model B right, Model A wrong

Each shuffle flips a coin for each of the 27 disagreements and records B wins minus A wins. Orange bars are differences at least as large as the real one (3 or more, in either direction).

Text description of the explorer

The top picture shows the 27 tickets where the models disagree. In the real result, B is right on 15 (orange, marked B) and A is right on 12 (blue, marked A). After a shuffle, each square shows who wins after a coin flip.

The histogram shows the difference “B wins minus A wins” for each shuffle. The differences are odd numbers, almost all between −13 and +13 and most between −7 and +7, with a peak around −1 and +1. The real result, +3, is in the middle of the histogram. Differences of 3 or more in either direction (orange bars) happen in about 70% of shuffles.

The readout shows the number of shuffles and the latest difference. It also shows how many shuffles gave a difference at least as large as the real one, and the exact p-value, 0.70.

About 70 of 100 shuffles give a difference of 3 or more, in one direction or the other. So a result like 15 to 12 is normal when the two models are equally good. The test result does not show that B is better.

The names for what you just did

• The null hypothesis is the assumption that there is no real difference. Here: A and B are equally good, so each disagreement is a coin flip.
• A hypothesis test checks if the data is surprising under the null hypothesis.
• The p-value is the share of chance-only results that are at least as extreme as the real result. Here it is 0.70: 70% of chance-only results are at least as uneven as 15 to 12. You can calculate it exactly from coin-flip probabilities (Distributions in practice); the explorer gets close by counting.

Many teams call a result statistically significant when p is below 0.05. Then chance alone gives such a result less than 1 time in 20. The 0.05 is a habit, not a law of nature.

Three things a p-value is not:

1. It is not the probability that B is better. p = 0.70 does not mean “30% chance that B is better”.
2. It is not the size of the effect. With a very large test set, a tiny, useless difference gets a very small p-value.
3. A large p-value does not prove that the models are equal. It says that this test cannot tell them apart. B could still be a little better, or a little worse.

In Python, this paired test on the disagreements is called McNemar’s test, for example in the statsmodels library.

An interval for the difference

A p-value answers only “could it be chance?”. An interval for the difference also shows how large the difference could be. For two models on the same tickets, it uses the two disagreement counts. Here b = 15 (B right, A wrong) and c = 12 (A right, B wrong).

$$\text{SE of the difference} = \frac{\sqrt{\,b + c - (b-c)^2 / n\,}}{n}$$

Read it as: “the square root of (the disagreements, minus the squared difference divided by n), divided by n”. You do not need to remember it; the steps are below.

Step	Value
Difference, (b − c) ÷ n	3 ÷ 300 = 0.010 = +1.0 point
b + c	27
(b − c)² ÷ n	9 ÷ 300 = 0.03
√(27 − 0.03)	√26.97 = 5.19
SE = 5.19 ÷ 300	0.0173 = 1.73 points
Half-width = 1.96 × SE	3.4 points
95% interval	1.0 − 3.4 to 1.0 + 3.4 = −2.4 to +4.4 points

Point estimates with 95% intervals: model A and model B on the same 300 tickets, and the difference B − A. The interval for the difference contains zero.

Text description of the chart

The chart has two panels. In the first, model A has 91.0% (87.8% to 94.2%) and model B has 92.0% (88.9% to 95.1%). The two intervals overlap almost completely.

In the second panel, the difference B − A is +1.0 point. On the same 300 tickets, its 95% interval runs from −2.4 to +4.4 points. A dashed line at zero marks “no difference”, and the interval crosses it. A grey row shows the wider interval for a test of the two models on different tickets: −3.5 to +5.5 points.

Read the second panel. The interval runs from −2.4 to +4.4 points, and it contains zero. The data fits “B is 4 points better”, “no difference” and “B is 2 points worse”. On 1,200 tickets a week, that is anything from about 29 more wrong-team tickets to about 53 fewer.

The grey row shows why testing both models on the same tickets helps. If A and B had different test tickets, part of the difference would come from easier or harder tickets. That interval would be wider, ±4.5 points instead of ±3.4. A paired comparison removes that part.

Practical significance: does it matter?

Suppose B is really 1 point better. Is that worth a switch? This is practical significance: whether a difference is large enough to matter for the decision. Statistics cannot answer it alone. Omar asks Grace and Tomás for the costs:

Item	Value
Gain if B is 1 point better	about 12 fewer wrong-team tickets a week
Work saved per wrong-team ticket	about 5 minutes of agent time to move it (and the customer waits less)
Work saved per week	12 × 5 = about 60 minutes
Cost to switch	about 30 hours: deploy, test, watch the first weeks
Weeks until the switch pays back	about 30 weeks, if the gain is real

Omar’s decision: “A 1-point gain is worth having, but only if it is real, and the test cannot show that. Do not switch on this evidence. Test B on live tickets in a fair way first.”

A difference can be statistically significant but not practically significant. With 100,000 test tickets, a 0.1-point gain gets a tiny p-value and still saves nobody any time. It can also be the other way round: a large gain that a small test cannot yet confirm. Report both parts.

Cause: correlation is not causation

Tomás has a second argument. In week 41, he switched the live system to model B for a trial. Grace’s weekly report shows the share of re-routed tickets, tickets that an agent had to move to another team:

• Model A in use
• Model B in use

Weekly share of re-routed tickets, weeks 35 to 44. Model B replaced model A in week 41, the same week that a delivery outage ended. Made-up data.

Text description of the chart

A line chart of the weekly share of re-routed tickets. With model A, weeks 35 to 37 are at about 9.2% (9.1, 9.4, 9.2). During a delivery outage in weeks 38 to 40, the share rises to 11.8%, 12.4% and 11.6%. Model B starts in week 41, when the outage also ends. Weeks 41 to 44 are at about 8.2% (8.1, 8.0, 8.4, 8.2).

“Re-routes fell from 11.6% to 8.1% when B started: 30% fewer,” says Tomás. The two things happened together: they are correlated. That does not show that B caused the fall. Causation means that changing one thing changes the other.

Grace knows what else happened in week 41: the delivery outage ended. During the outage, many confusing tickets arrived (“where is my order, and I want a refund”). Compare with the weeks before the outage instead: 9.2% before, 8.2% after. That is a fall of about 1 point, and even that could come from other changes, such as a new agent or the season.

A difference between two periods can have many causes: the ticket mix, the season, the staff, a change on the website. To find the effect of the model alone, the two models must run at the same time, on the same kind of tickets. That is an A/B test. Each new ticket goes to model A or model B at random, for example by a coin flip in the software. Random assignment makes the two groups alike in everything except the model. So the model causes any difference in re-routes, apart from chance, which the interval measures.

Omar decides to run an A/B test: for four weeks, each new ticket goes at random to A or B. That gives about 2,400 tickets per model.

Common mistake

A before-and-after comparison credited to the model

Symptom: a report says “after we deployed the new model, re-routed tickets fell by 30%”. Months later, the rate is back where it was, with the same model.

Cause: the report compared two periods. Something else changed at the same time: here, an outage ended. The fall correlated with the new model but was mostly caused by the ticket mix. A before-and-after comparison cannot separate the model from everything else that changed.

Fix: run both versions in the same period, and send each ticket (or user) to one of them at random. This is an A/B test. If an A/B test is not possible, compare with a period that had no special events. Write down every other change in the same weeks. Call the result “correlated with the switch”, not “caused by it”.

Check your understanding

Two models are tested on the same 300 tickets. Model A is right on 280, model B on 283. Which tickets carry the evidence about which model is better, and why do the others not?

Show the answer

Only the tickets where the two models disagree: one is right and the other wrong. On tickets where both are right, or both are wrong, the two models behave the same, so these tickets cannot show a difference. The 3-ticket difference is the number of B-only wins minus A-only wins. Whether 3 is surprising depends on how many disagreements there are: 3 of 9 disagreements is more uneven than 3 of 27.

Check your understanding

Tomás runs the same comparison on 100,000 tickets. B is 0.2 points better, with p = 0.001. Tomás says: 'Very significant, so B is much better.' What is right and what is wrong?

Show the answer

Right: with p = 0.001, chance alone is an unlikely explanation, so B is probably better on tickets like these. Wrong: “much better”. The p-value is small because the test set is very large, not because the effect is large. The effect size is 0.2 points: about 2 or 3 tickets a week at 1,200 tickets. That is probably not worth the cost of a switch. Statistically significant is not the same as practically significant.

Module practice

The task of this module: decide whether a small difference in performance supports a strong claim. First, a small worked example with every step visible. Then a new case for you.

Worked example: “B is better, let us use it”

Priya answers Tomás’s claim with five checks. Each check uses one idea from this module.

Check	Lesson	Result for A vs B
1. Is the test set fair, and the same for both?	Lesson 1	Yes: 300 random tickets from 12 months, split by customer; both models on the same tickets.
2. What is the observed effect?	Lesson 3	+1.0 point: 276 vs 273 of 300. About 12 tickets a week.
3. How uncertain is it?	Lessons 2 and 3	95% interval for the difference: −2.4 to +4.4 points. p = 0.70 (15 vs 12 of 27 disagreements).
4. Does it matter in practice?	Lesson 3	If real, it saves about 60 minutes a week; the switch costs about 30 hours.
5. What can we say about the cause?	Lesson 3	On the test set, the models saw the same tickets, so the difference comes from the models or from chance. For live effects (re-routes), only an A/B test can tell; the week-41 comparison cannot.

Priya’s conclusion: “On 300 tickets, B was right on 3 more than A. That difference is well within chance: the true difference could be from 2.4 points worse to 4.4 points better. The data does not support ‘B is better’. It also does not show that the models are equal. We run a 4-week A/B test before we decide.”

Your task: model C

Two months later, Tomás trains model C. Mei builds a new test set: 1,000 random tickets from the first two weeks of March, split by customer. These are the results:

	C correct	C wrong	Total
A correct	888	22	910
A wrong	41	49	90
Total	929	71	1,000

Tomás writes: “Model C is clearly better than A, and it will cut re-routed tickets by a third.”

1. In c03-notes.md, add a heading: ### Module 5 practice: model C.
2. Answer check 1: write what the test set represents, and what it does not.
3. Calculate the observed effect in points, in tickets per 1,000, and as a share of A’s errors.
4. Calculate the 95% interval for the difference. Use b = 41, c = 22 and n = 1,000 in the steps of the worked example.
5. Write whether 41 to 22 looks like a coin flip. The exact p-value is 0.023; say what that means in one sentence.
6. Check the two parts of the claim, “clearly better” and “a third fewer re-routes”, one by one.
7. Write a two-sentence conclusion for Omar.

Expected result: the effect is 1.9 points, and the interval is about +0.4 to +3.5 points. The data partly supports one part of the claim and does not support the other.

Show a model answer

1. Test set. 1,000 random tickets, split by customer, so no leakage. But they all come from two weeks in March. The result describes tickets like March tickets; other months (for example December, with many delivery tickets) are not tested.

2. Observed effect. A: 910 of 1,000 (91.0%). C: 929 of 1,000 (92.9%). Difference: +1.9 points, or 19 tickets per 1,000 (about 23 a week at 1,200 tickets). Errors: A 90, C 71, so 19 fewer errors: 19 ÷ 90 = 21% fewer errors.

3. Interval.

Step	Value
(b − c) ÷ n	19 ÷ 1,000 = +1.9 points
b + c	63
(b − c)² ÷ n	361 ÷ 1,000 = 0.361
√(63 − 0.361)	√62.639 = 7.91
SE = 7.91 ÷ 1,000	0.79 points
Half-width = 1.96 × 0.79	1.55 points
95% interval	+0.35 to +3.45 points (about +0.4 to +3.5)

4. Chance. 41 to 22 of 63 disagreements is more uneven than a coin usually gives. p = 0.023: if A and C were equally good, a split at least this uneven would happen in about 2 of 100 tests. Chance alone is an unlikely explanation.

5. The claim.

• “Clearly better”: partly supported. C is probably better on tickets like the March test set. But the gain could be as small as 0.4 points, and only two weeks of one month were tested. “Probably a little better” is honest; “clearly” is too strong.
• “A third fewer re-routed tickets”: not supported. Even the observed effect is 21% fewer errors on the test set, not a third. And re-routes in daily work are a causal claim about the live system: agents, ticket mix and season also change them. Only an A/B test on live tickets can measure that.

6. Conclusion for Omar. “On 1,000 March tickets, C was right on 1.9 points more tickets than A (95% interval +0.4 to +3.5). Chance alone is unlikely to explain this. Expect a small gain, not a third fewer re-routes, and confirm it with a 4-week A/B test.”

Module check

Check your understanding

Retrieval: in one or two sentences each, say what a standard error, a 95% confidence interval and a p-value are.

Show the answer

• Standard error: the typical distance between an estimate from one sample and the true value. For an accuracy, √(p(1 − p)/n).
• 95% confidence interval: a range around the estimate. Its method contains the true value in about 95 of 100 samples. It is about the estimate ± 1.96 SE.
• p-value: the share of chance-only results that are at least as extreme as the real result. “Chance-only” means under the null hypothesis: no real difference. It is not the probability that the hypothesis is true, and not the size of the effect.

Check your understanding

Predict: Mei builds a new random test set of 1,200 tickets. A and B disagree on 108 tickets, and B wins 60 of them, the same 5-to-4 split as before (15 to 12). What happens to the difference, its interval and the p-value, compared with the 300-ticket test?

Show the answer

The difference stays the same: (60 − 48) ÷ 1,200 = +1.0 point. The interval becomes about half as wide, because the test set is 4 times larger: about −0.7 to +2.7 points (SE 0.87 points). It still contains zero. The p-value becomes smaller, about 0.29: the same split over more disagreements is less likely by chance. It is still far above 0.05. A 1-point difference needs several thousand test tickets before a test can show it reliably.

Check your understanding

Completion check: a weekly report says: 'Model B cut re-routed tickets by 30% (11.6% to 8.1%) after we switched in week 41, and the offline test confirmed that B is better: 92.0% vs 91.0%.' Separate the observed effect, the uncertainty and the causal interpretation. What can Larkfield honestly say?

Show the answer

Observed effect. Offline: +1.0 point (276 vs 273 of the same 300 tickets). Live: re-routes went from 11.6% in week 40 to 8.1% in week 41. Compared with the weeks before the outage, they went from about 9.2% to about 8.2%.

Uncertainty. The offline difference has a 95% interval of −2.4 to +4.4 points and p = 0.70. It does not “confirm” that B is better. The weekly numbers also move by chance from week to week.

Causal interpretation. The live fall is a correlation with the switch. Nothing shows that the switch caused it. The delivery outage ended in the same week, and the weeks before the outage were already lower. Only an A/B test in the same weeks can show the effect of B.

Honest statement: “B and A performed about the same on our test set (B +1.0 point, interval −2.4 to +4.4). Re-routes fell after the switch, mostly because the outage ended. A 4-week A/B test will measure the effect of B.”

Recap

In this module, you learned to read a test result as evidence, not as a fact:

• Samples and populations: a test set is a sample. A random, representative sample, split by customer, can describe the population. Sampling bias, linked tickets and data leakage make results look better than they are, and more data does not fix them.
• Uncertainty: another sample gives another result. The standard error measures how much; a 95% confidence interval shows the range that the data supports. It describes the method, not one week or one ticket. The bootstrap gives the same range by resampling.
• Comparisons: give the effect size first, in units people understand. For two models on the same tickets, the evidence is in the disagreements. A p-value says if chance can explain the result; it is not the size of the effect. Practical significance is a separate question about costs.
• Cause: a change that happens together with another is a correlation. An A/B test with random assignment shows the cause.

For Larkfield: B’s 1-point gain on 300 tickets does not support “B is better”. The A/B test will decide.

Next, you turn a model’s scores into decisions with a threshold, and see why accuracy alone can mislead.

Next: Classification trade-offs