Estimate uncertainty

What you will be able to do

Report a result together with how much it can change by chance, so that nobody reads “92%” as an exact number.

NotebookOptional: check the numbers in a notebookColab · Kaggle · GitHub

The notebook is optional: this lesson works with a calculator. It has the 6,000 reply times of this page’s explorer, so you can count how many 95% intervals contain the true mean. Then you redo the worked examples, the bootstrap and the guided practice in Python, with checks. It uses only Python’s standard library, so it needs no install.

Open in ColabOpen in KaggleView on GitHub

Colab and Kaggle open your own copy. They need a free Google or Kaggle account. Stuck? See the solution notebook.

Before you startWhat to know and to have readyAbout 45 minutes

What you need to know

• A test set is a sample. Random samples scatter around the true value, from Samples and populations: See it: random samples and a biased sample.
• The standard deviation (SD), the square root of the variance, from Describe a dataset: Spread. Worked example 1 uses Monday’s 12 reply times from Describe a dataset: Monday’s 12 tickets.
• The normal curve, from Distributions in practice: Normal: the bell-shaped curve. This lesson uses its 95% rule.

What you need to have ready (the notebook needs none of it)

• A calculator with a square root key. A phone calculator is enough.
• Your c03-notes.md file, with the heading ## Sampling and statistical evidence from the “Before you start” box of Samples and populations. If you cannot find the file, make a new empty file with that name: this lesson does not use earlier answers.

The notebook was tested in Jupyter with Python 3.14.6 (standard library only).

Where you meet this in AI

• A leaderboard. Model P scores 81.2% on a benchmark and model Q scores 80.4%. The table ranks P first. With 500 questions, the two results can easily change places on another set of questions.
• A model report. “Accuracy 92% ± 3%” or “92% (95% CI 89–95)”. The second part tells you how far the true value can be from 92%.
• Error bars in a chart. A short line through each point shows the range that the data supports. Two bars that overlap a lot are a warning against a strong claim.
• A weekly dashboard. The accuracy of the ticket classifier was 91.4% last week and 92.6% this week. Did the model improve, or did chance change the tickets?

The idea: one test set is one sample

Model B is correct on 276 of Larkfield’s 300 test tickets: 92.0%. In the last lesson, Mei confirmed that the 300 tickets are a fair random sample. But they are still one sample. Another random 300 tickets from the same 12 months would give a slightly different accuracy: perhaps 90.7%, perhaps 93.3%.

Nobody can test on every future ticket. So the question is: how far from the true accuracy can a sample of 300 tickets be? You can see the answer when you know the population, as in the explorer below.

The population is the same 6,000 made-up tickets as in Lesson 1, with the minutes to the first reply. The true mean is 26.8 minutes. Each sample now also gets a 95% interval: a range around the sample mean, calculated only from the sample. The next sections explain how.

Explore

How often does a 95% interval contain the true mean?

Samples drawn

20

Intervals that contain the true mean

19 of 20 (95%)

Latest sample mean

22.2 min

Latest standard error

2.6 min

Latest 95% interval

16.8 to 27.5 min

19 of 20 intervals (95%) contain the true mean. With more samples, the share settles at about 93 to 95%: the long tail keeps it a little below 95% unless the samples are large.

• All 6,000 tickets
• Latest sample (one mark per ticket)

Each row is one sample: the dot is the sample mean, the line is its 95% interval. Dashed orange intervals miss the true mean. Made-up data, the same on every visit.

Text description of the explorer

The top chart is a histogram of the first-reply times of 6,000 made-up tickets. Most replies come after 10 to 40 minutes, and some take more than 100 minutes. The true mean is 26.8 minutes and the SD is 17.8 minutes. Orange marks along the bottom show the tickets of the latest sample.

The bottom chart has one row per sample, newest at the top. Each row shows the sample mean as a dot and the 95% interval as a line. Intervals that contain the true mean are blue; intervals that miss it are orange and dashed.

With 30 tickets per sample, about 93 to 95 of 100 intervals contain the true mean, and each interval is about 13 minutes wide. With 200 tickets, the intervals are about 5 minutes wide, and still about 95 of 100 contain the true mean. With only 5 tickets, the intervals are very wide, often wider than the chart. Only about 90 of 100 contain the true mean, because the data has a long tail.

The readout shows the number of samples and how many intervals contain the true mean. For the latest sample, it shows the mean, the standard error and the 95% interval.

What the explorer shows:

• Each sample mean is wrong by a different amount. Most are close to 26.8 minutes; a few are far.
• About 95 of 100 intervals contain the true mean. You can count it: the readout keeps the score. About 5 of 100 miss it, and nothing in the sample tells you which ones.
• Larger samples give narrower intervals. But the width does not shrink as fast as the sample grows: 4 times more tickets give an interval half as wide.

Standard error: the typical size of the chance error

The standard error (SE) is the typical distance between an estimate from one sample and the true value. In the explorer, it is the SD of the sample means: how much the dots scatter.

You can calculate it from one sample. For a mean:

$$\text{SE} = \frac{\text{SD}}{\sqrt{n}}$$

Read it as: “the standard error is the SD of the values, divided by the square root of the number of values”. The SD tells you how much single tickets vary. The square root of n tells you how much averaging calms that down.

With the population SD of 17.8 minutes:

Tickets in the sample (n)	√n	SE = 17.8 ÷ √n (minutes)
5	2.24	7.9
30	5.48	3.2
50	7.07	2.5
200	14.14	1.3

From 50 to 200 tickets, n grows 4 times, √n grows 2 times, and the SE halves: 2.5 to 1.3 minutes. This is the square-root rule. It explains why doubling a test set helps less than people expect.

Confidence interval: a range with a known success rate

A confidence interval is a range around an estimate, calculated with a method that contains the true value in a known share of samples. For a 95% confidence interval, that share is 95 of 100.

For a mean or an accuracy from a few hundred values or more, the rule is short:

$$\text{estimate} \pm 1.96 \times \text{SE}$$

Read it as: “the estimate, plus or minus about two standard errors”. The 1.96 comes from the normal distribution in Distributions in practice: 95% of a normal distribution is within 1.96 SD of its centre. For small samples, software uses a slightly larger multiplier, for example 2.20 for 12 values.

The 95% describes the method, not one interval. You saw this in the explorer: the method succeeds about 95 times in 100. Your one interval either contains the true value or not, and you cannot see which.

Worked example 1: the mean of 12 reply times

Priya starts with the 12 first-reply times from Module 1, in minutes: 12, 15, 18, 18, 20, 22, 24, 25, 27, 31, 38, 95. Grace wants to know the mean reply time on Monday mornings.

Step	Value
Number of tickets, n	12
Mean	345 ÷ 12 = 28.75 minutes
SD of the sample (software divides by n − 1)	22.0 minutes
√n	√12 = 3.46
SE = SD ÷ √n	22.0 ÷ 3.46 = 6.4 minutes
Multiplier for 12 values	2.20
Half-width = 2.20 × SE	2.20 × 6.4 = 14.0 minutes
95% interval	28.75 − 14.0 to 28.75 + 14.0 = 14.7 to 42.8 minutes

Why n − 1 here? In Module 1, you described these 12 tickets only, so you divided by n and got 21.1 minutes. Here, the 12 tickets are a sample, and Priya estimates the spread of all Monday tickets. The values in a sample are a little closer to their own mean than to the true mean. Dividing by n − 1 makes the SD slightly larger, to correct for this. With 12 values, the difference is 21.1 or 22.0 minutes. With hundreds of values, it almost disappears.

Priya tells Grace: “On these Monday mornings, the mean reply time is about 29 minutes. But 12 tickets cannot tell 15 minutes from 43 minutes. The one ticket from the outage, 95 minutes, makes the SD large. Collect a few weeks of Mondays before you change the staff plan.”

Worked example 2: the accuracy of model B

An accuracy is the mean of many 1s (correct) and 0s (wrong). For a share like this, the SE has its own short formula. It needs only the accuracy p and the number of tickets n:

$$\text{SE} = \sqrt{\frac{p\,(1-p)}{n}}$$

Read it as: “the square root of p times one minus p, divided by n”.

Step	Value
Correct tickets ÷ all tickets	276 ÷ 300 = 0.92
p × (1 − p)	0.92 × 0.08 = 0.0736
÷ n	0.0736 ÷ 300 = 0.000245
SE = √ of that	0.0157, so 1.57 percentage points
Half-width = 1.96 × SE	1.96 × 1.57 = 3.07 points
95% interval	92.0 − 3.1 to 92.0 + 3.1 = 88.9% to 95.1%

Priya writes in the report: “Model B: accuracy 92.0% on 300 random test tickets (95% confidence interval 88.9% to 95.1%).” Short form: 92% ± 3.1 points.

Note the unit. Accuracy is a percentage, and the ± is in percentage points: 92% ± 3.1 points means 88.9% to 95.1%. It does not mean “3.1% of 92%”.

The bootstrap: resample the test set

The formulas work for means and shares. Many AI metrics have no simple formula, for example the median reply time or a metric that combines precision and recall. The bootstrap works for all of them, with no formula.

The idea: you have only one test set, so use it as a model of the population. Build a new test set of 300 tickets: pick tickets from the 300 at random, with replacement. A ticket can come up twice, three times, or not at all. Calculate the accuracy. Repeat this a thousand times. The middle 95% of those accuracies is the bootstrap interval.

Explore

Bootstrap: resample model B's 300 test tickets

Resamples

200

Latest accuracy

92.0%

Middle 95% of resamples

89.0% to 95.0%

Formula interval (for comparison)

88.9% to 95.1%

The middle 95% of 200 resampled accuracies runs from 89.0% to 95.0%. This is the bootstrap interval.

• Correct ticket
• Wrong ticket

Each resample picks 300 tickets from the test set, with replacement, and records the accuracy. The histogram collects the accuracies; the bracket marks the middle 95%.

Text description of the explorer

The top picture shows the 300 test tickets as squares: 276 correct (blue) and 24 wrong (orange). After a resample, an empty square shows a ticket that the resample did not pick. A number in a square shows how many times the resample picked that ticket.

The histogram below shows the accuracy of each resampled test set, from about 87% to 97%. The results gather around 92.0%, the real test result. After a few hundred resamples, the middle 95% runs from about 89% to 95%. The formula interval from worked example 2, 88.9% to 95.1%, is almost the same.

The readout shows the number of resamples, the latest accuracy, the middle 95% of all resamples so far, and the formula interval for comparison.

The bootstrap and the formula agree: about 89% to 95%. That agreement is the point. The bootstrap gets there by simulation, so it also works when you have no formula. In Python, data scientists often use scipy.stats.bootstrap or a short loop with numpy.

Error bars on a leaderboard

Now read the leaderboard from the start of the lesson. It shows four made-up models on a public benchmark of 500 questions. Each row has the accuracy and its 95% interval.

A made-up leaderboard with error bars: the accuracy of four models on 500 questions, with 95% confidence intervals.

Text description of the chart

Four rows, each with a dot and an interval bar. Model P: 81.2% (77.8% to 84.6%). Model Q: 80.4% (76.9% to 83.9%). Model R: 78.6% (75.0% to 82.2%). Model S: 74.0% (70.2% to 77.8%). The intervals of P, Q and R overlap almost completely. The interval of S overlaps the others only a little.

Each interval is about ±3.5 points wide, because 500 questions is a small sample. The order of P, Q and R is not supported by the data: their intervals overlap almost completely. S is probably weaker than P, but the gap is less certain than the ranking suggests. Lesson 3 shows the right way to compare two models directly.

Guided practice: the interval for model A

Model A is correct on 273 of the same 300 tickets.

1. In c03-notes.md, add a heading: ### Interval for model A.
2. Calculate the accuracy p = 273 ÷ 300.
3. Calculate p × (1 − p).
4. Divide the result by 300.
5. Take the square root. This is the SE.
6. Multiply the SE by 1.96. This is the half-width.
7. Write the interval: p minus the half-width to p plus the half-width.
8. Write one sentence for a report, as Priya did.

Expected result: SE about 1.65 points, half-width about 3.2 points, interval about 87.8% to 94.2%.

Show an example answer

Step	Value
p	273 ÷ 300 = 0.91
p × (1 − p)	0.91 × 0.09 = 0.0819
÷ 300	0.000273
SE	√0.000273 = 0.0165 = 1.65 points
Half-width	1.96 × 1.65 = 3.24 points
Interval	91.0 − 3.2 to 91.0 + 3.2 = 87.8% to 94.2%

Report sentence: “Model A: accuracy 91.0% on 300 random test tickets (95% confidence interval 87.8% to 94.2%).”

Compare with model B, 88.9% to 95.1%. The two intervals overlap almost completely. That is a first warning against “B is better”. Lesson 3 does the full comparison.

Independent variation: a larger test set

Change one condition: the number of tickets. Mei can build a random test set of 1,200 tickets (one week’s worth, but sampled from 12 months). Suppose model B is again 92.0% correct on it.

In c03-notes.md:

1. Calculate the SE and the 95% interval.
2. Compare the width with the interval from 300 tickets. Use the square-root rule to explain the difference.
3. Omar wants ±0.75 points. Estimate how many tickets that needs.

Show an example answer

1. p × (1 − p) = 0.0736. ÷ 1,200 = 0.0000613. SE = √0.0000613 = 0.0078 = 0.78 points. Half-width = 1.96 × 0.78 = 1.53 points. Interval: 90.5% to 93.5%.
2. 1,200 is 4 times 300. √4 = 2, so the SE and the interval are half as wide: ±1.5 points instead of ±3.1 points.
3. ±0.75 points is half of ±1.5 points again, so 4 times more tickets: about 4,800 tickets. Each halving of the interval costs four times the data. Omar must decide if that labelling work is worth it.

Optional: check it in Python

You do not need code for this course. If you want, this box repeats worked example 2 and a bootstrap with the Python standard library.

Run Python in your browser

Try it: interval and bootstrap for model B

Run the code. The formula interval is 88.9 to 95.1. The bootstrap gives almost the same range. Then change correct to 273 (model A) and run it again: you get the interval from the guided practice.

Python code (you can change it)import math import random correct = 276 n = 300 p = correct / n se = math.sqrt(p * (1 - p) / n) print("accuracy:", round(100 * p, 1), "%") print("standard error:", round(100 * se, 2), "points") print("95% interval:", round(100 * (p - 1.96 * se), 1), "to", round(100 * (p + 1.96 * se), 1)) # Bootstrap: pick 300 results with replacement, 2,000 times. random.seed(1) results = [1] * correct + [0] * (n - correct) accs = sorted(sum(random.choices(results, k=n)) / n for _ in range(2000)) print("bootstrap middle 95%:", round(100 * accs[50], 1), "to", round(100 * accs[1949], 1))

Common mistake

An interval read as a promise about one week

Symptom: the model report says “accuracy 92% ± 3.1 points (88.9% to 95.1%)”. Omar tells the director of customer service: “Every week, at least 89% of tickets will go to the right team.” In the first week of December, Grace’s team checks a sample and finds 87.6%. The director asks what went wrong with the model.

Cause: the interval was read as certainty about one outcome. It is not. The interval estimates one number: the accuracy on tickets like the 300 test tickets, over the long run. It says nothing certain about one week. One week of 1,200 tickets has its own chance variation. December also has a different ticket mix (45% delivery tickets), so its tickets are not like the test tickets. And a 95% method misses 1 time in 20, without a warning. It also says nothing about one ticket: each ticket is either right or wrong.

Fix: write what the interval covers. For example: “On tickets like our test set, we estimate the accuracy at 92%. Values from 89% to 95% fit the test result.” Do not turn it into a promise for a week or a ticket. For weekly results, check a random sample of tickets each week and track the accuracy. Expect it to move. Check months with a different ticket mix separately.

Check your understanding

A test set of 300 tickets gives an interval of ±3.1 points. Explain why a test set of 1,200 tickets gives about ±1.5 points, not ±0.8 points.

Show the answer

The SE is the SD divided by the square root of n. Four times more tickets make √n two times larger, so the SE and the interval become half as wide: ±1.5 points. The interval would be a quarter as wide (±0.8 points) only if it shrank in proportion to n. It does not, because chance errors partly cancel out when you average, but only in proportion to √n.

Check your understanding

In the explorer, about 5 of 100 intervals miss the true mean. In real work, you have only one test set and one interval. Can you tell if your interval is one of the misses? What does the 95% tell you, then?

Show the answer

No. A missing interval looks like any other interval; nothing in the sample marks it. The 95% describes the method: if you use it on many samples, about 95 of 100 intervals contain the true value. So you can trust the method most of the time, but you cannot be certain about any one interval. This is why one result is never final evidence, and why a second, independent test set is valuable.

Recap

• One test set is one sample. Another sample gives a different result, by chance.
• The standard error is the typical size of that chance error. For a mean, SE = SD ÷ √n. For an accuracy p on n tickets, SE = √(p(1 − p)/n).
• A 95% confidence interval is about the estimate ± 1.96 SE. The method contains the true value in about 95 of 100 samples. It is not a promise about one week or one ticket.
• Four times more data gives an interval half as wide.
• The bootstrap resamples the test set with replacement. Its middle 95% gives an interval for any metric, without a formula.
• Model B: 92.0% (88.9% to 95.1%). Model A: 91.0% (87.8% to 94.2%).

Next, you compare A and B directly, on the same 300 tickets, and decide what the one-point difference supports.

Next: Compare alternatives