Samples and populations

What you will be able to do

Find out what a test result can tell you about the tickets that a model will see later. A result from the wrong tickets can look excellent and still fail in daily work.

NotebookOptional: check the numbers in a notebookColab · Kaggle · GitHub

The notebook is optional: this lesson needs no code and no calculator. It has the 6,000 reply times of this page’s explorer. You draw random and rated-only samples, see that more tickets do not remove the bias, and split tickets by customer, with checks. It uses only Python’s standard library, so it needs no install.

Open in ColabOpen in KaggleView on GitHub

Colab and Kaggle open your own copy. They need a free Google or Kaggle account. Stuck? See the solution notebook.

Before you startWhat to know and to have readyAbout 40 minutes

What you need to know

• The mean, from Describe a dataset: The mean and the median, and the standard deviation (SD), from Describe a dataset: Spread.
• How to read a histogram, from Read distributions: One dataset, three views.
• What independent events are, from Events and uncertainty: Independence. This lesson asks if tickets are independent.

What you need to have ready (the notebook needs none of it)

• Your c03-notes.md file. You made it in the “Before you start” box of Mathematical reading skills. Add a heading: ## Sampling and statistical evidence. If you cannot find the file, make a new empty file with that name: this module does not use earlier answers.
• No calculator is necessary for this lesson.

The notebook was tested in Jupyter with Python 3.14.6 (standard library only).

Larkfield so far, and the question of this module

Larkfield is an online shop for home and garden products. Customer messages become tickets, about 1,200 a week. Larkfield is building AI that suggests a team and a priority for each new ticket.

Tomás, the AI engineer, trained a new ticket classifier, model B. He tested it on 300 tickets that already have the correct team:

Model	Correct	Of	Accuracy
A (in use today)	273	300	91.0%
B (new)	276	300	92.0%

“B is better, let us use it,” says Tomás. Before Omar, the product manager, decides, Priya, the data scientist, asks three questions. This module answers them, one per lesson:

1. Which tickets are these 300? Do they look like the tickets that the model will see? (This lesson.)
2. How much can the result change by chance? Another 300 tickets give a different accuracy. (Lesson 2.)
3. Is the difference real, and does it matter? One percentage point is only 3 tickets. (Lesson 3.)

Where you meet this in AI

• A test set. A team reports “95% accuracy”. The number describes the test tickets only. If the test tickets are easier than real tickets, the real accuracy is lower.
• A public benchmark. A model scores well on a set of questions. Some of the questions were also on the internet before training, so the model may have seen the answers.
• User feedback. A chat assistant shows “87% of users liked the answer”. The number counts only the users who clicked a button. Unhappy users often leave without a click.
• Training data. A model learns from the data that was easy to collect, for example messages in one language. It then works less well for the customers who are missing from that data.

Population and sample

A population is the whole group that you want to know about. For Larkfield’s classifier, the population is every ticket that the model will sort in the coming months.

You cannot test on tickets that do not exist yet. So you take a sample: a smaller group that you can measure. The 300 test tickets are a sample.

The accuracy on the sample (92.0% for B) is an estimate. The value that you want is the accuracy on the population, the true value. Nobody knows it exactly. A good sample gives an estimate close to the true value. A bad sample can give an estimate far from it, and more tickets do not help.

A sample is representative when it contains the same kinds of tickets as the population, in about the same shares. The simplest way to get one is a random sample: every ticket in the population has the same chance to be chosen. Then no person and no process decides which tickets get in.

See it: random samples and a biased sample

The explorer below uses a population that you can see completely: 6,000 past tickets (about five weeks), with the minutes until the first reply. The page generates this made-up data. Because you have all 6,000, you know the true mean: 26.8 minutes.

Each sample takes 30 tickets. The explorer shows the mean of each sample. Then switch to Only rated tickets. After each reply, a customer can rate it. Customers rate more often when the reply came quickly.

Explore

Sample means from random tickets and from rated tickets

Samples drawn

20

Latest sample mean

22.2 min

Mean of all sample means

27.4 min

True mean (all 6,000)

26.8 min

SD of the sample means

3.8 min

The sample means scatter around 27.4 minutes. The true mean of all 6,000 tickets is 26.8.

• All 6,000 tickets
• Tickets that customers rated
• Latest sample (one mark per ticket)

Each row is one sample of 30 tickets; the dot is its mean. The vertical line is the true mean of all 6,000 tickets. Made-up data, the same on every visit.

Text description of the explorer

The top chart is a histogram of the first-reply times of 6,000 made-up tickets. Most replies come after 10 to 40 minutes, and a few take more than 100 minutes. The true mean is 26.8 minutes. Rated tickets (2,155 of them, shown as narrower bars) have more short times; their mean is 20.9 minutes.

The bottom chart shows one row per sample of 30 tickets, with a dot at the sample mean. With any ticket allowed, the means scatter around 26.8 minutes, most between about 20 and 33. With only rated tickets, the means scatter around 20.9 minutes. Drawing more samples does not move them closer to 26.8, because every sample comes from the same biased group.

The readout shows the number of samples and the latest sample mean. It also shows the mean of all sample means, the true mean and the SD of the sample means.

Two things happen in the explorer:

• Random samples scatter around the true value. No single sample is exact. Some are too high, some too low, but on average they are right. Lesson 2 measures this scatter.
• Rated-only samples scatter around the wrong value. Their mean is about 21 minutes. A report from them says “customers wait 21 minutes” when the true mean is 27. This error is in every sample, so more samples or larger samples cannot remove it.

Sampling bias is a difference between the sample and the population that comes from the way the sample was chosen. It moves the estimate in one direction, every time. Selection bias is the same idea for any data that a process selected, for example the tickets that customers chose to rate.

Three biased test sets

Mei, the data engineer, knows where Larkfield’s data comes from. She lists test sets that look easy to build and give a wrong picture:

Test set	Who is missing	What happens to the accuracy
All tickets from one week in December	Tickets from other months. In December, 45% of tickets are about delivery; over the year, 28% are.	It describes December. Teams that are rare in December are hardly tested.
Only tickets in English	About 1 ticket in 10 is in another language.	Usually too high: the hard tickets are missing.
Only tickets that customers rated	Customers who did not answer. They are often the unhappy ones.	Usually too high: rated tickets are often simple.

To show the effect, Mei rebuilds the test set in each wrong way, on purpose, from the same 12 months. Priya tests model B on each one:

• Random sample (the fair test)
• Built the wrong way

The same model B, tested on five test sets of 300 tickets that were built in different ways. Made-up results for illustration.

Text description of the chart

A dot chart with five rows. Model B has 92.0% accuracy on a random sample from the whole year. On tickets from one week in December, it has 89.3%. On English tickets only, 93.7%. On rated tickets only, 95.0%. On a test set that contains copies of training tickets, 97.3%. Four of the five numbers are wrong descriptions of the same model; only the random sample describes the whole year.

The model is the same in every row. Only the choice of tickets changed, and the result moved from 89.3% to 97.3%. That range is eight times larger than the difference between model A and model B.

Tickets that are not independent

A random sample also assumes that the tickets are independent: one ticket tells you nothing about the next one. Real tickets are often linked:

• Repeated measurements. One customer writes five times about the same order. That is five tickets, but one problem.
• One event, many tickets. A delivery outage on one day creates 60 tickets that all say “where is my order?”. If the model sorts one of them correctly, it sorts all 60 correctly.

Linked tickets carry less information than their number suggests. 300 tickets that include 60 copies of one problem are closer to 241 different problems (240 others plus the one problem). A result from them is less certain than “300 tickets” suggests.

Data leakage is the AI version of this problem. It happens when information from the test set is also in the training data. Example: a customer sends the same message twice. One copy goes to training and one to the test set. The model has seen the answer, so the test result is too high. Look at the last row of the chart above: 97.3%.

Worked example: Mei and Priya check the 300 test tickets

Before anybody compares model A and model B, Priya checks how Mei built the 300 test tickets. She asks four questions, and Mei answers from her records.

Question	Mei’s answer	Is it a problem?
1. What is the population?	The tickets that the model will sort next: all teams, all languages, all months.	—
2. How were the tickets chosen?	At random, from all tickets of the last 12 months, with a fixed seed.	No
3. Who is missing?	Tickets without a final team (about 2%), because there is no correct answer to compare.	Small; write it down
4. Are any tickets linked?	Mei split by customer: all tickets of one customer are in training or in test, never in both. She also removed exact copies.	No

Priya writes the result in one sentence: “The 300 tickets are a random sample of the last 12 months, split by customer. The accuracy describes tickets like those, except tickets that never got a final team.”

This answers question 1 of the module for Larkfield’s test set. Lessons 2 and 3 use the same 300 tickets.

Guided practice: check a test set

Before Mei joined, Tomás built a first test set. Check it with the four questions.

Test set v1: 500 tickets from 2 to 8 December. Only tickets with a customer rating. 120 of the tickets come from the delivery outage on 4 December. Tomás split training and test ticket by ticket, so some customers have tickets in both. Model A had 96% accuracy on it.

1. In c03-notes.md, add a heading: ### Check test set v1.
2. Write the population that the test should describe.
3. Find each problem in the description. Write one line for each.
4. For each problem, write the likely direction of the error: too high, too low, or unknown.
5. Write one fix for each problem.

Expected result: at least four problems. At least two of them make the 96% too high.

Show an example answer

Population: the tickets that the model will sort next, in all months, teams and languages.

Problem	Direction	Fix
One week in December	Unknown: the ticket mix is different (many delivery tickets)	Take a random sample from 12 months
Only rated tickets	Too high: rated tickets are often simple	Take tickets with and without a rating
120 tickets from one outage	Unknown, but the result is less certain than “500 tickets” suggests	Keep at most a few tickets from one event, or count the event once
Split ticket by ticket	Too high: data leakage from the same customers	Split by customer

The 96% is not a fair comparison with model B’s 92% on Mei’s test set. The two numbers describe different tickets.

Independent variation: user feedback

Change one condition: the data is now feedback, not a test set.

Larkfield’s help page has a chat assistant. Under each answer, there is a button “Was this helpful? Yes / No”. The weekly report says: “87% of users found the answers helpful.” Only 4 of 100 users click a button.

In c03-notes.md, write:

1. The population and the sample.
2. The likely bias, and its direction.
3. One better way to measure helpfulness.

Show an example answer

1. Population: all users who get an answer from the assistant. Sample: the 4% who click a button.
2. Selection bias. The users decide themselves if they answer. Users who did not get help often close the page without a click. So 87% is probably too high. The direction could also be the opposite (angry users click “No” more often), and you cannot tell from the report. That is the problem: the result depends on who chose to answer.
3. Take a random sample of conversations each week, for example 100. Grace’s team reads them and marks each answer as helpful or not. Also measure what users do next: did they still open a ticket about the same question?

Common mistake

The test result was 97%, the first live week was 88%

Symptom: a new classifier scores 97% on its test set. In the first week of real use, the support team checks a sample of tickets and finds 88% correct. Nobody changed the model.

Cause: data leakage. The data was split ticket by ticket. Many customers write more than once about the same problem, with almost the same words. One message went to training and another to the test set. The model recognised the test tickets instead of solving them. The test set also had no tickets from new customers, who are most of the live tickets.

Fix: split by customer, so that all tickets of one customer are on one side. Remove copies and near-copies across the split. Where possible, use a time-based split: train on older tickets and test on the newest weeks. That is the situation of a live model, which always sorts tickets from the future.

Check your understanding

Mei could test model B on 3,000 rated tickets instead of 300 random tickets. Is that better? Explain why or why not.

Show the answer

No. More tickets make the result less dependent on chance, but they do not remove sampling bias. Rated tickets are often simpler, so the result would be too high on every sample. You saw this in the explorer: rated-only sample means gather around 21 minutes, not 27, however many samples you draw. 300 random tickets describe all tickets better than 3,000 rated tickets.

Check your understanding

A test set has 300 tickets, but 90 of them come from one payment outage and are almost the same. Predict two effects on the test result.

Show the answer

First, the result depends a lot on one problem. If the model handles the outage tickets well, accuracy is high; if not, it is low. The other 210 tickets have less weight than they should. Second, the result is less certain than “300 tickets” suggests, because the 90 tickets give about the information of one ticket. The test describes outage weeks more than normal weeks.

Recap

• A population is the group that you want to know about. A sample is the part that you measure. A test result is an estimate for the population.
• A random sample is the simplest way to get a representative sample.
• Sampling bias moves every estimate in the same direction. More data does not fix it; a better way to choose the data does.
• Linked tickets (one customer, one outage) carry less information than their number. Data leakage puts test information into training and makes results too high.
• Before you compare two results, check that they come from the same, fair test set.

Next, you measure how much a result from a fair sample can still change by chance.

Next: Estimate uncertainty