Read distributions

What you will be able to do

See the whole shape of a dataset before you trust a summary of it. Notice when a chart or an average tells a different story from the data.

NotebookOptional: check the numbers in a notebookColab · Kaggle · GitHub

The notebook is optional: this lesson works with a calculator. It has the week’s 1,200 reply times from this page, so you can count the histogram with any bin width, find the outliers and redo the guided practice and the module practice in Python, with checks. It uses only Python’s standard library, so it needs no install.

Open in ColabOpen in KaggleView on GitHub

Colab and Kaggle open your own copy. They need a free Google or Kaggle account. Stuck? See the solution notebook.

Before you startWhat to know and to have readyAbout 50 minutes

What you need to know

• Monday’s 12 first-reply times, and their mean (28.75 minutes) and median (23 minutes), from Describe a dataset: The mean and the median.
• The quartiles (Q1 18, Q3 28 minutes) and the IQR (10 minutes), from Describe a dataset: Percentiles.
• The range, the variance and the SD (21.10 minutes), from Describe a dataset: Spread.

What you need to have ready (the notebook needs none of it)

• A calculator. A phone calculator is enough.
• Your c03-notes.md file. You made it in the “Before you start” box of Mathematical reading skills. If you cannot find it, make a new empty file with that name: this lesson does not use earlier answers.
• About 15 minutes at the end for the module practice and the module check.

The notebook was tested in Jupyter with Python 3.14.6 (standard library only).

Where you meet this in AI

A distribution is how the values of a dataset are spread: which values are common, which are rare, and where the extremes are. AI teams look at distributions all the time:

• Text length. Before Tomás chooses the longest ticket text that the model reads, he looks at the distribution of ticket lengths. A few very long tickets decide the setting.
• Model scores. A classifier gives each ticket a score from 0 to 1. A histogram of the scores shows if the model separates the classes or is unsure about most tickets. Module 6 uses this.
• Errors. A histogram of the model’s errors shows if most errors are small, with a few very large ones.
• Model comparisons. Dashboards compare two models in a column chart. The axis can make a small difference look large.

One dataset, three views

Here are Monday’s 12 first-reply times from the last lesson, in three views. Each view shows something that the others hide.

View 1: the table. It shows every exact value, and the ticket IDs.

Minutes	12	15	18	18	20	22	24	25	27	31	38	95

View 2: the histogram. A histogram puts the values into bins: ranges of equal width, such as 10 to under 20 minutes. The height of each bar is the number of values in that bin.

Most tickets are in the bins from 10 to 40 minutes. One ticket stands alone in the bin from 90 to 100 minutes.

Text description of the chart

Ten bins of 10 minutes from 0 to 100. The bin from 10 to under 20 minutes has 4 tickets, 20 to under 30 has 5, and 30 to under 40 has 2. The bins from 40 to 90 minutes are empty. The bin from 90 to 100 minutes has 1 ticket (T-2052, 95 minutes). Dashed lines mark the median at 23 minutes and the mean at 28.75 minutes.

View 3: the box plot. A box plot draws the summaries from the last lesson. The box goes from Q1 (18) to Q3 (28), and the line inside it is the median (23). The lines on each side, the whiskers, reach the furthest values that are not outliers. A common rule calls a value an outlier if it is more than 1.5 × IQR beyond the box. Here, the limit is 28 + 1.5 × 10 = 43 minutes. So the box plot shows T-2052 as a separate dot.

The box holds the middle half of the tickets, from 18 to 28 minutes. T-2052 is beyond the outlier limit of 43 minutes.

Text description of the chart

A horizontal box plot on an axis from 0 to 100 minutes. The box goes from 18 to 28 minutes, with the median line at 23. The left whisker reaches 12 minutes and the right whisker reaches 38 minutes. One dot at 95 minutes is an outlier. Under the box, small grey dots show all 12 values.

View	Shows	Hides
Table	Every exact value and its ticket	The shape, when there are more than about 20 values
Histogram	The shape: where values are common, gaps, extremes	Exact values. The picture also depends on the bin width
Box plot	Median, middle half, extremes, outliers. Good to compare groups side by side	Gaps and second peaks inside the box. Two very different datasets can have the same box

A full week: 1,200 tickets

Twelve tickets do not show much shape. Mei exports the first-reply times for all 1,200 tickets of one week. The histogram is the first thing that Priya draws. But how wide must the bins be?

Explore

Change the bin width of the same 1,200 tickets

Bin width

10

Bins

12

Empty bins

0

Tallest bin

414 (10 to under 20)

10-minute bins: a clear shape. After almost empty bins, the outage group shows as two bars at 90 to 110 minutes (21 and 14 tickets).

The data does not change, only the bins. Very narrow bins show noise; very wide bins hide the small group of 36 outage tickets at 90 to 110 minutes.

Text description of the explorer

A histogram of the first-reply times of 1,200 tickets from one week, from 0 to 120 minutes. A slider sets the bin width: 1, 2, 5, 10, 20 or 40 minutes. At the start, the bins are 10 minutes wide. The counts are 36, 414, 411, 194, 76, 17, 11, 4, 1, 21, 14 and 1 tickets. The tallest bins are 10 to 20 minutes (414) and 20 to 30 minutes (411). The counts fall to the right, nearly to zero at 70 to 90 minutes. Then they rise again to 21 and 14 tickets at 90 to 110 minutes: the outage group. Dashed lines mark the median at 22.5 minutes and the mean at 26.7 minutes. With 40-minute bins, the counts are 1,055, 108 and 37. Then you cannot see the outage group. With 1-minute bins, 37 of the 120 bins are empty and the bars are noisy.

There is no one correct bin width. Try a few, and choose the one that shows the shape without noise. When you show a histogram to a team, give the bin width in the caption, because a different width can tell a different story.

Skew: which side has the long tail

The week’s histogram is not symmetric. Most tickets are between 10 and 40 minutes, and a long tail of slower tickets goes far to the right. This shape is right-skewed, or skewed to the right: the tail points to the side of the large values.

In a right-skewed distribution, the mean is larger than the median, because the tail pulls the mean. For the week:

Summary	Value
Median	22.5 minutes
Mean	26.7 minutes
p90	42 minutes
Tickets below the mean	762 of 1,200 (63.5%)

Nearly two out of three customers waited less than the “average” wait. Waiting times, text lengths, prices and model latencies are usually right-skewed: they cannot go below zero, but they can be very large.

A left-skewed distribution has its tail on the side of the small values, and the mean is smaller than the median. For example, a good classifier gives most easy tickets a score close to 1 for the correct team. Hard tickets make a tail of lower scores.

Outliers: find the cause before you decide

An outlier is a flag, not a verdict. The 1.5 × IQR rule only says “look at this value”. For the week, Q1 is 17 and Q3 is 31 minutes, so the limit is 31 + 1.5 × 14 = 52 minutes. The rule flags 61 tickets.

In a right-skewed dataset, the 1.5 × IQR rule flags many ordinary slow tickets, not only the outage.

Text description of the chart

A horizontal box plot of 1,200 first-reply times on an axis from 0 to 120 minutes. The box goes from 17 to 31 minutes, with the median line at 22.5. The left whisker reaches 5 minutes and the right whisker reaches 52 minutes. Beyond 52 minutes, the chart shows 61 tickets as outlier dots. They form a scattered line from about 53 to 86 minutes and a dense group from 90 to 110 minutes.

Of the 61 flagged tickets, 36 are from the outage. The other 25 are ordinary slow tickets in the long tail. The rule cannot tell the two kinds apart; only a person who checks the cause can. Mei checks the log and finds the outage. Grace knows that some slow tickets are complex warranty cases.

Notice also what the box plot hides: it does not show the gap at 70 to 90 minutes. The histogram with 10-minute bins does.

Misleading axes

Tomás shows Omar a column chart of two classifiers on the same 300 test tickets. Model A is correct on 273 (91.0%), and model B on 276 (92.0%). His chart is the first one below. The second chart shows the same two numbers on an axis that starts at zero.

The same two numbers on two axes. In the first chart, the column for B is three times as tall as A's. In the second, the columns are almost equal, which matches a difference of 3 tickets in 300.

Text description of the chart

Two column charts of the same data: model A at 91.0% and model B at 92.0%. In the first chart, the axis starts at 90.5% and ends at 92%. The column for A is short and the column for B is three times as tall. In the second chart, the axis starts at 0% and ends at 100%, and the two columns are almost the same height.

In the first chart, the columns show only the part above 90.5%: 0.5 points for A and 1.5 points for B. So B looks three times as good. The real difference is 1 percentage point, which is 3 tickets out of 300.

A column chart compares lengths, so its axis must start at zero. If the difference is the point, show the difference itself, with its count: “+1 point, 3 tickets of 300”. Module 5 checks whether 3 tickets is more than chance. A line chart may start above zero. A log axis is honest when its label says so. You saw both in the first lesson.

Misleading averages

An average is misleading when the reader thinks that it describes the typical case, and it does not. You have seen two ways:

• Skew and outliers. The week’s mean of 26.7 minutes is above the wait of 63.5% of customers.
• Spread. Two groups with the same mean can be very different. One team answers everybody in about 25 minutes; another answers some in 8 minutes and some in 42. The module practice below shows this.

Before you report a mean, look at a histogram or a dot plot. Then report the median and a measure of spread with it. Or report a count that matters to the reader, such as “tickets that waited more than 40 minutes”.

Worked example: Grace’s weekly report

Grace’s weekly report has one line: “Mean first reply: 26.7 minutes.” Priya draws the histogram with 10-minute bins, then adds the values that the mean hides:

Value	Result	How Priya got it
Tickets	1,200	count
Median	22.5 minutes	half of the tickets at or below
p90	42 minutes	90% of the tickets at or below
Waited more than 40 minutes	132 tickets (11%)	count from the data
Outage group	36 tickets, 90 to 110 minutes	the bars at 90 to 110 minutes; Mei confirmed the cause
Mean	26.7 minutes	sum ÷ 1,200

Then she writes the new report line:

“Half of the customers got a first reply within 22.5 minutes, and 9 in 10 within 42 minutes. 132 tickets (11%) waited more than 40 minutes; 36 of them were held by Monday’s outage.”

Grace now knows two things that the mean did not show. Most customers wait less than 26.7 minutes. And the worst waits have one cause, which she can report to the IT team.

Guided practice: read a histogram from its counts

Use the counts of the week’s histogram with 10-minute bins:

Bin (minutes)	0–10	10–20	20–30	30–40	40–50	50–60	60–70	70–80	80–90	90–100	100–110	110–120
Tickets	36	414	411	194	76	17	11	4	1	21	14	1

Each bin includes its left edge and not its right edge: “10–20” means 10 to under 20 minutes.

1. In c03-notes.md, add the heading ## Read distributions.
2. Add up the counts. Check that the total is 1,200.
3. Count the tickets that waited under 40 minutes. Write it as a percentage of 1,200.
4. Count the tickets that waited 60 minutes or more.
5. Find the bin that contains the median. Add the counts from the left until the total passes 600.
6. Write the direction of the skew, and one sentence about the bins at 90 to 120 minutes.

Check: after step 5, the bin that you found agrees with the median of 22.5 minutes.

Show the example answer

2. 36 + 414 + 411 + 194 + 76 + 17 + 11 + 4 + 1 + 21 + 14 + 1 = 1,200.
3. 36 + 414 + 411 + 194 = 1,055 tickets, which is 1,055 ÷ 1,200 = 87.9%.
4. 11 + 4 + 1 + 21 + 14 + 1 = 52 tickets.
5. Running total: 36, then 450, then 861. It passes 600 in the 20–30 bin, which agrees with 22.5 minutes.
6. Right-skewed: the long tail is on the side of the large values. The bins at 90 to 120 minutes rise again after almost empty bins. So they are a separate group (the outage), not part of the normal tail.

Independent variation: without the outage

Mei removes the 36 outage tickets from the week, because the outage is now fixed. Change one thing: the data. In your notes, predict before you calculate:

1. Which bars of the 10-minute histogram change, and how?
2. Does the mean go up or down? Does it change more or less than the median?
3. Is the distribution still right-skewed?

Then check your predictions with these results for the 1,164 tickets that remain: mean 24.45 minutes, median 22 minutes.

Show the example answer

1. The bars from 90 to 120 minutes disappear: all 36 tickets in them were outage tickets. All other bars stay the same, because Mei removed only the outage tickets.
2. The mean falls by 2.25 minutes, from 26.7 to 24.45. The median falls by only 0.5 minutes, from 22.5 to 22. The removed tickets were all far above the median, so they move the mean much more.
3. Yes. The long tail of ordinary slow tickets, up to 86 minutes, is still there. The outage made the skew stronger, but it did not cause it.

Mei must record that she removed the tickets, and why. If the team trains a model on this data, it never sees tickets from an outage. That can be correct, or it can be a problem; see the common mistake below.

Common mistake

Deleting outliers without asking why

Symptom: Tomás trains the urgent-ticket model on data where somebody deleted every value beyond 1.5 × IQR, because “outliers are noise”. On the test set the model is fine. After release, it misses many urgent tickets during busy periods.

Cause: the deleted rows were not noise. Many were long, complex tickets from busy periods, and many of those were urgent. The model never learned from them, and the test set had the same rows removed, so the test could not show the problem.

Fix: for each group of outliers, find the cause before you decide. Remove a value only if it is wrong (for example, a broken clock) or out of scope, and write down the rule. Keep rare but real cases, and check the model on them separately. Apply the same rule to the training data and the test data, and report how many rows it removed.

Check your understanding

A histogram of ticket lengths has a long tail to the right. Which is larger, the mean length or the median length? Why does this matter when Tomás chooses the longest ticket that the model reads?

Show the answer

The mean is larger than the median, because the tail of long tickets pulls the mean up. For the setting, neither the mean nor the median is the right number: the question is how many tickets would be cut off. Tomás needs a high percentile, for example p99 of the length: then only about 1 ticket in 100 is longer than the setting.

Check your understanding

Two box plots of reply times look the same. Can the two datasets still be very different? Give an example.

Show the answer

Yes. A box plot shows only the five summaries and the outliers. One team can have waits that spread evenly from 15 to 35 minutes. Another team can have two groups: one at about 15 to 20 minutes and one at about 30 to 35, with nothing in between. Both can have the same Q1, median and Q3. A histogram or a dot plot shows the difference.

Module practice

The module practice compares two groups with the same mean and a different spread. First, a small worked example.

Worked example: two groups of three

Two agents answer three tickets each. Agent X answers in 9, 10 and 11 minutes. Agent Y answers in 0, 10 and 20 minutes. Both means are 30 ÷ 3 = 10 minutes. Priya calculates the spread step by step:

Agent	Values	Distances from 10	Squared distances	Sum	Variance (÷ 3)	SD	Range
X	9, 10, 11	−1, 0, 1	1, 0, 1	2	0.67	0.82 minutes	2 minutes
Y	0, 10, 20	−10, 0, 10	100, 0, 100	200	66.67	8.16 minutes	20 minutes

The mean says that the agents are equal. The SD says that Y’s waits are 10 times as spread out. For a customer, agent X is predictable; with agent Y, some customers wait twice as long as the mean.

Your task: returns and delivery

Grace compares eight tickets from each of two teams, in minutes to first reply:

• Returns
• Delivery

Both teams have a mean of 25 minutes. The returns team is close to 25 every time; the delivery team is not.

Text description of the chart

Two rows of eight dots on an axis from 0 to 50 minutes. Returns: 22, 23, 24, 25, 25, 26, 27 and 28 minutes, all close together. Delivery: 8, 12, 18, 25, 25, 32, 38 and 42 minutes, spread across most of the axis. Both rows have a mean mark at 25 minutes.

Team	Minutes
Returns	22, 23, 24, 25, 25, 26, 27, 28
Delivery	8, 12, 18, 25, 25, 32, 38, 42

1. In c03-notes.md, add the heading ## Module practice: Data summaries.
2. Calculate the mean and the median of each team.
3. Make a table of distances from the mean and squared distances for each team, like Priya’s.
4. Calculate the variance (divide by 8) and the SD of each team.
5. Calculate the range of each team.
6. Larkfield promises a first reply within 40 minutes. Count the tickets over 40 minutes in each team.
7. Write two or three sentences for Grace: are the teams equal? What does the mean hide?

Show the model answer

2. Returns: sum 200, mean 25, median (25 + 25) ÷ 2 = 25. Delivery: sum 200, mean 25, median 25.
3. Returns distances: −3, −2, −1, 0, 0, 1, 2, 3. Squares: 9, 4, 1, 0, 0, 1, 4, 9, sum 28. Delivery distances: −17, −13, −7, 0, 0, 7, 13, 17. Squares: 289, 169, 49, 0, 0, 49, 169, 289, sum 1,014.
4. Returns: variance 28 ÷ 8 = 3.5, SD 1.87 minutes. Delivery: variance 1,014 ÷ 8 = 126.75, SD 11.26 minutes.
5. Returns: 28 − 22 = 6 minutes. Delivery: 42 − 8 = 34 minutes.
6. Returns: 0 tickets. Delivery: 1 ticket (42 minutes).
7. “Both teams have a mean and a median of 25 minutes, so a report of the mean shows them as equal. They are not: the delivery team’s SD is about 6 times as large, and one of its eight tickets broke the 40-minute promise. Returns customers can expect about 25 minutes; delivery customers can wait anything from 8 to 42 minutes. The next question is why: the type of delivery problem, the agent, or the time of day.”

A model card has the same problem. Two models can have the same mean accuracy over five test runs and still differ a lot from run to run. Report the spread with the mean.

Module check

Check your understanding

Retrieval: name one summary of the centre and one summary of spread that do not change when the largest value gets much larger. Explain why.

Show the answer

The median and the IQR. Both depend only on values in the middle of the sorted list. If the largest value gets larger, it stays the largest, so the middle values do not change. The mean, the SD and the range all change.

Check your understanding

Predict: Larkfield adds 10 more outage tickets that waited about 100 minutes to the week. What happens to the mean, the median, p90 and the 10-minute histogram?

Show the answer

The mean goes up clearly: 10 tickets far above the mean add about 10 × (100 − 26.7) ÷ 1,210 ≈ 0.6 minutes. The median and p90 change very little. The 10 new values are above them, so each moves only a few places in the sorted list of 1,210 values. In the histogram, the bars at 90 to 110 minutes grow by 10 tickets in total. So the outage group is easier to see. The other bars do not change.

Check your understanding

Completion check: Omar asks for one number that describes how long customers wait for a first reply. Choose a summary for the week's 1,200 tickets, give its value, and explain what it hides. Then add one more value that shows what it hides.

Show the answer

A good choice is the median, 22.5 minutes: half of the customers waited this long or less, and the outage does not distort it. What it hides: the slow tail and the outage. The median does not change if the slowest tickets wait 2 hours or 2 days, so it cannot show customers who waited far too long. The added value: p90 (42 minutes), or the count over the limit. For example: “132 of 1,200 tickets (11%) waited more than 40 minutes. 36 of them were from the outage.”

You can also choose the mean, 26.7 minutes. Then explain that it is above the wait of 63.5% of customers, because the skew and the outage pull it up. It also does not show how many customers waited too long. Any summary is acceptable if you name correctly what it hides and add a value that shows it.

Recap

In this module you learned to read and summarise numbers:

• Reading: a percentage needs its count, and percentage points are not percent. 1e-3 is 0.001, and a log axis counts factors of 10. Σ means “add up”. Every number needs a unit.
• Summaries: the median and the IQR resist outliers; the mean, the SD and the range do not. Percentiles such as p90 and p99 describe the slow end, which matters for waits and for the speed of AI services.
• Shapes: a histogram shows the shape, and its bin width changes what you see. A box plot compares groups but hides gaps. Right-skewed data has a mean above the median.
• Honest charts and averages: column charts start at zero. Report a typical value with a spread, find the cause of an outlier before you remove it, and say what your summary hides.

Next, in Module 2, you see how an AI system stores a ticket as a list of numbers, a vector.

Next: Represent observations