Describe a dataset

What you will be able to do

Describe a set of numbers in two or three values that a team can act on. Choose the values that answer the team’s question. One unusual value can make a summary say something false.

NotebookOptional: check the numbers in a notebookColab · Kaggle · GitHub

The notebook is optional: this lesson works with a calculator. It has the lesson’s numbers, the worked example and the tasks in Python, with checks that say if your answer is right without showing it. It uses only Python’s standard library, so it needs no install.

Open in ColabOpen in KaggleView on GitHub

Colab and Kaggle open your own copy. They need a free Google or Kaggle account. Stuck? See the solution notebook.

Before you startWhat to know and to have readyAbout 45 minutes

What you need to know

• How to read the summation sign Σ, and the mean of four values (63 ÷ 4 = 15.75 minutes), from Mathematical reading skills: The summation sign, read aloud.
• How to turn a percentage back into counts, from Mathematical reading skills: Fractions, decimals and percentages. This lesson uses shares such as “82% of the total”.

What you need to have ready (the notebook needs none of it)

• A calculator. A phone calculator is enough.
• Your c03-notes.md file. You made it in the “Before you start” box of Mathematical reading skills. If you cannot find it, make a new empty file with that name: this lesson does not use earlier answers.

The notebook was tested in Jupyter with Python 3.14.6 (standard library only).

Where you meet this in AI

A summary is a few numbers that describe many. AI work uses the same few summaries everywhere:

• Describing training data. Before training, a data scientist prints a summary of each column: count, mean, standard deviation, minimum, percentiles, maximum. In Python, the pandas library prints it with one command, describe(). You build the same table by hand in this lesson.
• The speed of an AI service. A team that runs a model promises a speed, for example “95% of answers in under 300 ms”. That promise uses a percentile, p95, not the mean.
• Monitoring a model. When the mean or the spread of the input changes (for example, tickets suddenly get longer), the model may get worse. Teams watch these summaries every day.
• Preparing features. Many models need each input on a similar scale. A common method subtracts the mean and divides by the standard deviation. Module 2 shows why.

Two summaries of the centre, on three numbers

Three tickets got a first reply after 10, 20 and 30 minutes.

• The mean is the sum divided by the count: (10 + 20 + 30) ÷ 3 = 20 minutes. People also call it the average.
• The median is the middle value after you sort the values: 20 minutes.

Here they agree. Now a fourth ticket arrives, and it waits 100 minutes:

• Mean: (10 + 20 + 30 + 100) ÷ 4 = 160 ÷ 4 = 40 minutes.
• Median: with an even count, there are two middle values, 20 and 30. The median is halfway between them: 25 minutes.

One long wait moved the mean by 20 minutes and the median by only 5. The mean uses the size of every value. The median uses only the order, so a very large value counts as “the largest” and nothing more. A value far from the others is an outlier.

Monday’s 12 tickets

Grace wants to know how long customers wait for the first reply. Priya takes the 12 tickets that arrived on one Monday morning:

Ticket	T-2041	T-2042	T-2043	T-2044	T-2045	T-2046	T-2047	T-2048	T-2049	T-2050	T-2051	T-2052
Minutes	12	15	18	18	20	22	24	25	27	31	38	95

The values are already in order. Mei checks the log: T-2052 arrived during a system outage, so it waited 95 minutes.

Eleven tickets got a reply within 40 minutes. T-2052 waited 95 minutes and pulls the mean to the right of the median.

Text description of the chart

One dot for each of the 12 tickets, on an axis from 0 to 100 minutes. Eleven dots are between 12 and 38 minutes. Two tickets share 18 minutes, so one dot sits on top of the other. One dot, labelled 95, is far to the right. Below the dots, two short marks show the median at 23 minutes and the mean at 28.75 minutes.

The mean and the median

The mean, as a formula: add all the values, then divide by how many there are.

$$\bar{x} = \frac{1}{n}\sum_{i=1}^{n} x_i = \frac{345}{12} = 28.75 \text{ minutes}$$

Read it aloud as “x-bar equals one over n times the sum of x-i”. The symbol x̄ (x-bar) is the usual name for a mean.

The median: with 12 values, the middle is between the 6th and the 7th value, 22 and 24. So the median is (22 + 24) ÷ 2 = 23 minutes. Half of the tickets waited 23 minutes or less.

Only 3 of the 12 tickets waited longer than the mean of 28.75 minutes. The mean is not a typical wait here: T-2052 pulls it up. Use the explorer to see how much.

Explore

Move one ticket and watch the mean and the median

T-2052 (minutes)

95

Mean (minutes)

28.75

Median (minutes)

23

SD (minutes)

21.10

Mean − median (minutes)

5.75

T-2052 is the largest value. Each extra minute adds 1/12 = 0.083 to the mean, but the median stays at 23: the middle values did not change.

Each extra minute for T-2052 adds 1/12 of a minute to the mean. The median does not move while T-2052 stays the largest value.

Text description of the explorer

A number line from 0 to 200 minutes with the 12 reply times as dots. T-2052 is orange and you can move it from 0 to 200 minutes. Lines show the mean and the median, and a grey band shows the mean plus and minus one standard deviation. At the start, T-2052 is at 95 minutes: the mean is 28.75, the median 23 and the SD 21.10 minutes. At 195 minutes, the mean is 37.08 and the SD is 48.10 minutes. The median is still 23. At 0 minutes, T-2052 becomes the smallest value. Then the mean is 20.83, the median is 21 and the SD is 9.25 minutes. With T-2052 among the ordinary values, for example at 25 minutes, the mean and the median are both close to 23.

What the mean hides: the mean of 28.75 minutes describes no real customer. Eleven customers waited less, one waited much more. What the median hides: the median of 23 minutes does not change if T-2052 waits 95 minutes or 9 hours. If Grace’s question is “did anybody wait far too long?”, the median cannot answer it.

Percentiles: where a value stands

The median is the value with half of the data at or below it. A percentile extends this idea: the 90th percentile, written p90, is the value with 90% of the data at or below it. The median is p50. The quartiles split the data into four parts: Q1 is p25, the median is p50, and Q3 is p75. The general name for all of these is quantiles.

To find a percentile in a small sorted list:

1. Calculate the position: 1 + (n − 1) × p, where p is the percentile as a decimal.
2. If the position is a whole number, take that value.
3. If not, go between the two values around it, by the decimal part of the position.

For Monday’s tickets, n = 12:

Summary	Position	Values around it	Result
Q1 (p25)	1 + 11 × 0.25 = 3.75	3rd = 18, 4th = 18	18 minutes
Median (p50)	1 + 11 × 0.5 = 6.5	6th = 22, 7th = 24	22 + 0.5 × 2 = 23 minutes
Q3 (p75)	1 + 11 × 0.75 = 9.25	9th = 27, 10th = 31	27 + 0.25 × 4 = 28 minutes
p90	1 + 11 × 0.9 = 10.9	10th = 31, 11th = 38	31 + 0.9 × 7 = 37.3 minutes

This is the method that spreadsheets (PERCENTILE.INC) and numpy use by default. Other tools use slightly different methods, so small lists can give slightly different answers. With large datasets the methods agree.

The interquartile range (IQR) is Q3 − Q1 = 28 − 18 = 10 minutes. The middle half of the tickets got a reply within a 10-minute window. T-2052 does not change it.

Why AI services report p95 and p99. A model can answer in a mean time of 0.3 seconds and still make 1 customer in 100 wait 4 seconds. The mean hides those customers; p99 shows them. With only 12 values, though, a high percentile depends on one or two tickets. Here p95 lies between 38 and 95 minutes. Percentiles near 100 need many values.

Spread: range, variance and standard deviation

Two teams can have the same median and very different waits. A summary of spread says how far the values are from each other.

The range is the largest value minus the smallest: 95 − 12 = 83 minutes. It uses only two values, so one outlier controls it. Without T-2052 the range is 38 − 12 = 26 minutes.

The variance uses every value. It is the mean of the squared distances from the mean. Priya calculates it in a table: the distance from the mean for each ticket, then the square of that distance.

Ticket	Minutes, xᵢ	Distance, xᵢ − 28.75	Squared distance
T-2041	12	−16.75	280.56
T-2042	15	−13.75	189.06
T-2043	18	−10.75	115.56
T-2044	18	−10.75	115.56
T-2045	20	−8.75	76.56
T-2046	22	−6.75	45.56
T-2047	24	−4.75	22.56
T-2048	25	−3.75	14.06
T-2049	27	−1.75	3.06
T-2050	31	2.25	5.06
T-2051	38	9.25	85.56
T-2052	95	66.25	4,389.06
Sum	345	0	5,342.25

The distances add up to zero, as you saw in the last lesson. That is why the variance squares them first: a square is never negative, so distances on the two sides cannot cancel.

$$\text{variance} = \frac{1}{n}\sum_{i=1}^{n} (x_i - \bar{x})^2 = \frac{5342.25}{12} = 445.19 \text{ minutes}^2$$

Read it as “the mean of the squared distances from the mean”. The unit is a problem: “square minutes” means nothing to Grace. So take the square root, which returns to minutes. That is the standard deviation (SD):

$$\text{SD} = \sqrt{445.19} = 21.10 \text{ minutes}$$

In plain words, the SD is a typical distance between a value and the mean.

One ticket gives 82% of the total squared distance. Squaring makes large distances count much more than small ones.

Text description of the chart

Twelve columns, one for each ticket, labelled with its reply time in minutes. Each column is that ticket’s squared distance as a percentage of the total, 5,342.25. The eleven columns for 12 to 38 minutes are between 0.1% and 5.3%, and they are hard to see. The column for 95 minutes is 82.2%.

The chart shows what the SD hides. Squaring makes a large distance count much more than a small one. T-2052 is 66 minutes from the mean, and its square is 4,389. Without T-2052, the SD of the other 11 tickets is only 7.10 minutes. An SD of 21 minutes does not mean that customers’ waits usually differ by 21 minutes. It means that one wait was very different.

n or n − 1? Many tools divide by n − 1 instead of n when the data is a sample from a larger group. Module 5 explains why. For these 12 tickets, that gives an SD of 22.04 instead of 21.10 minutes. Python’s statistics.pstdev and numpy’s std divide by n. Python’s statistics.stdev and pandas divide by n − 1. Write in your notes which one you used.

Worked example: Priya’s summary for Grace

Grace asks: “How long do customers wait for a first reply, and is it the same for everybody?” Priya writes the full summary, in the same order as the pandas describe() table:

Summary	Value	What it answers
Count	12 tickets	How much data is there? Not much: one morning.
Mean	28.75 minutes	Total waiting time divided by tickets
SD	21.10 minutes	How far values are from the mean. Most of it comes from T-2052
Minimum	12 minutes	Fastest reply
Q1 (p25)	18 minutes	A quarter of tickets got a reply in this time or less
Median (p50)	23 minutes	A typical wait
Q3 (p75)	28 minutes	Three quarters got a reply in this time or less
Maximum	95 minutes	Slowest reply: T-2052, during the outage

Then she writes two sentences for Grace:

“A typical customer waited about 23 minutes for a first reply, and half of them waited between 18 and 28 minutes. One customer waited 95 minutes because of Monday’s outage. That one ticket raises the mean to 28.75 minutes and makes the SD look large.”

The sentences use the median for “typical”, the quartiles for “the same for everybody?”, and they name the outlier and its cause. They do not delete it.

Guided practice: summarise without the outage

Grace asks: “What do our replies look like on a normal morning, without the outage?” Use the 11 tickets T-2041 to T-2051.

1. In c03-notes.md, add the heading ## Describe a dataset.
2. Write the 11 values in order: 12, 15, 18, 18, 20, 22, 24, 25, 27, 31, 38.
3. Calculate the sum and the mean.
4. Find the median. With 11 values, it is the 6th value.
5. Calculate Q1 and Q3 with the position rule: 1 + 10 × 0.25, and 1 + 10 × 0.75.
6. Calculate the range and the IQR.
7. Optional: calculate the SD. Use 22.73 as the mean, and make a table of distances like Priya’s.
8. Write one sentence for Grace that compares a normal morning with Monday.

Check: your mean is lower than 28.75 minutes, and your median is close to 23 minutes.

Show the example answer

3. Sum = 250 minutes. Mean = 250 ÷ 11 = 22.73 minutes.
4. Median = 6th value = 22 minutes.
5. Q1: position 3.5, between 18 and 18, so 18 minutes. Q3: position 8.5, between 25 and 27, so 26 minutes.
6. Range = 38 − 12 = 26 minutes. IQR = 26 − 18 = 8 minutes.
7. The sum of squared distances is about 554.2. The variance is 554.2 ÷ 11 = 50.38, so the SD is about 7.10 minutes.
8. “On a normal morning, a typical customer waits about 22 minutes, and nobody waits more than 38. Monday’s typical wait was almost the same (23 minutes). The outage added one very long wait, not slower replies for everybody.”

Compare the changes when you remove T-2052. The mean fell by 6 minutes and the median by 1 minute. The range fell from 83 to 26 minutes, and the SD from 21.10 to 7.10 minutes. The median and the IQR almost did not move.

Run Python in your browser

Optional: check the summary

This box is optional. It prints the summaries of Monday’s 12 tickets. Remove 95 from the list and run it again to check your guided practice. Without method="inclusive", Python uses another method and gives slightly different quartiles.

Python code (you can change it)import statistics minutes = [12, 15, 18, 18, 20, 22, 24, 25, 27, 31, 38, 95] print("mean ", statistics.mean(minutes)) print("median", statistics.median(minutes)) print("SD (divide by n) ", round(statistics.pstdev(minutes), 2)) print("SD (divide by n - 1)", round(statistics.stdev(minutes), 2)) # Quartiles with the same method as the lesson (and numpy) print("Q1, median, Q3", statistics.quantiles(minutes, n=4, method="inclusive"))

Independent variation: Tuesday

On Tuesday morning, eight tickets have these first-reply times, in minutes: 10, 12, 15, 15, 18, 20, 24, 60.

In your notes, calculate the mean, the median, the range and the IQR. Then decide which one value you give Grace as “a typical wait on Tuesday”. Also decide which second value you add, so that she does not miss the 60-minute ticket.

Show the example answer

• Mean: 174 ÷ 8 = 21.75 minutes. Median: between 15 and 18, so 16.5 minutes.
• Range: 60 − 10 = 50 minutes. Q1: position 2.75, so 12 + 0.75 × 3 = 14.25. Q3: position 6.25, so 20 + 0.25 × 4 = 21. IQR = 6.75 minutes.
• Typical wait: the median, 16.5 minutes. Six of the eight tickets waited less than the mean, because the 60-minute ticket pulls the mean up.
• Second value: the maximum, 60 minutes (or “one ticket waited 60 minutes”). A count of tickets over a limit is also good, for example “1 of 8 tickets waited over 40 minutes”.

Common mistake

Reporting only the mean latency of a model

Symptom: Tomás reports that the new ticket classifier answers in a mean time of 0.3 seconds. The support tool waits up to 3 seconds for a suggestion. After release, agents say that suggestions are often missing. The dashboard still shows a mean of 0.3 seconds.

Cause: most answers take 0.2 seconds, but about 1 in 50 takes 5 seconds or more, when the model gets a very long ticket. The mean includes those slow answers, but it averages them away. The tool gives up after 3 seconds, so those tickets get no suggestion.

Fix: for any speed or waiting time, report percentiles with the mean: p50, p95 and p99. Also report the share of answers over the limit that matters (here, 3 seconds). Ask for the count: “How many requests out of how many were too slow?”

Check your understanding

T-2052 waited 95 minutes. Predict the mean and the median if it had waited 195 minutes. Explain why one changes and the other does not.

Show the answer

The sum grows by 100 minutes, from 345 to 445, so the mean is 445 ÷ 12 = 37.08 minutes (100 ÷ 12 = 8.33 minutes higher). The median stays at 23 minutes: T-2052 is still the largest value, and the 6th and 7th values (22 and 24) did not change. The mean uses the size of every value; the median uses only the order.

Check your understanding

A team that runs an AI service promises answers in under 1 second. Why does it measure p99 and not the mean?

Show the answer

The promise is about each customer’s answer, not about the total. The mean can be well under 1 second while some answers take much longer, because many fast answers hide a few slow ones. p99 says: 99 of 100 answers take this long or less. If p99 is under 1 second, at most 1 in 100 answers breaks the promise. The team also needs enough requests: with a few hundred, p99 depends on the slowest few.

Check your understanding

The SD of Monday's 12 tickets is 21.10 minutes, and without T-2052 it is 7.10 minutes. What does this tell you about the SD as a summary?

Show the answer

The SD is very sensitive to outliers, because it squares each distance from the mean. T-2052 alone gives 82% of the squared distances. So a large SD can mean “all values are spread out” or “one value is far away”. You must look at the data, or at a chart, to know which. The IQR (10 minutes with T-2052, 8 without) is less sensitive.

Recap

• The mean uses the size of every value, so one outlier can move it far. The median uses only the order.
• Percentiles say where a value stands: p90 has 90% of the data at or below it. Q1, the median and Q3 split the data into quarters. AI services report p95 and p99 for speed.
• The range uses only the two extreme values. The IQR is the spread of the middle half.
• The variance is the mean squared distance from the mean, and the SD is its square root, in the original unit. Squaring makes outliers count a lot.
• A good summary names the typical value, the spread and any outlier with its cause.

Next, you see the shape of a whole dataset in a histogram and a box plot, and how a chart can mislead.

Next: Read distributions