Z-scores, and why the mean is fooled
- Lesson 5
- 30 min
- Module 2: Simple baselines that are hard to beat
Where we are
We have the Larkfield data and an empty comparison table. Before any model, we try the simplest question: how far is this payment from the usual one? In this lesson, we measure that distance with a z-score, and we see how one huge payment moves the mean and hides the next strange one.
Lesson 5 of 38 · Anomaly Detection: Find What Is UnusualZ-scores, and why the mean is fooled
1. What "unusual" means
2. Simple baselines that are hard to beat
3. Statistical process control and distribution drift
4. Distance, density and isolation
5. Anomalies in time
6. Deep and embedding methods
7. Evaluate with few labels
8. Learn from the reviewers
9. Run a detector in production
What you will be able to do
Turn “this payment is far from usual” into a number, the z-score. Then see its weak point: one huge value moves the mean and the standard deviation, and the next strange payment hides behind them.
NotebookDo this lesson in a notebook
The notebook has every code cell, task and check of this lesson. Its setup cells install pandas, NumPy, SciPy, scikit-learn and matplotlib, and download the payments (about 7 MB). About 25 minutes.
Colab and Kaggle open your own copy. They need a free Google or Kaggle account. Stuck? See the solution notebook.
Before you startWhat to know and to have ready
What you need to know
- The comparison table, the split into reference and evaluation months, and the alert budget of 50 a day, from Meet the Larkfield data.
- The mean and the standard deviation are explained here. For more, read Describe a dataset in Essential Mathematics and Statistics for AI.
What you need on your computer
The same as in Meet the Larkfield data: Colab or Kaggle in a browser, or Python 3.12 with a virtual environment and the notebook kernel. The notebook’s first cell installs the rest.
Tested with Python 3.12.13 in Jupyter on macOS, with pandas 3.1.1 and NumPy 2.5.4. Colab, Kaggle and Windows were not tested.
The café owner who knows the usual bill
Section titled “The café owner who knows the usual bill”A café owner has a regular customer. Every morning, he buys a coffee and a croissant, about 14 dollars. One morning, the bill is 60 dollars. The owner notices at once. She does not need a rule for it. She knows the usual bill, and she knows how much it usually changes: a dollar or two.
Two things make her notice. The first is the usual value: about 14 dollars. The second is the usual variation: one or two dollars. A bill of 60 dollars is about 46 dollars away from usual. Compared with a variation of one or two dollars, that is enormous.
Larkfield has about 1,875 payments a day. Nobody can know every customer the way the café owner does. So we need her two ideas as numbers that a computer can compute for every payment.
The idea: count the distance in standard deviations
Section titled “The idea: count the distance in standard deviations”The mean is the average: add the values, then divide by how many there are. The standard deviation is the usual size of the differences from the mean. A small standard deviation means that the values stay close together.
The z-score of a new value combines the two:
z = (value − mean) ÷ standard deviation
It says how many standard deviations the value is from the mean. A z-score of 0 is exactly the mean. A z-score of 2 is two usual steps above it. A negative z-score is below the mean. Many teams call a value with a z-score above 3, or below −3, an outlier: a value far from the others. The size of z, without its sign, is written |z|.
Statistics books write the standard normal distribution with the letter Z, so the score got that name. The idea itself is the café owner’s judgement, written as a division.
A tiny example with every value visible
Section titled “A tiny example with every value visible”Here are the café customer’s last seven payments, in dollars. The table also shows each step of the standard deviation.
| Payment | Amount | Difference from the mean (14.14) | Difference squared |
|---|---|---|---|
| 1 | 12.00 | −2.14 | 4.59 |
| 2 | 15.00 | 0.86 | 0.73 |
| 3 | 14.00 | −0.14 | 0.02 |
| 4 | 13.00 | −1.14 | 1.31 |
| 5 | 16.00 | 1.86 | 3.45 |
| 6 | 15.00 | 0.86 | 0.73 |
| 7 | 14.00 | −0.14 | 0.02 |
| Mean 14.14 | Sum 10.86 |
Divide the sum of the squares by 6 (one less than the 7 payments): 10.86 ÷ 6 = 1.81. The square root of 1.81 is 1.35. So the standard deviation is 1.35 dollars.
Now the new payment of 60 dollars: (60 − 14.14) ÷ 1.35 = 34.0. With the values before rounding, the notebook prints 34.1. The new payment is 34 standard deviations above the mean. Like the café owner, the z-score says: very far from usual.
Run it in the notebook
Section titled “Run it in the notebook”- Open the notebook from the box Do this lesson in a notebook at the top of this page.
- Run the three setup cells. The second one ends with
Ready: payments.csv.gz (6.9 MB), sealed-key.txt (0.0 MB). - Run the first cell of section 1.
- Compare its output with the output below.
usual = [12.0, 15.0, 14.0, 13.0, 16.0, 15.0, 14.0] # one customer's last seven payments, in dollarsnew_payment = 60.0
mean = np.mean(usual)std = np.std(usual, ddof=1)z = (new_payment - mean) / stdprint(f"mean {mean:.2f}, standard deviation {std:.2f}, z-score of {new_payment:.0f} dollars: {z:.1f}")mean 14.14, standard deviation 1.35, z-score of 60 dollars: 34.1ddof=1 tells NumPy to divide by 6, not by 7, as in the table. This is the usual choice when the values are a sample of a customer’s habits.
One huge payment hides the next one
Section titled “One huge payment hides the next one”Now change one thing. Before the 60-dollar payment, one huge payment of 900 dollars joins the customer’s history. Perhaps it is a wedding gift. Perhaps it is a thief with the customer’s card. Then the same 60-dollar payment arrives.
Before you run the next cell, predict: is the z-score of the 60-dollar payment still above 3?
with_huge = usual + [900.0] # one huge payment joins the historymean2 = np.mean(with_huge)std2 = np.std(with_huge, ddof=1)z2 = (new_payment - mean2) / std2print(f"mean {mean2:.2f}, standard deviation {std2:.2f}, z-score of {new_payment:.0f} dollars: {z2:.2f}")mean 124.88, standard deviation 313.20, z-score of 60 dollars: -0.21The z-score is now −0.21. The 60-dollar payment looks completely normal, even a little below the usual payment. Nothing about the 60-dollar payment changed. Only the history changed:
Text description of the diagram
The diagram has two number lines. The top one goes from 0 to 70 dollars. It shows the seven usual payments as grey dots between 12 and 16 dollars, and the mean at 14.14. A narrow shaded band shows the mean plus or minus three standard deviations: 10.09 to 18.19 dollars. The new payment of 60 dollars is a blue dot far to the right of the band. Its z-score is 34.1. The bottom number line goes from 0 to 1,000 dollars, because the history now has a 900-dollar payment. The seven usual payments are squeezed together near 14. The mean is at 124.88. The shaded band now goes from −814.72 to 1,064.48 dollars and covers the whole line. The blue dot at 60 is inside the band. Its z-score is −0.21.
One value moved the mean by almost 111 dollars and made the standard deviation 232 times larger. Both use the size of every value, so one extreme value pulls them a long way. Statisticians say that the mean and the standard deviation are not robust. This is the weak point of the z-score, and the next lesson fixes it.
Why does it matter for fraud? A thief often makes one large payment first. If that payment enters the customer’s history, the next payments look normal. The first fraud hides the second.
Z-scores on Larkfield’s payments
Section titled “Z-scores on Larkfield’s payments”Now the same idea on the real data. The detector learns one mean and one standard deviation from the reference months, June to August 2025. Then it scores every payment of the evaluation months, September to November. Each day, Sofia’s team reviews the 50 payments with the highest |z|.
import comparepayments = pd.read_csv("data/payments.csv.gz", parse_dates=["ts"])reference, evaluation = compare.split(payments)mean, std = reference.amount.mean(), reference.amount.std()print(f"reference months: {len(reference):,} payments, mean {mean:.2f}, standard deviation {std:.2f}")print(f"evaluation months: {len(evaluation):,} payments")reference months: 173,271 payments, mean 119.36, standard deviation 223.07evaluation months: 169,803 paymentsThe next cell scores every evaluation payment with its |z| and adds a row to the comparison table:
z_global = ((evaluation.amount - mean) / std).abs()table = compare.Table()table.add(compare.score_payments(evaluation, z_global, "z-score", "|z| of the amount, one baseline for all", explain_minutes=1))table.show()detector setting alerts per day precision at budget recall cost per week ($) explain one alert (min) z-score |z| of the amount, one baseline for all 50.0 0.003 0.005 10264 1A precision at the budget of 0.003 means that fewer than 1 of the 50 daily alerts is a planted fraud. The explain one alert column is an assumption: a reviewer needs about one minute to read “this amount is far above the shop’s mean”.
Why so bad? Every payment here is compared with one baseline for the whole shop. The highest |z| belongs to the largest amounts. In the evaluation months, all 4,550 reviewed alerts were business customers’ payments of at least 478.28 dollars. These are the same 50 payments a day as the “biggest amounts” detector of Meet the Larkfield data. Even 50 random payments a day found more fraud (0.019).
Where the fraud is, and where z = 3 is
Section titled “Where the fraud is, and where z = 3 is”The chart compares the amounts of honest and planted payments in the evaluation months. Each line shows the share of its own group in each range of amounts.
- Honest payments
- Planted frauds
- z = 3: 789 dollars
Text description of the chart
The chart has two lines on a log scale of amounts, from 1 to 10,000 dollars. Each point is the share of its group in a range of amounts. The honest line rises to a peak of about 21% between 25 and 40 dollars. Then it falls slowly, to almost nothing above about 2,500 dollars. The planted line is high already between 1 and 6 dollars, where card tests are, with up to 11%. It stays between 6% and 15% up to about 60 dollars, then drops below 2%. A dashed vertical line at 789 dollars shows where z = 3 starts. Almost no planted payment is to the right of it: 10 of 2,817. In the evaluation months, 87.4% of the planted payments are below 50 dollars.
The picture explains the row. Most frauds are small payments: card tests of a few dollars, and stolen cards used for ordinary-looking baskets. A large amount is unusual for the shop, but a fraud is unusual for its customer. Customer C-01234 of the first lesson is a good example. Her eight payments in Portugal have a global |z| of at most 0.47.
One baseline per customer
Section titled “One baseline per customer”So give each customer their own mean and standard deviation, learned from their own reference payments. A customer with fewer than 5 reference payments keeps the shop’s baseline: there is too little to learn from.
Before you run the cell, predict the precision at the budget of the per-customer z-score.
stats = reference.groupby("customer_id").amount.agg(["count", "mean", "std"])known = evaluation.customer_id.map(stats["count"]).fillna(0) >= 5 # at least 5 payments to learn fromc_mean = evaluation.customer_id.map(stats["mean"]).where(known, mean)c_std = evaluation.customer_id.map(stats["std"]).where(known, std)z_customer = ((evaluation.amount - c_mean) / c_std).abs()row = table.add(compare.score_payments(evaluation, z_customer, "z-score per customer", "|z| against the customer's own mean (5+ payments)", explain_minutes=1))table.show()print("customers with 5 or more reference payments:", int((stats["count"] >= 5).sum())) detector setting alerts per day precision at budget recall cost per week ($) explain one alert (min) z-score |z| of the amount, one baseline for all 50.0 0.003 0.005 10264 1z-score per customer |z| against the customer's own mean (5+ payments) 50.0 0.045 0.072 9047 1customers with 5 or more reference payments: 12276The precision rises from 0.003 to 0.045: 15 times better. About 2 of the 50 daily alerts are now planted frauds, and the cost falls by about 1,200 dollars a week. The question changed from “large for the shop” to “large for this customer”. That change matters more than the formula.
It is still low, for two reasons. First, one past huge payment fools a customer’s own mean too, as in the tiny example. Second, many frauds are not about the amount at all. A stolen card used for 20-dollar baskets has an ordinary amount. The next two lessons fix one reason each.
Change one thing: a fixed limit instead of the budget
Section titled “Change one thing: a fixed limit instead of the budget”The habit “alert when |z| > 3” is common. What happens if the detector sends every payment with a global |z| above 3, in place of the top 50?
flags = (z_global > 3).astype(int)row3 = compare.score_payments(evaluation, flags, "z-score above 3", "every payment with |z| > 3", threshold=1, explain_minutes=1)print(f"alerts per day with |z| > 3: {row3['alerts per day']}, precision of the 50 reviewed: {row3['precision at budget']}")alerts per day with |z| > 3: 33.9, precision of the 50 reviewed: 0.003The limit sends 33.9 alerts a day, fewer than the team can review. The precision does not change: the alerts are still the largest amounts. A limit decides how many alerts you get, not which frauds you find. And 3 is a habit, not a law: with this data, z = 3 is an amount of 789 dollars, far above most frauds. The payment that did not fit explained why the course reviews the top 50 a day. Module 7 sets the threshold from the budget on purpose.
Your turn
Section titled “Your turn”Section 6 of the notebook has three tasks. Each has a check cell that says “Check passed” or “Not yet”.
- Write the z-score of the 60-dollar payment after the huge payment, as
z_after. - Count how many evaluation payments a day have a global |z| above 4. Write it as
alerts_per_day_above_4. - Score only the high side of the per-customer z-score: keep the sign, and set every negative value to 0. Keep the row as
up_row. Is it better than |z|?
Show the answers
z_after = -0.21, the value of the second cell.round(float((z_global > 4).sum() / evaluation.ts.dt.floor("D").nunique()), 1)gives 18.0 alerts a day. At z = 4 the limit is 1,011.65 dollars.up_only = ((evaluation.amount - c_mean) / c_std).clip(lower=0), thencompare.score_payments(evaluation, up_only, ...). Its precision at the budget is 0.024, worse than |z| (0.045). Many frauds are small payments, below the customer’s usual amount, so the high side alone misses them.
Common mistake
A customer who always pays the same amount
Symptom: a few payments get a z-score of inf (infinity), and NumPy prints RuntimeWarning: divide by zero. In a pandas column, the inf values sort first. They take places in the 50 daily alerts, and they are rarely fraud.
Cause: a customer who always paid 20 dollars has a standard deviation of 0. The z-score divides by 0. The notebook shows it in section 5:
same = [20.0, 20.0, 20.0, 20.0, 20.0] # a customer who always pays the same amountz_bad = (25.0 - np.mean(same)) / np.std(same, ddof=1)print("z-score of 25 dollars:", z_bad)z-score of 25 dollars: infFix: give the spread a floor, for example 1 dollar:
z_fixed = (25.0 - np.mean(same)) / max(np.std(same, ddof=1), 1.0) # a floor of 1 dollar for the spreadprint("z-score of 25 dollars, with a floor:", z_fixed)z-score of 25 dollars, with a floor: 5.0Consequence in AI work: a score that divides by a spread needs a floor for that spread. Otherwise a few customers with perfect habits fill the review budget with alerts that mean nothing.
When to use a z-score
Section titled “When to use a z-score”| Use it when | the values have one clear centre and no extreme values in the history. Sensor readings and daily counts in a stable period are good examples. |
| Do not use it when | the history can contain extreme values, or when one baseline covers groups that differ, such as personal and business customers. |
| What it costs | almost nothing to compute and one minute to explain. Its price is in what it misses. |
The comparison table so far:
| Detector | Setting | Precision at budget | Recall | Cost per week ($) |
|---|---|---|---|---|
| no detector | nothing is reviewed | 0.0 | 0.0 | 10,367 |
| z-score | |z| of the amount, one baseline for all | 0.003 | 0.005 | 10,264 |
| z-score per customer | |z| against the customer’s own mean | 0.045 | 0.072 | 9,047 |
Knowledge check
Section titled “Knowledge check”Check your understanding
A customer's payments have a mean of 40 dollars and a standard deviation of 5 dollars. What is the z-score of a payment of 60 dollars? And of 30 dollars? Which one is an outlier by the habit “|z| above 3”?
Show the answer
60 dollars: (60 − 40) ÷ 5 = 4.0. 30 dollars: (30 − 40) ÷ 5 = −2.0. Only the 60-dollar payment is above the habit’s limit of 3. The 30-dollar payment is 2 standard deviations below the mean, which the habit calls normal.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
Predict and explain: one huge payment joins a customer's history. What happens to the mean, to the standard deviation and to the z-score of the next unusual payment? Use the numbers of the tiny example.
Show the answer
The mean moves toward the huge value: from 14.14 to 124.88 dollars. The standard deviation grows a lot: from 1.35 to 313.20. The z-score of the next 60-dollar payment falls from 34.1 to −0.21, so it looks normal. The mean and the standard deviation use the size of every value, so one extreme value pulls them far. They are not robust.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
Practical: the global z-score has a precision at the budget of 0.003 on Larkfield's evaluation months, and the per-customer z-score 0.045. Explain the difference, and say one reason why 0.045 is still low.
Show the answer
With one baseline for the whole shop, the highest |z| are the largest amounts: business orders, which are honest. With one baseline per customer, the score asks “unusual for this customer”, which is closer to how fraud looks. It is still low because many frauds are small, ordinary amounts (stolen cards, card tests), so an amount-only score cannot see them. And one past huge payment can still fool a customer’s own mean.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Recap
- A z-score is the distance from the mean, counted in standard deviations: (value − mean) ÷ standard deviation.
- The mean and the standard deviation are not robust. One 900-dollar payment moved the z-score of a 60-dollar payment from 34.1 to −0.21.
- One baseline for the whole shop found almost nothing (0.003): the largest amounts are business orders. One baseline per customer did better (0.045).
- “|z| > 3” is a habit. A fixed limit changes how many alerts you get, not which frauds you find.
- A spread can be 0. Give it a floor.
You can now score a payment against its customer’s usual amount. But the mean is still fooled by one extreme value in the history. Next, you replace it with a centre and a spread that one value cannot move: the median and the MAD.
Next: Median, MAD and IQR