Skip to content
Français

Median, MAD and IQR

  • Lesson 6
  • 35 min
  • Module 2: Simple baselines that are hard to beat

Where we are

We saw that one extreme value moves the mean and the standard deviation, so the z-score misses the next anomaly. We need a centre and a spread that one value cannot move. In this lesson, we use the median, MAD and IQR, and we build a baseline for each customer, country and agent.

Plays from YouTube. Nothing loads from YouTube until you press play. Privacy
Lesson 6 of 38 · Anomaly Detection: Find What Is UnusualMedian, MAD and IQR
Course overview

What you will be able to do

Build a baseline that one extreme value cannot move, with the median, the MAD and the IQR. Then give every customer a baseline of their own, because a fraud is unusual for its customer, not for the shop.

NotebookDo this lesson in a notebookColab · Kaggle · GitHub

The notebook has every code cell, task and check of this lesson. Its setup cells install pandas, NumPy, SciPy, scikit-learn and matplotlib, and download the payments (about 7 MB). About 30 minutes.

Colab and Kaggle open your own copy. They need a free Google or Kaggle account. Stuck? See the solution notebook.

Before you startWhat to know and to have readyAbout 35 minutes

What you need to know

What you need on your computer

The same as in the last lesson: Colab or Kaggle in a browser, or Python 3.12 with the notebook kernel.

Tested with Python 3.12.13 in Jupyter on macOS, with pandas 3.1.1 and NumPy 2.5.4. Colab, Kaggle and Windows were not tested.

A teacher lines up seven children by height for a class photo. She puts the middle child in the centre. Now a basketball player, two metres tall, joins the line at the end. Who is in the middle now? Almost the same place: the middle moves by half a child. The average height, though, jumps up. One very tall person changed the average, but not the middle.

That is the whole idea of this lesson. In the last lesson, one 900-dollar payment moved the mean from 14.14 to 124.88 dollars. The next unusual payment then hid. We need a centre that behaves like the middle child, and a spread that behaves the same way.

The median is the middle value after you sort the values. Half of the values are below it and half are above it. To move the median a long way, you must move half of the values, not one.

The MAD, the median absolute deviation, is a spread built the same way. Take the distance of each value from the median. Then take the median of these distances. One huge value gives one huge distance, but the median of the distances ignores it.

The robust z-score uses the two in place of the mean and the standard deviation:

robust z = (value − median) ÷ (1.4826 × MAD)

Why 1.4826? When values have the normal bell shape, 1.4826 × MAD is about the same size as the standard deviation. So a robust z-score of 3 means about the same as a z-score of 3. A method that one extreme value does not change much is called robust.

The same seven payments as in the last lesson, sorted: 12, 13, 14, 14, 15, 15, 16. The middle one, the fourth, is 14. That is the median.

Payment 12 13 14 14 15 15 16
Distance from the median (14) 2 1 0 0 1 1 2

The distances, sorted, are 0, 0, 1, 1, 1, 2, 2. Their middle value is 1. That is the MAD. So the robust z-score of a 60-dollar payment is (60 − 14) ÷ (1.4826 × 1) = 31.0.

Now add the 900-dollar payment. The eight sorted values are 12, 13, 14, 14, 15, 15, 16, 900. With an even count, the median is halfway between the two middle values, 14 and 15: 14.5. The distances from 14.5 are 2.5, 1.5, 0.5, 0.5, 0.5, 0.5, 1.5 and 885.5. Their median is halfway between 0.5 and 1.5: 1.0. The MAD did not move. The robust z-score of 60 dollars is (60 − 14.5) ÷ 1.4826 = 30.7. The 60-dollar payment is still very far from usual.

One number line from 0 to 140 dollars, with five rows. Payments: seven dots between 12 and 16, a blue dot at 60, and a note that 900 is off the line. Mean plus or minus 1 standard deviation before: a narrow band at 14.14; z of 60 is 34.1. After the 900-dollar payment: an arrow shows the mean moving 110.74 dollars to 124.88, and the band of plus or minus 313.2 is wider than the line; z of 60 is minus 0.21. Median plus or minus 1.4826 times MAD before: a narrow band at 14.0; robust z of 60 is 31.0. After the 900-dollar payment: the band is at 14.5, it moved 0.50; robust z of 60 is 30.7.
One 900-dollar payment joins seven usual payments (a constructed example). The mean and the standard deviation move far; the median moves 0.50 dollars and the MAD does not move. Only the robust z-score still sees the 60-dollar payment.
Text description of the diagram

The diagram has one number line, from 0 to 140 dollars, and five rows. The first row shows the payments. Seven grey dots are between 12 and 16 dollars, and a blue dot is the new payment of 60 dollars. A note says that the 900-dollar payment is off the line, to the right. The second row shows the mean plus or minus one standard deviation before the huge payment: a narrow band at 14.14. The z-score of 60 dollars is 34.1. The third row shows the same after the 900-dollar payment. An arrow shows the mean moving 110.74 dollars, to 124.88. The band of plus or minus 313.2 dollars is wider than the whole line. The z-score of 60 dollars is −0.21. The fourth row shows the median plus or minus 1.4826 times the MAD before: a narrow band at 14.0. The robust z-score of 60 dollars is 31.0. The fifth row shows the same after the 900-dollar payment: the band is at 14.5. It moved 0.50 dollars and kept its width. The robust z-score is 30.7.

Run sections 1 and 2 of the notebook to check these numbers. Section 1 defines a small function, robust_z(x, values), that returns the median, the MAD and the robust z-score of x against values:

def robust_z(x, values):
values = np.array(values)
median = np.median(values)
mad = np.median(np.abs(values - median)) # the median distance from the median
return median, mad, (x - median) / (1.4826 * mad)

The output of the two sections:

distances from the median: [0.0, 0.0, 1.0, 1.0, 1.0, 2.0, 2.0]
median 14.00, MAD 1.00, robust z-score of 60 dollars: 31.0
median 14.50, MAD 1.00, robust z-score of 60 dollars: 30.7

There is a second robust spread, built from quartiles. Q1, the first quartile, has a quarter of the values below it. Q3, the third quartile, has three quarters below it. The IQR, the interquartile range, is Q3 − Q1: the width of the middle half of the values.

The boxplot rule calls a value an outlier when it is more than 1.5 IQR above Q3, or more than 1.5 IQR below Q1. The limit Q3 + 1.5 × IQR is the upper fence. The rule got its name from the boxplot, a chart that draws the middle half as a box and the outliers as dots.

For the eight payments with the 900-dollar one, the notebook prints:

Q1 13.75, Q3 15.25, IQR 1.50, upper fence 17.50
above the fence: [900.0, 60.0]

Q1 and Q3 fall between two values, so NumPy takes a point between them, as it did for the median. The fence is 15.25 + 1.5 × 1.50 = 17.50. Both 900 and 60 are above it.

Customer C-08459, a personal customer, made 7 payments in the reference months and 11 more until November. The box plot shows both sets:

Customer C-08459's payments. The box is the middle half, the line in it the median, the whiskers reach the last values inside the fences, and a dot on the line is an outlier. With all 18 payments, only the payment of 260.89 dollars is outside the fence.
Text description of the chart

The chart has two rows on an axis from 0 to 280 dollars. The first row uses the customer’s 7 payments of July and August 2025, between 25.84 and 64.42 dollars. The box covers the middle half, from about 30 to 58 dollars, and the median is 41.05. The second row uses all 18 payments, July to November. The box covers about 27 to 63 dollars. The whiskers reach from 19.18 to 116.52 dollars. One dot, at 260.89 dollars, lies outside the upper fence. It is a planted account takeover on 5 October 2025. Below each box, small dots show every payment.

Now the real payments. As before, the detector learns from the reference months and scores the evaluation months. Section 3 of the notebook computes the shop’s baseline:

g_median = reference.amount.median()
g_mad = (reference.amount - g_median).abs().median()
g_q1, g_q3 = reference.amount.quantile([0.25, 0.75])
reference months: median 44.78, MAD 25.02, Q1 25.33, Q3 102.62, upper fence 218.54

The mean of the same payments was 119.36 dollars. The median is 44.78. The large business orders pull the mean up, and they do not pull the median.

Before you run the next cell, predict: the robust z-score is not fooled by one huge payment. Will it find more fraud than the z-score of the last lesson, with a precision at the budget of 0.003?

detector setting alerts per day precision at budget recall cost per week ($) explain one alert (min)
MAD (global) |robust z| of the amount, one baseline 50.0 0.003 0.005 10264 1
IQR rule (global) amount above Q3 + 1.5 IQR 236.2 0.006 0.010 10169 1
the IQR rule flags 21,494 payments; 98.3% are business payments

No better. The global MAD reviews exactly the same 50 payments a day as the global z-score: the largest amounts. The boxplot rule flags 236.2 payments a day. That is far more than the 50 the team can review, so the first 50 of each day are reviewed and the rest wait. Of its flags, 98.3% are business payments, which are honest.

So robust statistics fix one problem, the extreme value in the history. They do not fix the other problem: one baseline for everybody asks the wrong question. A fraud is unusual for its customer.

The fix is a baseline per group: a set of rows that should share one normal. The notebook starts with the two account types. It keeps their baselines in a table called by_type, which the next cell uses as the fallback:

by_type = reference.groupby("account_type").amount.agg(
["median", lambda s: (s - s.median()).abs().median(), "size"])
by_type.columns = ["median", "mad", "payments"]
median mad payments
account_type
business 257.50 131.71 42255
personal 34.32 14.94 131016

A business’s usual payment is 257.50 dollars. A person’s is 34.32. So a 300-dollar payment is ordinary for a business and very unusual for a person. One shop-wide baseline cannot say both.

The same idea works for many groups:

  • Per customer: each customer’s own median and MAD. This is the café owner again.
  • Per country: a payment in Portugal against other payments in Portugal, if a shop sells there.
  • Per agent: in Module 3, each support agent’s refunds against their own usual rate.

A small group has a problem: too few rows to learn from. A median of two payments is not a habit. So the detector uses a fallback. A customer with at least 5 reference payments gets their own baseline. A customer with fewer uses the baseline of their account type.

g = reference.groupby("customer_id").amount
cust = pd.DataFrame({"n": g.size(), "median": g.median(),
"mad": g.apply(lambda s: (s - s.median()).abs().median())})
known = evaluation.customer_id.map(cust["n"]) >= 5 # at least 5 payments to learn from
c_median = evaluation.customer_id.map(cust["median"]).where(known).fillna(evaluation.account_type.map(by_type["median"]))
c_mad = evaluation.customer_id.map(cust["mad"]).where(known).fillna(evaluation.account_type.map(by_type["mad"]))
c_mad = np.maximum(c_mad, 1.0) # a floor of 1 dollar for the spread (section 5)
rz_customer = ((evaluation.amount - c_median) / (1.4826 * c_mad)).abs()

where(known) keeps a customer’s own value only when they have 5 or more reference payments. fillna(...) puts the account type’s value in the other places. Before you run it, predict the precision at the budget. The z-score per customer of the last lesson had 0.045.

detector setting alerts per day precision at budget recall cost per week ($) explain one alert (min)
MAD (global) |robust z| of the amount, one baseline 50.0 0.003 0.005 10264 1
IQR rule (global) amount above Q3 + 1.5 IQR 236.2 0.006 0.010 10169 1
MAD per customer |robust z| against the customer's own median 50.0 0.048 0.078 8280 1
customers with a baseline: 12,276; evaluation payments that use the account type's baseline: 50,778

MAD per customer has a precision at the budget of 0.048: 16 times the shop-wide baseline. It also beats the z-score per customer, 0.045, and it costs 8,280 dollars a week, against 9,047. Of the evaluation payments, 50,778 use the fallback, because their customer had fewer than 5 reference payments.

The chart shows customer C-08459 again, with the band that MAD per customer learned from the 7 reference payments:

  • Payments
  • Median, 41.05
  • Robust z = 3, 101.36
One customer's payments and their personal band, learned from July and August (median 41.05, MAD 13.56). The planted account takeover of 260.89 dollars has a robust z-score of 10.94 and is in the day's 50 reviewed alerts.
Text description of the chart

The chart shows 18 payments of customer C-08459 from July to November 2025, on an axis from 0 to 280 dollars. A dashed line at 41.05 dollars marks the customer’s median from the reference months. A second dashed line at 101.36 dollars marks a robust z-score of 3. Almost every payment is between 19 and 82 dollars, below the band. On 5 October there are two payments above it. One is 260.89 dollars, with a robust z-score of 10.94: a planted account takeover. The other, on the same day, is 116.52 dollars and honest.

Two things to see. First, the band is personal: 101.36 dollars is the edge for this customer, while the shop-wide fence was 218.54. With the shop-wide robust z, the 260.89-dollar payment scores 5.83, and it is not among the day’s 50 highest. With the personal band it scores 10.94, and it is reviewed.

Second, the honest payment of 116.52 dollars is also above the band, with a robust z-score of 3.75. A robust z-score of 3 is not a verdict. The team reviews the 50 highest scores of each day. That day, the 50th score was 4.42, so the 116.52-dollar payment was not reviewed.

The tiny example was constructed. Section 4 of the notebook shows the same effect in Larkfield’s data, with customer C-06547:

reference payments: [75.59, 69.38, 74.89, 248.87, 90.9, 69.39]
new payment: 1.40 dollars on 2025-09-16 06:14:59
mean 104.84, standard deviation 71.00, z-score -1.46
median 75.24, MAD 5.86, robust z-score -8.51

One past payment of 248.87 dollars made the standard deviation large. So a 1.40-dollar payment looks almost ordinary to the z-score: −1.46. The median and the MAD ignore the 248.87, and the robust z-score is −8.51. This payment is a planted card test: a thief tries a tiny payment to see if a stolen card works. Across the evaluation months, MAD per customer puts 93 planted payments in its daily 50 that the z-score per customer misses. The other way round, there are 77.

Section 6 of the notebook has three tasks.

  1. Write the robust z-score of the 60-dollar payment after the huge payment, as robust_after.
  2. Compute the upper fence of the boxplot rule for personal accounts only, in the reference months. Write it as personal_fence.
  3. Score only the high side of MAD per customer: keep the sign, so amounts below usual get low scores. Keep the row as up_row. Is it better?
Show the answers
  1. robust_after = 30.7.
  2. Take p = reference.amount[reference.account_type == "personal"], then p.quantile([0.25, 0.75]). The fence is 102.81 dollars, against 218.54 for the whole shop.
  3. signed = (evaluation.amount - c_median) / (1.4826 * c_mad), then compare.score_payments(evaluation, signed, ...). Its precision at the budget is 0.033, less than 0.048. Some frauds are far below the usual amount, like the 1.40-dollar card test. Keep both sides.

Common mistake

A MAD of zero

Symptom: a payment only 1 dollar above a customer’s usual amount gets a robust z-score of inf, and NumPy prints a RuntimeWarning. Such scores sort first and take places in the 50 daily alerts.

Cause: if more than half of a customer’s payments have exactly the same amount, the MAD is 0. This happens more easily than a standard deviation of 0: the values do not all need to be equal.

same = [20.0, 20.0, 20.0, 25.0, 20.0] # four of five payments are exactly 20 dollars
median0, mad0, rz0 = robust_z(21.0, same)
print(f"median {median0:.2f}, MAD {mad0:.2f}, robust z-score of 21 dollars: {rz0}")
median 20.00, MAD 0.00, robust z-score of 21 dollars: inf
customers with a baseline and a MAD below 1 dollar: 66

No Larkfield customer with a baseline has a MAD of exactly 0, but 66 have a MAD below 1 dollar. A tiny MAD gives the same problem in a weaker form: small changes get huge scores.

Fix: give the spread a floor. The detector uses 1 dollar, so the 21-dollar payment gets a robust z-score of 0.67.

Consequence in AI work: every score that divides by a spread needs a floor. Choose it in the units of the data, and say why.

Use it when the history can contain extreme values, which is almost always. Also when you can name the group that shares one normal: a customer, an account type, an agent.
Do not use it alone when the fraud is not about the value. A stolen card used for ordinary baskets has an ordinary amount.
What it costs a median per group and a fallback for small groups. It needs history: 50,778 evaluation payments, about 30%, have no baseline of their own. A baseline also gets old. When normal moves, as with the app launch of Module 1, it must be learned again.

The comparison table, with this lesson’s rows:

Detector Setting Alerts/day Precision at budget Recall Cost per week ($)
no detector nothing is reviewed 0.0 0.0 0.0 10,367
z-score |z| of the amount, one baseline for all 50.0 0.003 0.005 10,264
MAD (global) |robust z| of the amount, one baseline 50.0 0.003 0.005 10,264
IQR rule (global) amount above Q3 + 1.5 IQR 236.2 0.006 0.01 10,169
MAD per customer |robust z| against the customer’s own median 50.0 0.048 0.078 8,280
MAD per customer, high side robust z, only amounts above usual 50.0 0.033 0.054 8,348

Check your understanding

Five refunds of one agent, in dollars: 30, 35, 40, 45, 300. Compute the median and the MAD. What is the robust z-score of a refund of 70 dollars?

Show the answer

Sorted, the middle value is 40: that is the median. The distances from 40 are 10, 5, 0, 5 and 260. Sorted: 0, 5, 5, 10, 260. Their median is 5: that is the MAD. The robust z-score of 70 dollars is (70 − 40) ÷ (1.4826 × 5) = 30 ÷ 7.41 = about 4.0. The 300-dollar refund did not change the median or the MAD.

Check your understanding

Predict: Q1 of a customer's payments is 20 dollars and Q3 is 40 dollars. Where is the upper fence of the boxplot rule? Is a payment of 75 dollars an outlier?

Show the answer

The IQR is 40 − 20 = 20. The upper fence is Q3 + 1.5 × IQR = 40 + 30 = 70 dollars. A payment of 75 dollars is above the fence, so the boxplot rule calls it an outlier.

Check your understanding

Practical: on Larkfield's evaluation months, the global MAD has a precision at the budget of 0.003 and MAD per customer 0.048. Explain the difference. Then say what the detector does for a customer with only 2 reference payments, and why.

Show the answer

The global MAD compares every payment with the whole shop, so its highest scores are the largest amounts: honest business orders. MAD per customer asks if a payment is unusual for its own customer, which is how much of the fraud looks. A customer with only 2 reference payments has too little history for a baseline. So the detector uses the fallback: the median and the MAD of the customer’s account type.

Recap

  • The median and the MAD are robust: after a 900-dollar payment, the median moved 0.50 dollars and the MAD did not move.
  • The robust z-score is (value − median) ÷ (1.4826 × MAD). The boxplot rule flags values above Q3 + 1.5 × IQR.
  • Robust statistics do not fix a wrong question. One baseline for the whole shop still found almost nothing (0.003).
  • A baseline per group asks the right question. MAD per customer, with a fallback to the account type, had a precision at the budget of 0.048.
  • A MAD can be 0 or tiny. Give it a floor.

You can now score a payment against its own customer’s habits, and one strange past payment does not fool the score. But 0.048 means that only about 2 of the 50 daily alerts are fraud. Many frauds are not about the amount at all. Next, you write rules for the shape of a fraud, such as many payments on one device in one hour, and compare them with these scores.

Next: Rules, and when a rule is the right answer