Rules, and when a rule is the right answer
- Lesson 7
- 35 min
- Module 2: Simple baselines that are hard to beat
Where we are
Our robust scores find payments far from a customer's usual amount. But some fraud has a known shape, such as many payments in a few minutes. In this lesson, we write rules for these shapes, see what a rule does well and what it misses, and put the first rows in the comparison table.
Lesson 7 of 38 · Anomaly Detection: Find What Is UnusualRules, and when a rule is the right answer
1. What "unusual" means
2. Simple baselines that are hard to beat
3. Statistical process control and distribution drift
4. Distance, density and isolation
5. Anomalies in time
6. Deep and embedding methods
7. Evaluate with few labels
8. Learn from the reviewers
9. Run a detector in production
What you will be able to do
Write rules for the shape of a fraud, and score them like every other detector. Learn when a rule beats a statistic: when it captures how the fraud works. Also learn what a rule cannot do: find a fraud that nobody imagined.
NotebookDo this lesson in a notebook
The notebook has every code cell, task and check of this lesson. Its setup cells install pandas, NumPy, SciPy, scikit-learn and matplotlib, and download the payments (about 7 MB). About 30 minutes.
Colab and Kaggle open your own copy. They need a free Google or Kaggle account. Stuck? See the solution notebook.
Before you startWhat to know and to have ready
What you need to know
- MAD per customer and its row in the comparison table, from Median, MAD and IQR.
- The three kinds of planted fraud (stolen cards, card testing, account takeovers), from Meet the Larkfield data.
What you need on your computer
The same as in the last lesson: Colab or Kaggle in a browser, or Python 3.12 with the notebook kernel.
Tested with Python 3.12.13 in Jupyter on macOS, with pandas 3.1.1 and NumPy 2.5.4. Colab, Kaggle and Windows were not tested.
The bank that calls you after three payments
Section titled “The bank that calls you after three payments”Many people know this situation. You are on holiday, and you pay for a taxi, a coffee and a museum ticket within a few minutes. Then your bank sends a message: “Was this you?” Nobody at the bank looked at your payments. A rule did: “many payments on one card in a short time, in a new country”. Somebody wrote it down after they saw many thieves do exactly that.
A rule is a condition that a person writes in advance. When a payment meets the condition, the rule sends an alert. It needs no training data and no statistics, and it gives its reason in one sentence. That is its strength. Its weakness is the other side of the same thing: it finds only what somebody imagined.
The problem: frauds that are not about the amount
Section titled “The problem: frauds that are not about the amount”The last lesson ended with MAD per customer at a precision at the budget of 0.048. Only about 2 of the 50 daily alerts are planted frauds. The reason is in the data. In the evaluation months, the median planted payment is 14.31 dollars. Card tests are a few dollars; stolen cards buy ordinary baskets. Their amount is often normal. What is not normal is their shape: many payments, very fast, often on one device.
So we need a different kind of check. It does not ask “is this amount unusual?”. It asks “does this look like the way a known fraud works?”.
A velocity rule, in a tiny example
Section titled “A velocity rule, in a tiny example”A velocity rule counts events in a short window. Ours says: an alert when one card makes more than 5 payments in 10 minutes. Larkfield’s payments have no card number, so we count per customer.
Take customer C-01234 of the first lesson, on 3 October 2025. Section 2 of the notebook counts, for each payment, the payments of the same card in the last 10 minutes, this one included:
card = evaluation[(evaluation.customer_id == "C-01234") & (evaluation.ts >= "2025-10-03") & (evaluation.ts < "2025-10-04")].set_index("ts")card["in_last_10_min"] = card.amount.rolling("600s").count().astype(int) # this payment and the ones before itcard["rule"] = card.in_last_10_min > 5 # more than 5 payments in 10 minutesprint(card[["amount", "country", "device_id", "in_last_10_min", "rule"]].to_string())rolling("600s") looks back 600 seconds from each row. Before you run it, predict: at which payment does the rule fire first?
amount country device_id in_last_10_min rulets2025-10-03 02:14:07 24.27 PT D-9f3e1a2 1 False2025-10-03 02:15:05 19.26 PT D-9f3e1a2 2 False2025-10-03 02:16:16 20.66 PT D-9f3e1a2 3 False2025-10-03 02:17:17 18.88 PT D-9f3e1a2 4 False2025-10-03 02:18:18 24.39 PT D-9f3e1a2 5 False2025-10-03 02:19:13 19.16 PT D-9f3e1a2 6 True2025-10-03 02:20:23 24.36 PT D-9f3e1a2 7 True2025-10-03 02:21:31 19.25 PT D-9f3e1a2 8 TrueThe rule fires at the sixth payment. Its reason is clear: “this card paid 6 times in 10 minutes”. But the first five payments were already done. A rule needs the pattern to happen before it can see it. Earlier warning needs a lower limit, and a lower limit sends more alerts on honest customers.
The device rule: one device, many cards
Section titled “The device rule: one device, many cards”Card testing has another shape. A thief has a list of stolen card numbers and tries them one by one, with tiny payments, on one device. A device is the phone or the computer that pays. Each card pays only once, so the velocity rule cannot see it. The device rule counts per device: an alert when one device makes more than 3 payments in 1 hour.
The card test of Kinds of anomalies shows the difference. On 18 October 2025, device D-606ea539 paid 45 times in 45.8 minutes, each time with another customer’s card, for 1.02 to 4.89 dollars:
- This device, last hour
- Limit: more than 3
Text description of the chart
The chart shows 45 dots, one for each payment of device D-606ea539 on 18 October 2025, over 45.8 minutes. The vertical axis is the number of payments of this device in the last hour, from 0 to 50. The dots climb in a straight line from 1 to 45, about one payment a minute. A dashed line at 3 marks the limit of the device rule. The fourth payment, after 3.1 minutes, is the first above it. From then on, every payment is above the limit.
The per-card count of this episode never goes above 1. MAD per customer put none of its 45 payments in the daily 50. The device rule put 33 of them in the daily 50. The other 12 waited, because the day had more rule alerts than review places.
The diagram shows the device rule as a flow. It also has two parts that the rest of the lesson adds: an allow-list, and free review places for MAD per customer.
Text description of the diagram
The diagram starts with a payment that arrives. A first question asks: is it a business account, on the allow-list? If yes, the rule sends no alert. If no, the next box counts the payments of this device in the last hour. A second question asks: more than 3? If yes, the rule sends an alert with its reason, “this device paid N times in 1 hour”. If no, there is no rule alert. Every payment without a rule alert is scored with MAD per customer. At the end, today’s 50 reviews take the rule alerts first, then the highest MAD scores.
The two rules in the comparison table
Section titled “The two rules in the comparison table”Section 3 of the notebook counts for every evaluation payment: per card in 10 minutes, and per device in 1 hour. A small function, count_in_window, does the rolling count of section 2 for all payments at once:
per_card_10min = count_in_window("customer_id", "600s") # the velocity rule counts per cardper_device_1h = count_in_window("device_id", "3600s") # the device rule counts per deviceA rule gives 1 (alert) or 0 (no alert). In the comparison table, a rule’s alerts of one day are reviewed in time order, at most 50. Before you run the cell, predict which rule has the higher precision at the budget.
velocity_rule = (per_card_10min > 5).astype(int)device_rule = (per_device_1h > 3).astype(int)table.add(compare.score_payments(evaluation, velocity_rule, "velocity rule", "more than 5 payments in 10 minutes", threshold=1, explain_minutes=0.5))table.add(compare.score_payments(evaluation, device_rule, "device rule", "a device with more than 3 payments in 1 hour", threshold=1, explain_minutes=0.5))table.show() detector setting alerts per day precision at budget recall cost per week ($) explain one alert (min) no detector nothing is reviewed 0.0 0.000 0.000 10367 NaNvelocity rule more than 5 payments in 10 minutes 22.7 0.133 0.097 9919 0.5 device rule a device with more than 3 payments in 1 hour 53.5 0.405 0.539 6956 0.5The first row is the baseline of every comparison: no detector. Nobody reviews anything, and the missed frauds cost 10,367 dollars a week. Every detector must beat it. The explain one alert column is an assumption: a rule’s reason takes about half a minute to read.
The device rule has a precision at the budget of 0.405. About 20 of its daily reviews are planted frauds. MAD per customer had 0.048. The chart puts every row of this module side by side, with the ceiling of a detector that knows the answers:
- Statistics on the amount
- Rules
- Knows the answers
Text description of the chart
A column chart with six columns, on an axis from 0 to 0.6. The global z-score has 0.003, and MAD per customer 0.048. The velocity rule has 0.133, the device rule 0.405, and the two rules ranked 0.427. The last column, a detector that knows the answers, has 0.542.
A good rule beats a statistic when it captures how the fraud works. The device rule describes card testing and many stolen cards almost exactly. The statistics describe the amount, which these frauds do not change.
What the rules flag that is not fraud
Section titled “What the rules flag that is not fraud”The rules also flag honest payments. Section 3 counts them:
velocity rule: 1,788 honest payments flagged, 100.0% businessdevice rule: 2,857 honest payments flagged, 98.8% businessThe answer key also tags the honest payments that look unusual (the notebook prints only the account type). 1,785 of the velocity rule’s 1,788 false alerts are business bulk orders. A business that buys for an office pays many times in a few minutes, on one computer. For both rules, that looks like fraud.
Ranked rules
Section titled “Ranked rules”A rule says yes or no. A score can say how far over the limit a payment is. Section 4 adds the two counts above their limits and reviews the highest first. The row two rules, ranked has a precision at the budget of 0.427. It finds 180 of the 306 planted episodes, against 169 for the device rule alone. A device that paid 40 times is a clearer case than one that paid 4 times.
Which frauds does each method find?
Section titled “Which frauds does each method find?”Precision says how many alerts are right. It does not say which frauds a method finds. Section 5 computes the recall of each method for each kind of planted fraud:
MAD per customer device rule velocity rulekindaccount_takeover 0.534 0.000 0.000card_testing 0.050 0.522 0.000stolen_card 0.033 0.619 0.163- Device rule
- MAD per customer
Text description of the chart
A column chart with three pairs of columns, on an axis from 0 to 0.7. Stolen cards: the device rule finds 0.619 of the planted payments, MAD per customer 0.033. Card tests: the device rule 0.522, MAD per customer 0.050. Account takeovers: the device rule 0, MAD per customer 0.534.
The two methods find different frauds. In an account takeover, a thief uses the customer’s own account for one or two large payments. There is no burst, so no count goes over a limit, and the device rule finds none of them. But the amount is far from the customer’s usual one, so MAD per customer finds about half. Nobody wrote a rule for “one large payment”: a rule misses what nobody wrote down.
One fraud each method misses
Section titled “One fraud each method misses”The completion check of this module asks for one planted fraud that the rule finds and the statistic misses, and one the other way round. Section 7 of the notebook finds them with the answer key, used only to score:
found by the device rule, missed by MAD per customer: ts customer_id amount country device_id device_1h usual robust_zpayment_idP-0173288 2025-09-01 02:31:21 C-18943 1.64 ES D-f80a1694 4.0 34.32 1.5found by MAD per customer, missed by the device rule: ts customer_id amount country device_id device_1h usual robust_zpayment_idP-0176568 2025-09-03 03:09:47 C-08404 280.62 CA D-dc05bfb 3.0 56.74 5.5P-0173288is a card test: 1.64 dollars in Spain at 02:31. The customer has no baseline, so the usual amount is the personal accounts’ median, 34.32. Its robust z-score is only 1.5. But its device paid 4 times in that hour, so the device rule fires.P-0176568is 280.62 dollars from a customer whose median is 56.74: a robust z-score of 5.5. Its device count is 3, not more than 3, so the device rule stays silent.
Across the evaluation months, the device rule reviewed 1,455 planted payments that MAD per customer missed. MAD per customer reviewed 156 that the device rule missed, 117 of them account takeovers.
Allow-lists and business limits
Section titled “Allow-lists and business limits”Rules come in a few common forms:
- A velocity rule or count: more than N events in a window, per card, per device, per account.
- A business limit: a value above a fixed amount needs a second look. At Larkfield, a refund above 100 dollars needs a team lead’s approval.
- An allow-list: a list of cases that a rule skips on purpose, because they are known to be honest.
Most of the device rule’s false alerts are business bulk orders. So put business accounts on an allow-list: they skip the device rule. Section 6 measures it:
personal = (evaluation.account_type == "personal").astype(int) # business accounts are on the allow-listdevice_personal = device_rule * personaldevice rule, personal accounts only business accounts on an allow-list 20.4 0.981 0.594 6243 0.5The rule now sends 20.4 alerts a day, and 98.1% of the reviewed ones are planted. The false alerts fall to 2.5 a week. The recall rises to 0.594, because the bulk orders no longer take the review places of the frauds.
Two warnings come with this number. First, the allow-list works so well because, in this synthetic data, honest bursts come almost only from business bulk orders. A real shop has more kinds of honest bursts. Second, an allow-list opens a door. In the evaluation months, 9.5% of the planted payments were on business accounts. A thief with a stolen business card now passes the device rule. Business limits have the same weak point: people learn them. Module 3 finds an agent whose refunds cluster just below the 100-dollar approval limit.
A rule with a statistic
Section titled “A rule with a statistic”With the allow-list, the team has about 30 free review places each day. Fill them with the highest MAD per customer scores. The rule’s alerts come first:
rz_customer is the MAD per customer score of Median, MAD and IQR. The notebook computes it again in section 5.
combined = device_personal * 1000 + rz_customer # rule alerts first, then the highest MAD scoresA rule alert gets a score above 1,000, far above any robust z-score, so it always comes first. Before you run it, predict: will the precision go up or down? And the cost?
device rule, then MAD per customer rule alerts first, MAD per customer fills the rest 50.0 0.389 0.628 4615 1.0The precision falls to 0.389, because the MAD alerts are less often fraud than the rule alerts. But the recall rises to 0.628, and the detector finds 228 of the 306 planted episodes. The cost falls to 4,615 dollars a week. That is the lowest cost in this module, 2,341 dollars a week less than the device rule alone. The rule and the statistic find different frauds, so together they find more.
Your turn
Section titled “Your turn”Section 8 of the notebook has three tasks.
- Compute the recall of the device rule on
card_testing. Write it ascard_testing_recall. - Change one condition: make the device rule stricter, more than 5 payments in 1 hour. Keep its row as
strict_row. Does the precision go up or down? And the recall? - Count the planted payments that MAD per customer reviewed and the device rule did not. Write it as
mad_only_count.
Show the answers
compare.recall_by_kind(evaluation, device_rule, threshold=1)["card_testing"]gives 0.522.strict_rule = (per_device_1h > 5).astype(int). It sends 37.6 alerts a day, with a precision at the budget of 0.450 and a recall of 0.475. A stricter limit makes the alerts more often right and finds fewer frauds. Every limit trades one against the other.int((found.mad & ~found.rule).sum())gives 156.
Common mistake
A rule that floods on a busy day
Symptom: a new count rule works well for weeks. Then, on a sale day or a launch day, it sends hundreds of alerts, and almost all of them are honest. The team cannot review them, and the real frauds of that day wait in the queue.
Cause: a rule’s limit is fixed. It does not know that a busy day has more payments everywhere. In Larkfield’s data, the device rule’s honest flags include 5 Black Friday payments and 6 from the new region. They are few here, because the rule counts per device, not per shop. A rule that counts per shop, such as “more than 200 payments in one hour”, would fire every Black Friday.
Fix: count per card, per device or per account, never per shop. Review the top of a ranked list, not every flag. Read the false alerts of each rule every week, and change the limit or the allow-list when normal changes. A change of normal has its own lessons in Module 3.
Consequence in AI work: a rule is a decision that a person made once. It does not learn. When the business changes, somebody must change the rule, or it fails without any error message.
When a rule is the right answer
Section titled “When a rule is the right answer”| Use a rule when | you know how a fraud works and can write it as a condition: a count in a window, a limit, a list. Also when the reason must be clear to a customer or an auditor, or when the decision must be instant. |
| Do not rely on a rule alone when | the fraud can take shapes that nobody has seen. A rule finds nothing it was not written for, like the account takeovers here. |
| What it costs | no training and half a minute to explain. But someone must keep it up to date, and thieves learn the limits. |
The comparison table of this module, with the measured rows. The last two rows are new in this lesson:
| Detector | Setting | Alerts/day | Precision at budget | Recall | Cost per week ($) | Episodes found |
|---|---|---|---|---|---|---|
| no detector | nothing is reviewed | 0.0 | 0.0 | 0.0 | 10,367 | 0 of 306 |
| z-score | |z| of the amount, one baseline for all | 50.0 | 0.003 | 0.005 | 10,264 | 8 of 306 |
| MAD per customer | |robust z| against the customer’s own median | 50.0 | 0.048 | 0.078 | 8,280 | 115 of 306 |
| velocity rule | more than 5 payments in 10 minutes | 22.7 | 0.133 | 0.097 | 9,919 | 97 of 306 |
| device rule | a device with more than 3 payments in 1 hour | 53.5 | 0.405 | 0.539 | 6,956 | 169 of 306 |
| two rules, ranked | payments over the limits, most over first | 53.5 | 0.427 | 0.568 | 6,954 | 180 of 306 |
| device rule, personal accounts only | business accounts on an allow-list | 20.4 | 0.981 | 0.594 | 6,243 | 173 of 306 |
| device rule, then MAD per customer | rule alerts first, MAD per customer fills the rest | 50.0 | 0.389 | 0.628 | 4,615 | 228 of 306 |
The rows of the global MAD and the IQR rule are in Median, MAD and IQR. The row of the two rules without ranking is in the notebook. These are the rows that the other modules must beat.
Knowledge check
Section titled “Knowledge check”Check your understanding
A card makes payments at 10:00, 10:02, 10:03, 10:05, 10:07, 10:08 and 10:20. The velocity rule is “more than 5 payments in 10 minutes”, counting the current payment and the ones before it. At which payment does it fire first? Does it fire at 10:20?
Show the answer
At 10:08, the last 10 minutes hold 10:00, 10:02, 10:03, 10:05, 10:07 and 10:08: six payments, more than 5. So it fires at the sixth payment, 10:08. At 10:20, the last 10 minutes (after 10:10) hold only that payment, so it does not fire.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
Predict and explain: the device rule has a precision at the budget of 0.405 and MAD per customer 0.048. Yet the device rule finds no account takeover, and MAD per customer finds about half of them. Why?
Show the answer
The device rule describes how card tests and stolen cards work: many payments on one device in a short time. Those are most of the planted payments, so its alerts are often right. An account takeover is one or two large payments from the customer’s own account, with no burst, so no count goes over the limit. MAD per customer looks at the amount against the customer’s usual one, so it sees the large payment. A rule finds only the shape that it describes.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
Practical: Sofia wants one detector for tomorrow, with 50 reviews a day. Using the comparison table, choose between the device rule, MAD per customer and the device rule followed by MAD per customer. Say why, and name one risk of your choice.
Show the answer
Choose the device rule with business accounts on an allow-list, followed by MAD per customer. It has the lowest cost: 4,615 dollars a week. The device rule alone costs 6,956, and MAD per customer 8,280. It finds 228 of 306 episodes, because the rule finds the bursts and MAD finds the large single payments. Risks: a thief with a stolen business card passes the allow-list, and the allow-list works so well partly because the data is synthetic. Read the false alerts every week.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Recap
- A rule is a condition written in advance: a count in a window, a business limit, an allow-list. It is instant, and its reason is one sentence.
- The device rule had a precision at the budget of 0.405, against 0.048 for MAD per customer. A good rule beats a statistic when it captures how the fraud works.
- A rule misses what nobody wrote down: the device rule found no account takeover. MAD per customer found about half.
- An allow-list removes known honest cases, and opens a door for a thief who looks like them.
- The rule followed by MAD per customer cost 4,615 dollars a week, the lowest in this module.
You can now build three kinds of simple baseline and judge them in one table. But every check so far looks at single payments. Some problems are only visible in a process over time, such as Larkfield’s refund rate, which rose by 0.3 points and stayed there. Next, you watch a whole process with control charts, and ask if its variation is normal.