Skip to content
Français

Rules, and when a rule is the right answer

  • Lesson 7
  • 35 min
  • Module 2: Simple baselines that are hard to beat

Where we are

Our robust scores find payments far from a customer's usual amount. But some fraud has a known shape, such as many payments in a few minutes. In this lesson, we write rules for these shapes, see what a rule does well and what it misses, and put the first rows in the comparison table.

Lesson 7 of 38 · Anomaly Detection: Find What Is UnusualRules, and when a rule is the right answer
Course overview

What you will be able to do

Write rules for the shape of a fraud, and score them like every other detector. Learn when a rule beats a statistic: when it captures how the fraud works. Also learn what a rule cannot do: find a fraud that nobody imagined.

NotebookDo this lesson in a notebookColab · Kaggle · GitHub

The notebook has every code cell, task and check of this lesson. Its setup cells install pandas, NumPy, SciPy, scikit-learn and matplotlib, and download the payments (about 7 MB). About 30 minutes.

Colab and Kaggle open your own copy. They need a free Google or Kaggle account. Stuck? See the solution notebook.

Before you startWhat to know and to have readyAbout 35 minutes

What you need to know

What you need on your computer

The same as in the last lesson: Colab or Kaggle in a browser, or Python 3.12 with the notebook kernel.

Tested with Python 3.12.13 in Jupyter on macOS, with pandas 3.1.1 and NumPy 2.5.4. Colab, Kaggle and Windows were not tested.

The bank that calls you after three payments

Section titled “The bank that calls you after three payments”

Many people know this situation. You are on holiday, and you pay for a taxi, a coffee and a museum ticket within a few minutes. Then your bank sends a message: “Was this you?” Nobody at the bank looked at your payments. A rule did: “many payments on one card in a short time, in a new country”. Somebody wrote it down after they saw many thieves do exactly that.

A rule is a condition that a person writes in advance. When a payment meets the condition, the rule sends an alert. It needs no training data and no statistics, and it gives its reason in one sentence. That is its strength. Its weakness is the other side of the same thing: it finds only what somebody imagined.

The problem: frauds that are not about the amount

Section titled “The problem: frauds that are not about the amount”

The last lesson ended with MAD per customer at a precision at the budget of 0.048. Only about 2 of the 50 daily alerts are planted frauds. The reason is in the data. In the evaluation months, the median planted payment is 14.31 dollars. Card tests are a few dollars; stolen cards buy ordinary baskets. Their amount is often normal. What is not normal is their shape: many payments, very fast, often on one device.

So we need a different kind of check. It does not ask “is this amount unusual?”. It asks “does this look like the way a known fraud works?”.

A velocity rule counts events in a short window. Ours says: an alert when one card makes more than 5 payments in 10 minutes. Larkfield’s payments have no card number, so we count per customer.

Take customer C-01234 of the first lesson, on 3 October 2025. Section 2 of the notebook counts, for each payment, the payments of the same card in the last 10 minutes, this one included:

card = evaluation[(evaluation.customer_id == "C-01234") & (evaluation.ts >= "2025-10-03")
& (evaluation.ts < "2025-10-04")].set_index("ts")
card["in_last_10_min"] = card.amount.rolling("600s").count().astype(int) # this payment and the ones before it
card["rule"] = card.in_last_10_min > 5 # more than 5 payments in 10 minutes
print(card[["amount", "country", "device_id", "in_last_10_min", "rule"]].to_string())

rolling("600s") looks back 600 seconds from each row. Before you run it, predict: at which payment does the rule fire first?

amount country device_id in_last_10_min rule
ts
2025-10-03 02:14:07 24.27 PT D-9f3e1a2 1 False
2025-10-03 02:15:05 19.26 PT D-9f3e1a2 2 False
2025-10-03 02:16:16 20.66 PT D-9f3e1a2 3 False
2025-10-03 02:17:17 18.88 PT D-9f3e1a2 4 False
2025-10-03 02:18:18 24.39 PT D-9f3e1a2 5 False
2025-10-03 02:19:13 19.16 PT D-9f3e1a2 6 True
2025-10-03 02:20:23 24.36 PT D-9f3e1a2 7 True
2025-10-03 02:21:31 19.25 PT D-9f3e1a2 8 True

The rule fires at the sixth payment. Its reason is clear: “this card paid 6 times in 10 minutes”. But the first five payments were already done. A rule needs the pattern to happen before it can see it. Earlier warning needs a lower limit, and a lower limit sends more alerts on honest customers.

Card testing has another shape. A thief has a list of stolen card numbers and tries them one by one, with tiny payments, on one device. A device is the phone or the computer that pays. Each card pays only once, so the velocity rule cannot see it. The device rule counts per device: an alert when one device makes more than 3 payments in 1 hour.

The card test of Kinds of anomalies shows the difference. On 18 October 2025, device D-606ea539 paid 45 times in 45.8 minutes, each time with another customer’s card, for 1.02 to 4.89 dollars:

  • This device, last hour
  • Limit: more than 3
A planted card test on 18 October 2025 from 04:54: one device, 45 customers' cards, 45.8 minutes. The device rule fires at the 4th payment. Each card paid only once, so the velocity rule (per card) never fires, and the tiny amounts give MAD per customer nothing to see.
Text description of the chart

The chart shows 45 dots, one for each payment of device D-606ea539 on 18 October 2025, over 45.8 minutes. The vertical axis is the number of payments of this device in the last hour, from 0 to 50. The dots climb in a straight line from 1 to 45, about one payment a minute. A dashed line at 3 marks the limit of the device rule. The fourth payment, after 3.1 minutes, is the first above it. From then on, every payment is above the limit.

The per-card count of this episode never goes above 1. MAD per customer put none of its 45 payments in the daily 50. The device rule put 33 of them in the daily 50. The other 12 waited, because the day had more rule alerts than review places.

The diagram shows the device rule as a flow. It also has two parts that the rest of the lesson adds: an allow-list, and free review places for MAD per customer.

A flow diagram. A payment arrives. Business account, on the allow-list? Yes: no rule alert. No: count this device's payments in the last hour. More than 3? Yes: alert, with the reason 'this device paid N times in 1 hour'. No: no rule alert. A payment without a rule alert is scored with MAD per customer. Today's 50 reviews take the rule alerts first, then the highest MAD scores.
The device rule with its allow-list. Rule alerts go to review first; the free places of the 50 a day go to the highest MAD per customer scores.
Text description of the diagram

The diagram starts with a payment that arrives. A first question asks: is it a business account, on the allow-list? If yes, the rule sends no alert. If no, the next box counts the payments of this device in the last hour. A second question asks: more than 3? If yes, the rule sends an alert with its reason, “this device paid N times in 1 hour”. If no, there is no rule alert. Every payment without a rule alert is scored with MAD per customer. At the end, today’s 50 reviews take the rule alerts first, then the highest MAD scores.

Section 3 of the notebook counts for every evaluation payment: per card in 10 minutes, and per device in 1 hour. A small function, count_in_window, does the rolling count of section 2 for all payments at once:

per_card_10min = count_in_window("customer_id", "600s") # the velocity rule counts per card
per_device_1h = count_in_window("device_id", "3600s") # the device rule counts per device

A rule gives 1 (alert) or 0 (no alert). In the comparison table, a rule’s alerts of one day are reviewed in time order, at most 50. Before you run the cell, predict which rule has the higher precision at the budget.

velocity_rule = (per_card_10min > 5).astype(int)
device_rule = (per_device_1h > 3).astype(int)
table.add(compare.score_payments(evaluation, velocity_rule, "velocity rule", "more than 5 payments in 10 minutes",
threshold=1, explain_minutes=0.5))
table.add(compare.score_payments(evaluation, device_rule, "device rule", "a device with more than 3 payments in 1 hour",
threshold=1, explain_minutes=0.5))
table.show()
detector setting alerts per day precision at budget recall cost per week ($) explain one alert (min)
no detector nothing is reviewed 0.0 0.000 0.000 10367 NaN
velocity rule more than 5 payments in 10 minutes 22.7 0.133 0.097 9919 0.5
device rule a device with more than 3 payments in 1 hour 53.5 0.405 0.539 6956 0.5

The first row is the baseline of every comparison: no detector. Nobody reviews anything, and the missed frauds cost 10,367 dollars a week. Every detector must beat it. The explain one alert column is an assumption: a rule’s reason takes about half a minute to read.

The device rule has a precision at the budget of 0.405. About 20 of its daily reviews are planted frauds. MAD per customer had 0.048. The chart puts every row of this module side by side, with the ceiling of a detector that knows the answers:

  • Statistics on the amount
  • Rules
  • Knows the answers
Precision at the budget, evaluation months, from the comparison table. z: the global z-score. MAD: MAD per customer. Ranked: the two rules, ranked. Statistics on the amount stay below 0.05; the device rule reaches 0.405. A detector that knows the answers gets 0.542, because many days have fewer than 50 planted payments.
Text description of the chart

A column chart with six columns, on an axis from 0 to 0.6. The global z-score has 0.003, and MAD per customer 0.048. The velocity rule has 0.133, the device rule 0.405, and the two rules ranked 0.427. The last column, a detector that knows the answers, has 0.542.

A good rule beats a statistic when it captures how the fraud works. The device rule describes card testing and many stolen cards almost exactly. The statistics describe the amount, which these frauds do not change.

The rules also flag honest payments. Section 3 counts them:

velocity rule: 1,788 honest payments flagged, 100.0% business
device rule: 2,857 honest payments flagged, 98.8% business

The answer key also tags the honest payments that look unusual (the notebook prints only the account type). 1,785 of the velocity rule’s 1,788 false alerts are business bulk orders. A business that buys for an office pays many times in a few minutes, on one computer. For both rules, that looks like fraud.

A rule says yes or no. A score can say how far over the limit a payment is. Section 4 adds the two counts above their limits and reviews the highest first. The row two rules, ranked has a precision at the budget of 0.427. It finds 180 of the 306 planted episodes, against 169 for the device rule alone. A device that paid 40 times is a clearer case than one that paid 4 times.

Precision says how many alerts are right. It does not say which frauds a method finds. Section 5 computes the recall of each method for each kind of planted fraud:

MAD per customer device rule velocity rule
kind
account_takeover 0.534 0.000 0.000
card_testing 0.050 0.522 0.000
stolen_card 0.033 0.619 0.163
  • Device rule
  • MAD per customer
Recall by kind of planted fraud, evaluation months. The device rule finds stolen cards (0.619) and card tests (0.522) and no account takeover. MAD per customer finds account takeovers (0.534) and almost nothing else.
Text description of the chart

A column chart with three pairs of columns, on an axis from 0 to 0.7. Stolen cards: the device rule finds 0.619 of the planted payments, MAD per customer 0.033. Card tests: the device rule 0.522, MAD per customer 0.050. Account takeovers: the device rule 0, MAD per customer 0.534.

The two methods find different frauds. In an account takeover, a thief uses the customer’s own account for one or two large payments. There is no burst, so no count goes over a limit, and the device rule finds none of them. But the amount is far from the customer’s usual one, so MAD per customer finds about half. Nobody wrote a rule for “one large payment”: a rule misses what nobody wrote down.

The completion check of this module asks for one planted fraud that the rule finds and the statistic misses, and one the other way round. Section 7 of the notebook finds them with the answer key, used only to score:

found by the device rule, missed by MAD per customer:
ts customer_id amount country device_id device_1h usual robust_z
payment_id
P-0173288 2025-09-01 02:31:21 C-18943 1.64 ES D-f80a1694 4.0 34.32 1.5
found by MAD per customer, missed by the device rule:
ts customer_id amount country device_id device_1h usual robust_z
payment_id
P-0176568 2025-09-03 03:09:47 C-08404 280.62 CA D-dc05bfb 3.0 56.74 5.5
  • P-0173288 is a card test: 1.64 dollars in Spain at 02:31. The customer has no baseline, so the usual amount is the personal accounts’ median, 34.32. Its robust z-score is only 1.5. But its device paid 4 times in that hour, so the device rule fires.
  • P-0176568 is 280.62 dollars from a customer whose median is 56.74: a robust z-score of 5.5. Its device count is 3, not more than 3, so the device rule stays silent.

Across the evaluation months, the device rule reviewed 1,455 planted payments that MAD per customer missed. MAD per customer reviewed 156 that the device rule missed, 117 of them account takeovers.

Rules come in a few common forms:

  • A velocity rule or count: more than N events in a window, per card, per device, per account.
  • A business limit: a value above a fixed amount needs a second look. At Larkfield, a refund above 100 dollars needs a team lead’s approval.
  • An allow-list: a list of cases that a rule skips on purpose, because they are known to be honest.

Most of the device rule’s false alerts are business bulk orders. So put business accounts on an allow-list: they skip the device rule. Section 6 measures it:

personal = (evaluation.account_type == "personal").astype(int) # business accounts are on the allow-list
device_personal = device_rule * personal
device rule, personal accounts only business accounts on an allow-list 20.4 0.981 0.594 6243 0.5

The rule now sends 20.4 alerts a day, and 98.1% of the reviewed ones are planted. The false alerts fall to 2.5 a week. The recall rises to 0.594, because the bulk orders no longer take the review places of the frauds.

Two warnings come with this number. First, the allow-list works so well because, in this synthetic data, honest bursts come almost only from business bulk orders. A real shop has more kinds of honest bursts. Second, an allow-list opens a door. In the evaluation months, 9.5% of the planted payments were on business accounts. A thief with a stolen business card now passes the device rule. Business limits have the same weak point: people learn them. Module 3 finds an agent whose refunds cluster just below the 100-dollar approval limit.

With the allow-list, the team has about 30 free review places each day. Fill them with the highest MAD per customer scores. The rule’s alerts come first:

rz_customer is the MAD per customer score of Median, MAD and IQR. The notebook computes it again in section 5.

combined = device_personal * 1000 + rz_customer # rule alerts first, then the highest MAD scores

A rule alert gets a score above 1,000, far above any robust z-score, so it always comes first. Before you run it, predict: will the precision go up or down? And the cost?

device rule, then MAD per customer rule alerts first, MAD per customer fills the rest 50.0 0.389 0.628 4615 1.0

The precision falls to 0.389, because the MAD alerts are less often fraud than the rule alerts. But the recall rises to 0.628, and the detector finds 228 of the 306 planted episodes. The cost falls to 4,615 dollars a week. That is the lowest cost in this module, 2,341 dollars a week less than the device rule alone. The rule and the statistic find different frauds, so together they find more.

Section 8 of the notebook has three tasks.

  1. Compute the recall of the device rule on card_testing. Write it as card_testing_recall.
  2. Change one condition: make the device rule stricter, more than 5 payments in 1 hour. Keep its row as strict_row. Does the precision go up or down? And the recall?
  3. Count the planted payments that MAD per customer reviewed and the device rule did not. Write it as mad_only_count.
Show the answers
  1. compare.recall_by_kind(evaluation, device_rule, threshold=1)["card_testing"] gives 0.522.
  2. strict_rule = (per_device_1h > 5).astype(int). It sends 37.6 alerts a day, with a precision at the budget of 0.450 and a recall of 0.475. A stricter limit makes the alerts more often right and finds fewer frauds. Every limit trades one against the other.
  3. int((found.mad & ~found.rule).sum()) gives 156.

Common mistake

A rule that floods on a busy day

Symptom: a new count rule works well for weeks. Then, on a sale day or a launch day, it sends hundreds of alerts, and almost all of them are honest. The team cannot review them, and the real frauds of that day wait in the queue.

Cause: a rule’s limit is fixed. It does not know that a busy day has more payments everywhere. In Larkfield’s data, the device rule’s honest flags include 5 Black Friday payments and 6 from the new region. They are few here, because the rule counts per device, not per shop. A rule that counts per shop, such as “more than 200 payments in one hour”, would fire every Black Friday.

Fix: count per card, per device or per account, never per shop. Review the top of a ranked list, not every flag. Read the false alerts of each rule every week, and change the limit or the allow-list when normal changes. A change of normal has its own lessons in Module 3.

Consequence in AI work: a rule is a decision that a person made once. It does not learn. When the business changes, somebody must change the rule, or it fails without any error message.

Use a rule when you know how a fraud works and can write it as a condition: a count in a window, a limit, a list. Also when the reason must be clear to a customer or an auditor, or when the decision must be instant.
Do not rely on a rule alone when the fraud can take shapes that nobody has seen. A rule finds nothing it was not written for, like the account takeovers here.
What it costs no training and half a minute to explain. But someone must keep it up to date, and thieves learn the limits.

The comparison table of this module, with the measured rows. The last two rows are new in this lesson:

Detector Setting Alerts/day Precision at budget Recall Cost per week ($) Episodes found
no detector nothing is reviewed 0.0 0.0 0.0 10,367 0 of 306
z-score |z| of the amount, one baseline for all 50.0 0.003 0.005 10,264 8 of 306
MAD per customer |robust z| against the customer’s own median 50.0 0.048 0.078 8,280 115 of 306
velocity rule more than 5 payments in 10 minutes 22.7 0.133 0.097 9,919 97 of 306
device rule a device with more than 3 payments in 1 hour 53.5 0.405 0.539 6,956 169 of 306
two rules, ranked payments over the limits, most over first 53.5 0.427 0.568 6,954 180 of 306
device rule, personal accounts only business accounts on an allow-list 20.4 0.981 0.594 6,243 173 of 306
device rule, then MAD per customer rule alerts first, MAD per customer fills the rest 50.0 0.389 0.628 4,615 228 of 306

The rows of the global MAD and the IQR rule are in Median, MAD and IQR. The row of the two rules without ranking is in the notebook. These are the rows that the other modules must beat.

Check your understanding

A card makes payments at 10:00, 10:02, 10:03, 10:05, 10:07, 10:08 and 10:20. The velocity rule is “more than 5 payments in 10 minutes”, counting the current payment and the ones before it. At which payment does it fire first? Does it fire at 10:20?

Show the answer

At 10:08, the last 10 minutes hold 10:00, 10:02, 10:03, 10:05, 10:07 and 10:08: six payments, more than 5. So it fires at the sixth payment, 10:08. At 10:20, the last 10 minutes (after 10:10) hold only that payment, so it does not fire.

Check your understanding

Predict and explain: the device rule has a precision at the budget of 0.405 and MAD per customer 0.048. Yet the device rule finds no account takeover, and MAD per customer finds about half of them. Why?

Show the answer

The device rule describes how card tests and stolen cards work: many payments on one device in a short time. Those are most of the planted payments, so its alerts are often right. An account takeover is one or two large payments from the customer’s own account, with no burst, so no count goes over the limit. MAD per customer looks at the amount against the customer’s usual one, so it sees the large payment. A rule finds only the shape that it describes.

Check your understanding

Practical: Sofia wants one detector for tomorrow, with 50 reviews a day. Using the comparison table, choose between the device rule, MAD per customer and the device rule followed by MAD per customer. Say why, and name one risk of your choice.

Show the answer

Choose the device rule with business accounts on an allow-list, followed by MAD per customer. It has the lowest cost: 4,615 dollars a week. The device rule alone costs 6,956, and MAD per customer 8,280. It finds 228 of 306 episodes, because the rule finds the bursts and MAD finds the large single payments. Risks: a thief with a stolen business card passes the allow-list, and the allow-list works so well partly because the data is synthetic. Read the false alerts every week.

Recap

  • A rule is a condition written in advance: a count in a window, a business limit, an allow-list. It is instant, and its reason is one sentence.
  • The device rule had a precision at the budget of 0.405, against 0.048 for MAD per customer. A good rule beats a statistic when it captures how the fraud works.
  • A rule misses what nobody wrote down: the device rule found no account takeover. MAD per customer found about half.
  • An allow-list removes known honest cases, and opens a door for a thief who looks like them.
  • The rule followed by MAD per customer cost 4,615 dollars a week, the lowest in this module.

You can now build three kinds of simple baseline and judge them in one table. But every check so far looks at single payments. Some problems are only visible in a process over time, such as Larkfield’s refund rate, which rose by 0.3 points and stayed there. Next, you watch a whole process with control charts, and ask if its variation is normal.

Next: Control charts: is this variation normal?