Alert fatigue and ownership
- Lesson 38
- 35 min
- Module 9: Run a detector in production
Where we are
The job now follows the budget and survives the launch. People still stop reading alerts that never lead to action. In this lesson, we add rate limits, snooze and suppression rules, a first step in a runbook, and a weekly review of the alerts that nobody acted on.
Lesson 38 of 38 · Anomaly Detection: Find What Is UnusualAlert fatigue and ownership
1. What "unusual" means
2. Simple baselines that are hard to beat
3. Statistical process control and distribution drift
4. Distance, density and isolation
5. Anomalies in time
6. Deep and embedding methods
7. Evaluate with few labels
8. Learn from the reviewers
9. Run a detector in production
What you will be able to do
Keep the alerts worth reading. You choose a daily limit for grouped alerts and add a snooze rule after a reviewer’s answer. Each alert gets a first step. A weekly review looks at what was sent, what was real and what nobody read.
Before you startWhat to know and to have ready
What you need to know
- The detection job, its ledger and its two policies, from From score to alert and When “normal” moves. This lesson continues in the same folder.
- What one grouped alert is, from The alert budget.
What you need on your computer
The project folder detection-job from the last lesson, with the retrained model and the alerts up to 14 November 2025. Tested on macOS with Python 3.12.13. The PowerShell commands were not run.
The car alarm in the street
Section titled “The car alarm in the street”Think of a car alarm in a city street. The first time it rings, people look out of their windows. It was the wind. The second time, a cat. By the tenth time, nobody looks up. Then one night a real thief opens the car, the alarm rings, and nobody looks up.
The alarm did its job every time. The people stopped doing theirs, because the alarm taught them that it means nothing. This is alert fatigue: people stop reading alerts with care when too many of them lead to nothing. It is not a weakness of the reviewers. It is a property of the alerts.
The problem: 50 grouped alerts are mostly noise
Section titled “The problem: 50 grouped alerts are mostly noise”In the first lesson of this module, the job learned to group payments. One alert holds a whole event: all the payments of one customer on one day, or of one device that paid with many customers’ cards. That was a good change. But it changed what 50 alerts mean.
The course’s measurement replayed October 2025 with four daily budgets, and counted two things. First, the share of alerts that hold at least one planted fraud. Second, the share of all planted payments that are inside some alert:
- Alerts that hold a planted fraud
- Planted payments inside the alerts
Text description of the chart
A line chart with the daily budget across (5, 10, 20 and 50 grouped alerts a day) and a share from 0 to 100% up. One line is the share of alerts that hold a planted fraud: 56.1% at 5 a day, 31.9% at 10, 17.1% at 20 and 7.4% at 50. The other line is the share of planted payments inside the alerts: 90.3% at 5, 93.8% at 10, 95.4% at 20 and 96.4% at 50. The first line falls fast; the second line is almost flat.
Read the two lines together. Five alerts a day already hold 90.3% of the planted payments, because one group holds a whole fraud. Going from 5 to 50 alerts adds only 6 points of coverage. But it adds 45 alerts a day, and almost all of them are honest customers. At 50 a day, more than 9 alerts in 10 lead to nothing.
The weekly report of that replay shows the same thing. The week of 20 to 26 October, with reviewers who always answer right (an assumption: real reviewers make mistakes). The middle days are cut here:
Alert report, week 2025-10-20 to 2025-10-26
day sent snoozed real not unread2025-10-20 50 0 4 46 0...2025-10-26 50 0 4 46 0total 350 0 28 322 0
Precision of the reviewed alerts: 0.08 (28 of 350).Main reason of the alerts: new_device 170, night 121, payments_last_hour 37, new_country 21.28 of 350 alerts were real. The team read 322 alerts that led to nothing. The main reason of almost half of the alerts was new_device: the app launch again. That is the week before the drift check of 31 October asked for retraining.
This is not a bug. The budget of 50 was set in Module 7 for single payments. When the job started to send groups, nobody chose the budget again.
The idea: make each alert worth reading
Section titled “The idea: make each alert worth reading”There is no single fix. A team uses four tools together:
- A rate limit caps how many alerts the job sends in a period. The daily budget is one. A limit can also be per customer or per device.
- A snooze rule holds a customer’s alerts for some days after a reviewer said “not fraud”. The same honest change would most likely alert again tomorrow.
- A suppression rule stops alerts for a known, harmless pattern. For example, a business customer’s monthly bulk order. Module 2 showed that 1,785 of the velocity rule’s 1,788 false alerts were business bulk orders.
- A runbook first step in every alert, so the reviewer knows what to do in the first minute. The job adds it from the alert’s first reason.
And one habit holds them together: a weekly review with the people who own the alerts. They look at what was sent, what was real, and what nobody read.
Text description of the diagram
The diagram is a loop. It starts with the job, which sends at most the budget of alerts each day. A reviewer answers each alert with the command review, real or not. If the answer is not fraud, the snooze rule holds that customer’s alerts for 7 days, and this goes back to the job. If the answer is fraud, the reviewer acts, with the runbook’s first step. Alerts that have no answer after 2 days count as unread. The reviews and the unread alerts go into the weekly report. It counts the sent, snoozed, real, not and unread alerts, and gives the main reasons. The report goes to the weekly review. There, the team can change the budget, a reason, a rule or a runbook step. A dashed arrow goes back to the job.
A rate limit for grouped alerts
Section titled “A rate limit for grouped alerts”The cheapest fix is to choose the budget again. A budget of 10 grouped alerts a day held 93.8% of the planted payments in October, and 31.9% of its alerts were real. At 4 minutes an alert (the course’s assumption), 50 alerts take 200 minutes of the team’s day, and 10 take 40.
What does Sofia do with the 160 free minutes? She does not send them home. She uses them where alerts cannot help: a deeper look at each real alert and its customer, and calls to the customers. Or she keeps a budget of 20 for a while, and compares.
The setting DETECT_BUDGET changes the limit without a change to the code. The training also uses it: the fixed threshold is the score that gave budget alerts a day in the training days. So train again with the new budget.
Snooze, and what it cannot do
Section titled “Snooze, and what it cannot do”The job’s snooze rule is in detect/suppress.py. After a reviewer marks an alert “not” (not fraud), every customer of that alert is snoozed for 7 days (snooze_days in config.py). Two details matter:
- A device alert is never snoozed. A device that paid with many customers’ cards is not one honest customer.
- A snoozed alert is not deleted. The job writes it to
state/snoozed.jsonl, and the weekly report counts it.
How much does snooze help at Larkfield? The course’s measurement tested it. Reviewers who always answer right labelled all 1,561 October alerts. Then the first week of November ran with the rule. 5 alerts were snoozed, and the share of real alerts stayed at 0.066. So snooze helped very little here. Most honest customers in the alerts do not come back the next day. They are new each day: a new phone today, a bulk order tomorrow from someone else.
That is an honest result, not a failure. Snooze helps when the same honest customer alerts again and again, as Module 7 showed for business customers in Slices and fairness. It also has a risk: if a fraudster takes over an account after a “not fraud” review, the snooze hides that account for 7 days. Keep the snooze short, never snooze devices, and count the snoozed alerts.
Suppression rules: written down, with an owner
Section titled “Suppression rules: written down, with an owner”A suppression rule is stronger than a snooze: it stops a kind of alert for everyone, until someone removes it. “No alert for a business customer’s payment if it matches a standing bulk order” is an example. Each rule needs three things: the reason in one sentence, an owner, and an end date or a review date. A rule without an owner stays forever. A fraudster who learns the rule can use it as a door.
The course’s job has no suppression rule. That is on purpose: you have not seen a pattern yet that is harmless and stable enough. The weekly review is where such a pattern appears.
The runbook first step
Section titled “The runbook first step”Each alert carries a first step, chosen by its first reason. The job’s table, in detect/alerts.py:
| First reason | First step |
|---|---|
| many payments on one device in an hour | Block the device for 24 hours, then check its other payments of today. |
| a new device | Ask the customer (app message or call) if they used a new device today. |
| a new country | Check if the customer is travelling (earlier payments, the delivery address). |
| amount far above the customer’s usual | Compare the order with the customer’s earlier orders and the delivery address. |
| many payments by the customer in an hour | Look at the customer’s payments of the last hour together. |
| anything else, or no reason | Read the payments of the alert and the customer’s last month. |
A first step does not replace the reviewer’s judgement. It saves the first minute and makes two reviewers start in the same place. The full runbook, with what to do next and when to call someone, belongs to the team. The MLOps course shows how to write one in Runbooks.
Review, snooze and report
Section titled “Review, snooze and report”You review two alerts of 14 November, run the next day, and read the weekly report.
-
Look at the first alerts of 14 November:
Terminal window head -n 3 alerts/2025-11-14.jsonl | cut -c 1-120Terminal window Get-Content alerts\2025-11-14.jsonl -TotalCount 3 | ForEach-Object { $_.Substring(0, 120) }{"alert_id": "3e748a2f3c0aa7f3", "day": "2025-11-14", "group": "customer", "key": "C-00937", "customers": ["C-00937"], "{"alert_id": "436bdd63ad7f6085", "day": "2025-11-14", "group": "customer", "key": "C-10208", "customers": ["C-10208"], "{"alert_id": "e1a0a39b879a4b69", "day": "2025-11-14", "group": "customer", "key": "C-22827", "customers": ["C-22827"], " -
You play Sofia’s team and check two alerts. Customer
C-00937has six payments from a new country at night: a stolen card that we planted. CustomerC-13832has one payment on a new device. Her alert is further down the file:grep C-13832 alerts/2025-11-14.jsonlshows it (in PowerShell,Select-String C-13832 alerts\2025-11-14.jsonl). In this practice, she confirms in the app that it is her new phone; the answer key agrees, nothing is planted for her. Save both answers:Terminal window python -m detect review 3e748a2f3c0aa7f3 realpython -m detect review 3fec7c204496c223 notSaved: alert 3e748a2f3c0aa7f3 of 2025-11-14 is real.Saved: alert 3fec7c204496c223 of 2025-11-14 is not. -
Run 15 November:
Terminal window python -m detect run --day 2025-11-152025-11-15: 2,038 payments, 52 customers above the training threshold, 100 alert groups, 1 snoozed. Sent 50 new alerts to alerts/2025-11-15.jsonl; 0 already sent.Read
1 snoozed.C-13832came back on 15 November among the most unusual groups, and the rule held her alert. The job still sent 50 alerts: the next group took her place. -
Write the weekly report for the week that ends on 15 November:
Terminal window python -m detect report --week-ending 2025-11-15Alert report, week 2025-11-09 to 2025-11-15day sent snoozed real not unread2025-11-09 50 0 0 0 502025-11-10 50 0 0 0 502025-11-11 50 0 0 0 502025-11-12 50 0 0 0 502025-11-13 50 0 0 0 502025-11-14 50 0 1 1 482025-11-15 50 1 0 0 50total 350 1 1 1 348Precision of the reviewed alerts: 0.50 (1 of 2).Main reason of the alerts: new_device 107, night 104, payments_last_hour 44, new_country 29.Alerts nobody read after 2 days: 250. Ask why in the weekly review: too many alerts, or alerts that never lead to action?
In this practice, you reviewed 2 of 350 alerts, so 348 are unread. 250 of them are older than 2 days. In a real team, that line is the most important one of the report. Alerts that nobody reads are alerts that nobody needs, or alerts that came too late, or a team with too much work. The report cannot tell which. The weekly review must ask.
The precision line, 0.50, is from 2 reviews only, so it says almost nothing. Module 7’s lesson Evaluate without a full answer key showed how many reviews an estimate needs, and why the reviewed alerts are not a random sample.
Who owns what
Section titled “Who owns what”An alert that belongs to everyone belongs to no one. Larkfield writes the owners down, next to the job:
| What | Owner | What the owner decides |
|---|---|---|
| The alerts, the budget, the snooze rule | Sofia (payments risk) | how many alerts a day, approval of a new model or rule |
| The model, the features, the drift check | Priya (data science) | when to retrain, which window, the reasons’ text |
| The payments feed that the job reads | Mei (data engineering) | what happens when the feed is late or broken |
| The cost of a miss and of a false alert | Sofia (payments risk), with Omar (product) | the money values behind the budget |
The owner is the person who is called. If the job stops with “No payments on” a day, Mei is called, not Sofia. A broken feed is a data fault, as Module 3 showed for the refunds feed. The MLOps course has the general method in Ownership and support.
When to use these tools, and what they cost
Section titled “When to use these tools, and what they cost”Use a lower rate limit first: it costs nothing and saves the most time. Choose the budget from the measured curve, not from habit. Here, 10 to 20 grouped alerts a day keep more than 93% of the planted payments in the alerts.
Use snooze when the same honest customers alert again and again. It costs a little risk: a few days of blindness for a reviewed customer. Use a suppression rule only for a stable, harmless pattern, with an owner and a review date. It costs the most risk.
The weekly review costs an hour of several people’s time each week. Only the review finds what the other tools cannot. It finds reasons that do not help, runbook steps that nobody follows, and alerts that nobody reads.
Your turn: a budget of 10
Section titled “Your turn: a budget of 10”Run the job for the week of 8 to 14 November with a budget of 10, in its own folders. Train with the same budget, so that the fixed threshold matches it. Then write that week’s report. Before you run it, predict: will new_device still be the main reason of the alerts?
export DETECT_BUDGET=10 DETECT_STATE=state-10 DETECT_ALERTS=alerts-10python -m detect train --until 2025-11-07python -m detect replay --from 2025-11-08 --to 2025-11-14python -m detect report --week-ending 2025-11-14unset DETECT_BUDGET DETECT_STATE DETECT_ALERTS$env:DETECT_BUDGET = "10"; $env:DETECT_STATE = "state-10"; $env:DETECT_ALERTS = "alerts-10"python -m detect train --until 2025-11-07python -m detect replay --from 2025-11-08 --to 2025-11-14python -m detect report --week-ending 2025-11-14Remove-Item Env:DETECT_BUDGET, Env:DETECT_STATE, Env:DETECT_ALERTSShow the answer
The replay ends with Sent 70 alerts in 7 days (10.0 a day, budget 10, policy budget). The report’s last lines:
total 70 0 0 0 70
Main reason of the alerts: payments_last_hour 35, new_country 15, night 14, new_device 5.Alerts nobody read after 2 days: 50. Ask why in the weekly review: too many alerts, or alerts that never lead to action?No: new_device is the main reason of only 5 alerts, against 107 in the week with a budget of 50. The 10 most unusual groups of each day are mostly bursts of payments and new countries, which are the signs of a stolen card. The honest new phones were at the bottom of the 50, and a smaller budget cuts them first. In October, a budget of 10 held 93.8% of the planted payments, against 96.4% for a budget of 50. Write both numbers in your note to Sofia, with the review time: 40 minutes a day instead of 200.
Common mistake
Nobody reads the alerts any more
Symptom: after a month, the report says “Alerts nobody read after 2 days: 250” every week. Sofia’s team reads the first alerts of each morning and stops. A stolen card in alert number 31 gets through.
Cause: the budget of 50 was chosen for single payments. With grouped alerts, 50 a day are mostly honest customers: only 7.4% held a planted fraud in October. The team learned that most alerts lead to nothing.
Fix: choose the budget again from the measured curve, for example 10 grouped alerts a day (31.9% real, 93.8% of the planted payments). Keep the snooze rule short. Look at the unread line in every weekly review, and ask the owner why.
Consequence in AI work: a detector is only as good as the people who read its alerts. Alert fatigue turns a good detector into noise. The cause is the number of useless alerts, not the reviewers.
Knowledge check
Section titled “Knowledge check”Check your understanding
In October, 5 grouped alerts a day held 90.3% of the planted payments, and 50 a day held 96.4%. Explain why 45 more alerts add so little, and what that means for the budget.
Show the answer
One grouped alert holds a whole event: all the payments of one customer or one device. The most unusual groups are mostly the big frauds, so a few alerts already hold most planted payments. The next 45 alerts are mostly honest customers: only 7.4% of the alerts were real at 50 a day, against 56.1% at 5. So the budget of 50, chosen for single payments, is far too high for groups. Choose it again from this curve, for example 10 or 20 a day.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
Predict: a reviewer marks customer C-13832's new-device alert of 14 November as “not”. On 15 November she is among the most unusual groups again. What does the job do, and how many alerts does it send that day?
Show the answer
The snooze rule holds her alert, because a reviewer marked her “not fraud” less than 7 days before. The job writes the held alert to state/snoozed.jsonl and counts it: the run line says 1 snoozed. It still sends 50 alerts, because the next group takes her place. If her alert had been a device alert, it would not be snoozed.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
Omar asks: “Snooze held only 5 alerts in a week and the share of real alerts did not change. Why keep it?” Give one reason to keep it and one risk to watch.
Show the answer
Keep it because it costs little and it helps where the same honest customer alerts again and again, such as business customers with bulk orders. A reviewer should not check the same honest change every day. The risk: if a fraudster takes over an account after a “not fraud” review, the snooze hides that account for 7 days. So keep the snooze short, never snooze device alerts, and count the snoozed alerts in the weekly report.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Recap
- Alert fatigue: people stop reading alerts that lead to nothing. With grouped alerts in October, 50 a day were only 7.4% real. 5 a day were 56.1% real and held 90.3% of the planted payments.
- A rate limit is the first fix: choose the budget again for groups. Train with the same
DETECT_BUDGET. - A snooze rule holds a reviewed honest customer for 7 days, never a device. At Larkfield it held only 5 alerts in a week, because few honest customers alert two days in a row. A suppression rule needs a reason, an owner and a review date.
- Each alert carries a runbook first step from its first reason. The weekly report shows sent, snoozed, real, not and unread alerts; the weekly review asks why alerts were not read, with the owners written down.
You can now run a detector as a daily job that a team can trust. It sends no duplicates, keeps the budget, notices a moved normal and has owners. Next, the case studies apply the course’s methods to public data outside Larkfield, starting with simulated card payments and a review budget.