The alert budget
- Lesson 30
- 35 min
- Module 7: Evaluate with few labels
Where we are
In the last lesson, we measured a detector at any number of alerts a day. But Sofia's team can review only 50 alerts a day. In this lesson, we choose the threshold from that budget instead of a contamination guess, and we compute the cost per week of the frauds we miss and the false alerts we send. A clinic that sees the most urgent patients first shows the idea.
Lesson 30 of 38 · Anomaly Detection: Find What Is UnusualThe alert budget
1. What "unusual" means
2. Simple baselines that are hard to beat
3. Statistical process control and distribution drift
4. Distance, density and isolation
5. Anomalies in time
6. Deep and embedding methods
7. Evaluate with few labels
8. Learn from the reviewers
9. Run a detector in production
What you will be able to do
Choose a detector’s threshold from the number of alerts that the team can review each day. Then put a money value on the choice: the cost per week of missed frauds and false alerts.
NotebookDo this lesson in a notebook
The notebook has every code cell, task and check of this lesson. It computes the costs of a tiny example of 10 payments and compares the contamination setting with the budget. Then it finds the threshold of each day and draws the cost curve of Isolation Forest. Its setup cells install pandas and scikit-learn and download the course data (about 7 MB). About 35 minutes.
Colab and Kaggle open your own copy. They need a free Google or Kaggle account. Stuck? See the solution notebook.
Before you startWhat to know and to have ready
What you need to know
- Precision at k, recall and the precision–recall curve for anomaly scores, from Why accuracy lies here.
- Isolation Forest and its
contaminationsetting, from Isolation Forest. - Thresholds for a classifier’s scores, from Thresholds and calibration in the machine learning course. This lesson chooses a threshold in a different way, from a budget.
What you need on your computer
Nothing: the notebook runs in Colab or Kaggle. Tested with Python 3.12.13 in Jupyter on macOS, with pandas 3.1.1 and scikit-learn 1.9.1; Colab, Kaggle and Windows were not tested.
A clinic with a fixed number of places
Section titled “A clinic with a fixed number of places”Think of a small clinic. It has one doctor, and she can see 20 patients a day. On a quiet day, 15 people come, and she sees everyone. On a busy day in winter, 60 people come. She cannot see 60, so a nurse sorts them by how urgent they are, and the doctor sees the 20 most urgent first.
Notice what the clinic does not do. It does not say “we see everyone with a temperature above 38.5 degrees”. On a winter day, that rule would send 45 people to a doctor who can see 20, and some urgent patients would wait. On a summer day, it would leave the doctor with free time. The clinic starts from its capacity, the 20 places, and the most urgent patients fill them.
Sofia’s review team is the clinic. It can review 50 alerts a day. The course calls this number the alert budget: how many alerts the team can read with care each day. It is an assumption of the course, and it belongs to the team, not to the detector.
The problem: a threshold that nobody chose
Section titled “The problem: a threshold that nobody chose”Every anomaly detector gives a score. To send alerts, you must turn scores into “alert” or “no alert”. So you need a threshold: every payment with a score above it is an alert. Where does the threshold come from?
In Module 4, Isolation Forest had a setting called contamination: the share of the data that you expect to be anomalies. scikit-learn uses it to set the threshold. Its default, "auto", uses a fixed rule from the method’s paper. Priya, Larkfield’s data scientist, runs both the default and 1%:
flag_auto = forest.predict(X_eval) == -1 # contamination="auto", the defaultcut_1pct = np.quantile(-forest.score_samples(X_ref), 0.99) # contamination=0.01: the top 1% of the reference monthsflag_1pct = score > cut_1pctfor name, flag in [("auto", flag_auto), ("0.01", flag_1pct)]: per_day = pd.Series(flag).groupby(evaluation.ts.dt.floor("D").to_numpy()).sum() print(f"contamination {name:>4}: {flag.sum() / days:6.1f} alerts a day on average (lowest {per_day.min()}, highest {per_day.max()})")The output:
contamination auto: 386.3 alerts a day on average (lowest 187, highest 939)contamination 0.01: 25.7 alerts a day on average (lowest 0, highest 96)With the default, the team gets 386 alerts a day on average, almost eight times its budget. On one day it gets 939. With 1%, it gets 25.7 a day on average. On many days, half of its time stays free, and on the worst day 96 alerts arrive. The real share of planted frauds is 1.66%, about 31 a day, so neither setting even matches the data. And the right share does not help either: the team’s capacity is still 50, whatever the share of fraud.
The contamination setting answers the question “how much fraud is there?”. Nobody knows that number, and the team does not need it. The team needs an answer to another question: “which 50 payments should we read today?”
The idea: let the budget set the threshold
Section titled “The idea: let the budget set the threshold”The clinic’s rule gives the answer. Each day, sort the payments by score, from high to low, and send the top 50 to the team. The score of the 50th payment is that day’s threshold. So the budget sets the threshold, and the threshold moves from day to day. This is also why the last lesson measured precision at k: with a budget, k is the budget.
Text description of the diagram
The diagram has two columns. The left column, “From a guess”, has dashed boxes. First, you guess the share of fraud, with the contamination setting or with one fixed threshold. Then every payment above it is an alert. So the alerts a day are whatever crosses the threshold: 12 on a quiet day and 133 on a busy one. The right column, “From the budget”, has solid blue boxes. First, the team can read 50 alerts a day. Then, each day, you sort the scores and keep the top 50. The score of the 50th payment is that day’s threshold, which moved from 0.582 to 0.771 at Larkfield.
A tiny worked example: ten payments
Section titled “A tiny worked example: ten payments”Here are 10 payments of one day, sorted by score. Four of them are planted frauds. This is a constructed example, with the course’s costs. A missed fraud costs its amount plus a 15-dollar chargeback fee: the bank’s charge when a customer disputes a payment. A false alert costs 3 dollars, about 4 minutes of a reviewer’s time. These costs are assumptions of the course, the same in every lesson.
| Payment | P1 | P2 | P3 | P4 | P5 | P6 | P7 | P8 | P9 | P10 |
|---|---|---|---|---|---|---|---|---|---|---|
| Score | 0.91 | 0.88 | 0.80 | 0.77 | 0.70 | 0.64 | 0.60 | 0.52 | 0.47 | 0.41 |
| Planted fraud | yes | yes | no | yes | no | no | yes | no | no | no |
| Amount ($) | 40 | 25 | 60 | 120 | 15 | 30 | 18 | 45 | 22 | 70 |
With a budget of 3, the team reviews P1, P2 and P3. The threshold is 0.80, the score of P3. Two alerts are real and one is false. Two frauds are missed: P4 (120 + 15 = 135 dollars) and P7 (18 + 15 = 33 dollars). So the cost is 135 + 33 for the missed frauds and 3 for the false alert: 171 dollars. The function tiny_cost() in the notebook does this for four budgets:
budget threshold precision recall missed missed cost false alerts false-alert cost cost0 3 0.80 0.667 0.50 2 168 1 3 1711 5 0.70 0.600 0.75 1 33 2 6 392 8 0.52 0.500 1.00 0 0 4 12 123 10 0.41 0.400 1.00 0 0 6 18 18Read the cost column from top to bottom. A larger budget first saves a lot: each fraud that the team catches saves its amount and the fee. Then, at 8 alerts, every fraud is found, and the next alerts can only be false: 10 alerts cost more than 8. So the cost goes down and then up again. Its lowest point is the cheapest budget for these costs. Precision falls all the time, from 0.667 to 0.400, and that alone does not make a larger budget wrong.
The threshold that the budget sets at Larkfield
Section titled “The threshold that the budget sets at Larkfield”Priya applies the rule to Isolation Forest on the behaviour features. compare.daily_alerts() keeps the 50 highest scores of each day, and she reads the score of the 50th:
alerts = compare.daily_alerts(evaluation, score, budget=50)threshold = alerts[alerts.reviewed].groupby("day").score.min() # the score of the 50th payment of each dayprint(threshold.describe()[["min", "50%", "max"]].round(3).to_string())print("Black Friday 2025-11-28:", round(threshold["2025-11-28"], 3))The output:
min 0.58250% 0.653max 0.771Black Friday 2025-11-28: 0.665Over the 91 evaluation days, the threshold moves from 0.582 (6 October) to 0.771 (18 October). Tomás asks: “Why not take the median, 0.653, and keep it fixed? It is simpler.” Priya tries it:
fixed threshold 0.6530: lowest 12, median 50, highest 133 alerts a daydays with more than 50 alerts: 42 of 91On a typical day, the fixed threshold sends 50 alerts, as planned. But on 42 of the 91 days it sends more than 50, up to 133. On those days, the team cannot read them all, and the alerts that wait are the ones it misses. On quiet days, it sends as few as 12, and the team’s time is unused.
- One fixed threshold (0.653)
- The budget: top 50 of each day
Text description of the chart
A line chart over the 91 days from 1 September to 30 November 2025, with alerts a day from 0 to 140 up. The budget line is flat at 50. The fixed-threshold line moves around it: on 42 days it is above 50 and on the others below. Its lowest point is 12 alerts, and its highest point is 133 alerts, on 14 October. On Black Friday, 28 November, the fixed threshold sends 68 alerts.
The cost per week
Section titled “The cost per week”Precision at the budget tells Sofia how well her team’s time is used. Omar wants the business answer: “What does this choice cost us?” The course turns each choice into one number, the cost per week:
- each planted payment that the team does not review costs its amount plus the 15-dollar chargeback fee;
- each false alert costs 3 dollars of review time;
- the sum is divided by the number of weeks, 13 for the 91 evaluation days.
A blocked honest payment costs something too: a lost basket and some trust. The course does not invent a money value for it, and counts it apart. The last lesson of this module looks at who pays that cost.
Priya computes the cost of Isolation Forest at 50 a day by hand, and checks it against compare.score_payments():
{'missed payments a week': 41.6, 'their amounts a week': 2069, 'chargeback fees a week': 624, 'false alerts a week': 174.9, 'false-alert cost a week': 525, 'cost per week': 3218}compare.score_payments says: 3218So 2,069 + 624 + 525 = 3,218 dollars a week. Doing nothing costs 10,367 dollars a week: every planted payment gets through. Most of what remains is the money of the 41.6 frauds a week that the team does not see. The false alerts cost only 525 dollars, although they are half of the alerts.
Precision against the budget, for four detectors
Section titled “Precision against the budget, for four detectors”Before you choose a budget, compare the detectors at every budget. The chart shows precision at each daily budget, from 5 to 200 alerts a day:
- Isolation Forest
- kNN distance
- Autoencoder
- Two rules, ranked
- Perfect (dashed)
Text description of the chart
A line chart with the daily budget from 0 to 200 across and precision from 0 to 100% up, with one point for each of 11 budgets. Isolation Forest starts at 86.6% at 5 a day, is 50.0% at 50 a day (marked) and ends at 15.2% at 200 a day. kNN distance is almost on the same line: 82.4% at 5, 50.5% at 50, 15.3% at 200. The autoencoder starts lower: it stays between 46.7% and 48.6% up to 30 a day, then falls to 43.4% at 50 and 15.1% at 200. The two rules, ranked, stay between 41.5% and 44.5% up to 30 a day, then fall to 36.0% at 50 and 13.4% at 200. The perfect detector, dashed, starts at 92.7%, is 54.2% at 50 and 15.5% at 200.
At small budgets, Isolation Forest and kNN distance are far ahead. At 200 alerts a day, every detector is close to the perfect one, because no detector can find more frauds than exist. So a large budget hides the difference between detectors, and a small budget shows it.
The comparison table gives the two rules, ranked, 0.427. There, a rule sends every payment over its limits, 53.5 a day. Here each day has exactly the budget, so the rules’ 36.0% at 50 is a different measure, not a different result.
Explore the budget
Section titled “Explore the budget”In the explorer, pick a detector and move the budget. The left panel is the PR curve from the last lesson, with your budget marked. The right panel is the cost per week, with the lowest cost marked.
Explore
Choose the daily budget
Predict first: Sofia's team reviews 50 alerts a day with Isolation Forest. If it reviewed 100 a day, would the cost per week go up or down? And at 200 a day?
Solid line: the chosen detector. Dashed line: the perfect detector, the ceiling. The evaluation months, September to November 2025. Costs are the course’s assumptions: a missed planted payment costs its amount plus 15 dollars, a false alert 3 dollars.
- Precision at the budget
- 0.500
- Recall of planted payments
- 0.808
- False alerts a week
- 174.9
- Planted payments missed a week
- 41.6
- Cost per week (assumptions)
- $3,218
Isolation Forest at 50 alerts a day: 50% of the alerts are planted frauds (precision 0.500), and they find 80.8% of the planted payments (recall 0.808). Each week, the team reads 174.9 false alerts and misses 41.6 planted payments: $3,218 a week. The lowest cost for this detector is $2,802 a week, at 80 alerts a day.
Text description of the explorer
The explorer has two controls: six buttons for the detector, and a slider for the alerts that the team reviews each day: 5, 10, 20, 30, 40, 50, 60, 80, 100, 150 or 200. The left panel shows precision against recall for the chosen detector, one point per budget, with the chosen budget marked by a large dot. The perfect detector is a dashed line. The right panel shows the cost per week against the budget, with a dashed line at 10,367 dollars for no detector, a mark at the chosen budget and a ring at the lowest cost. A table below gives the exact values. For Isolation Forest at 50 a day: precision 0.500, recall 0.808, 174.9 false alerts and 41.6 missed planted payments a week, 3,218 dollars a week. The lowest cost for Isolation Forest is 2,802 dollars a week, at 80 a day. At 100 a day, it is 2,841 dollars; at 200, 4,175 dollars. For kNN distance, the lowest cost is 2,302 dollars at 80 a day. For the perfect detector, the lowest cost is 953 dollars at 50 a day. For MAD per customer, every budget costs more than 8,200 dollars a week.
The answer to the prediction: at 100 a day, Isolation Forest costs 2,841 dollars a week, less than 3,218. At 200 a day, it costs 4,175 dollars, more. The lowest cost is at 80 a day, 2,802 dollars. kNN distance has its lowest cost at 80 a day too, 2,302 dollars. The cost curve is flat near its bottom: from 60 to 100 alerts a day, Isolation Forest stays between 2,802 and 3,092 dollars.
Larkfield’s choice: 50, or 80?
Section titled “Larkfield’s choice: 50, or 80?”So should Sofia’s team review 80 alerts a day? Omar likes the number: 416 dollars a week less than at 50 (3,218 − 2,802). Sofia sees the other side. Her team can review 50, and 80 is 60% more work. And the 30 extra alerts a day are mostly false. At 50 a day, the team finds 25.0 real frauds a day; at 80 a day, it finds 28.2. So only about 3 of the 30 extra alerts are real.
The cost curve also rests on the assumptions. Suppose a false alert costs 6 dollars instead of 3, because the team needs more time per alert or pays overtime. The notebook’s second task adds 3 dollars per false alert to every point. With the new column, three rows of curve read:
budget cost per week ($) cost at 6 dollars4 50 3218 3742.75 60 3092 3799.46 80 2802 3891.6At 6 dollars a false alert, 50 a day is the cheapest budget, and 80 is the most expensive of the three. So the “best” budget is not a fact about the detector. It depends on what a false alert costs, and that is the business’s decision. The course keeps Sofia’s 50 a day, and reports the cost per week beside it. Priya writes both into the decision note: “50 a day costs 3,218 dollars a week; 80 a day would save about 416 dollars and needs 30 more reviews a day”.
What counts as one alert
Section titled “What counts as one alert”Module 9 builds a scheduled job that sends the alerts. There, one alert is a group. It holds all the payments of one customer on one day, or of one device that paid with 3 or more customers’ cards. A card test of 30 payments becomes one alert, not 30. So the same budget means something else:
Text description of the chart
A column chart with four columns on an axis from 0 to 60%. With grouped alerts (one per customer and day), 5 alerts a day are 56.1% real, 10 a day 31.9%, 20 a day 17.1% and 50 a day 7.4%. The share of planted payments that the alerts cover grows only a little: 90.3% at 5 a day, 93.8% at 10, 95.4% at 20 and 96.4% at 50.
Five grouped alerts a day already cover 90% of the planted payments, because one group holds a whole fraud. At 50 grouped alerts a day, more than 9 alerts in 10 are false. So when you change what one alert is, choose the budget again. Module 9 comes back to this as “alert fatigue”.
When to use a budget, and what it costs
Section titled “When to use a budget, and what it costs”Use a budget when people review the alerts and their time is fixed. That is the case for most fraud, risk and operations teams. The budget keeps the work steady on a busy day, and it uses the team’s time on a quiet day.
Do not use a pure budget when every alert above a level must be handled, whatever the number. A safety alarm on a machine is an example: if 200 machines overheat, all 200 need a person. Then use a fixed threshold, and plan the staff for the worst day. Many teams use both: the top 50 a day for review, and a fixed, very high threshold that always alerts.
A budget has a cost too. On a day with 120 real frauds, the team still reads only 50 alerts, and the rest get through. A budget does not tell you this happened. Watch the number of payments above the usual threshold each day, as the detection job of Module 9 does.
Guided practice: choose the threshold from the budget
Section titled “Guided practice: choose the threshold from the budget”- Open the notebook from the box Do this lesson in a notebook at the top of this page.
- Run the setup cells. The data cell ends with the
Ready:line. - In section 1, predict which budget costs the least. Then run the tiny example. You see
171,39,12and18. - In section 2, run the cell that fits Isolation Forest. You see
169,803 evaluation payments over 91 days. - In section 3, run the contamination cell. You see
386.3alerts a day forauto. - In section 4, run the threshold cell. You see a median of
0.653. - Run the fixed-threshold cell. You see
highest 133 alerts a day. - In section 5, run the cost cell. Both lines say
3218. - In section 6, predict the cost at 100 a day. Then run the cost curve. You see
lowest cost: 2802 dollars a week at 80 alerts a day.
Your turn: change the cost of a false alert
Section titled “Your turn: change the cost of a false alert”The notebook’s section 7 has two tasks. First, compute the cost per week of Isolation Forest at 30 alerts a day as cost_at_30. Second, suppose a false alert costs 6 dollars. Add a column cost at 6 dollars to curve, and find the budget with the lowest cost as best_budget_at_6. Before you run it, predict: will the cheapest budget move up, move down or stay at 80?
Show the answers
cost_at_30 = compare.score_payments(evaluation, score, "Isolation Forest", budget=30)["cost per week ($)"]# 3 more dollars for each false alert: add 3 x (false alerts per week) to each costcurve["cost at 6 dollars"] = curve["cost per week ($)"] + 3 * curve["false alerts per week"]best_budget_at_6 = int(curve.loc[curve["cost at 6 dollars"].idxmin(), "budget"])cost_at_30 is 4780: 30 a day misses more frauds than 50 a day (3,218). best_budget_at_6 is 50. When a false alert costs more, the cheapest budget moves down, from 80 to 50, because the extra alerts are mostly false.
Common mistake
The contamination guess decides the team's work
Symptom: Tomás deploys Isolation Forest with its default settings. On the first Monday, Sofia’s team finds 386 alerts on average waiting each day, and 939 on the worst day. The team reads the first 50 in time order and stops. Most alerts are never read.
Cause: contamination="auto" set the threshold from a fixed rule, not from the team. It flagged 20.7% of the payments, while 1.66% are planted. Even the right share, 1.66%, would not match the team’s capacity of 50 a day. And alerts read in time order are not the most unusual ones.
Fix: let the budget set the threshold. Each day, sort the scores and send the top 50, as compare.daily_alerts(evaluation, score, budget=50) does. Then report precision at the budget and the cost per week.
Consequence in AI work: a threshold that nobody chose floods a team on busy days and wastes it on quiet days. After a week of floods, people stop reading the alerts, and the detector becomes noise.
Knowledge check
Section titled “Knowledge check”Check your understanding
Explain why the threshold of Isolation Forest moves from 0.582 to 0.771 over the evaluation months, and why that is what Larkfield wants.
Show the answer
Each day, the team reviews the 50 highest scores, so the threshold is the score of the 50th payment of that day. On a day with many unusual payments, the 50th score is high; on a quiet day, it is low. The number of alerts stays at 50, the team’s capacity, and the threshold moves instead. One fixed threshold (0.653) would send from 12 to 133 alerts a day, more than 50 on 42 of the 91 days.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
Compute: at 50 alerts a day, Isolation Forest misses planted payments worth 2,069 dollars a week, misses 41.6 planted payments a week and sends 174.9 false alerts a week. What is the cost per week with the course's costs?
Show the answer
Missed frauds: 2,069 dollars plus 15 dollars of chargeback fee for each of the 41.6 payments, 624 dollars. False alerts: 3 dollars × 174.9 = 525 dollars. Total: 2,069 + 624 + 525 = 3,218 dollars a week. Doing nothing costs 10,367 dollars a week.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
Omar reads that 80 alerts a day is the cheapest budget for Isolation Forest (2,802 dollars a week). Should Larkfield switch from 50 to 80? Give one reason for and one against, with numbers.
Show the answer
For: it saves about 416 dollars a week (3,218 − 2,802), because the team catches more frauds. Against: it needs 30 more reviews a day, 60% more work than the team can do, and those extra alerts are mostly false. Also, the saving depends on the assumed 3 dollars per false alert. At 6 dollars, 50 a day becomes the cheapest budget. It is a business decision: report both numbers, and keep the budget that the team can really review.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Recap
- The alert budget is the number of alerts that the team can review each day: 50 for Sofia’s team. Let it set the threshold: each day, sort the scores and send the top 50. The threshold moves (0.582 to 0.771); the work stays steady.
- A contamination guess or one fixed threshold sends a number of alerts that nobody chose: 386 a day with the default, 12 to 133 with a fixed threshold.
- Cost per week = missed planted payments (amount + 15 dollars) + 3 dollars per false alert, per week. Isolation Forest at 50 a day: 3,218 dollars; doing nothing: 10,367 dollars.
- The cheapest budget (80 a day for Isolation Forest) depends on the cost assumptions; at 6 dollars per false alert it moves to 50. Report precision at the budget and the cost per week together.
You can now choose a budget and price it. But every number so far comes from the frauds that we planted, and real payments have no answer key. Next, you estimate precision with a sample that people review, and you see what each kind of evidence hides.