Skip to content
Français

From score to alert

  • Lesson 36
  • 45 min
  • Module 9: Run a detector in production

Where we are

So far, we ran every detector by hand in a notebook. Larkfield needs it to run every day, and a rerun must not send Sofia the same alert twice. In this lesson, we build a small scheduled job that reads new data, scores it, applies the budget, groups the alerts and writes them with reasons.

Plays from YouTube. Nothing loads from YouTube until you press play. Privacy
Lesson 36 of 38 · Anomaly Detection: Find What Is UnusualFrom score to alert
Course overview

What you will be able to do

Turn a detector’s scores into alerts that a team can act on. You run a small job each day. It scores the new payments, groups them, keeps the daily budget, explains each alert, and never sends the same alert twice.

Before you startWhat to know and to have readyAbout 45 minutes

What you need to know

What you need on your computer

  • Python 3.12 and a terminal. No notebook in this module: you work in a small project, in the terminal.
  • The course’s labs repository, with the folder C34. If you have it, run git -C nextia-learning-labs pull to get the latest files. The setup is in Meet the Larkfield data.
  • About 250 MB of disk for the virtual environment.

Tested on macOS with Python 3.12.13, pandas 3.1.1 and scikit-learn 1.9.1. The Windows (PowerShell) and Linux commands are written out but were not run.

Think of a post carrier. Every morning, she takes the new letters and sorts them by street. She puts all the letters for one house in one bundle, so the family opens the door once, not five times. On a busy day, she delivers the most urgent mail first. She has a list of the houses she has already visited. If a van breaks down and she must start the round again, she checks the list and does not ring the same bell twice.

Larkfield’s detection job does the same with payments. Each morning, it bundles the unusual payments of the day before. It keeps as many bundles as Sofia’s team can read. It adds a note to each: why it is unusual, and what to do first. It keeps a list of what it sent, so a rerun sends nothing twice.

In the earlier modules, Priya ran each detector in a notebook. That does not work for Sofia. Look at the most unusual payments of 3 October 2025, scored by the first version of the project (the snapshot start, which you set up below):

1,780 payments on 2025-10-03. The 8 most unusual:
payment_id ts customer_id amount country score
P-0229866 2025-10-03 02:21:31 C-01234 19.25 PT 0.775647
P-0229865 2025-10-03 02:20:23 C-01234 24.36 PT 0.771721
P-0229864 2025-10-03 02:19:13 C-01234 19.16 PT 0.768594
P-0229863 2025-10-03 02:18:18 C-01234 24.39 PT 0.768074
P-0230439 2025-10-03 13:03:28 C-15255 13.13 BR 0.759869
P-0229862 2025-10-03 02:17:17 C-01234 18.88 PT 0.756726
P-0229861 2025-10-03 02:16:16 C-01234 20.66 PT 0.754426
P-0230435 2025-10-03 13:01:35 C-15255 20.74 BR 0.745263

C-01234 is the customer of The payment that did not fit: eight payments in Portugal at 2 a.m., an anomaly we planted. The list shows three problems:

  1. One event fills many rows. Eight rows are only two customers. If each row is an alert, Sofia’s team reads C-01234 six times.
  2. A score explains nothing. The number 0.775647 does not tell a reviewer what to check. She must open the customer’s history and find the reason herself.
  3. Nothing remembers what was sent. A notebook that runs twice prints the same list twice. A job that sends alerts must not do that. Sofia’s team would review each alert again, and a customer might get two calls.

So a job needs more steps than “score and print”.

A detection job is a small program that runs on a schedule, for example every morning at 06:00. A schedule is a fixed time when a computer starts a program by itself. Each run does the same steps, in the same order:

Eight steps in two rows. The first row is the work of earlier modules, the second row is this lesson. 1, read the new day: 3 October 2025, 1,780 payments. 2, behaviour features, each payment against the customer's last 60 days. 3, score with Isolation Forest, learned on 2 August to 30 September. 4, group the 200 most unusual payments: one alert per customer, or per device with the cards of 3 or more customers; 160 groups. 5, keep the budget: the 50 most unusual groups. 6, add reasons and a runbook first step. 7, a question: sent before? Look up the alert ID in the ledger. If no, 8, write alerts/2025-10-03.jsonl: 50 new on the first run. If yes, skip it: 0 new and 50 already sent on the second run.
The detection job, with the numbers of 3 October 2025. Steps 1 to 3 are the work of earlier modules. Steps 4 to 8 turn scores into alerts.
Text description of the diagram

The diagram shows eight steps in two rows. The first row is the work of earlier modules; the second row is this lesson. Step 1 reads the new day: 1,780 payments on 3 October 2025. Step 2 computes the behaviour features, which compare each payment with the customer’s last 60 days. Step 3 scores each payment with Isolation Forest, learned on the 60 days from 2 August to 30 September. Step 4 groups the 200 most unusual payments into 160 groups. A group is one customer, or one device that paid with the cards of 3 or more customers. Step 5 keeps the budget, the 50 most unusual groups. Step 6 adds the reasons and a first step from the runbook. Step 7 asks a question: was this alert sent before? It looks up the alert’s ID in the ledger. If not, step 8 writes the alert to the file alerts/2025-10-03.jsonl; the first run writes 50 new alerts. If yes, the job skips it; the second run writes 0 new alerts and counts 50 as already sent.

Steps 1 to 3 come from earlier modules. This lesson adds steps 4 to 8. Each new step has a name:

  • Grouping puts the payments of one event into one alert.
  • The budget keeps the number of alerts that the team can read, 50 a day.
  • Reasons tell the reviewer, in words, what is unusual.
  • A runbook is a short written guide for the people who handle an alert. The job adds its first step: what to do first.
  • The ledger is the job’s list of alerts already sent. It makes the job idempotent: running it twice has the same effect as running it once.

The job groups the 200 most unusual payments of the day. It uses two rules:

  • By default, one alert holds all the payments of one customer on that day.
  • If one device paid with the cards of 3 or more customers on that day, one alert holds all the payments of that device. This is what card testing looks like: one phone or computer tries many stolen cards.

Here is a constructed example with five unusual payments:

Payment Customer Device Group Alert
P1 C-1 D-a customer C-1 A
P2 C-1 D-a customer C-1 A
P3 C-2 D-x device D-x (3 customers) B
P4 C-3 D-x device D-x (3 customers) B
P5 C-4 D-x device D-x (3 customers) B

Five payments become two alerts. Alert A is one customer’s two payments. Alert B is one device that paid with three customers’ cards. These alert groups are not the alert events of Many metrics at once. An alert event joins one metric’s alerts that are close in time. An alert group joins the payments of one customer, or one device, on one day. The score of an alert is the highest score of its payments, so the alerts are sorted by their most unusual payment.

On 12 October 2025, one alert held 39 payments: one device paid with the cards of 39 customers, in 35 minutes. It is a card test that we planted. Without grouping, that card test would fill 39 of Sofia’s 50 places. With grouping, it fills one.

After grouping, the job sorts the alerts by score and keeps the first 50. This is the rule from The alert budget: the team’s capacity sets the threshold, not a guess. On 3 October, the 200 most unusual payments made 160 groups, and the job kept 50. The setting DETECT_BUDGET holds the number, so the team can change it without a change to the code.

A reviewer needs to know why an alert is unusual. The model file keeps the median and the spread of each feature in the training days. For the most unusual payment of an alert, the job computes a robust z-score for each feature. It says how far the value is from the median, in units of the spread. You met this idea in Median, MAD and IQR. The job keeps the three features that are furthest above normal, if they are more than 2 spreads above it. It writes each one in words.

The first reason also chooses the first step of the runbook. A device with many payments: block the device for 24 hours. A new device: ask the customer. A new country: check if the customer is travelling. Here is the first alert of 3 October, the hook customer. The list of payments is cut short here:

{
"alert_id": "dbeea426d6dccd66",
"day": "2025-10-03",
"group": "customer",
"key": "C-01234",
"customers": ["C-01234"],
"payments": ["P-0229866", "P-0229865", "P-0229864", "..."],
"first": "2025-10-03 02:14:07",
"last": "2025-10-03 02:21:31",
"total_amount": 170.23,
"score": 0.7756,
"reasons": [
{"feature": "new_country", "deviation": 8.7, "text": "a country the customer never used before: PT"},
{"feature": "payments_last_hour", "deviation": 7.1, "text": "8 payments by the customer in one hour"},
{"feature": "night", "deviation": 5.5, "text": "paid at night (2:00)"}
],
"runbook": "Check if the customer is travelling (earlier payments, the delivery address).",
"model": "iforest-2025-08-02-2025-09-30"
}

Sofia reads it in a few seconds: eight payments, 170.23 dollars, a new country, at night. The alert also names the model that found it.

The reasons are simple on purpose: they show which features are far from normal, not how Isolation Forest combined them. Explain an alert goes further. 3 of the 50 alerts of 3 October had no reason above 2 spreads. Their runbook step says: “Read the payments of the alert and the customer’s last month.”

Steps 7 and 8: the ledger, so a rerun sends nothing twice

Section titled “Steps 7 and 8: the ledger, so a rerun sends nothing twice”

Jobs run again: a server restarts in the middle of a run, or a person reruns yesterday after a fix. Each time, the job must not send an alert that it sent before.

The job gives each alert an ID and keeps the IDs that it sent in a file, the ledger. Before it sends an alert, it looks for the ID in the ledger:

An alert of 2025-10-03, group customer, key C-01234, gets an ID: the first 16 characters of the SHA-256 of 2025-10-03|customer|C-01234, which is dbeea426d6dccd66. A question: is the ID in state/sent.jsonl or in the day's file? If no: send it, by writing a temporary file and renaming it in one step, then add the ID to state/sent.jsonl. If yes: do not send it, count it as already sent.
The alert ID comes from the day, the kind of group and its key, never from the model or the score. The same alert always gets the same ID.
Text description of the diagram

The diagram starts with an alert of 3 October 2025, for the customer C-01234. Its ID is made from the text “2025-10-03|customer|C-01234”. The job computes the SHA-256 hash of that text and keeps its first 16 characters, dbeea426d6dccd66. Then a question: is this ID already in the ledger, state/sent.jsonl, or in the day’s alert file? If not, the job sends the alert. It writes a temporary file and renames it in one step, and then it adds the ID to the ledger. If the ID is there already, the job does not send the alert and counts it as already sent.

A hash is a fixed-length code that a function computes from a text. The same text always gives the same code, and a different text gives a different one. SHA-256 is a common hash function. The job hashes the day, the kind of group and its key. It does not hash the model’s version or the score. So a retrained model that finds the same customer on the same day makes the same ID, and the ledger stops it.

If the job stops in the middle of a run, the rename in one step keeps all the new alerts in the file, or none. The job also reads the day’s alert file as sent, in case it stopped before the ledger.

Now you run all of this yourself. The project starts as the snapshot start: it can learn normal and score a day, as above. A snapshot is a copy of the project at one point of the course. You add the files of this module from the snapshot end-of-m09. This module has one snapshot for its three lessons, so you copy the files of the next two lessons now too. This lesson reads alerts.py and ledger.py.

  1. In the folder where you keep projects, copy the snapshot start to a new folder, detection-job:

    Terminal window
    cp -R nextia-learning-labs/C34/snapshots/start detection-job
    cd detection-job
  2. Make the virtual environment, install the packages and get the data. get_data.py copies the payments file from your labs folder and checks it:

    Terminal window
    python3.12 -m venv .venv
    source .venv/bin/activate
    python -m pip install -r requirements.txt
    python scripts/get_data.py --from ../nextia-learning-labs/C34/data
    python -m pytest -q

    You see OK payments.csv.gz (6.9 MB) and 7 passed. Without the labs folder, run python scripts/get_data.py. It downloads the same file from GitHub.

  3. Learn normal from the 60 days up to 30 September 2025:

    Terminal window
    python -m detect train --until 2025-09-30
    Trained iforest-2025-08-02-2025-09-30 on 107,793 payments (2025-08-02 to 2025-09-30). Saved to state/model.pkl.
  4. Score 3 October. You see the 8 rows at the top of this lesson:

    Terminal window
    python -m detect score --day 2025-10-03 --top 8
  5. Copy this module’s files from the snapshot end-of-m09. Seven files in detect/ are new or changed, and five test files are new:

    Terminal window
    S=../nextia-learning-labs/C34/snapshots/end-of-m09
    for f in alerts ledger drift suppress report __main__ config; do cp "$S/detect/$f.py" detect/; done
    for t in alerts ledger cli drift report_and_snooze; do cp "$S/tests/test_$t.py" tests/; done
    cp "$S/README.md" README.md
  6. Run the tests and the style check:

    Terminal window
    python -m pytest -q
    ruff check .

    You see 19 passed and All checks passed!.

  7. Send the alerts of 3 October:

    Terminal window
    python -m detect run --day 2025-10-03
    2025-10-03: 1,780 payments, 41 customers above the training threshold, 160 alert groups, 0 snoozed. Sent 50 new alerts to alerts/2025-10-03.jsonl; 0 already sent.
  8. Run the same day again:

    Terminal window
    python -m detect run --day 2025-10-03
    2025-10-03: 1,780 payments, 41 customers above the training threshold, 160 alert groups, 0 snoozed. Sent 0 new alerts to alerts/2025-10-03.jsonl; 50 already sent.
  9. Open alerts/2025-10-03.jsonl in your editor. It has 50 lines, one alert per line. The first line is the alert of C-01234 above.

The run line also counts customers above the training threshold: 41 on this day. The training threshold is the score that gave 50 alerts a day in the training days. This number does not change what the job sends. It measures the pressure on the team, and the next lesson watches it.

The file format is JSON lines (.jsonl): one JSON object per line, easy for a review screen or a dashboard to read. In this course, the file is the “send”. A real job would also post each new alert to the team’s queue.

The job is one command, so any scheduler can start it. For example, every day at 06:00, for the day before:

  • Linux or macOS (cron): 0 6 * * * cd /path/to/detection-job && .venv/bin/python -m detect run --day $(date -d yesterday +\%F). On macOS, date -v-1d +\%F gives yesterday instead.
  • Windows: a daily task in Task Scheduler that runs .venv\Scripts\python.exe -m detect run --day <yesterday> in the project folder.

The course has no scheduler: you run each day by hand, or a range of days with replay. The MLOps course shows how a team watches a job like this one, in Logs, metrics and traces. It shows who is called when the job fails, in Ownership and support.

A replay runs the job for every day of a period, in order, as the schedule would have done. It is how you test a job on past data. Run October 2025:

Terminal window
python -m detect replay --from 2025-10-01 --to 2025-10-31

The output has one row per day. Here are its first rows and its last line:

day payments above_threshold groups snoozed sent already_sent
2025-10-01 1651 53 109 0 50 0
2025-10-02 1763 43 143 0 50 0
2025-10-03 1780 40 159 0 2 48
2025-10-04 2173 58 81 0 50 0
...
2025-10-14 1782 138 49 0 49 0
...
Sent 1501 alerts in 31 days (48.4 a day, budget 50, policy budget).

Read the row of 3 October. You sent that day already, so 48 alerts were already sent. But 2 new alerts were sent. Why?

A replay computes the features of the whole month at once, which is faster. A single run computes them with exactly 60 days of history. So a few scores differ a little: 40 customers above the threshold instead of 41, and 159 groups instead of 160. Two groups moved into the top 50. They are new groups with new IDs, so the ledger lets them through. No alert was sent twice. But 3 October now has 52 alerts, two more than the budget.

So “idempotent” here means: the same alert is never sent twice. It does not mean that a day never gets more than 50. The Your-turn task below closes that gap. On 14 October, only 49 groups existed among the 200 most unusual payments, so the job sent 49. The next lesson shows why that day was unusual.

Your turn: never more than the budget on one day

Section titled “Your turn: never more than the budget on one day”

Change run_day() in detect/__main__.py so that a rerun never sends more alerts for a day than the budget allows. Count the alerts already in the day’s file. Then send at most budget minus that number. Keep the fixed policy as it is: it has no budget. Before you change it, predict: after your change, how many new alerts would a replay of 3 October send?

Show a solution

In run_day(), after the line chosen = alerts_mod.choose(...), add:

if s.policy == "budget":
day_file = s.alerts_dir / f"{day}.jsonl"
sent_today = len(day_file.read_text().splitlines()) if day_file.exists() else 0
done = ledger.sent_ids(s.state_dir, s.alerts_dir, day)
fresh = [a for a in chosen if a["alert_id"] not in done]
chosen = [a for a in chosen if a["alert_id"] in done] + fresh[:max(0, s.budget - sent_today)]

The already-sent alerts stay in chosen, so the job still counts them. Only the new ones are cut to the room that is left. The prediction: 0 new alerts. The day’s file already has 50 alerts, so there is no room. We ran it: first a run of 3 October, then a replay of 1 to 4 October. The replay gives 0 sent and 48 already sent for 3 October. The file keeps 50 lines. The 19 tests still pass. The course’s snapshot keeps the simpler version, and the outputs in these lessons come from it.

When to use a job like this, and what it costs

Section titled “When to use a job like this, and what it costs”

Use a scheduled job when the data arrives in batches, such as a day of payments. People then review the alerts later. It is simple and cheap. One run of a day used about 12 seconds of processor time on the test laptop. It needs no server.

Do not use a daily batch when a decision must be fast. A card test runs in minutes. To block payments while they happen, you need a score inside the payment’s path, in milliseconds: a real-time system, with other costs. Many shops use both: a few fast rules at payment time, and a batch job for the review team.

The costs are in the details. A fraud that spreads over two days becomes two alerts. Reasons from robust z-scores are easy to read, but they do not prove why the model scored the payment. And the ledger must live as long as the alerts: if you delete state/ and alerts/, the next run sends everything again.

Common mistake

A model version in the alert ID sends every alert again

Symptom: Tomás retrains the model and reruns the last week to check the new model. Sofia’s team gets every alert of that week again, under new IDs. They review them all a second time, and two customers get a second call. This is a constructed case: the course’s job does not do this.

Cause: the alert ID was made from the model’s version and the payment, for example iforest-2025-09-09-2025-11-07|P-0229866. A new model gives a new ID to the same event, so the ledger cannot see that it was sent.

Fix: make the ID from what the reviewer sees as “the same alert”: the day, the kind of group and its key (2025-10-03|customer|C-01234). Keep the model’s version inside the alert, not in its ID. In the next lesson, a retrained model reruns 15 October: 41 alerts already sent, 9 new, none twice.

Consequence in AI work: a job that sends duplicates wastes the team’s time and annoys customers. Worse, people learn that the alerts repeat, and they stop reading them with care.

Check your understanding

The 8 most unusual payments of 3 October belong to 2 customers. Explain what grouping does with them, and why it matters for a budget of 50 alerts a day.

Show the answer

Grouping puts all the unusual payments of one customer on one day into one alert. When one device pays with the cards of 3 or more customers, one alert holds that device’s payments. So the 8 payments become 2 alerts, one for C-01234 and one for C-15255. With a budget of 50, each place should hold a different event. Without grouping, one card test of 39 payments, as on 12 October, would fill 39 of the 50 places.

Check your understanding

Predict: you run the job for 3 October, then you retrain the model and run 3 October again. The new model also puts customer C-01234 in its top 50. Is C-01234's alert sent a second time? Explain with the alert ID.

Show the answer

No. The alert ID is the first 16 characters of the SHA-256 of “2025-10-03|customer|C-01234”. It does not contain the model or the score, so the retrained model gives the same ID, dbeea426d6dccd66. The ledger has that ID, so the job counts the alert as already sent. Only groups that the first model did not send can be new.

Check your understanding

After a run and a replay, the alert file of 3 October has 52 alerts, and the budget is 50. Is the job broken? Explain what happened and what idempotent means here.

Show the answer

No alert was sent twice, so the job is idempotent as it promises. The replay computed the features for the whole month at once, so a few scores differed a little. Two other groups moved into the top 50. They had new IDs, so the ledger let them through. For a hard daily limit, the job must count the day’s sent alerts and send at most the rest, as in the Your-turn task.

Recap

  • A detection job runs the same steps on a schedule. It reads the day, computes features and scores. It groups, keeps the budget, and adds reasons and a runbook first step. It sends only what the ledger does not have.
  • Grouping turns many payments of one event into one alert: per customer, or per device that paid with 3 or more customers’ cards. On 12 October 2025, one card test of 39 payments became one alert.
  • The alert ID is a hash of the day, the kind of group and its key, never of the model. A rerun of 3 October sent 0 new alerts, with 50 already sent.
  • Idempotent means no alert twice. A replay with slightly different scores can still add new groups to a day (52 on 3 October). Count the day’s alerts if you need a hard limit.

The job now sends 50 alerts a day with reasons. In October it learned normal from August and September. Next, a product launch moves normal for good, and you see what that does to the job.

Next: When "normal" moves