Explain an alert
- Lesson 35
- 30 min
- Module 8: Learn from the reviewers
Where we are
Our alerts are now better ranked, but each one is still only a score. Sofia needs to decide in one minute. In this lesson, we find which features made a score high, with per-feature deviations and SHAP for the re-ranker, and we write an alert card that a reviewer can act on.
Lesson 35 of 38 · Anomaly Detection: Find What Is UnusualExplain an alert
1. What "unusual" means
2. Simple baselines that are hard to beat
3. Statistical process control and distribution drift
4. Distance, density and isolation
5. Anomalies in time
6. Deep and embedding methods
7. Evaluate with few labels
8. Learn from the reviewers
9. Run a detector in production
What you will be able to do
Turn a score into a decision. Show which features made an alert’s score high, in a form that a reviewer can check in one minute. And know what such an explanation can and cannot tell you.
NotebookDo this lesson in a notebook
The notebook has every code cell, task and check of this lesson. It compares an alert with normal payments, splits the re-ranker’s score into contributions, computes the SHAP values, and writes an alert card. Its setup cells install the packages, SHAP among them, and download the data (about 7 MB). About 30 minutes.
Colab and Kaggle open your own copy. They need a free Google or Kaggle account. Stuck? See the solution notebook.
Before you startWhat to know and to have ready
What you need to know
- The re-ranker and its nine inputs, from From unsupervised to semi-supervised.
- The robust z (distance from the median, in robust sigmas), from Median, MAD and IQR. This lesson repeats the idea in one sentence.
- How one score of a linear model splits into contributions, from Explain predictions in Machine Learning: From Problem to Reliable Model. This lesson explains SHAP only for the re-ranker, and links there for the rest.
What you need on your computer
Nothing. The notebook runs in Colab, Kaggle or VS Code.
Tested with Python 3.12.13 in Jupyter on macOS, with pandas 3.1.1, scikit-learn 1.9.1 and SHAP 0.54.1. Colab and Kaggle were not tested.
A note from the family doctor
Section titled “A note from the family doctor”A family doctor sends a patient to a hospital. Compare two notes. The first says: “Risk score 0.93. Please see.” The second says: “Fever of 39.5 °C for three days, after a trip abroad. Normal for her: no fever, no travel. Please test for malaria.”
The hospital doctor can act on the second note in a minute. With the first, she must start again from nothing: ask the questions, run the checks, find out why the score is high. The score may be right. But it does not help her decide.
Sofia’s alerts today are the first note. This lesson writes the second.
The problem: a score costs time
Section titled “The problem: a score costs time”Take the alert at the top of the list on 3 October 2025. It is the customer of the course’s first lesson, C-01234. The re-ranker gives payment P-0229866 a score of 1.000: almost certainly real. That is all the alert says.
To decide, a reviewer opens the customer’s history. She looks for the usual amounts, the usual country, the devices, the times. The comparison table assumes 3 minutes to understand one Isolation Forest alert (an assumption of the course). With 50 alerts a day, that is 150 minutes, before any action.
A score alone also hides the easy cases. Later in this lesson, a split basket on a new phone is in the top 50 too, with a score of 0.387. Without reasons, the reviewer must check it as carefully as a theft.
The idea: two kinds of explanation
Section titled “The idea: two kinds of explanation”An explanation answers: which features made this score high? There are two kinds, and this course uses both.
- Per-feature deviations. Compare each feature of the payment with normal payments, one feature at a time. This works for any detector, because it does not look inside the model. It answers: “what is unusual about this payment?”
- Contributions, or SHAP values. Split the model’s score into one part per feature. Each part says how much that feature moved the score. This looks inside the model. It answers: “what moved the model’s score?”
The two answers are not the same. A payment can be unusual in a way that the model ignores. And the model can use a feature that is not unusual at all. A good alert card uses the first kind to describe the payment, and the second kind to choose which reasons to show.
Per-feature deviations
Section titled “Per-feature deviations”For a number, use the robust z of Module 2. It says how far the value is from the median of normal payments, in robust sigmas. A robust sigma is the spread measured with the median absolute deviation (MAD), so a few extreme values cannot stretch it. For a yes/no feature, give the share of normal payments that say yes. Normal means the reference months, June to August 2025.
The notebook does this for the hook customer’s alert:
log_amount_vs_usual 0.00 (normal 0.00, +0.0 robust sigma)new_country yes (1.1% of normal payments say yes)new_device yes (4.3% of normal payments say yes)log_minutes_since_last 0.76 (normal 8.75, -4.7 robust sigma)payments_last_hour 8 (normal 1; most payments have 1)device_payments_last_hour 8 (normal 1; most payments have 1)night yes (3.3% of normal payments say yes)is_business no (24.4% of normal payments say yes)Read it as a reviewer would. A new country: only 1.1% of normal payments have one. A new device: 4.3%. At night: 3.3%. Eight payments in the last hour, where normal is one. And very little time since the last payment. But the amount is about her usual amount.
Some features have a spread of 0: most normal payments have exactly the same value, here 1. A robust z would divide by 0. So the notebook writes the value beside the normal value instead.
Contributions, and what SHAP adds
Section titled “Contributions, and what SHAP adds”Now look inside the re-ranker. It is a logistic regression: a weighted sum of the scaled features, turned into a score from 0 to 1.
A tiny example with three features
Section titled “A tiny example with three features”Here is a constructed model, small enough to compute by hand. It has an intercept of −2.0 and three yes/no features:
| Feature | Weight | This payment | Contribution = weight × value |
|---|---|---|---|
| New device | 1.5 | yes (1) | 1.5 |
| New country | 1.0 | yes (1) | 1.0 |
| Night | 0.5 | no (0) | 0 |
The sum is −2.0 + 1.5 + 1.0 + 0 = 0.5. The logistic function turns it into a score: 1 ÷ (1 + e−0.5) = 0.62. Each contribution says how much one feature moved the sum. The new device moved it most. Night moved it not at all, because this payment was not at night.
The contributions are measured from a reference: here, a payment with every feature at 0. Change the reference, and every contribution changes. Remember this. It matters for the real model.
The re-ranker’s contributions
Section titled “The re-ranker’s contributions”The real re-ranker scales each feature first: a scaled value of 0 means “the average reviewed alert”. So its contributions compare the payment with the average alert, not with a normal payment. For the hook customer’s alert:
value scaled weight contributioniso 0.79 1.57 1.98 3.11is_business 0.00 -0.90 -1.72 1.56new_device 1.00 0.85 1.78 1.52new_country 1.00 1.17 1.22 1.42night 1.00 1.56 0.68 1.06log_minutes_since_last 0.76 -0.68 -0.91 0.62device_payments_last_hour 8.00 -0.10 1.08 -0.11log_amount_vs_usual 0.00 0.51 -0.72 -0.37payments_last_hour 8.00 0.89 -0.55 -0.48intercept 0.17 + contributions 8.34 = 8.50score from the sum: 0.9998 | predict_proba: 0.9998The split is exact: 0.17 + 8.34 = 8.50, and the logistic function of 8.50 is the model’s score.
Text description of the chart
A column chart with six columns, from the largest to the smallest. The detector score contributes 3.11. Not being a business account contributes 1.56. A new device 1.52. A new country 1.42. Night 1.06. A short time since the last payment 0.62. Three features, not shown, pushed the score down. They are the device’s payments in the last hour (−0.11), the amount compared with usual (−0.37), and the customer’s payments in the last hour (−0.48).
Two contributions need care. “Not a business account” pushes the score up by 1.56. Why? Among the reviewed alerts, 45% were business accounts, mostly honest bulk orders. So a personal account is more suspicious than the average alert. That is true about the model. It is no help to Sofia.
And eight payments in the last hour pushes the score down (−0.48). Many real alerts are card tests, one payment per stolen card. Among the alerts, many payments by one customer went with “not”. This is the sign trap of Explain predictions: a weight describes the training rows, not the world.
SHAP: the same numbers, with a name
Section titled “SHAP: the same numbers, with a name”SHAP (SHapley Additive exPlanations) is a method and a Python library. It splits any model’s score into one value per feature, so that the values add up to the score minus a base value. The base value is the model’s average output on a background data set. The machine learning course shows why contributions are not causes in Explain predictions. Read it before you show SHAP values to anyone.
For a linear model with independent features, the SHAP value of a feature is exactly its contribution, measured from the average of the background. Our background is the 1,400 reviewed alerts. Their scaled average is 0. So the base value is the intercept, and the SHAP values are the contributions above:
import shapbackground = re_scaler.transform(labels[FEATS])explainer = shap.LinearExplainer(re_ranker, shap.maskers.Independent(background, max_samples=len(background)))values = explainer.shap_values(re_scaler.transform(alert[FEATS].to_frame().T.astype(float)))[0]base value 0.17largest difference from the contributions: 1.8e-15The largest difference is 0.0000000000000018: the same numbers. One detail: by default, shap keeps only 100 rows of the background, chosen at random. The values then move a little, and the base value is no longer the intercept. The notebook uses all 1,400 rows.
For a linear re-ranker, you do not need the library. It becomes useful when the re-ranker is a model whose score does not split so easily, such as gradient-boosted trees. For the detector, this course uses the per-feature deviations: they are simpler to read.
Larkfield: the card that Sofia reads
Section titled “Larkfield: the card that Sofia reads”Now write the alert. A good card has five parts:
- The payment, in one line: who, when, how much, where.
- The customer’s normal, from the customer’s own history.
- The top reasons, in words, from the contributions, keeping only reasons that describe the payment.
- What we left out, so nobody wonders.
- The decision and its first action.
Here is the card for the hook customer:
Text description of the diagram
The picture shows two boxes. On the left, “A score alone”: alert P-0229866, customer C-01234, score 1.000. On the right, “The card”, in five parts. The header: alert P-0229866, customer C-01234, 3 October 2025 at 02:21. The payment: today, 8 payments in Portugal in euros, from 02:14 to 02:21, 170.23 dollars in all. This one is 19.25 dollars, on a mobile, for home goods. The normal: Saturday baskets of 11.15 to 17.98 dollars, in Canada. Then “Why it is high”, in a box with a strong border. Reason 1: first payment ever from Portugal, plus 1.42. Reason 2: a device this customer never used, plus 1.52. Reason 3: at 02:21, at night, plus 1.06. Reason 4: 8 payments in one hour, where the customer usually has 1. A dashed box lists what is not on the card: a high detector score, plus 3.11, and a personal account, plus 1.56. Both are true about the model, but no help to decide. The last box: decide. Real: block the card and call the customer. Not: mark “not” and write the reason, for example a trip.
Notice reason 4. Its contribution was negative, so the model did not use it as a reason. But it is the clearest sign for a person: eight payments in seven minutes, where the customer usually makes one. The deviations found it. A card can use both kinds of explanation.
With this card, Sofia can decide in under a minute: a stranger is using the card in Portugal at night.
The second card: a fast “not”
Section titled “The second card: a fast “not””The same day, payment P-0230910 is 27th in the re-ranked list, with a score of 0.387. Its largest contributions:
| Feature | Value | Contribution |
|---|---|---|
| Business account | no | +1.56 |
| New device | yes | +1.52 |
| Payments in the last hour | 3 | +0.29 |
| Detector score | 0.656 | -1.95 |
| New country | no | -1.04 |
Write its card:
- Payment: C-15551, 3 October at 17:25, 12.99 dollars, in Canada, on a mobile, garden products.
- Normal: about 24 dollars a payment, in Canada, on another mobile.
- Today: three payments in 13 minutes: 16.22, 11.55 and 12.99 dollars, 40.76 dollars in all. All are garden products, from the same new phone.
- Why it is in the list: a device this customer never used, and three payments in a short time.
A reviewer reads it in a minute: a customer with a new phone, paying for one basket in three parts, in the late afternoon. She marks it not. The answer key agrees: this is a split basket, one of the normal-but-unusual cases of the course data. Without the card, the score of 0.387 says only “maybe”.
The minute, as a cost
Section titled “The minute, as a cost”The comparison table assumes 3 minutes to understand one Isolation Forest alert. If a card brings that down to 1 minute, the team saves 2 minutes an alert: 100 minutes a day at 50 alerts. The one minute is a goal, an assumption of this lesson, not a measurement. To know the real number, time the reviews for a week with and without cards.
When to explain, and what it costs
Section titled “When to explain, and what it costs”Show when to use each explanation, when not, and the cost
Use per-feature deviations when:
- You want to describe the payment for a person. They work for any detector, Isolation Forest included.
- The features have a clear “normal”: the customer’s own past, or the reference months.
Use contributions or SHAP when:
- You want to know what moved the model’s score, to choose which reasons to show.
- You want to find a weight that has learned something wrong.
- The model is the re-ranker, or another model whose features a person can read.
Do not:
- Show a contribution as a cause. “A personal account” moved the score. It did not make the payment fraud.
- Show reasons that are true only about the model. They cost the reviewer time and trust.
- Write a card from SHAP values of features that nobody can read, such as an autoencoder’s hidden numbers.
What it costs:
- A function that writes the card, and a test that its numbers match the data. The next module adds reasons to every alert of the detection job.
- Words for each feature, written once by the team, in Sofia’s language.
- A check, every few months, that the cards still say true things after the model is retrained.
Guided practice: the card in the notebook
Section titled “Guided practice: the card in the notebook”- Open the notebook from the box Do this lesson in a notebook at the top of this page.
- Run the setup cells. The data cell ends with the
Ready:line. - In section 1, run the cell. You see
P-0229866and customerC-01234. - In section 2, run the deviations cell.
new_countrysaysyes (1.1% of normal payments say yes). - In section 3, run the contributions cell. You see
intercept 0.17 + contributions 8.34 = 8.50. - In section 4, run the SHAP cell. You see
base value 0.17. - In section 5, run the card cell. Read the three reasons, and find the one that is true only about the model.
- In section 6, write the two answers, and run each check cell. Each prints
Check passed.
Your turn: explain the split basket
Section titled “Your turn: explain the split basket”Change one thing: explain payment P-0230910 instead of the hook customer’s payment. Before you run it, predict: which feature pushes its score up the most?
Show the answer
second = day[day.payment_id == "P-0230910"].iloc[0]largest_push = contributions(second).contribution.idxmax()t2 = contributions(second)score_from_sum = round(float(1 / (1 + np.exp(-(re_ranker.intercept_[0] + t2.contribution.sum())))), 3)Check passed: largest_push is right.Check passed: score_from_sum is 0.387.The largest push is is_business: “not a business account”, +1.56. The new device is a close second, +1.52. So the automatic card would start with a reason that is true only about the model. That is why a person chooses the words of the card, once, for every feature. Then the card keeps the new device and the three payments, and drops “personal account”.
Common mistake
A card that lists the model's reasons
Symptom: Tomás writes the card from the three largest SHAP values, with no other step. For the hook customer, the card says: “unusual overall (detector score 0.79); a personal account; a device this customer never used”. Sofia reads “a personal account” on almost every alert. After a week, she stops reading the reasons.
Cause: SHAP values explain the model, measured from the average reviewed alert. A personal account is more suspicious than the average alert, because so many alerts were business bulk orders. That makes it a large SHAP value. But it says nothing about this payment that Sofia can check.
Fix: write a card from both explanations. Use the contributions to choose the candidates for reasons. Keep only the ones that compare the payment with the customer’s normal, in words. Add strong deviations that the model does not use, such as eight payments in one hour. Say which model reasons you left out.
Consequence in AI work: an explanation that is true about the model but useless to the person teaches the person to ignore all explanations. Then a wrong alert passes as easily as a right one.
Knowledge check
Section titled “Knowledge check”Check your understanding
What is the difference between a per-feature deviation and a SHAP value (or contribution)? Give one example from the hook customer's alert where they disagree.
Show the answer
A per-feature deviation compares the payment with normal payments, one feature at a time. It works for any detector, and it says what is unusual about the payment. A SHAP value says how much a feature moved the model’s score, measured from the average reviewed alert. For the linear re-ranker, it is the same as the contribution. They disagree on the payments in the last hour. 8 against a normal of 1 is a strong deviation, but its contribution is negative (−0.48). Among the alerts, many real frauds were single card tests. They also disagree on “not a business account”: not unusual at all, but a contribution of +1.56.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
For the re-ranker, show how the score of the hook customer's alert is built from its parts, and say why the SHAP values equal the contributions.
Show the answer
The intercept, 0.17, plus the nine contributions (weight times scaled value), 8.34, gives a sum of 8.50. The logistic function 1 ÷ (1 + e−8.50) gives 0.9998, the model’s score. For a linear model with independent features, a SHAP value is the weight times the distance from the background average. The background is the 1,400 reviewed alerts, and their scaled average is 0. So each SHAP value is the contribution, and the base value is the intercept. The notebook’s largest difference is 1.8e-15.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
Write the five parts of an alert card that a reviewer can decide on in one minute, and say which reasons you leave out, and why.
Show the answer
The five parts: the payment in one line (who, when, how much, where); the customer’s normal; the top reasons in words; the reasons left out; the decision with its first action. Leave out reasons that are true only about the model, such as “a personal account” or “a high detector score”. The reviewer cannot check them against the payment, and they waste her minute. Keep reasons that compare the payment with the customer’s normal. Add strong deviations that the model does not use, such as eight payments in one hour.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Recap
- Per-feature deviations compare a payment with normal, one feature at a time, for any detector.
- For the linear re-ranker, the score splits exactly into contributions: 0.17 + 8.34 = 8.50 for the hook customer. SHAP values are the same numbers, measured from the average reviewed alert.
- A contribution is the model’s view, not a cause. “A personal account” moved the score; it is no help to the reviewer.
- A good alert card shows the payment, the customer’s normal, the top reasons in words, what was left out, and the first action. The goal: a decision in one minute instead of three (an assumption until you time it).
Module 8 is complete: the reviewers’ labels re-rank the alerts, a random sample keeps the labels honest, and each alert explains itself. Next, in Module 9, we put the detector into a scheduled job. It reads new payments, scores them, groups them and sends alerts with their reasons. And it never sends an alert twice.
Next: From score to alert