Skip to content
Français

Explain an alert

  • Lesson 35
  • 30 min
  • Module 8: Learn from the reviewers

Where we are

Our alerts are now better ranked, but each one is still only a score. Sofia needs to decide in one minute. In this lesson, we find which features made a score high, with per-feature deviations and SHAP for the re-ranker, and we write an alert card that a reviewer can act on.

Plays from YouTube. Nothing loads from YouTube until you press play. Privacy
Lesson 35 of 38 · Anomaly Detection: Find What Is UnusualExplain an alert
Course overview

What you will be able to do

Turn a score into a decision. Show which features made an alert’s score high, in a form that a reviewer can check in one minute. And know what such an explanation can and cannot tell you.

NotebookDo this lesson in a notebookColab · Kaggle · GitHub

The notebook has every code cell, task and check of this lesson. It compares an alert with normal payments, splits the re-ranker’s score into contributions, computes the SHAP values, and writes an alert card. Its setup cells install the packages, SHAP among them, and download the data (about 7 MB). About 30 minutes.

Colab and Kaggle open your own copy. They need a free Google or Kaggle account. Stuck? See the solution notebook.

Before you startWhat to know and to have readyAbout 30 minutes

What you need to know

What you need on your computer

Nothing. The notebook runs in Colab, Kaggle or VS Code.

Tested with Python 3.12.13 in Jupyter on macOS, with pandas 3.1.1, scikit-learn 1.9.1 and SHAP 0.54.1. Colab and Kaggle were not tested.

A family doctor sends a patient to a hospital. Compare two notes. The first says: “Risk score 0.93. Please see.” The second says: “Fever of 39.5 °C for three days, after a trip abroad. Normal for her: no fever, no travel. Please test for malaria.”

The hospital doctor can act on the second note in a minute. With the first, she must start again from nothing: ask the questions, run the checks, find out why the score is high. The score may be right. But it does not help her decide.

Sofia’s alerts today are the first note. This lesson writes the second.

Take the alert at the top of the list on 3 October 2025. It is the customer of the course’s first lesson, C-01234. The re-ranker gives payment P-0229866 a score of 1.000: almost certainly real. That is all the alert says.

To decide, a reviewer opens the customer’s history. She looks for the usual amounts, the usual country, the devices, the times. The comparison table assumes 3 minutes to understand one Isolation Forest alert (an assumption of the course). With 50 alerts a day, that is 150 minutes, before any action.

A score alone also hides the easy cases. Later in this lesson, a split basket on a new phone is in the top 50 too, with a score of 0.387. Without reasons, the reviewer must check it as carefully as a theft.

An explanation answers: which features made this score high? There are two kinds, and this course uses both.

  1. Per-feature deviations. Compare each feature of the payment with normal payments, one feature at a time. This works for any detector, because it does not look inside the model. It answers: “what is unusual about this payment?”
  2. Contributions, or SHAP values. Split the model’s score into one part per feature. Each part says how much that feature moved the score. This looks inside the model. It answers: “what moved the model’s score?”

The two answers are not the same. A payment can be unusual in a way that the model ignores. And the model can use a feature that is not unusual at all. A good alert card uses the first kind to describe the payment, and the second kind to choose which reasons to show.

For a number, use the robust z of Module 2. It says how far the value is from the median of normal payments, in robust sigmas. A robust sigma is the spread measured with the median absolute deviation (MAD), so a few extreme values cannot stretch it. For a yes/no feature, give the share of normal payments that say yes. Normal means the reference months, June to August 2025.

The notebook does this for the hook customer’s alert:

log_amount_vs_usual 0.00 (normal 0.00, +0.0 robust sigma)
new_country yes (1.1% of normal payments say yes)
new_device yes (4.3% of normal payments say yes)
log_minutes_since_last 0.76 (normal 8.75, -4.7 robust sigma)
payments_last_hour 8 (normal 1; most payments have 1)
device_payments_last_hour 8 (normal 1; most payments have 1)
night yes (3.3% of normal payments say yes)
is_business no (24.4% of normal payments say yes)

Read it as a reviewer would. A new country: only 1.1% of normal payments have one. A new device: 4.3%. At night: 3.3%. Eight payments in the last hour, where normal is one. And very little time since the last payment. But the amount is about her usual amount.

Some features have a spread of 0: most normal payments have exactly the same value, here 1. A robust z would divide by 0. So the notebook writes the value beside the normal value instead.

Now look inside the re-ranker. It is a logistic regression: a weighted sum of the scaled features, turned into a score from 0 to 1.

Here is a constructed model, small enough to compute by hand. It has an intercept of −2.0 and three yes/no features:

Feature Weight This payment Contribution = weight × value
New device 1.5 yes (1) 1.5
New country 1.0 yes (1) 1.0
Night 0.5 no (0) 0

The sum is −2.0 + 1.5 + 1.0 + 0 = 0.5. The logistic function turns it into a score: 1 ÷ (1 + e−0.5) = 0.62. Each contribution says how much one feature moved the sum. The new device moved it most. Night moved it not at all, because this payment was not at night.

The contributions are measured from a reference: here, a payment with every feature at 0. Change the reference, and every contribution changes. Remember this. It matters for the real model.

The real re-ranker scales each feature first: a scaled value of 0 means “the average reviewed alert”. So its contributions compare the payment with the average alert, not with a normal payment. For the hook customer’s alert:

value scaled weight contribution
iso 0.79 1.57 1.98 3.11
is_business 0.00 -0.90 -1.72 1.56
new_device 1.00 0.85 1.78 1.52
new_country 1.00 1.17 1.22 1.42
night 1.00 1.56 0.68 1.06
log_minutes_since_last 0.76 -0.68 -0.91 0.62
device_payments_last_hour 8.00 -0.10 1.08 -0.11
log_amount_vs_usual 0.00 0.51 -0.72 -0.37
payments_last_hour 8.00 0.89 -0.55 -0.48
intercept 0.17 + contributions 8.34 = 8.50
score from the sum: 0.9998 | predict_proba: 0.9998

The split is exact: 0.17 + 8.34 = 8.50, and the logistic function of 8.50 is the model’s score.

The six features that pushed the re-ranker's score up for the hook customer's alert on 3 October 2025. Score: the Isolation Forest score. Personal: not a business account. Device: a new device. Country: a new country. Gap: a short time since the last payment. The other three features pushed it down a little (−0.11, −0.37, −0.48).
Text description of the chart

A column chart with six columns, from the largest to the smallest. The detector score contributes 3.11. Not being a business account contributes 1.56. A new device 1.52. A new country 1.42. Night 1.06. A short time since the last payment 0.62. Three features, not shown, pushed the score down. They are the device’s payments in the last hour (−0.11), the amount compared with usual (−0.37), and the customer’s payments in the last hour (−0.48).

Two contributions need care. “Not a business account” pushes the score up by 1.56. Why? Among the reviewed alerts, 45% were business accounts, mostly honest bulk orders. So a personal account is more suspicious than the average alert. That is true about the model. It is no help to Sofia.

And eight payments in the last hour pushes the score down (−0.48). Many real alerts are card tests, one payment per stolen card. Among the alerts, many payments by one customer went with “not”. This is the sign trap of Explain predictions: a weight describes the training rows, not the world.

SHAP (SHapley Additive exPlanations) is a method and a Python library. It splits any model’s score into one value per feature, so that the values add up to the score minus a base value. The base value is the model’s average output on a background data set. The machine learning course shows why contributions are not causes in Explain predictions. Read it before you show SHAP values to anyone.

For a linear model with independent features, the SHAP value of a feature is exactly its contribution, measured from the average of the background. Our background is the 1,400 reviewed alerts. Their scaled average is 0. So the base value is the intercept, and the SHAP values are the contributions above:

import shap
background = re_scaler.transform(labels[FEATS])
explainer = shap.LinearExplainer(re_ranker, shap.maskers.Independent(background, max_samples=len(background)))
values = explainer.shap_values(re_scaler.transform(alert[FEATS].to_frame().T.astype(float)))[0]
base value 0.17
largest difference from the contributions: 1.8e-15

The largest difference is 0.0000000000000018: the same numbers. One detail: by default, shap keeps only 100 rows of the background, chosen at random. The values then move a little, and the base value is no longer the intercept. The notebook uses all 1,400 rows.

For a linear re-ranker, you do not need the library. It becomes useful when the re-ranker is a model whose score does not split so easily, such as gradient-boosted trees. For the detector, this course uses the per-feature deviations: they are simpler to read.

Now write the alert. A good card has five parts:

  1. The payment, in one line: who, when, how much, where.
  2. The customer’s normal, from the customer’s own history.
  3. The top reasons, in words, from the contributions, keeping only reasons that describe the payment.
  4. What we left out, so nobody wonders.
  5. The decision and its first action.

Here is the card for the hook customer:

Two versions of the same alert. On the left, a score alone: alert P-0229866, customer C-01234, score 1.000. On the right, the card. Header: alert P-0229866, C-01234, 3 October 2025 at 02:21. Today: 8 payments in Portugal in euros, from 02:14 to 02:21, 170.23 dollars in all. This one: 19.25 dollars, mobile, home. Usual: Saturday baskets of 11.15 to 17.98 dollars, in Canada. Why it is high: first payment ever from Portugal, plus 1.42; a device this customer never used, plus 1.52; at 02:21, at night, plus 1.06; 8 payments in one hour, usually 1. Not on the card: high detector score, plus 3.11, and a personal account, plus 1.56. Decide: real, block the card and call the customer; not, mark not and write the reason.
The same alert, as a score alone and as a card. The card keeps the reasons that compare the payment with the customer's normal, and says which model reasons it left out.
Text description of the diagram

The picture shows two boxes. On the left, “A score alone”: alert P-0229866, customer C-01234, score 1.000. On the right, “The card”, in five parts. The header: alert P-0229866, customer C-01234, 3 October 2025 at 02:21. The payment: today, 8 payments in Portugal in euros, from 02:14 to 02:21, 170.23 dollars in all. This one is 19.25 dollars, on a mobile, for home goods. The normal: Saturday baskets of 11.15 to 17.98 dollars, in Canada. Then “Why it is high”, in a box with a strong border. Reason 1: first payment ever from Portugal, plus 1.42. Reason 2: a device this customer never used, plus 1.52. Reason 3: at 02:21, at night, plus 1.06. Reason 4: 8 payments in one hour, where the customer usually has 1. A dashed box lists what is not on the card: a high detector score, plus 3.11, and a personal account, plus 1.56. Both are true about the model, but no help to decide. The last box: decide. Real: block the card and call the customer. Not: mark “not” and write the reason, for example a trip.

Notice reason 4. Its contribution was negative, so the model did not use it as a reason. But it is the clearest sign for a person: eight payments in seven minutes, where the customer usually makes one. The deviations found it. A card can use both kinds of explanation.

With this card, Sofia can decide in under a minute: a stranger is using the card in Portugal at night.

The same day, payment P-0230910 is 27th in the re-ranked list, with a score of 0.387. Its largest contributions:

Feature Value Contribution
Business account no +1.56
New device yes +1.52
Payments in the last hour 3 +0.29
Detector score 0.656 -1.95
New country no -1.04

Write its card:

  • Payment: C-15551, 3 October at 17:25, 12.99 dollars, in Canada, on a mobile, garden products.
  • Normal: about 24 dollars a payment, in Canada, on another mobile.
  • Today: three payments in 13 minutes: 16.22, 11.55 and 12.99 dollars, 40.76 dollars in all. All are garden products, from the same new phone.
  • Why it is in the list: a device this customer never used, and three payments in a short time.

A reviewer reads it in a minute: a customer with a new phone, paying for one basket in three parts, in the late afternoon. She marks it not. The answer key agrees: this is a split basket, one of the normal-but-unusual cases of the course data. Without the card, the score of 0.387 says only “maybe”.

The comparison table assumes 3 minutes to understand one Isolation Forest alert. If a card brings that down to 1 minute, the team saves 2 minutes an alert: 100 minutes a day at 50 alerts. The one minute is a goal, an assumption of this lesson, not a measurement. To know the real number, time the reviews for a week with and without cards.

Show when to use each explanation, when not, and the cost

Use per-feature deviations when:

  • You want to describe the payment for a person. They work for any detector, Isolation Forest included.
  • The features have a clear “normal”: the customer’s own past, or the reference months.

Use contributions or SHAP when:

  • You want to know what moved the model’s score, to choose which reasons to show.
  • You want to find a weight that has learned something wrong.
  • The model is the re-ranker, or another model whose features a person can read.

Do not:

  • Show a contribution as a cause. “A personal account” moved the score. It did not make the payment fraud.
  • Show reasons that are true only about the model. They cost the reviewer time and trust.
  • Write a card from SHAP values of features that nobody can read, such as an autoencoder’s hidden numbers.

What it costs:

  • A function that writes the card, and a test that its numbers match the data. The next module adds reasons to every alert of the detection job.
  • Words for each feature, written once by the team, in Sofia’s language.
  • A check, every few months, that the cards still say true things after the model is retrained.
  1. Open the notebook from the box Do this lesson in a notebook at the top of this page.
  2. Run the setup cells. The data cell ends with the Ready: line.
  3. In section 1, run the cell. You see P-0229866 and customer C-01234.
  4. In section 2, run the deviations cell. new_country says yes (1.1% of normal payments say yes).
  5. In section 3, run the contributions cell. You see intercept 0.17 + contributions 8.34 = 8.50.
  6. In section 4, run the SHAP cell. You see base value 0.17.
  7. In section 5, run the card cell. Read the three reasons, and find the one that is true only about the model.
  8. In section 6, write the two answers, and run each check cell. Each prints Check passed.

Change one thing: explain payment P-0230910 instead of the hook customer’s payment. Before you run it, predict: which feature pushes its score up the most?

Show the answer
second = day[day.payment_id == "P-0230910"].iloc[0]
largest_push = contributions(second).contribution.idxmax()
t2 = contributions(second)
score_from_sum = round(float(1 / (1 + np.exp(-(re_ranker.intercept_[0] + t2.contribution.sum())))), 3)
Check passed: largest_push is right.
Check passed: score_from_sum is 0.387.

The largest push is is_business: “not a business account”, +1.56. The new device is a close second, +1.52. So the automatic card would start with a reason that is true only about the model. That is why a person chooses the words of the card, once, for every feature. Then the card keeps the new device and the three payments, and drops “personal account”.

Common mistake

A card that lists the model's reasons

Symptom: Tomás writes the card from the three largest SHAP values, with no other step. For the hook customer, the card says: “unusual overall (detector score 0.79); a personal account; a device this customer never used”. Sofia reads “a personal account” on almost every alert. After a week, she stops reading the reasons.

Cause: SHAP values explain the model, measured from the average reviewed alert. A personal account is more suspicious than the average alert, because so many alerts were business bulk orders. That makes it a large SHAP value. But it says nothing about this payment that Sofia can check.

Fix: write a card from both explanations. Use the contributions to choose the candidates for reasons. Keep only the ones that compare the payment with the customer’s normal, in words. Add strong deviations that the model does not use, such as eight payments in one hour. Say which model reasons you left out.

Consequence in AI work: an explanation that is true about the model but useless to the person teaches the person to ignore all explanations. Then a wrong alert passes as easily as a right one.

Check your understanding

What is the difference between a per-feature deviation and a SHAP value (or contribution)? Give one example from the hook customer's alert where they disagree.

Show the answer

A per-feature deviation compares the payment with normal payments, one feature at a time. It works for any detector, and it says what is unusual about the payment. A SHAP value says how much a feature moved the model’s score, measured from the average reviewed alert. For the linear re-ranker, it is the same as the contribution. They disagree on the payments in the last hour. 8 against a normal of 1 is a strong deviation, but its contribution is negative (−0.48). Among the alerts, many real frauds were single card tests. They also disagree on “not a business account”: not unusual at all, but a contribution of +1.56.

Check your understanding

For the re-ranker, show how the score of the hook customer's alert is built from its parts, and say why the SHAP values equal the contributions.

Show the answer

The intercept, 0.17, plus the nine contributions (weight times scaled value), 8.34, gives a sum of 8.50. The logistic function 1 ÷ (1 + e−8.50) gives 0.9998, the model’s score. For a linear model with independent features, a SHAP value is the weight times the distance from the background average. The background is the 1,400 reviewed alerts, and their scaled average is 0. So each SHAP value is the contribution, and the base value is the intercept. The notebook’s largest difference is 1.8e-15.

Check your understanding

Write the five parts of an alert card that a reviewer can decide on in one minute, and say which reasons you leave out, and why.

Show the answer

The five parts: the payment in one line (who, when, how much, where); the customer’s normal; the top reasons in words; the reasons left out; the decision with its first action. Leave out reasons that are true only about the model, such as “a personal account” or “a high detector score”. The reviewer cannot check them against the payment, and they waste her minute. Keep reasons that compare the payment with the customer’s normal. Add strong deviations that the model does not use, such as eight payments in one hour.

Recap

  • Per-feature deviations compare a payment with normal, one feature at a time, for any detector.
  • For the linear re-ranker, the score splits exactly into contributions: 0.17 + 8.34 = 8.50 for the hook customer. SHAP values are the same numbers, measured from the average reviewed alert.
  • A contribution is the model’s view, not a cause. “A personal account” moved the score; it is no help to the reviewer.
  • A good alert card shows the payment, the customer’s normal, the top reasons in words, what was left out, and the first action. The goal: a decision in one minute instead of three (an assumption until you time it).

Module 8 is complete: the reviewers’ labels re-rank the alerts, a random sample keeps the labels honest, and each alert explains itself. Next, in Module 9, we put the detector into a scheduled job. It reads new payments, scores them, groups them and sends alerts with their reasons. And it never sends an alert twice.

Next: From score to alert