From unsupervised to semi-supervised
- Lesson 33
- 35 min
- Module 8: Learn from the reviewers
Where we are
Sofia's team reviews 50 alerts a day and marks each one as real or not. Our detectors ignore those answers. In this lesson, we use the reviewers' labels to train a small classifier on top of the anomaly scores, and we see how much it improves precision at the budget.
Lesson 33 of 38 · Anomaly Detection: Find What Is UnusualFrom unsupervised to semi-supervised
1. What "unusual" means
2. Simple baselines that are hard to beat
3. Statistical process control and distribution drift
4. Distance, density and isolation
5. Anomalies in time
6. Deep and embedding methods
7. Evaluate with few labels
8. Learn from the reviewers
9. Run a detector in production
What you will be able to do
Use the answers that reviewers already give every day. A small second model learns from them and puts the real alerts first. Then measure, honestly, how much it helps at the budget of 50 alerts a day.
NotebookDo this lesson in a notebook
The notebook has every code cell, task and check of this lesson. It scores the payments with Isolation Forest, plays four weeks of reviews, trains the re-ranker and measures it on the weeks after. Its setup cells install the packages and download the data (about 7 MB). About 35 minutes.
Colab and Kaggle open your own copy. They need a free Google or Kaggle account. Stuck? See the solution notebook.
Before you startWhat to know and to have ready
What you need to know
- The alert budget of 50 a day and precision at the budget, from The payment that did not fit.
- The semi-supervised setting and why reviewed labels are rare, from Labels are rare, and that changes everything.
- A logistic regression turns a weighted sum of features into a score from 0 to 1. Classification intuition in Machine Learning: From Problem to Reliable Model explains it. This lesson explains the parts that it uses.
What you need on your computer
Nothing. The notebook runs in Colab, Kaggle or VS Code.
Tested with Python 3.12.13 in Jupyter on macOS, with pandas 3.1.1 and scikit-learn 1.9.1. Colab and Kaggle were not tested.
The new colleague at the returns desk
Section titled “The new colleague at the returns desk”Think of a new person at a shop’s returns desk. A rule tells her to call a manager for every return without a receipt. In her first week, she calls the manager twenty times a day. The manager comes, looks, and says “that one is fine” most of the time.
After a month, she has learned something. A regular customer who returns a gift without a receipt is almost always fine. A stranger who returns five new phones is not. She still follows the rule, but she puts the calls in a better order. The manager now sees the worrying cases first.
Nobody gave her a new rule. She learned from the manager’s answers. This lesson does the same for Larkfield’s alerts.
The problem: the detector never learns
Section titled “The problem: the detector never learns”Isolation Forest is the detector of this module. You built it in Isolation Forest. Here you need only one fact. It gives each payment a score: the higher the score, the more unusual the payment. Each day, Sofia’s team reviews the 50 payments with the highest scores.
On good features, about half of those 50 alerts are planted frauds. The other half are honest payments that look unusual. Sofia’s team marks each alert real or not. Then the next day comes, and the detector scores the new payments in exactly the same way. It does not know what the team said yesterday.
So the same kinds of false alert come back every day. Business customers who buy in bulk are one example. Slices and fairness measured false alerts per 10,000 honest payments. For business accounts, it was 444.1; for personal accounts, 47.5. Sofia’s team says “not” to these bulk orders again and again. Each “not” costs 4 minutes, about 3 dollars (an assumption of the course). And the detector ignores every one of those answers.
The idea: a second model that re-orders the alerts
Section titled “The idea: a second model that re-orders the alerts”The reviewers’ answers are labels: the right answer for one example, written down. We cannot use them to replace the detector, because labels exist only for the alerts that the team saw. But we can use them for a smaller job: to re-order the detector’s top candidates.
This is a two-stage design:
- Stage one, the detector. Isolation Forest scores every payment. Each day, its 200 highest scores are the candidates. It needs no labels.
- Stage two, the re-ranker. A small classifier learns from the reviewed alerts. It gives each candidate a new score: how likely it is to be real. The team reviews the 50 candidates with the highest new scores.
The second model is called a re-ranker, because it changes the ranking, the order of the list. It never looks at all 1,875 payments of a day. It looks only at the detector’s candidates.
Text description of the diagram
The diagram is a flow from left to right. All payments of a day, about 1,875, go into stage one: Isolation Forest, which needs no labels. Its output is the 200 highest scores of the day, the candidates. They go into stage two: the re-ranker, a logistic regression on the forest score and eight behaviour features. Its output is the top 50, which go to Sofia’s team. The team marks each alert real or not. A dashed arrow goes from these labels back to the re-ranker, labelled “four weeks of labels train it”.
Why the name “semi-supervised”
Section titled “Why the name “semi-supervised””In Labels are rare, and that changes everything, “semi-supervised” meant: learn from data that you trust to be normal. The word has a second common meaning: a lot of data without labels, plus a few rows with labels. The two-stage design is this second kind. The detector learns from all payments without labels. The re-ranker learns from the few that were reviewed. The diagram in that lesson had an arrow for this step: “reviews add a few labels”.
A tiny example: six alerts and a budget of three
Section titled “A tiny example: six alerts and a budget of three”Here is a constructed example, small enough to check by hand. A detector gives six candidates. The team can review three. Each candidate has two facts: was the device new for this customer, and is it a business account? The last column is the reviewers’ answer, from earlier days.
| Alert | Forest score | New device | Business account | Reviewers said |
|---|---|---|---|---|
| A | 0.80 | no | yes | not |
| B | 0.78 | yes | no | real |
| C | 0.76 | no | yes | not |
| D | 0.74 | yes | no | real |
| E | 0.72 | yes | no | real |
| F | 0.70 | no | no | not |
The detector’s top three are A, B and C. Only B is real: a precision of 1 ÷ 3 = 0.33.
From earlier days, the reviewers learned two patterns. A new device often meant fraud. A business account often meant a bulk order. So we give each candidate a new score: the forest score, plus 0.10 for a new device, minus 0.10 for a business account.
| Alert | Forest score | + new device | − business | New score | New rank |
|---|---|---|---|---|---|
| A | 0.80 | 0 | −0.10 | 0.70 | 4 |
| B | 0.78 | +0.10 | 0 | 0.88 | 1 |
| C | 0.76 | 0 | −0.10 | 0.66 | 6 |
| D | 0.74 | +0.10 | 0 | 0.84 | 2 |
| E | 0.72 | +0.10 | 0 | 0.82 | 3 |
| F | 0.70 | 0 | 0 | 0.70 | 4 |
The new top three are B, D and E. All three are real: a precision of 3 ÷ 3 = 1.00. The detector found the right candidates. It only put them in the wrong order.
Here, we chose the weights 0.10 by hand. A real re-ranker learns its weights from the labels. Logistic regression does exactly that: it finds the weights that best separate “real” from “not” in the reviewed alerts. Then it turns the weighted sum into a number between 0 and 1.
Larkfield: four weeks of reviews
Section titled “Larkfield: four weeks of reviews”Now the real data. The course plays Sofia’s team for the four weeks from 1 to 28 September 2025. Each day, Isolation Forest scores the payments, the 200 highest scores are the candidates, and the team reviews the top 50. Every reviewed alert gets its answer from the answer key.
This is an assumption: real reviewers make mistakes, and some answers arrive weeks later. Here, every answer is right and immediate.
18,200 candidates in 91 days1,400 reviewed alerts in 4 weeks, 708 real (50.6%)So the re-ranker learns from 1,400 labels: 708 real and 692 not.
What the re-ranker looks at
Section titled “What the re-ranker looks at”The re-ranker uses nine numbers for each candidate: the forest score and the eight behaviour features. Each feature compares a payment with the customer’s own past, as Module 4 builds them:
| Feature | What it says |
|---|---|
iso |
The Isolation Forest score: how unusual the payment is overall |
log_amount_vs_usual |
The amount compared with the customer’s usual amount (0 = the same, on a log scale) |
new_country |
1 if the customer never paid from this country before today |
new_device |
1 if the customer never used this device before today |
log_minutes_since_last |
The time since the customer’s last payment (log scale) |
payments_last_hour |
The customer’s payments in the last hour |
device_payments_last_hour |
The device’s payments in the last hour, for all customers |
night |
1 if the payment was between midnight and 6 in the morning |
is_business |
1 for a business account |
Train it
Section titled “Train it”The notebook scales the nine numbers so that each has a mean of 0 and the same spread. Then it fits a logistic regression on the reviewed alerts only:
FEATS = ["iso"] + list(behaviour.columns)re_scaler = StandardScaler().fit(labels[FEATS])re_ranker = LogisticRegression(max_iter=2000).fit(re_scaler.transform(labels[FEATS]), labels.label)Each feature gets a weight. A positive weight pushes a candidate up the list. A negative weight pushes it down. Because the features are scaled, you can compare the sizes:
| Feature | Weight |
|---|---|
iso |
1.98 |
new_device |
1.78 |
is_business |
-1.72 |
new_country |
1.22 |
device_payments_last_hour |
1.08 |
log_minutes_since_last |
-0.91 |
log_amount_vs_usual |
-0.72 |
night |
0.68 |
payments_last_hour |
-0.55 |
The three largest weights are the forest score, a new device and a business account. The business weight is negative: among the alerts, business accounts were mostly bulk orders, so the re-ranker moves them down. That is the returns-desk colleague’s lesson, learned from 1,400 answers.
Two weights look strange. A higher amount than usual has a negative weight, and so do more payments in the last hour. Why? The re-ranker learned from alerts, not from all payments. Among the alerts, many real frauds are card tests: tiny amounts, far below the customer’s usual amount. A weight describes what separates real from not inside the candidates. It does not describe fraud in general.
The result, told honestly
Section titled “The result, told honestly”Now the test. The re-ranker re-orders the candidates of the following weeks, 29 September to 30 November 2025 (63 days). It never saw these days. The team still reviews 50 a day.
test weeks: Isolation Forest alone 0.498 precision, 0.834 of the real candidates found re-ranked 0.514 precision, 0.862 of the real candidates foundPrecision at the budget goes from 0.498 to 0.514. That is a real gain, and a small one. Count the real payments over the 63 days. The team finds 1,568 with the forest alone and 1,619 with the re-ranker: 51 more, less than one a day. False alerts fall from 1,582 to 1,531. Most bulk orders leave the top 50, but other false alerts take their slots: the failure case below shows the numbers.
The course’s own measurement gives a re-ranker three more scores: the autoencoder of Module 6, MAD per customer and the two rules of Module 2. It reaches 0.516 (recall 0.834 → 0.865). More inputs, almost the same result.
Why the gain is small
Section titled “Why the gain is small”Is 0.514 good or bad? Compare it with the best possible order. Suppose a re-ranker knew the answers and put every real candidate first. In the test weeks, its precision at the budget would be 0.533, not 1. On 55 of the 63 test days, the 200 candidates hold fewer than 50 real payments; the median day has 28. No order can fill 50 slots with 28 frauds.
Text description of the chart
A column chart with three columns, on an axis from 0 to 0.6. Isolation Forest alone has 0.498. The re-ranker has 0.514. A re-ranker that knows the answers, and so puts every real candidate first, has 0.533. The three columns are close: the forest already puts most real candidates in the top 50.
So the gap between the forest and the best order is only 0.035. The re-ranker closes 0.016 of it, about half. Isolation Forest on good features already puts most real candidates in the top 50. A re-ranker helps most where the detector’s top alerts hold many false alerts. It helps even more when those false alerts repeat a pattern that labels can teach.
Not a row in the comparison table
Section titled “Not a row in the comparison table”The comparison table scores every detector on the same three months, September to November. The re-ranker cannot use September: it learns from it. So its numbers come from the later weeks only, and they do not go into the table. Compare it only with the forest on the same weeks, as above.
When to use a re-ranker, and what it costs
Section titled “When to use a re-ranker, and what it costs”Show when to use it, when not, and the cost
Use it when:
- The team gives labels every day, and they arrive soon.
- The same kinds of false alert come back, like the bulk orders.
- You have a few hundred labels, with both answers. Here: 708 real and 692 not.
Do not use it when:
- You have only a few dozen labels. The weights then follow chance.
- The reviewers’ answers are often wrong or late. The re-ranker copies their mistakes.
- New kinds of fraud matter most. The re-ranker learns the kinds it has seen. The next lesson shows what it does to the kinds it has rarely seen.
What it costs:
- A second model to train, store and check. Logistic regression trains in under a second here.
- A schedule: train again every few weeks, on the newest labels, and measure on the weeks after.
- Keep the detector as stage one. Without it, nothing finds the candidates that nobody has labelled yet.
Guided practice: the re-ranker in the notebook
Section titled “Guided practice: the re-ranker in the notebook”- Open the notebook from the box Do this lesson in a notebook at the top of this page.
- Run the setup cells. The data cell ends with the
Ready:line. - In section 2, run the first cell. You see
18,200 candidates in 91 days. - Run the review cell. You see
1,400 reviewed alerts in 4 weeks, 708 real (50.6%). - In section 3, run the training cell. The largest weights are
iso,new_deviceandis_business. - Write your prediction for the test weeks. Then run the test cell. You see
0.498and0.514. - In section 5, write the two answers, and run each check cell. Each prints
Check passed.
Your turn: fewer weeks of reviews
Section titled “Your turn: fewer weeks of reviews”Change one thing: train the re-ranker on two weeks of reviews (1 to 14 September) instead of four. Test it on the same weeks. Before you run it, predict: does precision at the budget fall a lot, a little, or not at all?
Show the answer
labels_2w = candidates[(candidates.day <= "2025-09-14") & candidates.shown]s2 = StandardScaler().fit(labels_2w[FEATS])r2 = LogisticRegression(max_iter=2000).fit(s2.transform(labels_2w[FEATS]), labels_2w.label)test["re_rank_2w"] = r2.predict_proba(s2.transform(test[FEATS]))[:, 1]precision_two_weeks = precision_at_50(test, "re_rank_2w")[0]Check passed: precision_two_weeks is right.With two weeks of reviews (700 labels), the precision at the budget is 0.511. With four weeks, it is 0.514. Here, two weeks of labels already teach most of what four weeks teach. The patterns are few and strong: a new device, a business account. With weaker patterns, you would need more weeks.
Common mistake
85% fewer false alerts, and almost the same precision
Symptom: Omar reads the new weekly numbers. In the test weeks, false alerts from business accounts fell from 1,102 to 168. He tells the team: “The re-ranker removed 85% of the false alerts.” But Sofia’s team does not feel the difference.
false alerts in the top 50, 63 days: business accounts 1102 -> 168 all 1582 -> 1531Cause: the budget is fixed at 50 alerts a day. When the re-ranker moves a bulk order down, the next candidate takes its slot. Most of the time, that candidate is another false alert, from a personal account. So one kind of false alert almost disappeared, and the total fell by only 51, about 3%.
Fix: measure the whole top 50, every day: precision at the budget, recall and cost per week. A count of one kind of alert is not a result. Report it beside the totals, as here.
Consequence in AI work: a number about one slice can sound like a big win while the team’s workload stays the same. With a fixed budget, ask: what took the empty slots?
Knowledge check
Section titled “Knowledge check”Check your understanding
Explain the two stages of a re-ranker design. What does each stage learn from, and why do we not replace the detector with a classifier trained on the reviews?
Show the answer
Stage one, the detector, scores every payment without labels and keeps the top candidates (here, 200 a day). Stage two, the re-ranker, is a small classifier trained on the reviewed alerts. It re-orders the candidates, and the team reviews the top 50. We keep the detector because labels exist only for alerts that the team saw. A classifier trained on them knows nothing about payments that no detector showed. The detector is what finds new and unlabelled cases.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
The re-ranker raised precision at the budget from 0.498 to 0.514 on the test weeks. A perfect order of the candidates would give 0.533. What do these three numbers tell you, and why is the gain small?
Show the answer
The gain is real but small: about 51 more real payments in 63 days, less than one a day. The best possible order of the same candidates reaches only 0.533, because on most days the candidates hold fewer than 50 real payments. So the forest was already close to the ceiling: the gap was 0.035, and the re-ranker closed about half of it. A re-ranker helps more when the detector’s top alerts mix many false alerts that repeat a pattern.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
The weight of “amount compared with usual” is negative in the re-ranker. Does this mean that higher amounts are less likely to be fraud? Explain.
Show the answer
No, not in general. The re-ranker learned only from the detector’s alerts. Among the alerts, many real frauds are card tests with tiny amounts, far below the customer’s usual one. So inside the candidates, a lower amount goes with “real”. A weight describes what separates real from not among the alerts that the model learned from. It says nothing about all payments.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Recap
- Reviewers’ answers are labels. A re-ranker, a small classifier on top of the detector, can learn from them and put the real alerts first.
- Two stages: the detector finds the candidates without labels; the re-ranker orders them. Keep both.
- On Larkfield’s test weeks, precision at the budget went from 0.498 to 0.514. The best possible order of the candidates gives 0.533. A small gain, because the forest was already close to the ceiling.
- Train on earlier weeks and measure on later weeks that the re-ranker never saw.
- With a fixed budget, a removed false alert leaves a slot for the next candidate. Measure the whole top 50.
The re-ranker learned only from the alerts that the team saw. It never saw a fraud that the detector ranked 120th. Next, we measure what this does to the labels, and to the kinds of fraud that the detector rarely shows.