The feedback loop and its traps
- Lesson 34
- 35 min
- Module 8: Learn from the reviewers
Where we are
Our re-ranker learns from the alerts that people reviewed. But people review only what the detector shows, so the labels repeat its choices. In this lesson, we see this selection bias in numbers, and we reduce it with a small random sample for review.
Lesson 34 of 38 · Anomaly Detection: Find What Is UnusualThe feedback loop and its traps
1. What "unusual" means
2. Simple baselines that are hard to beat
3. Statistical process control and distribution drift
4. Distance, density and isolation
5. Anomalies in time
6. Deep and embedding methods
7. Evaluate with few labels
8. Learn from the reviewers
9. Run a detector in production
What you will be able to do
See the trap in every system that learns from its own alerts: the labels exist only where the detector looked. Measure it, and reduce it with a small random sample of reviews and an honest “unknown”.
NotebookDo this lesson in a notebook
The notebook has every code cell, task and check of this lesson. It measures the share of real alerts in the labels and the labels of each kind of fraud. Then it counts the re-ranker’s account takeovers, takes a random sample of reviews and tests “unknown”. Its setup cells install the packages and download the data (about 7 MB). About 35 minutes.
Colab and Kaggle open your own copy. They need a free Google or Kaggle account. Stuck? See the solution notebook.
Before you startWhat to know and to have ready
What you need to know
- The two stages, the candidates and the re-ranker, from From unsupervised to semi-supervised.
- Why reviewed labels are not a fair sample, first met in Labels are rare, and that changes everything. This lesson measures it.
What you need on your computer
Nothing. The notebook runs in Colab, Kaggle or VS Code.
Tested with Python 3.12.13 in Jupyter on macOS, with pandas 3.1.1 and scikit-learn 1.9.1. Colab and Kaggle were not tested.
The keys under the street light
Section titled “The keys under the street light”An old story: a man looks for his keys under a street light at night. A friend helps him search. After some minutes, the friend asks: “Are you sure you lost them here?” The man says: “No, I lost them in the park. But this is where the light is.”
Every detector has a street light. Sofia’s team looks only at the 50 alerts that the detector shows. Every label comes from that small lit area. A fraud in the dark part of the list never gets a label. If a model learns only from the labels, it learns to search under the same light, a little better each time.
The problem: the labels repeat the detector’s choices
Section titled “The problem: the labels repeat the detector’s choices”In the last lesson, the re-ranker learned from 1,400 reviewed alerts. It raised precision at the budget. But look at one kind of fraud. In an account takeover, a thief gets into a customer’s account. The thief often uses the customer’s own device, in the customer’s own country. These frauds look more normal than a stolen card used abroad.
In the test weeks, the forest alone put 76 account-takeover payments in the top 50 of each day. After re-ranking, 71. The re-ranker raised precision overall, but it found fewer account takeovers. It learned mostly from the kinds that the team saw most often, and account takeovers looked like the “not” alerts.
The idea, and its name
Section titled “The idea, and its name”When the rows that get labels are not chosen at random, the labels do not look like the whole data. This is selection bias. Here, the detector chooses the rows, and a model learns from them and then chooses the next rows. The loop feeds itself. This is a feedback loop: the output of the system becomes its next input.
Text description of the diagram
The diagram shows a loop of four steps. Detect: the detector scores every payment. Review: the team reviews the top 50 a day. Label: each alert gets real, not or unknown. Re-rank: a model learns from the labels and orders tomorrow’s alerts, which goes back to Review. A side branch leaves Detect: 150 candidates a day that are never shown, with no label. From them, a box “Random sample, 5 a day from the unseen candidates” leads to Label. A dashed arrow from the unseen candidates to Re-rank says “frauds here are never learned”.
A tiny example: ten payments, three reviews
Section titled “A tiny example: ten payments, three reviews”Here is a constructed example. A detector scores ten payments. The team can review three. Three payments are fraud: two stolen cards and one account takeover.
| Rank | Payment | Truth | Reviewed? | Label |
|---|---|---|---|---|
| 1 | P1 | stolen card | yes | real |
| 2 | P2 | stolen card | yes | real |
| 3 | P3 | honest | yes | not |
| 4 | P4 | honest | no | none |
| 5 | P5 | honest | no | none |
| 6 | P6 | account takeover | no | none |
| 7 | P7 | honest | no | none |
| 8 | P8 | honest | no | none |
| 9 | P9 | honest | no | none |
| 10 | P10 | honest | no | none |
Count the shares:
- In the labels, 2 of 3 are real: 67%.
- In all ten payments, 3 of 10 are fraud: 30%.
- Account takeovers with a label: 0 of 1.
A model that learns from these three labels learns “stolen cards are real”. It has no example of an account takeover. So it cannot push P6 up. The next day, P6’s kind stays in the dark again.
Now add one review chosen at random from the seven unseen payments. With a chance of 1 in 7, it is P6. Over many days, the random reviews find some account takeovers, and they also tell you how much fraud the top three miss.
Larkfield: the bias in numbers
Section titled “Larkfield: the bias in numbers”Now the real data: the four weeks of simulated reviews from the last lesson. The notebook counts the share of real fraud in four groups of payments:
reviewed alerts (the labels) 0.506all candidates (top 200) 0.157candidates not shown (51st to 200th) 0.041all payments 0.018Text description of the chart
A column chart with four columns. The reviewed alerts, the labels, are 50.6% real fraud. All candidates, the top 200 of each day, are 15.7%. A random sample of the candidates that the team did not see is 2.9%. All payments of the four weeks are 1.8%. The first column is 28 times as high as the last.
The labels are 50.6% fraud. All payments are 1.8% fraud. The labels are full of fraud because the detector chose them. That is fine for Sofia: she wants her 50 slots full of fraud. But a model that learns from the labels sees a world where half of all unusual payments are fraud. That world does not exist.
Which kinds get labels
Section titled “Which kinds get labels”The bias is not the same for every kind of fraud. Here are the planted payments of each kind in the four weeks, and how many got a label:
| Kind | Planted payments | With a label | Share labelled |
|---|---|---|---|
| Stolen card | 440 | 390 | 88.6% |
| Card testing | 400 | 297 | 74.3% |
| Account takeover | 54 | 21 | 38.9% |
Text description of the chart
A column chart with three columns. Of the stolen-card payments, 88.6% got a label. Of the card tests, 74.3%. Of the account takeovers, 38.9%: only 21 of 54.
The detector sees stolen cards well: a new country, a new device, many payments. It sees account takeovers less well. So the labels hold 390 stolen-card payments and only 21 account takeovers. From 21 examples, the re-ranker learns little about account takeovers. That is why it found 71 instead of 76 in the test weeks.
The loop over months
Section titled “The loop over months”Now imagine that the loop runs for a year. Each month, the re-ranker trains on the labels of the month before. Those labels hold fewer account takeovers, because the last re-ranker pushed them down. So the next re-ranker learns even less about them. Nothing in the precision at the budget warns you: it stays high, because the slots are full of the kinds that the model knows. The kinds that it does not know disappear from the labels, from the reports and from the team’s attention.
The step that reduces it: a small random sample
Section titled “The step that reduces it: a small random sample”Each day, the team reviews a few more candidates, chosen at random from those it did not see. Nobody chose these payments because they looked like fraud. So their labels are a fair sample of the unseen candidates. Here: 5 a day, from the 150 candidates ranked 51st to 200th, for the four weeks.
random sample: 140 reviews, 4 real (2.9%); 95% interval 1.1% to 7.1%estimate: about 4.3 real payments a day among the 150 candidates that the team did not seeThe sample says that about 4 real frauds a day sit just below the top 50. The interval is wide, because 140 reviews is a small sample. The 95% interval of Evaluate without a full answer key goes from 1.1% to 7.1%.
The course data has an answer key, so we can check. The true share is 4.1%: 6.2 real payments a day among the unseen candidates. The estimate is low, but the truth is inside its interval. A real team has no answer key. The random sample is the only honest number it can get.
What the random sample does not do
Section titled “What the random sample does not do”Does the random sample make the re-ranker better? Not here. Retrained with the 140 extra labels, its precision at the budget is 0.514, the same as before. The sample added only 4 real payments: too few to teach anything.
So be clear about its job. The random sample is for measuring: how much fraud does the top 50 miss, and which kinds? Over months, it also gives labels to kinds that the detector rarely shows. It is not a trick to raise this week’s precision.
What it costs
Section titled “What it costs”Five extra reviews a day, at 4 minutes each, is 20 minutes a day of Sofia’s team. That is a tenth of the time of the 50 normal reviews (an assumption of the course: 4 minutes a review). In exchange, the team learns how much it misses. Without the sample, nobody knows.
“Not sure” is not “not”
Section titled ““Not sure” is not “not””A reviewer cannot always decide. An account takeover can look like the customer on a new phone. The honest record then has three answers: real, not and unknown. Unknown means: we do not know yet, ask later or ask the customer.
What happens if the team writes “not sure” as “not”? The notebook pretends that the team was unsure about every account takeover. It trains the re-ranker in two ways:
written as not precision 0.509, account takeovers found 59left out precision 0.510, account takeovers found 61With the right answers, the re-ranker found 71 account takeovers. Written as “not”, the 21 account takeovers teach it that this kind is honest: it finds 59. Left out as unknown, the re-ranker loses their information, but it learns nothing wrong: 61. The difference is small here, because there were only 21 such labels. With more labels, a wrong “not” does more harm.
So keep “unknown” as its own answer. Leave unknown alerts out of training. Then come back to them when the answer arrives: a chargeback from the bank, or a call with the customer.
When to worry, and what to do
Section titled “When to worry, and what to do”Show when selection bias matters, and the steps against it
It matters when:
- A model learns from labels that its own alerts produced. Re-rankers, fraud classifiers and spam filters all do this.
- Some kinds of anomaly look normal to the detector. They get few labels, and the next model sees them even less.
- You report precision at the budget only. It stays high while whole kinds disappear.
The steps:
- Review a small random sample every day, from the candidates that the team did not see. Here: 5 a day.
- Count labels by kind, every month. A kind whose labels fall is a warning.
- Keep unknown as its own answer. Never write it as “not”.
- Keep the unsupervised detector as stage one. It does not learn from the labels, so the loop cannot narrow it.
Guided practice: the bias in the notebook
Section titled “Guided practice: the bias in the notebook”- Open the notebook from the box Do this lesson in a notebook at the top of this page.
- Run the setup cells. The data cell ends with the
Ready:line. - In section 1, run the cell. You see
1,400 reviewed alerts, 708 real. - In section 2, write your prediction, then run the shares cell. You see
0.506and0.018. - Run the kinds cell. Account takeovers have
54planted payments and21with a label. - In section 3, run the re-ranker cell. You see
76and71account takeovers. - In section 4, run the random sample cells. You see
4 real (2.9%), then4.1%from the answer key. - In section 5, run the “unknown” cell. You see
59and61. - In section 6, write the two answers, and run each check cell. Each prints
Check passed.
Your turn: a larger random sample
Section titled “Your turn: a larger random sample”Change one thing: review 10 random candidates a day instead of 5. Before you run it, predict: will the estimate of the missed frauds a day move closer to the true 6.2?
Show the answer
sample_10 = unseen.groupby("day", group_keys=False).apply(lambda g: g.sample(10, random_state=1))missed_per_day_10 = round(float(sample_10.label.mean() * 150), 1)Check passed: missed_per_day_10 is 6.4.With 10 reviews a day (280 in all), the estimate is 6.4 real payments a day. The truth is 6.2. A larger sample gives a narrower interval, so a single estimate is less likely to be far off. It costs twice the time: 40 minutes a day instead of 20. Your random numbers may give another value if you change the seed. That is the point of the interval.
Common mistake
The precision that stayed high while a kind disappeared
Symptom: after the re-ranker goes live, Sofia’s weekly report looks better: precision at the budget is up, from 0.498 to 0.514. Three months later, the bank reports a wave of chargebacks from account takeovers. None of them was in the team’s alerts. (The story is constructed; the numbers below are measured.)
Cause: the re-ranker learned from labels that held 390 stolen-card payments and only 21 account takeovers. It moved account takeovers down: 76 in the top 50 with the forest alone, 71 after re-ranking. Precision stayed high, because the freed slots went to stolen cards. The report counted all real alerts together, so it hid the loss.
Fix: report recall and labels by kind, every week. Keep a random sample of reviews, so that the team can estimate what it misses. And retrain only on labels that include a fair share of each kind, or weight the rare kinds.
Consequence in AI work: a model that learns from its own outputs narrows its view a little each time. The usual scores do not show it. Only a sample that the model did not choose shows what is outside the light.
Knowledge check
Section titled “Knowledge check”Check your understanding
Show the selection-bias trap with numbers. What share of the reviewed alerts is real, what share of all payments, and why do they differ?
Show the answer
In the four weeks of reviews, 50.6% of the 1,400 reviewed alerts are real (708). Of all payments, only 1.8% are planted fraud. Of all 200 candidates a day, 15.7%. They differ because the detector chose which payments got a label: it showed its most suspicious payments, so the labels are full of fraud. They are not a random sample of payments, and a model trained on them learns a world where half of the unusual payments are fraud.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
Name the step that reduces the bias, say what it measured at Larkfield, and say what it did not do.
Show the answer
Review a small random sample every day from the candidates that the team did not see: 5 a day from ranks 51 to 200. In four weeks, 140 random reviews found 4 real payments: 2.9%, with a 95% interval from 1.1% to 7.1%. That estimates about 4 missed frauds a day just below the top 50 (the answer key says 6.2, inside the interval). It did not raise precision at the budget: the re-ranker with the extra labels had 0.514, the same as without. Its job is to measure what is missed, not to raise this week’s score.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
A reviewer is not sure about an alert. Why should she record “unknown” and not “not”? Use the account-takeover numbers.
Show the answer
“Not” is a label that teaches the model that this kind of payment is honest. When every unsure account takeover was written as “not”, the re-ranker found 59 account takeovers in the test weeks. Left out as unknown, it found 61. With the right answers, 71. “Unknown” loses some information, but it does not teach the wrong answer. Keep unknown alerts out of training, and add them back when the true answer arrives.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Recap
- Labels exist only where the detector looked. That is selection bias, and a model that learns from its own alerts makes it a feedback loop.
- At Larkfield: 50.6% of the reviewed alerts are real, against 1.8% of all payments. Only 38.9% of the account takeovers got a label, and the re-ranker found fewer of them (76 → 71).
- A small random sample of reviews, 5 a day, measures what the top 50 misses: here about 4 a day (truth 6.2). It does not raise precision.
- Record unknown honestly. Written as “not”, it teaches the model the wrong answer.
The re-ranker now puts the real alerts first, and the random sample keeps it honest. But each alert is still only a score. Next, we find which features made a score high, and we write an alert that Sofia can decide on in a minute.
Next: Explain an alert