Skip to content
Français

The feedback loop and its traps

  • Lesson 34
  • 35 min
  • Module 8: Learn from the reviewers

Where we are

Our re-ranker learns from the alerts that people reviewed. But people review only what the detector shows, so the labels repeat its choices. In this lesson, we see this selection bias in numbers, and we reduce it with a small random sample for review.

Plays from YouTube. Nothing loads from YouTube until you press play. Privacy
Lesson 34 of 38 · Anomaly Detection: Find What Is UnusualThe feedback loop and its traps
Course overview

What you will be able to do

See the trap in every system that learns from its own alerts: the labels exist only where the detector looked. Measure it, and reduce it with a small random sample of reviews and an honest “unknown”.

NotebookDo this lesson in a notebookColab · Kaggle · GitHub

The notebook has every code cell, task and check of this lesson. It measures the share of real alerts in the labels and the labels of each kind of fraud. Then it counts the re-ranker’s account takeovers, takes a random sample of reviews and tests “unknown”. Its setup cells install the packages and download the data (about 7 MB). About 35 minutes.

Colab and Kaggle open your own copy. They need a free Google or Kaggle account. Stuck? See the solution notebook.

Before you startWhat to know and to have readyAbout 35 minutes

What you need to know

What you need on your computer

Nothing. The notebook runs in Colab, Kaggle or VS Code.

Tested with Python 3.12.13 in Jupyter on macOS, with pandas 3.1.1 and scikit-learn 1.9.1. Colab and Kaggle were not tested.

An old story: a man looks for his keys under a street light at night. A friend helps him search. After some minutes, the friend asks: “Are you sure you lost them here?” The man says: “No, I lost them in the park. But this is where the light is.”

Every detector has a street light. Sofia’s team looks only at the 50 alerts that the detector shows. Every label comes from that small lit area. A fraud in the dark part of the list never gets a label. If a model learns only from the labels, it learns to search under the same light, a little better each time.

The problem: the labels repeat the detector’s choices

Section titled “The problem: the labels repeat the detector’s choices”

In the last lesson, the re-ranker learned from 1,400 reviewed alerts. It raised precision at the budget. But look at one kind of fraud. In an account takeover, a thief gets into a customer’s account. The thief often uses the customer’s own device, in the customer’s own country. These frauds look more normal than a stolen card used abroad.

In the test weeks, the forest alone put 76 account-takeover payments in the top 50 of each day. After re-ranking, 71. The re-ranker raised precision overall, but it found fewer account takeovers. It learned mostly from the kinds that the team saw most often, and account takeovers looked like the “not” alerts.

When the rows that get labels are not chosen at random, the labels do not look like the whole data. This is selection bias. Here, the detector chooses the rows, and a model learns from them and then chooses the next rows. The loop feeds itself. This is a feedback loop: the output of the system becomes its next input.

A loop: detect, review the top 50 a day, label real, not or unknown, re-rank, and back to review. A side branch: the detector's 150 other candidates a day are never shown and get no label. A random sample of 5 a day from them goes to review and to the labels. A dashed arrow from the unseen candidates to the re-ranker says: frauds here are never learned.
The feedback loop. The labels come only from what the team reviews. The random sample is the only path from the unseen candidates to the labels.
Text description of the diagram

The diagram shows a loop of four steps. Detect: the detector scores every payment. Review: the team reviews the top 50 a day. Label: each alert gets real, not or unknown. Re-rank: a model learns from the labels and orders tomorrow’s alerts, which goes back to Review. A side branch leaves Detect: 150 candidates a day that are never shown, with no label. From them, a box “Random sample, 5 a day from the unseen candidates” leads to Label. A dashed arrow from the unseen candidates to Re-rank says “frauds here are never learned”.

A tiny example: ten payments, three reviews

Section titled “A tiny example: ten payments, three reviews”

Here is a constructed example. A detector scores ten payments. The team can review three. Three payments are fraud: two stolen cards and one account takeover.

Rank Payment Truth Reviewed? Label
1 P1 stolen card yes real
2 P2 stolen card yes real
3 P3 honest yes not
4 P4 honest no none
5 P5 honest no none
6 P6 account takeover no none
7 P7 honest no none
8 P8 honest no none
9 P9 honest no none
10 P10 honest no none

Count the shares:

  • In the labels, 2 of 3 are real: 67%.
  • In all ten payments, 3 of 10 are fraud: 30%.
  • Account takeovers with a label: 0 of 1.

A model that learns from these three labels learns “stolen cards are real”. It has no example of an account takeover. So it cannot push P6 up. The next day, P6’s kind stays in the dark again.

Now add one review chosen at random from the seven unseen payments. With a chance of 1 in 7, it is P6. Over many days, the random reviews find some account takeovers, and they also tell you how much fraud the top three miss.

Now the real data: the four weeks of simulated reviews from the last lesson. The notebook counts the share of real fraud in four groups of payments:

reviewed alerts (the labels) 0.506
all candidates (top 200) 0.157
candidates not shown (51st to 200th) 0.041
all payments 0.018
The share of real fraud, 1 to 28 September 2025. Labels: the 1,400 reviewed alerts. Top 200: all candidates, the 200 highest scores of each day. Sample: 140 reviews chosen at random from the candidates that the team did not see. All: every payment of the four weeks.
Text description of the chart

A column chart with four columns. The reviewed alerts, the labels, are 50.6% real fraud. All candidates, the top 200 of each day, are 15.7%. A random sample of the candidates that the team did not see is 2.9%. All payments of the four weeks are 1.8%. The first column is 28 times as high as the last.

The labels are 50.6% fraud. All payments are 1.8% fraud. The labels are full of fraud because the detector chose them. That is fine for Sofia: she wants her 50 slots full of fraud. But a model that learns from the labels sees a world where half of all unusual payments are fraud. That world does not exist.

The bias is not the same for every kind of fraud. Here are the planted payments of each kind in the four weeks, and how many got a label:

Kind Planted payments With a label Share labelled
Stolen card 440 390 88.6%
Card testing 400 297 74.3%
Account takeover 54 21 38.9%
The share of the planted payments of each kind that the team reviewed in the four weeks. The kind that looks most normal, the account takeover, is the kind with the fewest labels.
Text description of the chart

A column chart with three columns. Of the stolen-card payments, 88.6% got a label. Of the card tests, 74.3%. Of the account takeovers, 38.9%: only 21 of 54.

The detector sees stolen cards well: a new country, a new device, many payments. It sees account takeovers less well. So the labels hold 390 stolen-card payments and only 21 account takeovers. From 21 examples, the re-ranker learns little about account takeovers. That is why it found 71 instead of 76 in the test weeks.

Now imagine that the loop runs for a year. Each month, the re-ranker trains on the labels of the month before. Those labels hold fewer account takeovers, because the last re-ranker pushed them down. So the next re-ranker learns even less about them. Nothing in the precision at the budget warns you: it stays high, because the slots are full of the kinds that the model knows. The kinds that it does not know disappear from the labels, from the reports and from the team’s attention.

The step that reduces it: a small random sample

Section titled “The step that reduces it: a small random sample”

Each day, the team reviews a few more candidates, chosen at random from those it did not see. Nobody chose these payments because they looked like fraud. So their labels are a fair sample of the unseen candidates. Here: 5 a day, from the 150 candidates ranked 51st to 200th, for the four weeks.

random sample: 140 reviews, 4 real (2.9%); 95% interval 1.1% to 7.1%
estimate: about 4.3 real payments a day among the 150 candidates that the team did not see

The sample says that about 4 real frauds a day sit just below the top 50. The interval is wide, because 140 reviews is a small sample. The 95% interval of Evaluate without a full answer key goes from 1.1% to 7.1%.

The course data has an answer key, so we can check. The true share is 4.1%: 6.2 real payments a day among the unseen candidates. The estimate is low, but the truth is inside its interval. A real team has no answer key. The random sample is the only honest number it can get.

Does the random sample make the re-ranker better? Not here. Retrained with the 140 extra labels, its precision at the budget is 0.514, the same as before. The sample added only 4 real payments: too few to teach anything.

So be clear about its job. The random sample is for measuring: how much fraud does the top 50 miss, and which kinds? Over months, it also gives labels to kinds that the detector rarely shows. It is not a trick to raise this week’s precision.

Five extra reviews a day, at 4 minutes each, is 20 minutes a day of Sofia’s team. That is a tenth of the time of the 50 normal reviews (an assumption of the course: 4 minutes a review). In exchange, the team learns how much it misses. Without the sample, nobody knows.

A reviewer cannot always decide. An account takeover can look like the customer on a new phone. The honest record then has three answers: real, not and unknown. Unknown means: we do not know yet, ask later or ask the customer.

What happens if the team writes “not sure” as “not”? The notebook pretends that the team was unsure about every account takeover. It trains the re-ranker in two ways:

written as not precision 0.509, account takeovers found 59
left out precision 0.510, account takeovers found 61

With the right answers, the re-ranker found 71 account takeovers. Written as “not”, the 21 account takeovers teach it that this kind is honest: it finds 59. Left out as unknown, the re-ranker loses their information, but it learns nothing wrong: 61. The difference is small here, because there were only 21 such labels. With more labels, a wrong “not” does more harm.

So keep “unknown” as its own answer. Leave unknown alerts out of training. Then come back to them when the answer arrives: a chargeback from the bank, or a call with the customer.

Show when selection bias matters, and the steps against it

It matters when:

  • A model learns from labels that its own alerts produced. Re-rankers, fraud classifiers and spam filters all do this.
  • Some kinds of anomaly look normal to the detector. They get few labels, and the next model sees them even less.
  • You report precision at the budget only. It stays high while whole kinds disappear.

The steps:

  • Review a small random sample every day, from the candidates that the team did not see. Here: 5 a day.
  • Count labels by kind, every month. A kind whose labels fall is a warning.
  • Keep unknown as its own answer. Never write it as “not”.
  • Keep the unsupervised detector as stage one. It does not learn from the labels, so the loop cannot narrow it.
  1. Open the notebook from the box Do this lesson in a notebook at the top of this page.
  2. Run the setup cells. The data cell ends with the Ready: line.
  3. In section 1, run the cell. You see 1,400 reviewed alerts, 708 real.
  4. In section 2, write your prediction, then run the shares cell. You see 0.506 and 0.018.
  5. Run the kinds cell. Account takeovers have 54 planted payments and 21 with a label.
  6. In section 3, run the re-ranker cell. You see 76 and 71 account takeovers.
  7. In section 4, run the random sample cells. You see 4 real (2.9%), then 4.1% from the answer key.
  8. In section 5, run the “unknown” cell. You see 59 and 61.
  9. In section 6, write the two answers, and run each check cell. Each prints Check passed.

Change one thing: review 10 random candidates a day instead of 5. Before you run it, predict: will the estimate of the missed frauds a day move closer to the true 6.2?

Show the answer
sample_10 = unseen.groupby("day", group_keys=False).apply(lambda g: g.sample(10, random_state=1))
missed_per_day_10 = round(float(sample_10.label.mean() * 150), 1)
Check passed: missed_per_day_10 is 6.4.

With 10 reviews a day (280 in all), the estimate is 6.4 real payments a day. The truth is 6.2. A larger sample gives a narrower interval, so a single estimate is less likely to be far off. It costs twice the time: 40 minutes a day instead of 20. Your random numbers may give another value if you change the seed. That is the point of the interval.

Common mistake

The precision that stayed high while a kind disappeared

Symptom: after the re-ranker goes live, Sofia’s weekly report looks better: precision at the budget is up, from 0.498 to 0.514. Three months later, the bank reports a wave of chargebacks from account takeovers. None of them was in the team’s alerts. (The story is constructed; the numbers below are measured.)

Cause: the re-ranker learned from labels that held 390 stolen-card payments and only 21 account takeovers. It moved account takeovers down: 76 in the top 50 with the forest alone, 71 after re-ranking. Precision stayed high, because the freed slots went to stolen cards. The report counted all real alerts together, so it hid the loss.

Fix: report recall and labels by kind, every week. Keep a random sample of reviews, so that the team can estimate what it misses. And retrain only on labels that include a fair share of each kind, or weight the rare kinds.

Consequence in AI work: a model that learns from its own outputs narrows its view a little each time. The usual scores do not show it. Only a sample that the model did not choose shows what is outside the light.

Check your understanding

Show the selection-bias trap with numbers. What share of the reviewed alerts is real, what share of all payments, and why do they differ?

Show the answer

In the four weeks of reviews, 50.6% of the 1,400 reviewed alerts are real (708). Of all payments, only 1.8% are planted fraud. Of all 200 candidates a day, 15.7%. They differ because the detector chose which payments got a label: it showed its most suspicious payments, so the labels are full of fraud. They are not a random sample of payments, and a model trained on them learns a world where half of the unusual payments are fraud.

Check your understanding

Name the step that reduces the bias, say what it measured at Larkfield, and say what it did not do.

Show the answer

Review a small random sample every day from the candidates that the team did not see: 5 a day from ranks 51 to 200. In four weeks, 140 random reviews found 4 real payments: 2.9%, with a 95% interval from 1.1% to 7.1%. That estimates about 4 missed frauds a day just below the top 50 (the answer key says 6.2, inside the interval). It did not raise precision at the budget: the re-ranker with the extra labels had 0.514, the same as without. Its job is to measure what is missed, not to raise this week’s score.

Check your understanding

A reviewer is not sure about an alert. Why should she record “unknown” and not “not”? Use the account-takeover numbers.

Show the answer

“Not” is a label that teaches the model that this kind of payment is honest. When every unsure account takeover was written as “not”, the re-ranker found 59 account takeovers in the test weeks. Left out as unknown, it found 61. With the right answers, 71. “Unknown” loses some information, but it does not teach the wrong answer. Keep unknown alerts out of training, and add them back when the true answer arrives.

Recap

  • Labels exist only where the detector looked. That is selection bias, and a model that learns from its own alerts makes it a feedback loop.
  • At Larkfield: 50.6% of the reviewed alerts are real, against 1.8% of all payments. Only 38.9% of the account takeovers got a label, and the re-ranker found fewer of them (76 → 71).
  • A small random sample of reviews, 5 a day, measures what the top 50 misses: here about 4 a day (truth 6.2). It does not raise precision.
  • Record unknown honestly. Written as “not”, it teaches the model the wrong answer.

The re-ranker now puts the real alerts first, and the random sample keeps it honest. But each alert is still only a score. Next, we find which features made a score high, and we write an alert that Sofia can decide on in a minute.

Next: Explain an alert