Labels are rare, and that changes everything
- Lesson 3
- 30 min
- Module 1: What "unusual" means
Where we are
We can now name the kind of anomaly that we look for. The next question decides the method: how many examples do we have with the right answer? In this lesson, we see why labels are rare here, and we meet the supervised, semi-supervised and unsupervised settings, and the cases where a plain rule wins.
Lesson 3 of 38 · Anomaly Detection: Find What Is UnusualLabels are rare, and that changes everything
1. What "unusual" means
2. Simple baselines that are hard to beat
3. Statistical process control and distribution drift
4. Distance, density and isolation
5. Anomalies in time
6. Deep and embedding methods
7. Evaluate with few labels
8. Learn from the reviewers
9. Run a detector in production
What you will be able to do
Choose a method from the labels that you really have, not from the labels that you wish you had. And stop judging a detector by its accuracy: when anomalies are rare, “nothing is wrong” is almost always right, and it finds nothing.
Before you startWhat to know and to have ready
What you need to know
- The alert budget of 50 a day and precision at the budget, from The payment that did not fit.
- The four kinds of anomaly, from Kinds of anomalies.
- Accuracy and precision are explained here. If you want more, read Classification metrics in Machine Learning: From Problem to Reliable Model.
What you need on your computer
Nothing. This lesson has no code.
The bag screener who never says “stop”
Section titled “The bag screener who never says “stop””Think of a person who checks bags at an airport. Every day, thousands of bags go past the screen. Almost all of them hold clothes, books and shoes. A dangerous bag is so rare that a screener may work for years without seeing a real one.
Two things follow. First, the screener cannot learn mainly from dangerous bags, because there are almost none. She learns what normal bags look like, very well, and she stops a bag that does not fit. Second, imagine a lazy screener who says “fine” to every bag. He is right on almost every bag, every day. By the count of right answers, he looks excellent. He is also useless.
Larkfield’s fraud team is in the same position as the screener. This lesson is about both problems. How do you learn when there are few known examples? And how do you avoid the lazy screener’s trap?
What a label is, and why fraud has so few
Section titled “What a label is, and why fraud has so few”A label is the right answer for one example, written down. For a payment, the label says “fraud” or “honest”. In Machine Learning: From Problem to Reliable Model, every ticket had a label: escalated or not. A model learned from thousands of labelled tickets. Here, labels are rare, for four reasons:
- The event is rare. Most payments are honest. In Larkfield’s data, 4,971 of 343,074 payments are planted fraud: 1.45%, about 1 in 70. A real shop has far less fraud than this. We planted more so that the exercises have enough cases.
- Labels arrive late. The card owner may see a strange payment on a bank statement weeks later. Until the bank reports it, nobody knows.
- Labels exist only where someone looked. A payment gets a label when Sofia’s team reviews it. Her team reviews 50 alerts a day. Almost every other payment never gets a label at all.
- New kinds appear. Thieves change their methods. The frauds of next month may not look like any labelled fraud of last month.
The base rate: 1 in 70
Section titled “The base rate: 1 in 70”The share of anomalies among all the rows is called the base rate. It is the most important number to know before you judge any detector. The picture shows 70 payments, with the Larkfield base rate:
Text description of the diagram
The picture is a grid of 70 small squares. Sixty-nine are empty: honest payments. One square, near the middle, is filled: a planted fraud. Below the grid, the text says: 70 payments, 69 honest, 1 planted fraud. All Larkfield payments: 4,971 planted in 343,074, which is 1.45%.
Why accuracy lies here
Section titled “Why accuracy lies here”Accuracy is the share of all answers that are right. It is the first score most people think of, and here it misleads.
A tiny example with 70 payments
Section titled “A tiny example with 70 payments”Take the 70 payments of the picture: 69 honest and 1 fraud. Compare two detectors:
| Detector | Alerts | Fraud found | False alerts | Right answers | Accuracy |
|---|---|---|---|---|---|
| “Nothing is fraud” | 0 | 0 | 0 | 69 of 70 | 98.6% |
| A detector that alerts on 5 payments, the fraud among them | 5 | 1 | 4 | 66 of 70 | 94.3% |
Count the right answers of the second detector. It is right on the fraud and on the 65 honest payments without an alert: 66. It is wrong on its 4 false alerts. So its accuracy is 66 ÷ 70 = 94.3%. The detector that does nothing scores higher than the one that finds the fraud. Accuracy rewards the lazy screener.
The same on Larkfield’s data
Section titled “The same on Larkfield’s data”On all 343,074 payments, “nothing is fraud” is right on the 338,103 honest ones: an accuracy of 98.55%. The course scores detectors on its evaluation months, September to November 2025, where 1.66% of the payments are planted. There, the measured accuracy of “nothing is fraud” is 98.34%.
Now take a good detector. Isolation Forest, a method that you build in Module 4, sends 50 alerts a day. Half of its alerts are planted frauds. Its accuracy is also 98.34%. Even a detector that knows the answers gets only 98.57%. On quiet days, it must fill some of its 50 slots with honest payments. On busy days, 50 slots cannot hold every fraud. The two charts compare accuracy and precision at the budget for the same three detectors:
Text description of the chart
A column chart with three columns, on an axis from 0 to 100%. “Nothing is fraud” has 98.34%. Isolation Forest, with 50 alerts a day, has 98.34%. A detector that knows the answers has 98.57%. The three columns look identical.
Text description of the chart
A column chart with three columns, on an axis from 0 to 1. “Nothing is fraud” reviews nothing, so its precision at the budget is 0. Isolation Forest has 0.500: 25 of its 50 daily alerts are planted frauds, on average. A detector that knows the answers has 0.542. It cannot reach 1, because most days have fewer than 50 planted payments.
This is why the course never ranks detectors by accuracy. It uses precision at the budget, recall (the share of all planted anomalies that the reviewed alerts find) and cost per week. These numbers move when a detector gets better. Module 7 goes deeper into evaluation with few labels.
Three settings, from the labels you have
Section titled “Three settings, from the labels you have”The labels you have decide the setting of the problem: the kind of learning that is possible. There are three settings, and a fourth answer that needs no learning at all.
Text description of the diagram
The diagram starts with one question: which right answers, or labels, do we have? It has four branches. First: many labelled examples of each kind of fraud, and the kinds stay the same. This leads to the supervised setting, where a classifier learns the labels. Second: only data that we trust to be normal. This leads to the semi-supervised setting: learn what normal is, and flag what does not fit. Third: no labels at all. This leads to the unsupervised setting: score how unusual each row is. Fourth: a pattern that a person can write down exactly. This leads to a rule, such as a limit, a count or an allow-list. A dashed arrow goes from unsupervised to semi-supervised, labelled “reviews add a few labels”.
Supervised: learn from labelled examples
Section titled “Supervised: learn from labelled examples”In the supervised setting, you have many labelled examples, both fraud and honest. A model learns to tell them apart. This is the classifier of the machine learning course. It works well when you have enough labels of each kind and the frauds of tomorrow look like the frauds of yesterday.
It fails when a new kind of fraud appears. A classifier learns the patterns in its labels. A fraud that looks like none of them gets a low score. And with 1 fraud in 70 payments, a classifier needs great care, as the machine learning course showed for rare escalations.
Semi-supervised: learn only what normal looks like
Section titled “Semi-supervised: learn only what normal looks like”In the semi-supervised setting, you have data that you trust to be normal, and no fraud labels. A method learns the shape of normal and flags what does not fit it. This is the bag screener: she knows normal bags very well.
There is a catch. “Trusted normal” data is rarely perfectly clean. Larkfield’s reference months, June to August 2025, also contain planted fraud, because fraud is in every month. The method still works when the anomalies are a small share, as here. A One-Class SVM (Module 4) and an autoencoder (Module 6) are methods of this setting.
Unsupervised: no labels at all
Section titled “Unsupervised: no labels at all”In the unsupervised setting, you have no labels. A method gives each row a score: how unusual is it, compared with the other rows? Robust statistics (Module 2), distance to neighbours and Isolation Forest (Module 4) work this way. This is the most common start, and most of this course lives here.
Even without labels for training, you still need some labels to check the detector. Somebody must review the top alerts and say which ones were real. The course uses its answer key for that. A real team uses its reviews.
A rule: when you know the exact pattern
Section titled “A rule: when you know the exact pattern”Sometimes a person can write the pattern down exactly. Two examples: “more than 5 payments in 10 minutes on one card” and “a refund above 100 dollars needs approval”. Then a rule is often the best answer. It is instant, it needs no training data, and anyone can read it. Its weakness is that it finds only what somebody imagined. Rules, and when a rule is the right answer measures two rules against the other methods.
The four answers side by side
Section titled “The four answers side by side”| You have | Setting | What the method learns | Larkfield example | In this course |
|---|---|---|---|---|
| Many labels of each kind, and the kinds repeat | Supervised | The difference between labelled fraud and honest | Few such cases: frauds are rare and change | The machine learning course |
| Data trusted to be normal | Semi-supervised | The shape of normal | The reference months, June to August | Modules 4 and 6 |
| No labels | Unsupervised | How unusual each row is | Every table, on its first day | Modules 2, 4 and 5 |
| An exact pattern | A rule | Nothing: a person writes it | The velocity and device rules | Module 2 |
The labels you do get are not a fair sample
Section titled “The labels you do get are not a fair sample”Over time, Sofia’s reviews add labels. That sounds like a way out: after a few weeks, train a classifier on the reviewed alerts. But these labels have a problem. They exist only for the payments that a detector put in front of the team.
Module 8 measures this with four weeks of simulated reviews. The reviewers are a simulation, an assumption of the course, but the effect is real. In 1,400 reviewed alerts, 708 were real fraud: 50.6%. In a random sample of the payments the team did not see (those ranked just below the top 50 each day), only 2.9% were planted. In all payments, 1.8%. The reviewed alerts are full of fraud because the detector chose them.
A model trained only on these labels learns “what the detector already found”. The frauds that the detector misses never get a label, so the next model misses them too. Module 8 shows how to learn from reviewers anyway: add a small random sample of reviews, and treat the labels as a biased sample.
What the answer key is, and what it is not
Section titled “What the answer key is, and what it is not”The course data is synthetic, so we know every planted anomaly. That list is the answer key. A real team never has one. The course uses the key only to score detectors in the comparison table, never to train them. It is packed in a file called sealed-key.txt so that you do not read it by accident. You can unpack it if you want: it is there to score your own work, not to keep a secret. Meet the Larkfield data shows how the comparison table uses it.
Guided practice: choose the setting
Section titled “Guided practice: choose the setting”For each situation, choose one answer: supervised, semi-supervised, unsupervised, or a rule. Then say why in one sentence.
- Read all five situations first.
- Write your answer and your reason for each one.
- Open the answers and compare your reasons, not only your choices.
- Larkfield’s finance team says: “Refunds above 100 dollars need a team lead’s approval. Find refunds above 100 dollars that have no approval.”
- Amira has 60 days of
ticket-apimetrics. Nobody has marked any minute as good or bad. - A bank has five years of card payments, with 40,000 confirmed frauds of the same few kinds, and the kinds have not changed.
- Larkfield’s new region, east, has been live for one week. Grace has two months of refunds from a stable period in the old regions.
- Sofia’s team has reviewed 1,400 alerts. Tomás wants to train a classifier on them and stop using the unsupervised detector.
Show the answers
- A rule. The pattern is exact and written in a policy. No data is needed to learn it.
- Unsupervised. There are no labels. Score how unusual each minute is, then let Amira review the top alerts.
- Supervised is a good choice here: many labels of each kind, and the kinds repeat. Keep an unsupervised check beside it for new kinds.
- Semi-supervised. The stable period is data that Grace trusts as normal. Learn its shape, and check the new region against it. But a new region may have a new normal, as the next modules show.
- Not yet. The 1,400 labels come only from alerts that the detector chose. So 50.6% are real, against 2.9% in a random sample of the payments the team did not see. A classifier trained on them misses what the detector never showed. Keep the unsupervised detector, and add labels from a random sample. Module 8 does this.
Your turn: a lower base rate
Section titled “Your turn: a lower base rate”Change one condition. In a real shop, suppose 1 payment in 1,000 is fraud instead of 1 in 70. Predict the accuracy of “nothing is fraud”. Then predict what happens to precision at the budget, if a detector still sends 50 alerts a day out of about 1,875 payments.
Show the answer
“Nothing is fraud” is right on 999 of 1,000 payments: 99.9% accuracy, even higher than before, and still useless. With about 1,875 payments a day, there are only about 1.9 frauds a day. So even a perfect detector can fill at most about 2 of its 50 slots with fraud: a precision at the budget of about 0.04. A lower base rate makes accuracy look better and precision look worse. So always report the base rate next to precision. And the budget must fit the number of real cases.
Common mistake
A detector with 98.34% accuracy
Symptom: Tomás reports: “My new detector is 98.34% accurate on the evaluation months.” The number sounds excellent. Omar is ready to put it into use.
Cause: in those months, 1.66% of the payments are planted fraud. A detector that never sends an alert is right on the other 98.34%, so it has exactly the same accuracy. A score that the lazy screener also gets says nothing about finding fraud. In the course data, Isolation Forest at 50 alerts a day also has 98.34%, and so does doing nothing (both measured).
Fix: report the base rate first, then precision at the budget, recall and cost per week. Compare every detector with “nothing is reviewed” in the same table: 0 precision and 10,367 dollars a week of missed fraud.
Consequence in AI work: with a rare event, a high accuracy can hide a detector that finds nothing. A team that trusts it learns about the failure from the losses, weeks later.
Knowledge check
Section titled “Knowledge check”Check your understanding
Name the three learning settings for anomaly detection, plus the answer that needs no learning. For each one, say what you must have, and give one example.
Show the answer
- Supervised: many labelled examples of each kind, and kinds that repeat. A classifier learns them. Example: a bank with years of confirmed frauds of a few stable kinds.
- Semi-supervised: data that you trust to be normal, and no fraud labels. The method learns the shape of normal. Example: Larkfield’s reference months, June to August.
- Unsupervised: no labels. The method scores how unusual each row is. Example: 60 days of
ticket-apimetrics that nobody has marked. - A rule: an exact pattern that a person can write down. Example: refunds above 100 dollars without a team lead’s approval.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
Predict: 1.45% of Larkfield's payments are planted fraud. What is the accuracy of a detector that says “honest” to every payment? Why does that make accuracy a bad score here?
Show the answer
It is right on every honest payment: 338,103 of 343,074, or 98.55%. It finds no fraud at all. Because the base rate is so low, doing nothing already scores near 100%. A good detector scores almost the same. In the evaluation months, doing nothing and Isolation Forest at 50 alerts a day both have 98.34%. Accuracy cannot tell them apart. Precision at the budget can: 0 against 0.500.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
Sofia's reviewed alerts are 50.6% real fraud, but a random sample of the payments they did not see is only 2.9%. Explain why, and what it means for a model trained only on reviewed alerts.
Show the answer
Labels exist only for the alerts that a detector put in front of the team, and the detector chose the most suspicious payments. So the reviewed alerts are full of fraud, far more than payments in general. A model trained only on them learns what the old detector already found. Frauds that the old detector missed never get a label, so the new model misses them too. A small random sample of reviews gives labels that the detector did not choose.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Recap
- A label is the right answer for one example. Fraud labels are rare: the event is rare, labels come late, they exist only where someone looked, and new kinds appear.
- The base rate is the share of anomalies: 4,971 in 343,074 payments, about 1 in 70.
- Accuracy lies when anomalies are rare. “Nothing is fraud” has 98.55% on all payments. In the evaluation months, it has 98.34%, the same as a good detector. Use precision at the budget, recall and cost per week.
- The labels you have decide the setting: supervised (many labels of each kind), semi-supervised (trusted normal data), unsupervised (no labels), or a rule (an exact pattern).
- Reviewed alerts are not a fair sample: 50.6% real against 2.9% in a random sample of the unseen payments.
You can now choose a setting from the labels you have, and you know which numbers to trust. But so far, you have only read about Larkfield’s data. Next, you open it in a notebook, check every table, and score your first detectors in the comparison table.
Next: Meet the Larkfield data