Evaluate without a full answer key
- Lesson 31
- 35 min
- Module 7: Evaluate with few labels
Where we are
In the last two lessons, we measured precision at the budget and the cost per week. Every one of those numbers came from the anomalies that we planted, and real payments have no full answer key. In this lesson, we compare three ways to evaluate without one: planted anomalies, a sample that people review, and agreement between detectors, with what each one hides. A light-bulb factory shows all three.
Lesson 31 of 38 · Anomaly Detection: Find What Is UnusualEvaluate without a full answer key
1. What "unusual" means
2. Simple baselines that are hard to beat
3. Statistical process control and distribution drift
4. Distance, density and isolation
5. Anomalies in time
6. Deep and embedding methods
7. Evaluate with few labels
8. Learn from the reviewers
9. Run a detector in production
What you will be able to do
Estimate how good a detector is when nobody knows all the answers, and know what each kind of evidence cannot show. Real payments come with no answer key, and a number from planted cases alone can be confidently wrong about the frauds that matter.
NotebookDo this lesson in a notebook
The notebook has every code cell, task and check of this lesson. It computes the interval of a reviewed sample and plays the reviewer on 200 random alerts and on 1,000 more samples. Then it compares the alerts of Isolation Forest and kNN distance. Its setup cells install pandas and scikit-learn and download the course data (about 7 MB). About 35 minutes.
Colab and Kaggle open your own copy. They need a free Google or Kaggle account. Stuck? See the solution notebook.
Before you startWhat to know and to have ready
What you need to know
- Precision at the budget and the cost per week, from The alert budget.
- Recall by kind of fraud, from Why accuracy lies here.
- kNN distance and Isolation Forest on the behaviour features, from Far from the neighbours: kNN distance and Isolation Forest.
- A share from a sample and its spread, from Essential Mathematics and Statistics for AI. This lesson shows the one formula that it needs.
What you need on your computer
Nothing: the notebook runs in Colab or Kaggle. Tested with Python 3.12.13 in Jupyter on macOS, with pandas 3.1.1 and scikit-learn 1.9.1; Colab, Kaggle and Windows were not tested.
A light-bulb factory
Section titled “A light-bulb factory”Think of a factory that makes 10,000 light bulbs a day. A machine at the end of the line tests each bulb and throws out the ones that look broken. How does the factory know that the machine works? Nobody can test 10,000 bulbs again by hand. The factory uses three tricks.
First, a worker puts 20 bulbs into the line that she broke on purpose. At the end of the day, she counts how many of her 20 the machine threw out. Second, a quality inspector takes 50 bulbs at random from the “good” pile and tests each one by hand. Third, the factory buys a second machine from another company, and checks where the two machines agree.
Each trick tells the factory something true, and each one has a blind spot. The broken bulbs only test the ways the worker knew how to break a bulb. The inspector’s 50 bulbs give a good estimate, but not an exact count. And two machines can both miss the same fault. A fraud detector has the same three tricks and the same three blind spots.
The problem: the answer key ends where the course data ends
Section titled “The problem: the answer key ends where the course data ends”Every number so far in this module came from the answer key: the list of the frauds that we planted in the course data. Precision at 50 a day is 0.500 because the key says which of the 4,550 alerts are real. In real work, Larkfield’s next month of payments comes with no key.
What does Larkfield have instead? Sofia’s team marks each alert that it reads as “real” or “not”. But Tomás now wants to try his autoencoder in shadow mode: it scores every payment and sends nothing to the team. Nobody reads its alerts, so nobody knows its precision. And a confirmed fraud, a chargeback from the bank, can arrive weeks after the payment.
Tomás has a quick answer: “We tested it on the planted frauds. Its precision at 50 a day was 0.434. Done.” Priya, Larkfield’s data scientist, is not sure. The planted frauds are the ones that we imagined. Is a number from our own test a number about the real fraud?
Three ways to evaluate, and what each one hides
Section titled “Three ways to evaluate, and what each one hides”Text description of the diagram
The diagram has three columns, one for each way to evaluate. Each column has a blue box, “tells you”, above a dashed box, “hides”. Planted anomalies tell you exact precision and recall, by kind of fraud; they hide every kind of fraud that nobody planted. A reviewed random sample tells you the precision of real alerts, with an interval; it hides the frauds that got no alert, so recall stays unknown. The agreement of two detectors tells you which alerts to read first, where both detectors agree; it hides the frauds that both detectors miss.
Way 1: planted anomalies
Section titled “Way 1: planted anomalies”Planted anomalies, also called injected anomalies, are the factory worker’s broken bulbs. You add cases that you made yourself to real or synthetic data, and you count how many the detector finds. The course data is built this way. Its 4,971 planted payments are stolen cards, card tests and account takeovers, and the answer key lists every one.
The strength of planted cases is that the numbers are exact. Precision and recall need no estimate, and you can split them by kind:
print("precision at 50 from the planted key:", compare.score_payments(evaluation, iforest, "Isolation Forest")["precision at budget"])compare.recall_by_kind(evaluation, iforest)The output:
precision at 50 from the planted key: 0.5
kindaccount_takeover 0.443card_testing 0.718stolen_card 0.905Name: reviewed, dtype: float64Isolation Forest finds 90.5% of the stolen-card payments and 44.3% of the account-takeover payments. You can only see this because you know the kind of every planted case.
What do planted cases hide? Everything that you did not plant. A fraudster’s new trick is not in the key, so it adds nothing to recall and nothing to precision. Worse, a detector can be tuned until it finds your planted cases very well, and learn your way of planting instead of fraud. Planted cases also have the share that you chose. Larkfield’s data has far more fraud than a real shop, so its precision is higher than real life would give.
So use planted cases to compare methods on the same test, as the comparison table does. Do not report them as the detector’s real precision.
Way 2: a sample that people review
Section titled “Way 2: a sample that people review”The second trick is the quality inspector. Take a random sample of the alerts, and ask a person to check each one. The share of real frauds in the sample is your estimate. Random means that every alert has the same chance to be picked, so the sample is a small, fair copy of all the alerts.
A tiny example: 20 reviewed alerts
Section titled “A tiny example: 20 reviewed alerts”A reviewer checked 20 alerts, and 11 were real frauds. The estimate is 11 ÷ 20 = 0.55. But a different 20 alerts would give a different share. How far from the truth can we be?
The usual answer is a 95% interval. It is a range around the estimate, and it holds the true share in about 95 of 100 such samples. For a share, its half-width, the margin, is 1.96 × √(share × (1 − share) ÷ number reviewed). The square root shrinks as you review more alerts, so the margin shrinks too:
real, reviewed = 11, 20 # a reviewer checked 20 alerts, and 11 were real fraudsestimate = real / reviewedmargin = 1.96 * np.sqrt(estimate * (1 - estimate) / reviewed) # the 95% interval of a shareprint(f"estimate {estimate:.2f}, 95% interval {estimate - margin:.2f} to {estimate + margin:.2f}")for n in (20, 50, 200, 800): print(f"{n:>4} reviewed alerts: plus or minus {1.96 * np.sqrt(0.5 * 0.5 / n):.3f} around 0.50")The output:
estimate 0.55, 95% interval 0.33 to 0.77 20 reviewed alerts: plus or minus 0.219 around 0.50 50 reviewed alerts: plus or minus 0.139 around 0.50 200 reviewed alerts: plus or minus 0.069 around 0.50 800 reviewed alerts: plus or minus 0.035 around 0.50With 20 reviews, the precision is somewhere between 0.33 and 0.77: not very useful. With 200 reviews, the margin is about 0.07. Four times more reviews make the margin only half as wide, so each extra review helps less than the one before.
Larkfield: 200 reviewed alerts
Section titled “Larkfield: 200 reviewed alerts”Priya tests the idea on Isolation Forest, where she can check the result. Its 4,550 alerts (50 a day for 91 days) are the “pile”. She picks 200 at random. The notebook then plays the reviewer. It uses the answer key only to say “real” or “not” for the 200 picked alerts, as a person would.
# Pretend the key does not exist: a reviewer checks 200 alerts chosen at random.planted_ids = set(compare.planted_payments().payment_id) # the reveal: only to play the reviewerreviewed_alerts = sorted(alerts_if.payment_id[alerts_if.reviewed])is_real = np.array([p in planted_ids for p in reviewed_alerts])rng = np.random.default_rng(7)picked = rng.choice(len(reviewed_alerts), 200, replace=False)estimate = is_real[picked].mean()margin = 1.96 * np.sqrt(estimate * (1 - estimate) / 200)print(f"{is_real[picked].sum()} of 200 are real: estimate {estimate:.3f}, 95% interval {estimate - margin:.3f} to {estimate + margin:.3f}")print(f"the true precision of all {len(reviewed_alerts):,} alerts: {is_real.mean():.3f}")The output:
95 of 200 are real: estimate 0.475, 95% interval 0.406 to 0.544the true precision of all 4,550 alerts: 0.500The estimate is 0.475, and the interval, 0.406 to 0.544, holds the true precision, 0.500.
How much does one sample move? The notebook draws 1,000 samples of 200 and computes each estimate:
Text description of the chart
A histogram of 1,000 estimates of Isolation Forest’s precision, each from a random sample of 200 of its 4,550 alerts, in bins of 2.5 points from 38.75% to 63.75%. The bars form a hill with its top at the true precision, 50.0%, marked with a line. Our first sample, 47.5%, is marked just left of the top. The lowest estimate is 39.5% and the highest 61.0%. 90% of the estimates are between 44.0% and 56.0%. In 941 of the 1,000 samples, the 95% interval holds the true precision.
The estimates form a hill around the truth. Most samples land within 0.06 of 0.500, and a few unlucky ones are more than 0.10 away. The 95% interval holds the truth in 941 of the 1,000 samples, close to the 95 in 100 that it promises.
What a reviewed sample hides
Section titled “What a reviewed sample hides”A sample of alerts tells you about the alerts. It says nothing about the frauds that got no alert, because nobody reviews payments that are not alerts. So it gives precision and never recall.
To estimate recall, you would need a random sample of all payments, and frauds are rare. A random 200 payments contain on average 200 × 0.0166 = 3.3 planted frauds at Larkfield, and fewer in a real shop. That is far too few to count how many the detector missed. This is why recall in this course comes from the planted cases. Module 8 keeps a small random sample for a related reason.
A reviewed sample also has a price. 200 reviews are four days of Sofia’s budget of 50 a day. At the course’s assumed 3 dollars per review, that is 600 dollars of review time. And people make mistakes: a reviewer who calls a real fraud “not” moves the estimate. In this course, the reviewers are simulated and always right, which real reviewers are not.
Way 3: agreement between detectors
Section titled “Way 3: agreement between detectors”The third trick is the second machine. Run two detectors, and look at where their alerts agree. Agreement needs no key at all, so you can compute it on any day of real payments.
Priya compares Isolation Forest and kNN distance, both at 50 a day on the behaviour features. Each has 4,550 alerts. The Jaccard index measures their overlap: the alerts in both lists, divided by the alerts in either list. 0 means no alert in common, 1 means the same alerts.
a = set(alerts_if.payment_id[alerts_if.reviewed])k = set(alerts_knn.payment_id[alerts_knn.reviewed])groups = {"both": a & k, "only Isolation Forest": a - k, "only kNN": k - a}print(f"agreement (Jaccard): {len(a & k) / len(a | k):.2f}")pd.DataFrame.from_dict({name: {"alerts": len(ids), "real (from the key)": sum(p in planted_ids for p in ids), "precision": round(np.mean([p in planted_ids for p in ids]), 3)} for name, ids in groups.items()}, orient="index")The output:
agreement (Jaccard): 0.53 alerts real (from the key) precisionboth 3154 2060 0.653only Isolation Forest 1396 216 0.155only kNN 1396 238 0.170The two detectors share 3,154 of their 5,946 different alerts: a Jaccard index of 0.53. The column “real” uses the key, to show what agreement means. Where both detectors alert, 65.3% of the alerts are real. Where only one alerts, fewer than 1 in 5 is real.
Text description of the chart
A column chart with three columns on an axis from 0 to 100%. “Both”: 65.3% of the 3,154 alerts that both detectors send are real frauds. “Only Isolation Forest”: 15.5% of its 1,396 own alerts are real. “Only kNN”: 17.0% of its 1,396 own alerts are real. The first column is about four times as high as the other two.
So agreement is a good way to order the work: read the alerts that both detectors send first. It does not measure precision by itself: without the key, you would not know that “both” means 65.3% and not 30%. And it is blind where both detectors are blind. 303 planted payments are in neither list, and agreement cannot show a single one of them. Two detectors on the same features often share their blind spots, so a high agreement can also mean “they make the same mistakes”.
Which way, when
Section titled “Which way, when”| Way | It tells you | Use it to | It costs |
|---|---|---|---|
| Planted anomalies | Exact precision and recall, by kind | Compare methods on the same test, before launch | Time to build realistic cases; no new kinds |
| A reviewed random sample | Precision of the real alerts, with an interval | Check a detector in production or in shadow mode | Reviewer time: 200 reviews for about ±0.07 |
| Agreement of two detectors | Where to look first | Order the alerts; spot a detector that changed | A second detector to run; nothing about misses |
Larkfield uses all three. The comparison table uses planted cases to choose a method. Before Tomás’s autoencoder sends a single alert, a reviewer checks 200 random alerts from its shadow run. And the alerts that two detectors share go to the top of Sofia’s list.
Guided practice: three ways in the notebook
Section titled “Guided practice: three ways in the notebook”- Open the notebook from the box Do this lesson in a notebook at the top of this page.
- Run the setup cells. The data cell ends with the
Ready:line. - In section 1, run the tiny example. You see
estimate 0.55, 95% interval 0.33 to 0.77. - In section 2, run the cell that fits both detectors. You see
4,550 alertsfor each one. - In section 3, run the planted cell. You see
account_takeover 0.443. - In section 4, predict the estimate from 200 reviews. Then run the sample cell. You see
95 of 200 are real. - Run the 1,000 samples. You see
the interval holds the truth in 94.1%. - In section 5, run the agreement cell. You see
agreement (Jaccard): 0.53.
Your turn: fewer reviews, and what agreement misses
Section titled “Your turn: fewer reviews, and what agreement misses”The notebook’s section 6 has two tasks. First, repeat the reviewed sample with only 50 alerts: use np.random.default_rng(7) again, and write the margin of the 95% interval, rounded to 3 decimals, as margin_50. Before you run it, predict the margin from the tiny example. Second, count the planted payments that are in neither detector’s alerts, as missed_by_both.
Show the answers
rng = np.random.default_rng(7)picked_50 = rng.choice(len(reviewed_alerts), 50, replace=False)estimate_50 = is_real[picked_50].mean()margin_50 = round(float(1.96 * np.sqrt(estimate_50 * (1 - estimate_50) / 50)), 3)missed_by_both = len(planted_ids & set(evaluation.payment_id) - (a | k))The 50 alerts give an estimate of 0.52 and margin_50 is 0.138, about twice the margin of 200 alerts. missed_by_both is 303: 303 of the 2,817 planted payments of the evaluation months are in neither list. Agreement between the two detectors can never point at them.
Common mistake
A sample that is not random
Symptom: Tomás needs a quick estimate. He asks a reviewer to check “the first 200 alerts” of the evaluation months. They come from 1 to 4 September, and 65.0% of them are real. Another time, he asks for “the 200 most suspicious alerts”, and 100% of them are real. The true precision of all 4,550 alerts is 50.0%.
Cause: neither sample is random. The first 200 alerts are four days in early September, and those days are not a fair copy of three months. The 200 highest scores are the easiest cases, so they are always real. The formula for the 95% interval assumes that every alert had the same chance to be picked.
Fix: pick the sample with a random generator, from all the alerts of the period, as rng.choice(len(reviewed_alerts), 200, replace=False) does. Write down the seed and the period, so that another person can check the sample.
Consequence in AI work: a sample that is easy to pick is often the most flattering one. The detector looks better than it is, the team trusts the number, and the gap appears later as missed fraud.
Knowledge check
Section titled “Knowledge check”Check your understanding
For each way to evaluate without a full answer key (planted anomalies, a reviewed random sample, agreement between two detectors), say what it tells you and what it hides.
Show the answer
- Planted anomalies give exact precision and recall, by kind of fraud. They hide every kind of fraud that nobody planted, and a detector can learn the way you plant instead of fraud.
- A reviewed random sample of alerts estimates the precision of the real alerts, with an interval (200 reviews: about plus or minus 0.07). It hides the frauds that got no alert, so it cannot give recall.
- Agreement between two detectors shows which alerts to read first (both detectors: 65.3% real at Larkfield). It hides what both detectors miss, 303 planted payments here, and it cannot give precision without labels.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
Calculate: a reviewer checks 100 random alerts of the autoencoder's shadow run and finds 40 real frauds. Give the estimate and its 95% interval. How many reviews would make the margin about half as wide?
Show the answer
The estimate is 40 ÷ 100 = 0.40. The margin is 1.96 × √(0.40 × 0.60 ÷ 100) = 1.96 × 0.049 = 0.096, so the interval is about 0.30 to 0.50. To make the margin half as wide, review four times as many alerts: about 400.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
Tomás says: “Isolation Forest and kNN distance agree on 53% of their alerts, so they are both about 53% right.” What is wrong with this?
Show the answer
Agreement is not precision. The Jaccard index of 0.53 says how much the two lists overlap, not how many alerts are real. With the key, the shared alerts are 65.3% real and the others about 16%. Two detectors could also agree completely and both be wrong, because they share blind spots. To know the precision, you need labels: planted cases or a random sample that people review.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Recap
- Real payments have no full answer key, so every evaluation needs evidence of another kind, and each kind has a blind spot.
- Planted anomalies give exact numbers by kind of fraud. Isolation Forest finds 90.5% of stolen-card payments and 44.3% of account-takeover payments. They say nothing about kinds that nobody planted.
- A random sample of reviewed alerts estimates precision with a 95% interval: 200 reviews gave 0.475 (0.406 to 0.544) when the truth was 0.500. It cannot give recall. A sample that is not random, such as the first 200 alerts, flatters the detector.
- Agreement between two detectors (Jaccard 0.53) orders the work: shared alerts were 65.3% real. It cannot see the 303 planted payments that both missed.
You can now judge a detector with and without a key. But all these numbers are averages over every customer. Next, you check the false alerts group by group, and you see who pays when an honest payment is blocked.
Next: Slices and fairness