Imbalance and rare classes
- Lesson 28
- 35 min
- Module 7: Harder label shapes
Where we are
So far, we read the table by overall accuracy and cost. One class has only about 40 training examples: account takeover. In this lesson, we see how accuracy hides a rare class, and we raise its recall with class weights, resampling and a threshold per class.
Lesson 28 of 37 · Text Classification and Routing: Send Each Message to the Right PlaceImbalance and rare classes
1. The routing problem and its cost
2. Labels you can trust
3. Baselines: rules and classical models
4. Embeddings and small fine-tuned models
5. LLMs as classifiers
6. Decision models, cascades and distillation
7. Harder label shapes
8. Confidence and people in the loop
9. Run it in production
What you will be able to do
Find the rare messages that cost the most. A router can be right 99.4% of the time and still miss every account takeover. This lesson shows why, and three ways to fix it.
NotebookDo this lesson in a notebook
The notebook has every code cell, task and check of this lesson. It counts the takeover messages, compares accuracy with macro-F1 and trains a takeover flag three ways. Then you choose the flag’s threshold on the validation messages. Its setup cells install pandas and scikit-learn and download the practice data, so it needs no local setup. About 35 minutes.
Colab and Kaggle open your own copy. They need a free Google or Kaggle account. Stuck? See the solution notebook.
Before you startWhat to know and to have ready
What you need to know
- Precision and recall, from Classification metrics in the machine learning course. Recall of a class: the share of its messages that the model finds. Precision: the share of the model’s “yes” answers that are right.
- A threshold turns a score into a decision, from The precision–recall trade-off.
- The cost table, from The cost of each mistake. A missed account takeover costs US$200.
- TF-IDF and logistic regression, from Text as numbers: bag of words and TF-IDF.
What you need on your computer: nothing for this lesson. The notebook runs in Colab, Kaggle or VS Code. The project does not change in this module.
Tested on macOS with Python 3.12, pandas 3.1.1 and scikit-learn 1.9.1.
The red stamp at the post office
Section titled “The red stamp at the post office”Go back to the post office. Most letters are birthday cards, bills and catalogues. About one letter in two hundred has a red stamp: urgent, from a court. If it arrives late, someone loses a case.
A new worker is judged by one number: the share of letters that she puts in the right tray. She soon learns a trick. She never looks for the red stamp. She puts every letter in the normal tray, and her score is 99.5%. Her manager is happy with the number. The people who wait for the court letters are not.
The number was not wrong. It answered the wrong question.
The problem: accuracy hides a rare class
Section titled “The problem: accuracy hides a rare class”Larkfield has the same problem. An account takeover message says that someone else is using the customer’s account. Examples: a changed password, or an order that the customer did not make. It is urgent. The cost table prices a missed takeover at US$200, fifty times a normal wrong queue.
It is also rare. The inbox has 68 takeover messages in 10,000: 44 in train, 12 in validation and 12 in test. A class with far fewer examples than the others is called a rare class, and data like this is called imbalanced. We made this rare class on purpose for practice. Real inboxes have their own: fraud, safety, legal threats.
Text description of the diagram
The picture is a grid of 2,000 small squares: 80 columns and 25 rows on a wide screen, 40 columns and 50 rows on a phone. Each square is one test message, in the order of the test file. 1,988 squares are light grey: every other kind of message. Twelve squares are red, with a dark border: the account takeovers. They are scattered across the grid, one or two in a row, with large grey areas between them.
A tiny example: the model that always says “no”
Section titled “A tiny example: the model that always says “no””Take 1,000 messages. Six of them are takeovers. A lazy model answers “not a takeover” for every message. Count what it gets right:
| Takeover (6) | Not a takeover (994) | |
|---|---|---|
| The model says “no” | 6, all wrong | 994, all right |
Its accuracy is 994 ÷ 1,000 = 99.4%. Its recall of takeovers is 0 ÷ 6 = 0. It finds no takeover at all. At US$200 each, the six missed takeovers cost US$1,200.
On Larkfield’s test set, the same lazy model would be right on 1,988 of 2,000 messages: accuracy 0.994. Any model that misses a few takeovers loses almost no accuracy. So accuracy cannot tell you whether a model finds them.
Measures that see the rare class
Section titled “Measures that see the rare class”The fix is to look at each class on its own. Two measures do this.
- Recall by class. For each class, the share of its messages that the model finds. The lazy model has recall 1.000 on “not a takeover” and 0 on “takeover”. The zero is now visible.
- Macro-F1. The F1 score of a class combines its precision and recall into one number. “Macro” means that we take the F1 of each class and average them, with the same weight for every class. A class of 12 messages counts as much as a class of 247.
Compute it for the lazy model. For “not a takeover”, precision is 994 ÷ 1,000 = 0.994 and recall is 1.000, so its F1 is 0.997. For “takeover”, the model never says “yes”, so its F1 is 0. The macro-F1 is (0.997 + 0) ÷ 2 = 0.50. Accuracy said 99.4%; macro-F1 says “half of the job is not done”.
What Larkfield’s models do
Section titled “What Larkfield’s models do”Priya checks the router first. TF-IDF with logistic regression has accuracy 0.962 and macro-F1 0.965 on this inbox. It sends 11 of the 12 test takeovers to the account team and 1 to billing.
Then she trains a model on the 27 topics, one of which is account_takeover. Its accuracy is 0.954 and its macro-F1 is 0.955. Its recall of takeovers is 0.833: 10 of 12. Here the rare topic is not far behind. But look at the count: with 12 messages, one message more or less moves the recall by 8 points. Always write the count next to the recall of a rare class.
A flag for takeovers
Section titled “A flag for takeovers”Grace wants more than a route. She wants a takeover flag: a separate yes/no model that marks a possible takeover, so the account team reads it first. Priya trains it on the same 6,000 training messages, with the label “takeover: yes or no”. The model gives each message a score from 0 to 1. By default, it says “yes” when the score is at least 0.5.
The flag finds only 5 of the 12 test takeovers: recall 0.417. Its precision is perfect, because the 5 that it flags are all takeovers. But it misses 7, and 7 × US$200 = US$1,400.
Why? In training, 5,956 messages say “no” and 44 say “yes”. The model learns that “no” is almost always right, so it needs a lot of evidence before it says “yes”. Its scores for real takeovers stay low. The chart shows the score of each of the 12 test takeovers.
Text description of the diagram
The horizontal axis is the flag’s score, from 0 to 1. The 12 red dots are the scores of the 12 test takeovers: 0.071, 0.106, 0.156, 0.322 twice, 0.380, 0.492, 0.717, 0.734, 0.822, 0.854 and 0.944. A blue dashed line at 0.50 marks the default threshold: 5 dots are to its right. A green dashed line at 0.12 marks a lower threshold: 10 dots are to its right. A grey dot at 0.057 is the highest score of the other 1,988 test messages. Every other message scores below both lines.
Read the chart. The takeovers are not mixed with the other messages: the highest score of any other message is 0.057. The model ranks the takeovers above everything else. Only its threshold of 0.5 is too high for such a rare class.
Three ways to raise the recall
Section titled “Three ways to raise the recall”There are three common fixes. Each one tells the model, in a different way, that a missed takeover matters more.
Class weights: each takeover counts more
Section titled “Class weights: each takeover counts more”During training, a model adds up its errors and makes the sum small. A class weight multiplies the error of each message of a class. With a large weight on takeovers, missing one costs the model as much as missing many normal messages.
scikit-learn computes “balanced” weights for you: class_weight="balanced". Each class gets the weight messages ÷ (classes × messages of that class):
| Class | Training messages | Weight |
|---|---|---|
| takeover | 44 | 6,000 ÷ (2 × 44) = 68.2 |
| not a takeover | 5,956 | 6,000 ÷ (2 × 5,956) = 0.504 |
So one takeover counts about 135 times as much as one other message. Both classes now weigh the same in total. With class weights, the flag finds 11 of 12 test takeovers (recall 0.917), and it flags no other message.
Resampling: show the rare messages more often
Section titled “Resampling: show the rare messages more often”Resampling changes the training data, not the model. Oversampling copies the rare messages: here, each takeover appears 10 times, so the model sees 440 takeover rows. Undersampling does the opposite: it drops part of the common class. We use oversampling, because we have few messages to lose.
With 10 copies, the flag finds 9 of 12 (recall 0.750). Copies add no new information. The model sees the same 44 messages again and can learn their exact words. That is a risk: it may not recognise a takeover written in new words.
A threshold of its own
Section titled “A threshold of its own”The chart above suggests the simplest fix. Keep the default model, and lower its threshold. Choose the threshold on the validation messages, never on the test messages. Grace’s target: find at least 90% of the takeovers.
Priya tries thresholds from 0.50 down to 0.01, in steps of 0.01. She keeps the highest one that reaches 90% recall on the 12 validation takeovers. That is 0.12: it finds 11 of the 12 there. On the test messages, the flag at 0.12 finds 10 of 12 (recall 0.833). It flags no other message.
Notice that the target was 90% on validation and the result is 83% on test. A target chosen on 12 examples is a hope, not a promise. Say so next to the number.
Compare the four flags
Section titled “Compare the four flags”Now put the four versions side by side, on the same 12 test takeovers. Before you look: which one finds the most?
Text description of the chart
A column chart with four columns, the recall of the takeover flag on the 12 test takeovers. Default, threshold 0.50: 0.417 (5 found). Weights, the class weights: 0.917 (11). 10 copies, the resampling: 0.750 (9). At 0.12, the default model with its own threshold chosen on validation: 0.833 (10).
The table adds what the chart leaves out: what each version flags, and what its misses cost with the cost table.
| Flag | Found | Flagged | Missed | Cost of the misses |
|---|---|---|---|---|
| Default, threshold 0.50 | 5 | 5 | 7 | US$1,400 |
| Class weights | 11 | 11 | 1 | US$200 |
| Resampling ×10 | 9 | 9 | 3 | US$600 |
| Own threshold 0.12 | 10 | 10 | 2 | US$400 |
Two honest notes. First, no version flagged a single message that was not a takeover. That is a sign of this inbox: one language model wrote all its messages, and the takeover messages use very clear words. In a real inbox, a lower threshold also flags normal password questions. Each extra flag costs a person a minute. Second, the differences between 9, 10 and 11 are one or two messages. With 12 test examples, do not crown a winner. Check again on next month’s messages.
When to use which fix
Section titled “When to use which fix”- Look at recall by class and macro-F1 first. They cost nothing, and they show the problem. Report them with the count of each rare class.
- A threshold of its own is the cheapest fix. You keep the model and change one number. Use it when the model already ranks the rare messages high, as here. You need enough validation examples to choose it.
- Class weights are one setting in most scikit-learn models. Use them when the rare class is mixed with others and the model must learn its words better. The model’s scores are no longer true probabilities after this change, so do not read them as such.
- Resampling works with any model, even one with no weight setting. Copying adds no information, so watch for a model that only remembers the rare examples.
- None of them replaces more examples. 44 training examples is very few. A person who labels 50 more takeovers often helps more than any setting.
Each fix trains in seconds. The real cost is the review time of extra flags.
Guided practice: the takeover flag in the notebook
Section titled “Guided practice: the takeover flag in the notebook”- Open the notebook from the box Do this lesson in a notebook at the top of this page.
- Run the setup cells. The data cell ends with
Ready: 10000 messages (6000 train, 2000 validation, 2000 test), all synthetic. - In section 1, run the cell. You see
train 44 | validation 12 | test 12. - In section 2, predict the takeover recall of the topic model. Then run the cell. You see accuracy
0.954, macro-F10.955and takeover recall0.833. - In section 3, run the cell. The table shows recall
0.417for the default flag and0.917for class weights. - In section 4, write the threshold search. The check says
Check passed: threshold is 0.12. - Run the next cell. You see test recall
0.833(10 of 12) and no other message flagged.
Your turn: aim for every takeover
Section titled “Your turn: aim for every takeover”Grace asks for more: “Find every takeover on validation.” Change the target in your threshold search from 0.9 to 1.0. Before you run it, predict: will the flag find more test takeovers?
Show the answer
for threshold in np.round(np.arange(0.50, 0.0, -0.01), 2): if (p_val >= threshold)[ato(validation)].mean() >= 1.0: breakThe search stops at 0.11: the lowest validation takeover scores 0.111. On the test messages, the flag at 0.11 still finds 10 of 12. The two missed test takeovers score 0.071 and 0.106, below 0.11 too. A stricter target on validation did not find more on test. Here, no other test message scores above 0.057, so the lower threshold costs nothing. On a real inbox, it would add flags.
Common mistake
A threshold chosen on the test messages
Symptom: Tomás searches for the threshold on the test messages and reports “the flag finds 100% of the takeovers”. Next month, it misses some.
Cause: he chose the number that fits these 12 test messages best. The test score then measures how well he fitted the test set, not how well the flag works on new messages. With 12 examples, one lucky message changes the result by 8 points.
Fix: choose every threshold on the validation messages. Report the test score once, with its count: “10 of 12 on test”. If you need a better choice, collect more validation examples of the rare class.
Consequence in AI work: a recall of a rare class is the easiest number to make look good by accident. It is also the number that matters most when the class is expensive. Choose on validation, report on test, and give the count.
Knowledge check
Section titled “Knowledge check”Check your understanding
A takeover flag is right on 99.4% of 2,000 test messages. Explain why this number does not tell Grace whether the flag works, and what two numbers she should ask for instead.
Show the answer
Only 12 of the 2,000 test messages are takeovers. A flag that never says “yes” is right on the other 1,988, so its accuracy is 0.994, and it finds no takeover. Accuracy is almost the same whether the flag finds 0 or 12 takeovers. Grace should ask for the recall of the takeover class, with its count, for example “10 of 12”. She should also ask for the number of messages flagged. Macro-F1 also helps, because it gives the rare class the same weight as the others.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
Predict: the default flag ranks every takeover above every other test message, but finds only 5 of 12 at the threshold 0.5. Which fix would you try first, and why?
Show the answer
A threshold of its own. The model already puts the takeovers above every other message: the highest other score is 0.057. Only the threshold is too high for such a rare class. Lowering it, chosen on validation (0.12), finds 10 of 12 on test with no extra flags. It needs no retraining. Class weights (11 of 12) also work, but they change the model and its scores.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
In practice: you train a fraud flag with 30 fraud messages in training and 8 in validation. You choose a threshold that finds 8 of 8 on validation. What do you report, and what do you do before you trust it?
Show the answer
Report the result on the test messages once, with the counts (“6 of 8 found, 14 flagged”), not the validation result. One message more or less moves a recall by 12 points with 8 examples, so say that the number is uncertain. Before you trust it, collect more fraud examples for validation and test, and check the flag again on new messages. Also report how many messages it flags, because each flag costs review time.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Recap
- Accuracy hides a rare class. A model that never finds a takeover is right on 1,988 of 2,000 test messages (0.994).
- Recall by class and macro-F1 show the rare class. Always give the count next to its recall.
- The default takeover flag finds 5 of 12 (0.417). Class weights find 11, resampling ×10 finds 9, and a threshold of its own (0.12, chosen on validation) finds 10. On this inbox, none flags another message.
- Choose thresholds on validation, report on test once. With 12 examples, one message is 8 points.
You can now find a rare message that one route hides. But some messages hold two requests, and a single route can only answer one of them. Next, we give one message two routes.
Next: Multi-label messages