Classification trade-offs

What you will be able to do

Judge a model that flags urgent tickets by counting its four kinds of result. Then one number, such as accuracy, cannot hide the urgent tickets that it misses.

NotebookOptional: check the numbers in a notebookColab · Kaggle · GitHub

The notebook is optional: this lesson works with a calculator. It has the scores of the 1,000 test tickets from this page’s explorer. Count the confusion matrix at any threshold, and redo the worked example and the tasks in Python, with checks. It uses only Python’s standard library, so it needs no install.

Open in ColabOpen in KaggleView on GitHub

Colab and Kaggle open your own copy. They need a free Google or Kaggle account. Stuck? See the solution notebook.

Before you startWhat to know and to have readyAbout 45 minutes

What you need to know

• How to read a count as a share inside a group (“3 of 4 is 75%”), from Conditional probability: Probability inside a group.
• How a rare problem makes an alert look better than it is, from Conditional probability: How “90% accurate” misleads. This lesson uses the same idea for accuracy.
• That a share from a small test set is uncertain, from Estimate uncertainty: Worked example 2.

What you need to have ready (the notebook needs none of it)

• A calculator. A phone calculator is enough.
• Your c03-notes.md file. You made it in the “Before you start” box of Mathematical reading skills. Add a heading: ## Statistics for model decisions. If you cannot find the file, make a new empty file with that name: this lesson does not use earlier answers.

The notebook was tested in Jupyter with Python 3.14.6 (standard library only).

Larkfield, again

Larkfield is a fictional online shop for home and garden products. About 1,200 customer messages arrive each week as tickets. Some tickets are urgent: a customer was charged twice, or a delivery with medicine for a pet is lost. Grace, the support lead, wants a senior agent to see these tickets first.

Tomás, the AI engineer, trained a model for this. For each new ticket, the model gives a score from 0 to 1. A high score means “this looks urgent”. In this module, you decide how Larkfield turns that score into an action, and what the action costs.

Where you meet this in AI

Many AI systems give a score, and a person or a program acts when the score is high enough:

• A spam filter moves an email to the spam folder. Sometimes a real invoice goes to spam, and sometimes spam stays in the inbox.
• A bank’s fraud alert blocks a card payment. A wrong block annoys a customer. A missed fraud costs money.
• A screening model in a hospital marks scans for a doctor to check first. A missed scan can delay treatment.
• A content moderation queue sends posts to human reviewers. Too many flags overload the reviewers.

Each system makes two kinds of mistake, and the two kinds cost different amounts. To see them, you count.

From a score to a flag

A score is not a decision. Larkfield chooses a threshold: if the score is at least the threshold, the ticket is flagged and a senior agent reviews it. If the score is lower, the ticket goes into the normal queue.

Here are ten tickets with their scores and what they really were. Grace checked each ticket by hand, so “Really urgent?” is the correct answer. The threshold is 0.50.

Ticket	Score	Flagged at 0.50?	Really urgent?	What happened
T-5001	0.91	yes	yes	caught
T-5002	0.74	yes	no	false alarm
T-5003	0.66	yes	yes	caught
T-5004	0.58	yes	no	false alarm
T-5005	0.52	yes	yes	caught
T-5006	0.45	no	yes	missed
T-5007	0.33	no	no	correctly left
T-5008	0.21	no	no	correctly left
T-5009	0.12	no	no	correctly left
T-5010	0.05	no	no	correctly left

Every ticket lands in one of four groups. Count them:

	Flagged	Not flagged	Total
Really urgent	3 caught	1 missed	4
Not urgent	2 false alarms	4 correctly left	6
Total	5	5	10

This table of four counts is a confusion matrix. It shows where the model is right and where it confuses one class with the other. A class is one of the possible answers, here “urgent” or “not urgent”.

Three questions you can answer from the counts

Grace, Omar and Tomás each ask a different question. Each answer uses a different row or column of the table.

1. Grace: “When my senior agent opens a flagged ticket, how often is it really urgent?” Look at the Flagged column: 3 of 5 flags are urgent. That is 60%.
2. Omar: “Of all the urgent tickets, how many does the model catch?” Look at the Really urgent row: 3 of 4. That is 75%.
3. Tomás: “Of the tickets that are not urgent, how many does the model flag by mistake?” Look at the Not urgent row: 2 of 6. That is about 33%.

The three questions have different answers, because they divide by different totals. Always ask: “out of what?”

Now the names

Professionals use fixed names for the four groups and the three shares. Positive means “flagged as urgent” and negative means “not flagged”. True means the flag was right, and false means it was wrong.

What you counted	Name	Short
Urgent and flagged (caught)	true positive	TP
Urgent and not flagged (missed)	false negative	FN
Not urgent and flagged (false alarm)	false positive	FP
Not urgent and not flagged	true negative	TN

• Precision is Grace’s question: the share of flags that are really urgent. It is TP divided by all flags.
• Recall is Omar’s question: the share of urgent tickets that are flagged. It is TP divided by all urgent tickets.
• False-positive rate is Tomás’s question: the share of tickets that are not urgent but flagged. It is FP divided by all tickets that are not urgent.

As formulas, read from left to right:

$$\text{precision} = \frac{TP}{TP + FP} \qquad \text{recall} = \frac{TP}{TP + FN} \qquad \text{false-positive rate} = \frac{FP}{FP + TN}$$

“Precision is the true positives divided by everything that was flagged. Recall is the true positives divided by everything that is really urgent. The false-positive rate is the false alarms divided by everything that is not urgent.”

You do not need to learn the formulas. If you can find the right row or column in the confusion matrix, you can always work them out again.

Worked example: the 1,000 test tickets

Ten tickets are too few to judge a model. Priya, the data scientist, uses a test set of 1,000 past tickets that the model did not see in training. Grace’s team marked each one by hand: 80 are urgent (8%) and 920 are not urgent. The model scored all 1,000. Priya counts at threshold 0.50:

	Flagged	Not flagged	Total
Urgent	65 (TP)	15 (FN)	80
Not urgent	47 (FP)	873 (TN)	920
Total	112	888	1,000

Before she calculates anything, Priya checks the totals: 65 + 15 = 80, 47 + 873 = 920, and 112 + 888 = 1,000. A wrong total means a wrong count somewhere.

Measure	Out of what?	Calculation	Result
Precision	the 112 flagged tickets	65 ÷ 112	58.0%
Recall	the 80 urgent tickets	65 ÷ 80	81.3%
False-positive rate	the 920 tickets that are not urgent	47 ÷ 920	5.1%
Accuracy	all 1,000 tickets	(65 + 873) ÷ 1,000	93.8%

Accuracy is the share of all tickets where the model is right: the true positives plus the true negatives, out of everything.

Priya writes one plain sentence for each number, because Omar and Grace do not use the names:

• “Of every 10 flagged tickets, about 6 are really urgent.” (precision)
• “The model catches about 8 of every 10 urgent tickets, and misses 15 of the 80.” (recall)
• “About 5 of every 100 normal tickets are flagged by mistake.” (false-positive rate)

How accuracy misleads with a rare class

Tomás reads the table and says: “93.8% accurate. That is very good.” Priya shows him a second “model” that needs no training at all: flag nothing.

• It flags 0 tickets, so it misses all 80 urgent tickets.
• It is right about all 920 tickets that are not urgent.
• Its accuracy is 920 ÷ 1,000 = 92.0%.

A model that does nothing useful is 92% accurate, because 92% of tickets are not urgent. When one class is rare, accuracy mostly measures how well the model handles the common class. The share of the rare class in the data is the base rate, here 8%. Always state the base rate next to accuracy.

Priya also tries a stricter threshold, 0.70. Accuracy goes up to 94.2%, because the model makes fewer false alarms. But it now misses 53 of the 80 urgent tickets.

Accuracy of three ways to flag the 1,000 test tickets. All three look almost the same.

Text description of the chart

A column chart with three columns on an axis from 0 to 100%. Flag nothing: 92.0%. Threshold 0.50: 93.8%. Threshold 0.70: 94.2%. The columns are almost the same height, and the strictest rule has the highest accuracy.

Urgent tickets missed by the same three rules. Now the difference is large.

Text description of the chart

A column chart of the number of urgent tickets that each rule misses, out of 80. Flag nothing misses 80. Threshold 0.50 misses 15. Threshold 0.70 misses 53. The rule with the highest accuracy misses more than three times as many urgent tickets as threshold 0.50.

The two charts show the same three rules. Accuracy says they are almost equal. The count of missed urgent tickets says they are very different. For Grace’s team, the second chart is the one that matters.

Explore: move the threshold

The explorer shows all 1,000 test tickets. The upper panel shows the scores of the 920 tickets that are not urgent. The lower panel shows the scores of the 80 urgent tickets. Each panel has its own count scale, because one class is more than 11 times bigger than the other. The scores are generated with a fixed seed, so you see the same tickets on every visit.

Explore

One threshold, four counts

Solid bars are flagged (score at or above the threshold). Pale bars are not flagged. You can also drag the round handle on the line, or focus it and press the arrow keys (Shift for steps of 0.10).

At threshold 0.50, the model flags 112: 65 urgent and 47 false alarms. It misses 15 of the 80 urgent tickets.

Confusion matrix at threshold 0.50

Real class	Flagged	Not flagged	Total
Urgent	65true positive	15false negative (missed)	80
Not urgent	47false positive (false alarm)	873true negative	920
Total	112	888	1,000

Flagged

112

of 1,000 tickets

Precision

58.0%

65 of 112 flags

Recall

81.3%

65 of 80

False-positive rate

5.1%

47 of 920

Score distributions of the 1,000 test tickets (920 not urgent, 80 urgent) with the threshold, the confusion matrix and the three shares.

Text description of the explorer

The upper panel is a histogram of the scores of the 920 tickets that are not urgent. Their scores go from near 0 to about 0.75. Most are from 0.05 to 0.45, with a peak near 0.25. The lower panel is a histogram of the 80 urgent tickets. Their scores go from about 0.3 to 1. Most are from 0.5 to 0.8, with a peak near 0.57. The two distributions overlap between about 0.35 and 0.65. A vertical line marks the threshold. Tickets at or above it are flagged.

At the starting threshold of 0.50, the model flags 112 tickets: 65 urgent (true positives) and 47 not urgent (false positives). It misses 15 urgent tickets (false negatives), and 873 tickets are correctly left (true negatives). Precision is 58.0%, recall is 81.3% and the false-positive rate is 5.1%.

At 0.30: 79 true positives, 1 false negative, 363 false positives, 557 true negatives. Precision 17.9%, recall 98.8%. At 0.70: 27 true positives, 53 false negatives, 5 false positives, 915 true negatives. Precision 84.4%, recall 33.8%. At 0.00, every ticket is flagged: recall is 100% and precision is 8.0%. At 1.00, nothing is flagged: recall is 0%, and precision is undefined, because there are no flags to check.

When you raise the threshold, the line moves right. Fewer tickets are flagged, so there are fewer false alarms and precision goes up. But more urgent tickets fall to the left of the line, so recall goes down. You cannot move the line to make both better. That is the trade-off. Only a better model, with less overlap between the two panels, improves both at the same time.

Guided practice: a lower threshold

Grace asks: “What if we flag more, at threshold 0.30?” Use the counts below. You can also check them in the explorer.

At threshold 0.30: TP = 79, FN = 1, FP = 363, TN = 557.

1. In c03-notes.md, add a heading: ### Threshold 0.30.
2. Draw the confusion matrix with the four counts and the row and column totals.
3. Check the totals. The rows must give 80 and 920. All four counts must give 1,000.
4. Calculate precision: TP out of all flagged tickets.
5. Calculate recall: TP out of all urgent tickets.
6. Calculate the false-positive rate: FP out of all tickets that are not urgent.
7. Calculate accuracy.
8. Write one sentence for Grace: how many flags will her senior agent see, and how many are urgent?

Expected results: 442 flagged; precision 17.9%; recall 98.8%; false-positive rate 39.5%; accuracy 63.6%.

Show an example answer

	Flagged	Not flagged	Total
Urgent	79	1	80
Not urgent	363	557	920
Total	442	558	1,000

• Precision: 79 ÷ 442 = 0.179, so 17.9%.
• Recall: 79 ÷ 80 = 0.988, so 98.8%.
• False-positive rate: 363 ÷ 920 = 0.395, so 39.5%.
• Accuracy: (79 + 557) ÷ 1,000 = 0.636, so 63.6%.

For Grace: “At 0.30, the model flags 442 of every 1,000 tickets, and fewer than 1 in 5 flags is urgent. It misses only 1 of the 80 urgent tickets.”

Accuracy fell from 93.8% to 63.6%, but the model now misses almost nothing. Whether that is better depends on what a review costs and what a missed urgent ticket costs. That is the next lesson.

Independent variation: a stricter threshold

Now Tomás wants fewer flags. At threshold 0.70, the counts are TP = 27, FN = 53, FP = 5, TN = 915.

Calculate precision, recall, the false-positive rate and accuracy. Then write one sentence for Omar that does not use the word “accuracy”.

Show an example answer

• Flagged: 27 + 5 = 32.
• Precision: 27 ÷ 32 = 84.4%.
• Recall: 27 ÷ 80 = 33.8%.
• False-positive rate: 5 ÷ 920 = 0.5%.
• Accuracy: (27 + 915) ÷ 1,000 = 94.2%.

For Omar: “At 0.70, almost every flag is real (27 of 32), but the model misses 53 of the 80 urgent tickets. That is two out of every three.”

Run Python in your browser

Optional: check the four shares

This is optional. It checks the worked example at threshold 0.50. Change the four counts to the numbers for 0.30 or 0.70 and run it again. In professional work, scikit-learn functions such as confusion_matrix and precision_score count and divide for you.

Python code (you can change it)# The four counts at threshold 0.50 (1,000 test tickets) tp = 65 # urgent, flagged fn = 15 # urgent, not flagged (missed) fp = 47 # not urgent, flagged (false alarm) tn = 873 # not urgent, not flagged precision = tp / (tp + fp) recall = tp / (tp + fn) false_positive_rate = fp / (fp + tn) accuracy = (tp + tn) / (tp + fn + fp + tn) print("precision:", round(precision * 100, 1), "%") print("recall:", round(recall * 100, 1), "%") print("false-positive rate:", round(false_positive_rate * 100, 1), "%") print("accuracy:", round(accuracy * 100, 1), "%")

Common mistake

Accuracy without class balance

Symptom: a model report says “94% accurate”, and the team releases the model. After two weeks, Grace’s team finds urgent tickets in the normal queue every day. Customers who were charged twice wait two days for an answer.

Cause: only 8% of tickets are urgent. A model that never flags anything is already 92% accurate. The report gave one number and did not show the base rate or the confusion matrix. So nobody saw that the model missed 53 of 80 urgent tickets.

Fix: report the base rate first (“80 of 1,000 test tickets are urgent”). Then show the confusion matrix and recall and precision for the rare class. Compare the model with the “flag nothing” rule. If a model is not clearly better than flagging nothing on the class that matters, it is not ready.

Check your understanding

A model flags 50 tickets. 40 of them are urgent. There are 100 urgent tickets in the test set. What are precision and recall?

Show the answer

Precision is 40 of the 50 flags: 80%. Recall is 40 of the 100 urgent tickets: 40%. The model is usually right when it flags, but it misses 60 of the 100 urgent tickets. The two numbers divide by different totals: the flags for precision, the urgent tickets for recall.

Check your understanding

A colleague says: 'Our new model is 95% accurate, so it is better than the old rule.' Which two numbers do you ask for first, and why?

Show the answer

Ask for the base rate (how common the important class is) and the recall for that class, or the full confusion matrix. If only 5% of cases are positive, a rule that never flags anything is also 95% accurate. Accuracy alone cannot show whether the model catches the rare cases.

Check your understanding

Predict: Tomás lowers the threshold from 0.50 to 0.40. What happens to the number of false negatives, the number of false positives and the precision?

Show the answer

More tickets are flagged. False negatives go down, because more urgent tickets are now above the line (from 15 to 7). False positives go up, because more normal tickets are also above the line (from 47 to 155). Precision goes down, from 58.0% to 32.0%, because the extra flags are mostly false alarms. You can check this in the explorer.

Recap

• A threshold turns a score into a flag. Each threshold gives four counts: true positives, false negatives, false positives and true negatives. Together they are the confusion matrix.
• Precision divides by the flags. Recall divides by the really urgent tickets. The false-positive rate divides by the tickets that are not urgent. Always ask “out of what?”.
• With a rare class, accuracy misleads. Flagging nothing is 92% accurate on Larkfield’s tickets.
• Moving the threshold trades precision against recall. Only a better model improves both.

Next, you give each mistake a cost, and you find the threshold that costs Larkfield the least.

Next: Thresholds and costs