Classification metrics

What you will be able to do

Measure a model that answers yes or no by counting what really happened to its answers. One number, such as accuracy, can look excellent while the model catches almost none of the cases that matter.

NotebookDo this lesson in a notebookColab · Kaggle · GitHub

The notebook has every code cell, task and check of this lesson, with the instructions. Its setup cells install pandas and scikit-learn and download the practice data, so it works without the local project. Read the explanations here, and run the code there.

Open in ColabOpen in KaggleView on GitHub

Colab and Kaggle open your own copy. They need a free Google or Kaggle account. Stuck? See the solution notebook.

Before you startWhat to know and to have readyAbout 60 minutes

What you need to know

• Why a regression error is real − predicted, from Regression metrics.
• Features and the label escalated_72h, from Supervised learning: Features and labels.
• The baselines “never escalate” and “flag priority 1”, the senior team’s capacity of 20% and the improvement target, from Define a baseline.
• The validation set, the data we keep to check our work while we build, from Train, validation and test sets: Three sets, three jobs.
• You met the confusion matrix, precision and recall briefly in Classification trade-offs: Now the names in Essential Mathematics and Statistics for AI. This lesson builds on them.

What you need on your computer (only for the local route; the notebook needs none of it)

• The project folder ~/projects/ticket-model with its .venv (pandas 3.0.6, scikit-learn 1.9.1, matplotlib 3.11.2) and the data in data/, including extra_features.csv. You made them in Define a baseline: Set up your project. If a data file is missing or changed, run python get_data.py --reset. The last line must be Ready: train 16939 rows (1983 escalated), valid 1179 (183), test 1002 (157).
• To run the notebook locally, open it in VS Code with the Jupyter extension and the .venv kernel, as in Define a baseline: Open a notebook in VS Code.

Tested on macOS with Python 3.14.6, pandas 3.0.6, scikit-learn 1.9.1 and matplotlib 3.11.2. The Windows commands are not tested. The notebook was tested in Jupyter.

Why subtraction does not work here

In Regression metrics, every prediction was a number, so the error was real − predicted: 40 − 30 = 10 tickets. Larkfield’s main question is different. For each new ticket, Grace, the support lead, wants a yes or no: will this ticket be escalated to a senior agent within 72 hours? The answer is a class, not a number.

What is “escalated” minus “not escalated”? The question has no meaning. You could write the classes as 1 and 0 and subtract, but the difference can only be 0 (right), 1 or −1 (wrong). It has no size: a wrong “no” is not “a little” wrong. All that you can do with a yes-or-no answer is ask whether it was right. And because there are two kinds of answer and two kinds of truth, there are exactly four things that can happen. So for classification, we do not measure distances. We count the four outcomes, and compute every metric from the counts.

Four things can happen: a smoke alarm

A smoke alarm is a classifier. Every minute, it answers one question: is there a fire? It answers by ringing or staying silent. Compare its answer with what really happened, and you get one of four outcomes:

1. There is a fire, and the alarm rings. It caught the fire.
2. There is a fire, and the alarm stays silent. It missed the fire. This is the worst outcome.
3. There is no fire, only smoke from the toaster, and the alarm rings. A false alarm: annoying, but nobody is hurt.
4. There is no fire, and the alarm is silent. The room was correctly left alone, as it is almost every minute of the year.

Two truths (fire or no fire) and two answers (the alarm rings or is silent) give four outcomes. The green cells are right answers, the orange cells are mistakes, and the two kinds of mistake cost very different amounts.

Text description of the chart

A two-by-two grid. The columns are the alarm’s answer: rings, or silent. The rows are the truth: a real fire, or no fire. Top left, a flame and a ringing bell: caught, a true positive (TP). Top right, a flame and a silent bell: missed, a false negative (FN). Bottom left, smoking toast and a ringing bell: a false alarm, a false positive (FP). Bottom right, toast without smoke and a silent bell: correctly left, a true negative (TN). The two right answers, TP and TN, have a green background. The two mistakes, FN and FP, have an orange background.

You already used these four words for Larkfield’s flags in Define a baseline: caught, false alarms, missed and correctly left. Professionals use fixed technical names for them, and you meet these names in every library and paper. The class that we are looking for, the fire or the escalation, is the positive class. “Positive” does not mean “good”: a fire is not good news. The other class is the negative class. In the four names, positive or negative is the answer that the classifier gave, and true or false says whether that answer was right:

Plain word	At Larkfield	Technical name	Short
caught	flagged, and it escalated	true positive	TP
missed	not flagged, but it escalated	false negative	FN
false alarm	flagged, but it did not escalate: an unnecessary review	false positive	FP
correctly left	not flagged, and it did not escalate	true negative	TN

So a false negative is a “no” that was wrong: a missed escalation.

At Larkfield, the two mistakes cost different amounts. A missed escalation means a customer with a serious problem waits in the normal queue, often a day more, and some customers leave. An unnecessary review costs about 10 minutes of a senior agent. Keep both costs in mind: every metric below looks at them in a different way.

A first yes-or-no flag: the points score

To count outcomes, we need a classifier. No machine learning model exists yet in this course, and we do not need one. Hospitals have used a simple tool for decades: an early-warning score. A nurse checks a few signs at the bedside, such as the heart rate and the breathing rate. Each sign outside the normal range adds points. A patient with many points gets a doctor’s attention at once. Banks have used similar scorecards for loans. Such a points score is a checklist: each fact that makes the bad outcome more likely adds points. Anyone can compute it by hand, and anyone can check it.

Priya, the data scientist, builds one for Larkfield. She uses only the train set, the tickets before May, and only facts that are known when a ticket is created. On train, 11.7% of tickets escalate. She looks for yes-or-no facts that make an escalation clearly more likely:

Fact about the ticket (train set)	Tickets with it	Escalated, with the fact	Escalated, without it	Points
Priority 1 (the most urgent)	2,385 (14.1%)	29.1%	8.9%	2
Team is payment or warranty	5,000 (29.5%)	18.5%	8.9%	1
At least one escalation of the customer in the last 90 days	7,809 (46.1%)	14.8%	9.1%	1

Priority 1 is the strongest fact: with it, a ticket escalates about three times as often as without it. So it gets 2 points. Each of the other two facts makes an escalation about 1.6 to 2 times as likely, so each gets 1 point. Priya keeps the checklist short on purpose: three questions that an agent can answer in a few seconds from the ticket and the customer’s history. Other facts help too, such as a long message (18.8% with 150 words or more, against 10.0%) or the phone channel (16.5% against 10.9%). A longer checklist could add them, but every question makes it slower to use and harder to check. The last fact, prior_escalations_90d, is in extra_features.csv; Module 6 tests it again as a model feature.

A ticket’s score is the sum of its points, from 0 to 4. In Vector operations in the mathematics course, you met this as a weighted sum. Does the score sort the train tickets well? It does: the more points, the higher the escalation rate.

Points	0	1	2	3	4
Train tickets	5,643	7,037	2,645	1,204	410
Escalated	4.7%	10.0%	17.4%	29.7%	46.8%

A score is not yet a yes-or-no answer. To get one, choose a cut-off: flag every ticket with at least that many points. Priya starts with 3 points or more: a ticket needs priority 1 and one more fact. On train, that flags 9.5% of the tickets, a little fewer than the priority-1 rule (14.1%). The next lesson moves the cut-off and shows what changes.

The confusion matrix

Here are ten made-up tickets with their points. Four of them escalated.

Ticket	Priority	Team	Earlier escalations	Points	Flagged (3 or more)?	Escalated?	Outcome
T-1	1	payment	1	4	yes	yes	caught (TP)
T-2	1	delivery	0	2	no	no	correctly left (TN)
T-3	1	warranty	0	3	yes	yes	caught (TP)
T-4	1	returns	1	3	yes	no	false alarm (FP)
T-5	3	payment	0	1	no	yes	missed (FN)
T-6	2	account	0	0	no	no	correctly left (TN)
T-7	3	delivery	2	1	no	no	correctly left (TN)
T-8	2	warranty	1	2	no	yes	missed (FN)
T-9	3	returns	0	0	no	no	correctly left (TN)
T-10	3	account	1	1	no	no	correctly left (TN)

Check one row yourself. T-3 has priority 1 (2 points), team warranty (1 point) and no earlier escalation (0 points). That makes 3 points, so it is flagged. It escalated, so it was caught: a true positive.

Now count each outcome and put the counts in a table with two rows (what really happened) and two columns (the flag). This table is the confusion matrix. It shows where the classifier is right and where it confuses one class with the other.

	Flagged	Not flagged	Total
Escalated	TP = 2 (T-1, T-3)	FN = 2 (T-5, T-8)	4
Not escalated	FP = 1 (T-4)	TN = 5	6
Total	3	7	10

Always check the totals: the rows give the real classes (4 escalated, 6 not), and all four counts give the 10 tickets. Every metric in this lesson is one way to combine these four numbers.

Accuracy, and its trap

The first idea is the share of right answers. Right answers are the tickets caught (TP) and the tickets correctly left (TN): (2 + 5) ÷ 10 = 0.7. This is accuracy.

Now a test. Take a smoke alarm that never rings, not even in a fire. On a normal day there is no fire, so the alarm is right almost every minute of the year. Its accuracy is close to 100%, and it is useless: it fails at the one moment that matters.

The same happens with the ten tickets. A rule that never flags is right on the 6 tickets that did not escalate: accuracy 0.6. It catches none of the 4 escalations. Its accuracy is only a little below the points score’s 0.7.

On the real May tickets, the effect is much stronger. In Define a baseline, “never escalate” had an accuracy of 84.5%, and it catches nothing. Why so high? Because 84.5% of the May tickets did not escalate. When one class is rare, accuracy mostly measures how well a classifier handles the common class. A rare positive class is normal in practice: fraud, disease, defects, spam on some sites, and escalations are all rare.

So accuracy alone is never enough when the classes are unbalanced. Always read it next to the base rate, the share of the rare class. And read it next to the two metrics below, which look only at the class that matters.

Precision and recall

Go back to the smoke alarm. Two people ask two different questions about it.

The person who must leave the building at every alarm asks: “When the alarm rings, how often is there a real fire?” If the answer is “1 time in 50”, people stop leaving the building. This is precision: of the flags, the share that were right. At Larkfield, it is Grace’s question: of the tickets that go to her senior team, how many really escalate?

The fire safety officer asks: “Of the real fires, how many did the alarm ring for?” If the answer is “half of them”, the alarm is not safe. This is recall: of the real positives, the share that the classifier caught. At Larkfield, it is the question for the customers: of the tickets that escalate, how many did we send to a senior agent at once?

The two questions divide the same number, the true positives, by different totals. Precision looks down the Flagged column; recall looks along the Escalated row. For the ten tickets:

• Precision = caught ÷ flagged = TP ÷ (TP + FP) = 2 ÷ 3 = 0.667. Two of the three flags were right.
• Recall = caught ÷ escalated = TP ÷ (TP + FN) = 2 ÷ 4 = 0.5. The flag caught two of the four escalations.

$$\text{precision} = \frac{TP}{TP + FP} \qquad \text{recall} = \frac{TP}{TP + FN} \qquad \text{accuracy} = \frac{TP + TN}{TP + FP + FN + TN}$$

Each metric is hurt by one kind of mistake. False alarms (FP) lower precision: each unnecessary review is a flag that was wrong. Misses (FN) lower recall: each missed escalation is a positive that was not caught. So when someone asks “which number should I watch?”, the answer is: the one that is hurt by the mistake that costs more. Notice also what neither metric uses: the true negatives. That is why precision and recall do not grow when the common class grows, and accuracy does.

You cannot read one without the other. A smoke alarm that rings all the time has a recall of 100%: it rings for every fire. Its precision is terrible. An alarm that rings only when the room is full of flames has a precision near 100%, and a poor recall. A useful classifier needs both to be good enough for its job. Recall is also called sensitivity or the true positive rate in medicine.

What does the “never escalate” rule get? Its recall is 0: it catches nothing. Its precision has no value, because it flags nothing and 0 ÷ 0 is not a number. The two metrics show at once what accuracy hid.

F1: one number for both

Sometimes you need one number, for example to sort many experiments. The plain mean of precision and recall is a poor choice. A classifier with precision 0.9 and recall 0.1 has a mean of 0.5. One with 0.5 and 0.5 has the same mean. But the first one misses 9 of every 10 positives.

The F1 score combines the two so that it is high only when both are high. It is the harmonic mean: 2 × precision × recall ÷ (precision + recall). For precision 0.9 and recall 0.1: 2 × 0.9 × 0.1 ÷ 1.0 = 0.18, much lower than 0.5. The weaker of the two numbers pulls F1 down. For the ten tickets: 2 × 0.667 × 0.5 ÷ 1.167 = 0.571.

$$F_1 = \frac{2 \times \text{precision} \times \text{recall}}{\text{precision} + \text{recall}}$$

F1 is useful when a false alarm and a miss cost about the same, and when nothing else limits the number of flags. It is less useful when one mistake costs much more than the other, because F1 treats them alike. And it knows nothing about a capacity: Larkfield’s senior team can review at most 20% of the tickets, and F1 cannot see that limit.

Worked example: the baselines and the points score on May

Priya compares three classifiers on the validation set, the 1,179 tickets of May 2026. They are never escalate, the priority-1 rule, and the points score at 3 or more. Each code block is a cell of the lesson notebook, where the setup cells have already loaded pandas as pd and numpy as np. First, she loads the data with the extra column, and writes the points score as a function:

TARGET = "escalated_72h"

def load(part):
    """One part of the split, with prior_escalations_90d from extra_features.csv."""
    tickets = pd.read_csv(f"data/{part}.csv", keep_default_na=False, na_values=[""])
    extra = pd.read_csv("data/extra_features.csv", usecols=["ticket_id", "prior_escalations_90d"])
    return tickets.merge(extra, on="ticket_id", how="left")

def points_score(tickets):
    """Larkfield's points score: 0 to 4 points per ticket."""
    points = 2 * (tickets["priority"] == 1)
    points += tickets["team"].isin(["payment", "warranty"])
    points += tickets["prior_escalations_90d"] >= 1
    return points.astype(int).rename("points")

train, valid = load("train"), load("valid")
print(valid.groupby(points_score(valid))[TARGET].agg(tickets="size", rate="mean").round(3))

The output:

Output

        tickets   rate
points
0           320  0.053
1           532  0.118
2           211  0.237
3            84  0.417
4            32  0.562

On May, the score sorts the tickets as it did on train: 5.3% of the tickets with 0 points escalate, and 56.2% of those with 4 points. Next comes a function that counts the four outcomes and computes the metrics. It is the flag_report() of Define a baseline, with the same plain words for the four counts, and with F1 added:

def flag_report(y, flags):
    """Compare 0/1 flags with the real 0/1 answers: four counts and five measures."""
    y, flags = np.asarray(y, dtype=bool), np.asarray(flags, dtype=bool)
    caught, false_alarms = int((flags & y).sum()), int((flags & ~y).sum())
    missed, correctly_left = int((~flags & y).sum()), int((~flags & ~y).sum())
    precision = caught / (caught + false_alarms) if caught + false_alarms else None
    recall = caught / (caught + missed)
    f1 = 2 * precision * recall / (precision + recall) if caught else 0.0
    return {"flagged": caught + false_alarms, "share flagged": round((caught + false_alarms) / len(y), 3),
            "caught": caught, "false alarms": false_alarms, "missed": missed, "correctly left": correctly_left,
            "precision": round(precision, 3) if precision is not None else None, "recall": round(recall, 3),
            "F1": round(f1, 3), "accuracy": round((caught + correctly_left) / len(y), 3)}

Before you run the next cell, predict: which of the three has the highest accuracy?

y = valid[TARGET]
points = points_score(valid)
reports = pd.DataFrame([flag_report(y, np.zeros(len(valid))),
                        flag_report(y, valid["priority"] == 1),
                        flag_report(y, points >= 3)],
                       index=["never escalate", "rule: priority 1", "points: 3 or more"])
print(reports.to_string())

Output

                   flagged  share flagged  caught  false alarms  missed  correctly left  precision  recall     F1  accuracy
never escalate           0          0.000       0             0     183             996        NaN   0.000  0.000     0.845
rule: priority 1       163          0.138      61           102     122             894      0.374   0.333  0.353     0.810
points: 3 or more      116          0.098      53            63     130             933      0.457   0.290  0.355     0.836

Read it row by row, as Grace would:

• Never escalate has the highest accuracy, 0.845, and catches none of the 183 escalations. It is the most “accurate” and the most useless.
• The rule flags 163 tickets. 61 of them escalate (precision 0.374), and it catches 61 of the 183 escalations (recall 0.333).
• The points score flags 116 tickets, fewer than the rule. 53 of them escalate: precision 0.457, almost half of the reviews find a real escalation. It catches 53 of the 183: recall 0.290.

Escalations caught on the 1,179 May tickets. Accuracy puts never escalate first; by escalations caught, it is last.

Text description of the chart

A column chart of the escalations that each classifier flags, out of 183 in May. Never escalate: 0. Flag priority 1: 61. The points score at 3 or more: 53. The accuracies are 84.5%, 81.0% and 83.6%: never escalate has the highest accuracy and catches nothing.

So which is better, the rule or the points score? The points score has the better precision, and the rule has the better recall. But the two flag different numbers of tickets: 163 against 116. A classifier that flags fewer tickets usually has a higher precision and a lower recall, so this comparison is not fair to either. The next lesson shows how to compare them at the same workload, and what happens when you move the cut-off.

Choose the metric

No metric is right for every classifier. The choice comes from the decision that the flags drive, and from what each mistake costs there. Three questions help.

First: which mistake is worse? If a false alarm is expensive, watch precision. If a miss is expensive, watch recall. Second: is there a limit on the flags? A team that can check only so many cases needs precision and recall measured at that number of flags. Third: what happens after a flag? If a second check follows each flag, false alarms are cheap, and recall matters most.

A spam filter is a classic case for precision. A false positive hides a real email, such as an invoice or a job offer, in the spam folder, and you may never see it. A missed spam costs you one second to delete it. So spam filters are tuned for very high precision, and accept that some spam reaches the inbox.

Cancer screening is the opposite. A missed cancer can cost a life, so the first test is tuned for high recall. Its false alarms are many, but each positive result leads to a second, more exact test, which removes most of them. The second test is what makes low precision acceptable.

Card fraud alerts sit in between. A missed fraud costs money; a false alarm blocks an honest customer’s card at the till. Banks watch recall, but only at a precision that their customers accept, and a phone call or a text message checks each alert.

Larkfield has a hard limit. The senior team can review at most 20% of new tickets, so the number of flags is fixed by the capacity, not by the metric. The question becomes: with at most 20% of tickets flagged, how many escalations do we catch (recall), and how many reviews are useful (precision)? That is why the improvement target is written as a recall at a flag rate.

Case	The worse mistake	What limits the flags	Metric to watch
Spam filter	False positive: a real email is hidden	Nothing; the user checks the folder rarely	Precision, kept very high
Cancer screening, first test	False negative: a missed cancer	A second test checks every positive	Recall, kept very high
Card fraud alerts	Both: lost money, blocked customers	Customers accept only a few blocks	Recall at an acceptable precision
Larkfield’s escalation flags	A miss (a day’s wait), but a review costs 10 minutes	The senior team’s capacity: at most 20%	Precision and recall at a flag rate of at most 20%
A shop’s “top 10 products for you”	Showing an item the customer does not want	Only 10 places on the page	Precision of the top 10 (next lesson)
Accuracy of a balanced task, such as cat or dog photos	Both about the same	Nothing	Accuracy, or F1

The common mistake is to choose the metric that is easiest to compute, or the one with the nicest number. Accuracy is both, and it is often unrelated to the decision. Optimising a metric that is unrelated to the decision is a quiet failure: the number in the report goes up, and the decision gets worse. The failure case below shows it.

More than two classes

Some classifiers choose between more than two classes. For example, a model could send each new ticket to one of Larkfield’s five teams. Then there are no single “positive” and “negative” classes. Instead, compute precision and recall for each class in turn. For “warranty”, the warranty tickets are the positives, and all other tickets are the negatives. To get one number, take the plain mean over the classes: the macro average. The macro average gives a rare class the same weight as a common one. Suppose that the model routes 95% of tickets correctly, but sends almost every warranty ticket to the wrong team. Then the warranty recall is low, and the macro average shows it. Accuracy does not.

Guided practice: the counts with scikit-learn

Use the notebook, sections 3 and 4. You check the points score’s counts with scikit-learn, and compute the metrics yourself.

1. Open the notebook from the box at the top of this page, in Colab, Kaggle or VS Code.
2. Run the setup cells. You should see pandas 3.0.6, scikit-learn 1.9.1 and the Ready: line.
3. Run sections 1 and 2. In section 2, you should see the ten tickets with their points and caught 2, false alarms 1, missed 2, correctly left 5.
4. In section 3, run the worked example. You should see the three rows of the table above.
5. In section 4, run confusion_matrix(y, points >= 3). You should see [[933 63] and [130 53]].
6. Read the matrix. scikit-learn puts the real class in the rows and the answer in the columns, with class 0 first: [[TN, FP], [FN, TP]].
7. Compute precision_3 and recall_3 from the four numbers, each rounded to 3 decimals.
8. Run the check cell. It prints Check passed: (precision_3, recall_3) is right. when both are right.

Show the answer

from sklearn.metrics import confusion_matrix

tn, fp, fn, tp = confusion_matrix(y, points >= 3).ravel()
precision_3 = round(tp / (tp + fp), 3)
recall_3 = round(tp / (tp + fn), 3)
print(precision_3, recall_3)

Output

0.457 0.29

Precision is 53 ÷ (53 + 63) = 0.457, and recall is 53 ÷ (53 + 130) = 0.290. precision_score(y, points >= 3) and recall_score(y, points >= 3) from sklearn.metrics give the same numbers. Note the order: scikit-learn’s matrix starts with class 0 (not escalated), so the true negatives are top left. This lesson’s tables start with the escalated row. Always read the labels before you read a confusion matrix.

Your turn: a wider cut-off

Change one condition: the cut-off. Flag the tickets with 2 points or more. Before you run section 5 of the notebook, predict: will precision go up or down? Recall? F1? Will the flags fit the capacity of 20%? Then compute flag_report() for the new flags.

Show the answer

print(flag_report(y, points >= 2))

Output

{'flagged': 327, 'share flagged': 0.277, 'caught': 103, 'false alarms': 224, 'missed': 80, 'correctly left': 772, 'precision': 0.315, 'recall': 0.563, 'F1': 0.404, 'accuracy': 0.742}

Now 327 tickets are flagged. Recall goes up, from 0.290 to 0.563: 103 escalations are caught instead of 53. Precision goes down, from 0.457 to 0.315: most of the extra flags are false alarms. F1 goes up, from 0.355 to 0.404, because recall rose more than precision fell. But the flags are 27.7% of the tickets, more than the senior team can review. F1 cannot see the capacity, so it prefers a cut-off that Grace cannot use. A wider cut-off trades precision for recall. The next lesson is about that trade-off.

Common mistake

The most accurate cut-off

Symptom: Tomás, the AI engineer, wants “the most accurate checklist”. He tries every cut-off from 0 to 5 points on the May tickets and keeps the one with the best accuracy:

for cut_off in range(6):
    report = flag_report(y, points >= cut_off)
    print(cut_off, "| flagged", report["flagged"], "| recall", report["recall"], "| accuracy", report["accuracy"])

Output

0 | flagged 1179 | recall 1.0 | accuracy 0.155
1 | flagged 859 | recall 0.907 | accuracy 0.398
2 | flagged 327 | recall 0.563 | accuracy 0.742
3 | flagged 116 | recall 0.29 | accuracy 0.836
4 | flagged 32 | recall 0.098 | accuracy 0.848
5 | flagged 0 | recall 0.0 | accuracy 0.845

He reports: “4 points or more is 84.8% accurate, our best result.” The senior team gets 32 tickets in a month, and 165 of the 183 escalations wait in the normal queue.

Cause: accuracy rewards each correct “no”. With 84.5% of the tickets not escalated, the fewer flags, the higher the accuracy, almost down to zero flags. The metric does not match the decision. Grace’s decision is about catching escalations with at most 20% of the tickets flagged; accuracy counts something else.

Fix: write down the decision and its limits first, then choose the metric from them. Here: “flag at most 20% of tickets, and catch as many escalations as possible”. Measure recall and precision at that flag rate. Report accuracy only next to the base rate, if at all.

Consequence in AI work: a metric that does not match the decision optimises the wrong thing, and nothing warns you. The report shows a higher number, Omar, the product manager, sees “84.8% accurate”, and customers with serious problems wait longer than before.

See it end to endPredict credit-card defaultCase study

The credit case study turns a cost table for missed defaults and false alarms into a decision.

See it end to endRank a call listCase study

The call-list case study shows why accuracy is useless when 89% of answers are no.

Knowledge check

Check your understanding

A fraud model flags 80 card payments in a day. 20 of them are fraud. The day had 50 frauds in total. Compute precision and recall, and say in one sentence each what they mean for the bank.

Show the answer

TP = 20, FP = 80 − 20 = 60, FN = 50 − 20 = 30. Precision = 20 ÷ 80 = 0.25: one in four blocked payments is really fraud, so three in four blocks annoy an honest customer. Recall = 20 ÷ 50 = 0.4: the model stops 2 of every 5 frauds, and 30 frauds get through. The two numbers divide by different totals: the flags for precision, the real frauds for recall.

Check your understanding

For each case, which metric matters most, and why? (a) A filter that moves emails to a spam folder. (b) A first screening test for a serious disease, where every positive gets a second test. (c) Larkfield's flags for a senior team that can review at most 20% of tickets.

Show the answer

(a) Precision: a false positive hides a real email, while a missed spam costs a second. (b) Recall: a miss can cost a life, and the second test removes most false alarms, so low precision is acceptable. (c) Precision and recall at a flag rate of at most 20%. The capacity fixes how many tickets can be flagged. So the question is how many escalations those flags catch (recall), and how many reviews are useful (precision). Accuracy is not the answer in any of the three.

Check your understanding

Omar sees that never escalate has a higher accuracy than the points score at 3 or more points (84.5% against 83.6% on May), and asks why we do not use it. Explain why accuracy is the wrong number here, and give the numbers that you show him instead.

Show the answer

Accuracy says that “never escalate” is better (84.5% against 83.6%). The reason is that 84.5% of May tickets do not escalate, and accuracy counts every correct “no”. It cannot see that “never escalate” catches nothing. Show recall and precision: the points score catches 53 of 183 escalations (recall 0.290), and 53 of its 116 flags escalate (precision 0.457). “Never escalate” has recall 0 and no flags at all. Also show the share flagged, 9.8%, against the capacity of 20%.

Recap

• A class has no size, so classification metrics count outcomes. Two answers and two truths give four counts: caught (TP), false alarms (FP), missed (FN) and correctly left (TN). Together they are the confusion matrix.
• Accuracy is the share of right answers. With a rare positive class, it rewards saying “no”: never escalate has 84.5% on May and catches nothing.
• Precision asks “when it flags, how often is it right?” and is lowered by false alarms. Recall asks “of the real positives, how many did it catch?” and is lowered by misses. F1 is high only when both are high, and knows nothing about costs or capacity.
• Choose the metric from the decision: which mistake costs more, and what limits the number of flags. For Larkfield: precision and recall at a flag rate of at most 20%.
• The points score at 3 or more points, on May: 116 flagged (9.8%), precision 0.457, recall 0.290. The priority-1 rule flags more (163) and has the better recall (0.333) and the worse precision (0.374). Comparing two classifiers at different workloads is not fair to either.

Next, you move the cut-off and see precision and recall trade against each other. You also learn metrics that summarise every cut-off at once.

Next: The precision–recall trade-off