The precision–recall trade-off

What you will be able to do

Choose where to cut a ranked list of cases, and compare two rankings without choosing a cut. Every flag rule trades the cases it catches against the false alarms it raises; the decision, not the data, says where to stand.

NotebookDo this lesson in a notebookColab · Kaggle · GitHub

The notebook has every code cell, task and check of this lesson, the module practice and the module check, with the instructions. Its setup cells install pandas and scikit-learn and download the practice data, so it works without the local project. Use the explorer and the charts here, and run the code there.

Open in ColabOpen in KaggleView on GitHub

Colab and Kaggle open your own copy. They need a free Google or Kaggle account. Stuck? See the solution notebook.

Before you startWhat to know and to have readyAbout 70 minutes

What you need to know

• The confusion matrix, precision and recall, from Classification metrics: The confusion matrix and Precision and recall.
• Larkfield’s points score (0 to 4 points per ticket) and the ten tickets T-1 to T-10, from Classification metrics: A first yes-or-no flag: the points score.
• The senior team’s capacity (at most 20% of tickets) and the improvement target, from Define a baseline.
• The validation and test sets, from Train, validation and test sets: Three sets, three jobs.
• You moved a threshold on made-up scores in Classification trade-offs and gave each mistake a cost in Thresholds and costs, both in Essential Mathematics and Statistics for AI. This lesson does it on Larkfield’s data.

What you need on your computer (only for the local route; the notebook needs none of it)

• The project folder ~/projects/ticket-model with its .venv (pandas 3.0.6, scikit-learn 1.9.1, matplotlib 3.11.2) and the data in data/, including extra_features.csv. You made them in Define a baseline: Set up your project. If a data file is missing or changed, run python get_data.py --reset. The last line must be Ready: train 16939 rows (1983 escalated), valid 1179 (183), test 1002 (157).
• To run the notebook locally, open it in VS Code with the Jupyter extension and the .venv kernel, as in Define a baseline: Open a notebook in VS Code.

Tested on macOS with Python 3.14.6, pandas 3.0.6, scikit-learn 1.9.1 and matplotlib 3.11.2. The Windows commands are not tested. The notebook was tested in Jupyter.

A score and a threshold

Most classifiers do not answer “yes” or “no” directly. They give each case a score: a number that is higher when the positive class is more likely. Larkfield’s points score gives 0 to 4 points. The machine learning models of Module 4 give a score from 0 to 1. A spam filter gives each email a spam score; a search engine gives each page a relevance score.

A score becomes a decision only when you choose a threshold: flag every case with a score at or above it. For the points score, the threshold is a whole number of points, and the last lesson called it the cut-off: “3 points or more”. The two words mean the same thing here.

The score sorts the cases. The threshold says how far down the sorted list you act. In the last lesson, you saw that a wider cut-off (2 points instead of 3) raised recall and lowered precision. This lesson shows that this always happens, why, and how to choose.

A fishing net

Think of fishing in a pond with a net. The pond has the fish you want, and also things you do not want: old boots, cans, weeds. A small net in the best spot catches a few fish and almost no junk. A wide net catches every fish, and a lot of junk with them.

The same pond, two nets. The small net catches 3 of the 6 fish and 1 piece of junk: 3 of its 4 catches are fish (precision 75%), but it gets only half of the fish (recall 50%). The wide net catches all 6 fish and 6 pieces of junk: recall 100%, precision 50%.

Text description of the chart

Two panels show the same pond with 6 fish (blue) and 9 pieces of junk such as old boots (grey). In the left panel, a small dashed circle on the left side of the pond holds 3 fish and 1 boot. In the right panel, a wide dashed oval covers most of the pond and holds all 6 fish and 6 boots. Three boots near the edges are outside even the wide net.

Now use the names of the last lesson. The fish are the positives, and the junk is the negatives. Recall is the share of all fish that the net caught. Precision is the share of the catch that is fish. The small net has high precision and low recall. The wide net has high recall and low precision.

Making the net wider is moving the threshold down. You catch more of what you want, and more of what you do not want. You cannot make the same net catch more fish and less junk at the same time. That is the precision–recall trade-off. Only a better net, one that catches fish and lets junk through, improves both. For a classifier, that means a better score: one that puts the positives higher in the list.

A search engine: precision@k and recall@k

A search engine faces the same choice. You search for “repair a garden hose”, and the engine ranks pages from the most to the least relevant. How many results should the first page show?

If it shows 3 results, most of them are probably useful, but you may miss a good page further down. If it shows 100, the good pages are in there somewhere, but you must read through a lot of noise. Showing more results is the wide net.

For a ranked list, the cut is a number of results, k. Two metrics measure the top k:

• Precision@k (“precision at k”): of the top k results, the share that are relevant.
• Recall@k: of all the relevant results that exist, the share that are in the top k.

Here is one search with 10 results. Results 1, 2, 4 and 7 are relevant. One more relevant page exists, but the engine ranked it below 10.

Ten search results in rank order. The blue results with a tick are relevant (ranks 1, 2, 4 and 7). A fifth relevant page exists but is not in the top 10. The brackets mark the top 3 and the top 10.

Text description of the chart

A vertical list of ten results, numbered 1 to 10 from the top. Results 1, 2, 4 and 7 are blue with a tick: they are relevant. The others are pale grey. A bracket beside results 1 to 3 is labelled k = 3, with 2 relevant. A bracket beside results 4 to 10 is labelled k = 10, with 4 relevant in total. Below the list, a dashed empty box stands for the fifth relevant page, which is not in the top 10.

k	Relevant in the top k	Precision@k	Recall@k
1	1	1 ÷ 1 = 1.00	1 ÷ 5 = 0.20
3	2	2 ÷ 3 = 0.67	2 ÷ 5 = 0.40
5	3	3 ÷ 5 = 0.60	3 ÷ 5 = 0.60
10	4	4 ÷ 10 = 0.40	4 ÷ 5 = 0.80

As k grows, recall@k can only go up or stay the same: a longer list never loses a relevant page. Precision@k usually goes down, because the best results are at the top. It is the same trade-off, with k in place of a threshold.

You meet these two metrics again in RAG: Building AI That Uses Your Data. RAG means retrieval-augmented generation. There, a program searches a collection of documents for the passages that answer a question, and gives the top k passages to a language model. Recall@k asks: is the passage with the answer among the k? If not, the model cannot answer well. Precision@k asks: how many of the k passages are useful, and how many only add noise? Every retrieval system is tuned with this trade-off.

A threshold and a k are two ways to say where to cut the same ranked list. At Larkfield, “flag 3 points or more” and “review the top 116 tickets of May” are the same 116 tickets.

Move the threshold on ten tickets

Take the ten tickets of the last lesson again. Here they are, sorted by points: T-1 (4 points, escalated), T-3 and T-4 (3 points; T-3 escalated), T-2 and T-8 (2 points; T-8 escalated), T-5, T-7 and T-10 (1 point; T-5 escalated), and T-6 and T-9 (0 points). Four escalated. Move the threshold down, one point at a time:

Flag at	Flagged	TP	FP	FN	Precision	Recall
4 points or more	1	1	0	3	1 ÷ 1 = 1.00	1 ÷ 4 = 0.25
3 or more	3	2	1	2	2 ÷ 3 = 0.67	2 ÷ 4 = 0.50
2 or more	5	3	2	1	3 ÷ 5 = 0.60	3 ÷ 4 = 0.75
1 or more	8	4	4	0	4 ÷ 8 = 0.50	4 ÷ 4 = 1.00
0 or more (all)	10	4	6	0	4 ÷ 10 = 0.40	1.00

Read the two right-hand columns from top to bottom. Recall rises from 0.25 to 1.00, because each lower threshold catches more escalations. Precision falls from 1.00 to 0.40, because each lower threshold also lets in more tickets that do not escalate. At the bottom, everything is flagged: recall is 1.00 and precision equals the share of escalated tickets, 4 in 10. At the top, the few flags are the clearest cases.

Notice also that T-2 and T-8 both have 2 points. No threshold can flag one of them without the other. A points score has only a few values, so many tickets tie, and the threshold can only move in steps. Remember this for the real data.

The trade-off on May

Now the 1,179 May tickets of the validation set. Use the explorer before you read the table. The upper panel shows the tickets that did not escalate, the lower panel the escalated ones, both by points. Move the cut-off and watch the confusion matrix, the two metrics and the dot on the curve.

Explore

Move the cut-off of the points score

Solid bars are flagged (at or above the cut-off). Pale bars are not flagged. Each panel has its own count scale, because escalated tickets are much fewer.

At 4 points or more, 32 tickets are flagged: 18 escalated and 14 false alarms. 165 of the 183 escalations are missed. The senior team can review this many.

Confusion matrix at 4 points or more

Real class	Flagged	Not flagged	Total
Escalated	18true positive	165false negative (missed)	183
Not escalated	14false positive (false alarm)	982true negative	996
Total	32	1147	1,179

Flagged

32

2.7% of 1,179 tickets

Precision

56.3%

18 of 32 flags

Recall

9.8%

18 of 183 escalations

Capacity

Within capacity

at most 20.0% (235 tickets)

Each dot on the curve is one cut-off. The large dot is the cut-off you chose. The dashed line is the precision of random flags: the share of tickets that escalate.

The 1,179 May tickets of the validation set by points (996 not escalated, 183 escalated), with the cut-off, the confusion matrix, precision, recall, the senior team's capacity of 20%, and the precision–recall curve.

Text description of the explorer

The upper panel is a column chart of the 996 May tickets that did not escalate, by points: 303 with 0 points, 469 with 1, 161 with 2, 49 with 3 and 14 with 4. The lower panel shows the 183 escalated tickets: 17 with 0 points, 63 with 1, 50 with 2, 35 with 3 and 18 with 4. The escalated tickets have more points on average, but the two groups overlap at every value. A vertical line marks the cut-off; columns to its right are flagged.

At the starting cut-off, 4 points (the most), 32 tickets are flagged: 18 escalated and 14 not. Precision is 56.2% and recall 9.8%. At 3 or more: 116 flagged (9.8%), precision 45.7%, recall 29.0%, within the capacity of 235 tickets. At 2 or more: 327 flagged (27.7%), precision 31.5%, recall 56.3%, over the capacity. At 1 or more: 859 flagged, precision 19.3%, recall 90.7%. At 0 or more, every ticket is flagged: precision 15.5% (the share of escalated tickets) and recall 100%. At 5 or more, nothing is flagged and precision is not defined. The precision–recall curve below joins these points: precision falls from 56.2% to 15.5% as recall rises from 9.8% to 100%.

The same numbers as a table, from the strictest cut-off to the widest:

Flag at	Flagged	Share flagged	TP	FP	FN	Precision	Recall
4 points	32	2.7%	18	14	165	0.562	0.098
3 or more	116	9.8%	53	63	130	0.457	0.290
2 or more	327	27.7%	103	224	80	0.315	0.563
1 or more	859	72.9%	166	693	17	0.193	0.907
0 or more (all)	1,179	100%	183	996	0	0.155	1.000

• Precision
• Recall
• Share of tickets flagged
• Capacity: 20% flagged (dashed)

Precision, recall and the share of tickets flagged at each cut-off of the points score, on the 1,179 May tickets. A higher cut-off raises precision and lowers recall.

Text description of the chart

A line chart with the cut-off from 0+ to 5+ points on the horizontal axis and a value from 0% to 100% on the vertical axis. A dashed horizontal line marks the capacity, 20% of tickets flagged. Precision rises from 15.5% at 0+ to 19.3%, 31.5%, 45.7% and 56.2% at 4+; it has no value at 5+, where nothing is flagged. Recall falls from 100% at 0+ to 90.7%, 56.3%, 29.0%, 9.8% and 0% at 5+. The share flagged falls from 100% to 72.9%, 27.7%, 9.8%, 2.7% and 0%. It crosses the capacity line between 2+ and 3+.

Three things to see:

1. Precision and recall move in opposite directions. Every step to the right raises precision and lowers recall. No cut-off is best at both.
2. The decision chooses the cut-off, not the data. Grace’s team can review at most 20% of the tickets, 235 of these 1,179. A lower cut-off catches more escalations, so the best cut-off is the lowest one that fits. That is 3 points or more: 116 tickets flagged, precision 0.457 and recall 0.290.
3. The cut-off alone cannot use the whole capacity. The next step down, 2 points or more, flags 27.7% of the tickets, over the capacity. So 119 of the 235 possible reviews stay unused. The reason is that 211 tickets tie at 2 points, and the score cannot tell them apart. Recall stays at 0.290, below the target of 0.40.

Breaking the tie

Grace does not want to waste 119 reviews. The fix is a tie rule: a fair way to choose among tickets with the same score. Larkfield’s rule is simple. Flag every ticket with 3 or more points. Then flag tickets with 2 points in arrival order, first come, first flagged, until the capacity is full. Priya applies it to May: the 116 tickets with 3 or more points, then the first 119 tickets with 2 points.

May, 1,179 tickets	Flagged	Share flagged	Caught	Precision	Recall
3 or more points	116	9.8%	53	0.457	0.290
3 or more, then 2 points in arrival order	235	19.9%	84	0.357	0.459
Priority-1 rule (for comparison)	163	13.8%	61	0.374	0.333

With the full capacity, the points score catches 84 of the 183 escalations, a recall of 0.459, and meets the improvement target on May. Precision falls to 0.357, because the 2-point tickets escalate less often (23.7%) than the 3-point ones. That is the trade-off again: more reviews, more escalations caught, a lower share of useful reviews. The arrival order adds nothing about risk; it only shares the reviews fairly among tickets that look the same. A score with many different values, such as the models of Module 4, needs no tie rule.

The trade-off is not a flaw of the points score. Every score that is not perfect has it. If Grace had twice the capacity, the cut-off could move to 2 points, with recall 0.563 and precision 0.315. The same score, a different decision, a different threshold.

The precision–recall curve

The table has five rows, and a model has hundreds of possible thresholds. To see them all at once, draw each threshold as one point, with its recall across and its precision up. The line through the points is the precision–recall curve (PR curve). The large dot in the explorer moves along this curve.

• Points score (one dot per cut-off)
• Random flags (dashed)

The precision–recall curve of the points score on the 1,179 May tickets, with the two rules of Define a baseline as single points. Both rules lie below the curve.

Text description of the chart

A chart with recall from 0% to 100% across and precision from 0% to 100% up. The points-score curve has five dots: 56.2% precision at 9.8% recall (4 points), 45.7% at 29.0% (3+), 31.5% at 56.3% (2+), 19.3% at 90.7% (1+) and 15.5% at 100% (all tickets). It falls from the top left to the bottom right. A dashed horizontal line at 15.5% shows random flags: at any recall, random flags have the precision of the share escalated. Two single points mark the rules. The priority-1 rule is at 33.3% recall and 37.4% precision, below the curve. Grace’s rule (priority 1 or warranty) is at 53.6% recall and 31.4% precision, also below the curve.

A curve that is higher is better: at the same recall, more of the flags are right. The priority-1 rule and Grace’s rule from Define a baseline are single points, because a rule has no score to cut. Both lie below the points-score curve: at the same recall, the points score has a better precision. The dashed line is the floor: if you flag tickets at random, about 15.5% of the flags escalate, whatever the recall.

Average precision

A curve is hard to put in a table. To compare two scores with one number, summarise the whole curve: take the area under it. The usual way to compute it is the average precision (AP).

The idea, on the ten tickets: go down the ranked list, cut-off by cut-off. Each time you catch one more escalation, write down the precision at that point. Then take the mean of those precisions.

• At 4 points, you catch the first escalation (T-1). Precision: 1 ÷ 1 = 1.00.
• At 3 or more, you catch the second (T-3). Precision: 2 ÷ 3 = 0.67.
• At 2 or more, the third (T-8). Precision: 3 ÷ 5 = 0.60.
• At 1 or more, the fourth (T-5). Precision: 4 ÷ 8 = 0.50.

AP = (1.00 + 0.67 + 0.60 + 0.50) ÷ 4 = 0.692. In general, each precision is weighted by how much recall grew at that step. Here each step catches exactly one of the four escalations, so the weights are equal and AP is a plain mean.

$$\text{AP} = \sum_{n} (R_n - R_{n-1}) \, P_n$$

Read it as: for each threshold n, the increase in recall times the precision there, added up. AP is high when the escalations are near the top of the list, where precision is high. A perfect score, with every escalation above every other ticket, has AP = 1.

What is the floor? A score that cannot rank, such as the same score for every ticket, flags everything at once. Its precision is the share of positives, so its AP equals the share of positives: 0.155 on May. Always read AP next to that floor. On May, the points score has AP 0.310, about twice the floor.

AP suits Larkfield because escalations are rare (15.5%) and the decision is about the top of the list, where the senior team works. AP is built from precisions, and precision changes a lot at the top. AP is also called the area under the PR curve, or PR AUC.

The ROC curve

There is a second classic curve. It also draws one point per threshold, but with two different rates:

• across, the false-positive rate: of the tickets that did not escalate, the share that was flagged. You met it in Classification trade-offs in the mathematics course. It is FP ÷ (FP + TN).
• up, the true positive rate, which is another name for recall: TP ÷ (TP + FN).

The curve is the ROC curve. The name comes from radar engineers in the Second World War: “receiver operating characteristic”. It shows how many real positives you catch for each share of negatives that you flag by mistake.

• Points score (one dot per cut-off)
• Guessing (dashed)

The ROC curve of the points score on the 1,179 May tickets. The dashed diagonal is guessing. The area under the curve is the ROC AUC, 0.715.

Text description of the chart

A chart with the false-positive rate from 0% to 100% across and the true positive rate (recall) from 0% to 100% up. The points-score curve starts at the bottom left (nothing flagged) and goes through 1.4% and 9.8% (4 points), 6.3% and 29.0% (3+), 22.5% and 56.3% (2+), 69.6% and 90.7% (1+), to 100% and 100% (all tickets). It stays above the dashed diagonal, which is guessing. The priority-1 rule is a single point at 10.2% and 33.3%, just below the curve.

The dashed diagonal is guessing: if you flag tickets at random, you flag the same share of escalated and other tickets. A useful score bends up toward the top left corner: it catches many escalations while it flags few other tickets.

ROC AUC

The area under the ROC curve is the ROC AUC (area under the curve). It has a plain meaning that is easier to remember than the curve: ROC AUC is the chance that a random escalated ticket gets a higher score than a random other ticket. Ties count as half.

Check it on the ten tickets. There are 4 escalated and 6 other tickets, so 4 × 6 = 24 pairs.

• T-1 (4 points) is above all 6 others: 6 pairs.
• T-3 (3 points) is above 5 of them and ties with T-4: 5 + 0.5 = 5.5.
• T-8 (2 points) is above 4 and ties with T-2: 4.5.
• T-5 (1 point) is above the 2 tickets with 0 points and ties with T-7 and T-10: 2 + 1 = 3.

ROC AUC = (6 + 5.5 + 4.5 + 3) ÷ 24 = 19 ÷ 24 = 0.792. A perfect ranking gives 1.0, and guessing gives 0.5: a coin flip for each pair. On May, the points score has ROC AUC 0.715.

ROC AUC is common in reports, and its meaning does not depend on how rare the positives are. That is a strength and a weakness. With rare positives, most of the pairs are easy: a quiet ticket with 0 points below an escalation. Those pairs raise ROC AUC, but they say nothing about the top of the list, where the decision is. Also, the false-positive rate divides by all the negatives, which are many. Flagging 63 tickets by mistake is only 6.3% of the 996 other tickets, a small step across the chart. But those 63 are more than half of the 116 flags. So with rare positives, ROC AUC can look good while precision at the top is poor. Use AP, or precision and recall at the threshold, for that.

Choose a summary

You now have three ways to measure a score: at one threshold, at the top k, and over every threshold. Choose by what the decision needs.

If the threshold is already fixed, measure precision and recall at that threshold, and the share flagged. If a capacity fixes the number of cases you can act on, measure recall at that flag rate; Larkfield’s target is of this kind. For a list of k items, measure precision@k and recall@k. If you compare two scores before anyone chooses a threshold, summarise every threshold. Use average precision when the positives are rare and the top of the list matters. Use ROC AUC when the classes are more balanced, or when a report needs a number that does not depend on the base rate.

What you need to decide	Metric	Larkfield example
Is this threshold good enough?	Precision and recall at the threshold, and the share flagged	3+ points: precision 0.457, recall 0.290, 9.8% flagged
What do we catch with the capacity we have?	Recall at a fixed flag rate, or recall@k	The target: recall at least 0.40 with at most 20% flagged; with the tie rule, 0.459
How good is the top of a list of k?	Precision@k (and recall@k)	The 32 tickets with 4 points: precision@32 = 0.562
Which of two scores ranks better, with rare positives?	Average precision, next to its floor (the share of positives)	Points score AP 0.310; floor 0.155
Which of two scores ranks better, in a common report?	ROC AUC, next to its floor (0.5)	Points score ROC AUC 0.715
Retrieval for RAG	Recall@k, then precision@k	Is the passage with the answer in the top k?

Worked example: Priya summarises the points score

Priya computes the two summaries and the top of the list for the May tickets. The setup cells, load(), points_score() and flag_report() are the same as in Classification metrics, and valid holds the May tickets. First, the summaries next to their floors:

from sklearn.metrics import average_precision_score, roc_auc_score

y = valid[TARGET]
points = points_score(valid)
print("average precision:", round(average_precision_score(y, points), 3), "| floor:", round(y.mean(), 3))
print("ROC AUC:", round(roc_auc_score(y, points), 3), "| floor: 0.5")

The output:

Output

average precision: 0.31 | floor: 0.155
ROC AUC: 0.715 | floor: 0.5

scikit-learn handles the ties for you: all tickets with the same points enter the list together, as one step of the threshold. Next, Priya computes precision@k and recall@k for the top of the ranked list. She sorts by points, and keeps the arrival order among tickets with the same points: that is the tie rule. So k = 32 and k = 116 are the cut-offs 4 and 3, and k = 235 is the full capacity with the tie rule:

def at_k(y, scores, k):
    """Precision and recall of the k highest scores."""
    top = np.argsort(-np.asarray(scores), kind="stable")[:k]
    caught = int(np.asarray(y)[top].sum())
    return round(caught / k, 3), round(caught / int(np.sum(y)), 3)

for k in (32, 116, 235):
    print(f"k = {k}: precision@k, recall@k =", at_k(y, points, k))

Output

k = 32: precision@k, recall@k = (0.562, 0.098)
k = 116: precision@k, recall@k = (0.457, 0.29)
k = 235: precision@k, recall@k = (0.357, 0.459)

The first two rows are the cut-offs 4 and 3 of the table above, and the last row is the tie rule. A k and a threshold are two ways to cut the same list. Priya’s summary for Grace: “The points score ranks escalations about twice as well as chance (AP 0.310 against 0.155). For your capacity, flag 3 or more points, then 2-point tickets in arrival order until 235 a month. About 1 in 3 reviews finds a real escalation, and we catch 46% of the escalations, more than the target of 40%. The test month will confirm it later.”

Guided practice: the curve in code

Use the notebook, sections 1 to 4.

1. Open the notebook from the box at the top of this page, in Colab, Kaggle or VS Code.
2. Run the setup cells. You should see pandas 3.0.6, scikit-learn 1.9.1 and the Ready: line.
3. In section 1, run the ten tickets. You should see AP 0.692 and ROC AUC 0.792.
4. In section 2, compute precision@3 and recall@3 for the search example. The check cell prints Check passed: (precision_at_3, recall_at_3) is right.
5. In section 3, run the cut-off table for May. You should see the five rows of the table above.
6. In section 4, call precision_recall_curve(y, points) from sklearn.metrics. It returns the precision and recall at every threshold, and the thresholds.
7. Plot recall across and precision up. You should see the curve of the chart above.
8. Run the summaries. You should see average precision: 0.31 and ROC AUC: 0.715.

Show the answer

from sklearn.metrics import precision_recall_curve

precision, recall, thresholds = precision_recall_curve(y, points)
for p, r, t in zip(precision, recall, thresholds):
    print(f"{t} points or more: precision {p:.3f}, recall {r:.3f}")

Output

0 points or more: precision 0.155, recall 1.000
1 points or more: precision 0.193, recall 0.907
2 points or more: precision 0.315, recall 0.563
3 points or more: precision 0.457, recall 0.290
4 points or more: precision 0.562, recall 0.098

precision_recall_curve returns one more precision and recall than thresholds: the last pair is (1.0, 0.0), the point where nothing is flagged, which zip leaves out. The thresholds come from low to high, so the table reads from the widest net to the smallest.

Your turn: rarer escalations

Change one condition: how rare the positive class is. Keep all 996 May tickets that did not escalate, but only 61 of the 183 escalations, picked at random. The points of each ticket do not change. Before you run section 5 of the notebook, predict: does ROC AUC go up, down or stay about the same? And AP?

Show the answer

rng = np.random.default_rng(0)
escalated = np.where(y == 1)[0]
others = np.where(y == 0)[0]
keep = np.concatenate([rng.choice(escalated, 61, replace=False), others])
y_rare, points_rare = y.to_numpy()[keep], points.to_numpy()[keep]
print(f"escalated: {y_rare.mean():.1%}")
print("ROC AUC:", round(roc_auc_score(y_rare, points_rare), 3))
print("average precision:", round(average_precision_score(y_rare, points_rare), 3))

Output

escalated: 5.8%
ROC AUC: 0.704
average precision: 0.135

ROC AUC stays about the same (0.704 against 0.715). It compares pairs, and an escalated ticket is still above a random other ticket as often as before. AP falls from 0.310 to 0.135. With fewer escalations, each flag finds an escalation less often, so precision falls at every threshold. Its floor fell too, from 0.155 to 0.058. Both metrics are correct; they answer different questions. So compare AP only between scores on the same tickets, always next to its floor. And when positives are rare, do not let a good ROC AUC hide a poor precision.

Common mistake

Precision compared at different workloads

Symptom: Tomás compares the points score with the priority-1 rule. He reports: “At 3 points or more, the points score has a precision of 0.457. The rule has 0.374. The points score is 22% more precise.” Omar is pleased and asks Grace to switch.

for name, flags in [("rule: priority 1", valid["priority"] == 1), ("points: 3 or more", points >= 3)]:
    report = flag_report(y, flags)
    print(f"{name:18s} flagged {report['flagged']:3d} | precision {report['precision']} | recall {report['recall']}")

Output

rule: priority 1   flagged 163 | precision 0.374 | recall 0.333
points: 3 or more  flagged 116 | precision 0.457 | recall 0.29

Cause: the two flag sets have different sizes. The points score at 3 or more flags only 116 tickets, its clearest cases, and catches 53 escalations. The rule flags 163 and catches 61. A stricter cut-off almost always raises precision, so precision at different workloads compares two different decisions, not two classifiers. With Tomás’s argument alone, Grace would switch to a flag that catches 8 fewer escalations a month.

Fix: compare classifiers at the same workload, or across all thresholds. Give the points score the rule’s 163 reviews, with the tie rule: at_k(y, points, 163) gives precision 0.405 and recall 0.361. At the same workload, the points score beats the rule on both numbers. On the precision–recall curve, the rule is a point below the points-score curve. Or compare the AP of two scores.

Consequence in AI work: a precision or a recall alone, without the threshold and the share flagged, says almost nothing. Teams that report “95% precision” for a retrieval system or a fraud filter often mean “at a threshold that flags very little”. Ask for the other number, and for the workload, every time.

See it end to endRank a call listCase study

The call-list case study turns precision at a fixed capacity into a hit rate and lift for a call centre.

Knowledge check

Check your understanding

A search for a support article returns 5 results. Results 1, 3 and 4 are useful. The knowledge base has 6 useful articles for this question. What are precision@5 and recall@5? If the search returned 20 results, which of the two could go down?

Show the answer

Precision@5 = 3 ÷ 5 = 0.6: three of the five results are useful. Recall@5 = 3 ÷ 6 = 0.5: half of the useful articles are in the top 5. With 20 results, recall@20 cannot be lower than 0.5, because the top 20 includes the top 5. Precision@20 can go down, and usually does: the 15 extra results are ranked lower, so fewer of them are useful.

Check your understanding

Predict: Grace's capacity grows from 20% to 40% of the tickets. Which cut-off of the points score do you choose now, and what happens to precision and recall?

Show the answer

Choose the lowest cut-off that flags at most 40%. 2 points or more flags 27.7% of the May tickets; 1 or more flags 72.9%, too many. At 2 or more, recall rises from 0.290 to 0.563: 103 of 183 escalations are caught. Precision falls from 0.457 to 0.315: about 1 in 3 reviews finds an escalation. With the tie rule, Grace could also add 1-point tickets in arrival order up to 40%. The score is the same; the decision moved the threshold.

Module practice

This module taught you how to judge a supervised model before you build one. Use separate data, use the metric that matches the decision, and keep the trade-off in view. The module practice puts it together for Larkfield’s escalation flags.

Guided: evaluate two flag rules for the capacity

Use the notebook, section 7.

1. Compute flag_report() on the validation set for the priority-1 rule.
2. Find the lowest cut-off of the points score that flags at most 20% of the May tickets. Put it in cut_20.
3. Run the check cell for cut_20.
4. Compute flag_report() for the points score at cut_20.
5. Fill the rest of the capacity with the tie rule: at_k(y, points, 235).
6. Write a short evaluation note for Grace: for each way to flag, the share flagged, precision and recall, and whether it meets the target (at most 20% flagged, recall at least 0.40).

Show a model answer

cut_20 = min(c for c in range(6) if (points >= c).mean() <= 0.20)
print("cut-off:", cut_20)
for name, flags in [("rule: priority 1", valid["priority"] == 1), (f"points: {cut_20} or more", points >= cut_20)]:
    report = flag_report(y, flags)
    print(f"{name:18s} {report['share flagged']:.1%} flagged | precision {report['precision']} | recall {report['recall']}")
print("tie rule, 235 flags: precision, recall =", at_k(y, points, 235))

Output

cut-off: 3
rule: priority 1   13.8% flagged | precision 0.374 | recall 0.333
points: 3 or more  9.8% flagged | precision 0.457 | recall 0.29
tie rule, 235 flags: precision, recall = (0.357, 0.459)

Note for Grace (validation set, May 2026, 1,179 tickets, 183 escalated): “The priority-1 rule flags 13.8% of the tickets. 37% of its flags escalate, and it catches 33% of the escalations: below the target. The points score at 3 or more points flags only 9.8%: 46% of its flags escalate, but it catches only 29%. It leaves 119 reviews unused, because the next cut-off, 2 points, flags 28%. With the tie rule (then 2-point tickets in arrival order, up to 235 a month), it flags 19.9%. 36% of its flags escalate, and it catches 46% of the escalations. That meets the target on May. We confirm it on the test month only at the end. Module 4 builds models with finer scores, which need no tie rule.”

Your task: a different decision

Larkfield’s warehouse team asks for a new flag: “Which orders will be returned?” A returned order costs about 15 dollars to handle. Before shipping, the team can call at most 50 customers a day, out of about 400 orders. A call costs a few minutes. A call to a customer who would not return the order costs nothing more. Choose the main metric for this decision, say which number limits the flags, and say how you would compare two scores. Write four or five sentences.

Show a model answer

The team can make 50 calls a day. So the decision is a ranked list with a fixed k = 50, out of about 400 orders (12.5%). The main metric is recall@50: of all the orders that customers return, how many are in the 50 calls. Precision@50 goes next to it, because it says how many calls are useful. But a wasted call costs little, so recall matters more. Accuracy is not useful: customers return only a few orders, so a rule that predicts “no return” is very accurate and useless. To compare two scores, compute recall@50 and precision@50 for both on the same validation days. Add AP as a summary of the whole ranking, next to its floor, the share of returned orders.

Module check

Check your understanding

Retrieval: in one sentence each, what do precision and recall measure, and which kind of mistake lowers each one?

Show the answer

Precision is the share of the flags (or of the top k results) that are really positive; false positives, the false alarms, lower it. Recall is the share of all real positives that were flagged (or are in the top k); false negatives, the misses, lower it. Neither uses the true negatives, which is why both stay honest when the positive class is rare.

Check your understanding

Explain: Tomás wants to choose the cut-off of the points score on the test set (June), because it is the newest data. Why must the cut-off be chosen on the validation set, and what is the test set for?

Show the answer

A cut-off is a choice, like a setting of a model. If you choose it by looking at the test set, the test score is no longer an honest estimate. You picked the option that happened to work best on those tickets. So the score is too good, and no new data is left to check it. Choose the cut-off on the validation set (May), which exists for choosing and comparing. Use the test set (June) once, at the end, to report how the final choice works on new tickets. After you look at it, change nothing. This is the sealed final exam of Train, validation and test sets.

Check your understanding

Completion: a hospital uses a model to choose which chest X-rays a radiologist reads first. Every X-ray is read on the same day anyway; the first 30 are read within an hour. A missed urgent case read late can harm the patient. Choose the metric for this decision and justify it.

Show the answer

The model ranks the X-rays, and the decision is the top 30 each day. So the metric is a top-k metric: recall@30, the share of the day’s urgent cases that are in the first 30. A missed urgent case is the costly mistake, so recall matters most. Precision@30 goes next to it: it shows how much of the radiologist’s first hour goes to urgent cases. Accuracy is wrong here, because urgent cases are rare. To compare two models before choosing anything, use AP next to the share of urgent cases.

End of Module 3

You can now judge a supervised model before you build one. Module 3 gave you four tools: three sets of data with three jobs, regression metrics for numbers, classification metrics for classes, and the trade-off of a score with its threshold. Module 4 builds the first models, and you measure them with exactly these tools.

Reset to a known state

Module 3 does not add files to your project. If your project folder is broken, or a data file is missing or changed:

1. Open a terminal in ~/projects/ticket-model, and activate the .venv.
2. Run python get_data.py --reset.
3. Check the last line: Ready: train 16939 rows (1983 escalated), valid 1179 (183), test 1002 (157).

The notebooks of this module need none of this: each one downloads its own copy of the data.

Recap

• A score ranks the cases, and a threshold (or a k) cuts the list. A lower threshold catches more positives (recall up) and more false alarms (precision down). Only a better score improves both.
• The decision chooses the threshold. For Larkfield, it is the lowest cut-off that fits the 20% capacity: 3 points or more, with precision 0.457 and recall 0.290. A tie rule (2-point tickets in arrival order) fills the capacity: precision 0.357, recall 0.459.
• Precision@k and recall@k measure the top k of a ranked list; the RAG course uses them to judge retrieval.
• The PR curve and average precision summarise every threshold. The floor of AP is the share of positives (points score: AP 0.310, floor 0.155). The ROC curve and ROC AUC use the false-positive rate, with a floor of 0.5 (points score: 0.715). With rare positives, prefer AP.
• Module 3 in short: split the data so that the judge has never seen the answers. Choose the metric from the decision. Read every number next to its baseline, its floor and its threshold.

Next, Module 4 builds the first models: a regression for the daily tickets and a classifier for the escalations, measured with these metrics.

Next: Regression intuition