Thresholds and costs

What you will be able to do

Choose a threshold by counting what each choice costs. Count the urgent tickets that are missed, and the work of the senior agents who review every flag.

NotebookOptional: check the numbers in a notebookColab · Kaggle · GitHub

The notebook is optional: this lesson works with a calculator and the explorer. It has the scores of the 1,000 test tickets from this page’s explorer. Calculate the cost at all 101 thresholds, find the cheapest one for other costs and base rates, and redo the tasks in Python, with checks. It uses only Python’s standard library, so it needs no install.

Open in ColabOpen in KaggleView on GitHub

Colab and Kaggle open your own copy. They need a free Google or Kaggle account. Stuck? See the solution notebook.

Before you startWhat to know and to have readyAbout 45 minutes

What you need to know

• The four counts of a confusion matrix, and precision and recall, from Classification trade-offs: Now the names.
• The test counts at 0.30, 0.50 and 0.70, from Classification trade-offs: Worked example and Guided practice. This lesson repeats them.
• That a result from a small sample is uncertain, from Estimate uncertainty: The idea. The test set has only 80 urgent tickets.

What you need to have ready (the notebook needs none of it)

• A calculator. A phone calculator is enough.
• Your c03-notes.md file. You made it in the “Before you start” box of Mathematical reading skills. If you cannot find it, make a new empty file with that name: this lesson does not use earlier answers.

The notebook was tested in Jupyter with Python 3.14.6 (standard library only).

Where you meet this in AI

Every system that acts on a score has a threshold, and somebody chose it. Usually, they chose it by counting costs:

• A bank’s fraud model blocks a payment above a threshold. The bank compares the cost of a missed fraud with the cost of a phone call to a customer whose payment was blocked by mistake.
• A content moderation team has a fixed number of reviewers. The threshold must not send them more posts than they can check in a day.
• A screening model in a hospital often has a recall target, for example “find at least 95 of 100 cases”. The threshold is set to meet the target, and the team plans staff for the extra checks.
• A spam filter uses a high threshold, because a lost invoice costs much more than one spam email in the inbox.

In each case, the threshold is a business decision, not a property of the model. The model gives the scores. People decide where to draw the line.

One threshold, three consequences

In the last lesson, you counted the 1,000 test tickets (80 urgent, 920 not urgent) at three thresholds. Now ask what each threshold means for Larkfield’s week.

The test set has 1,000 tickets, and Larkfield gets about 1,200 tickets a week. If new tickets look like the test tickets, multiply a count per 1,000 by 1.2 to get a count per week. For a day, divide the week by 7.

Threshold	Flagged per 1,000	Missed urgent	False alarms	Reviews per week	Reviews per day
0.30	442	1	363	442 × 1.2 = 530	about 76
0.50	112	15	47	112 × 1.2 = 134	about 19
0.70	32	53	5	32 × 1.2 = 38	about 5

Grace reads the table for her team. At 0.30, her senior agents review about 76 tickets a day, and most are false alarms. At 0.70, they review only 5 a day, but 53 of every 80 urgent tickets wait in the normal queue.

Neither end is good. To compare the thresholds between them, Larkfield needs one number that includes both kinds of mistake.

Give each mistake a cost

Omar, the product manager, works out two costs with Grace. He uses cost points, so that work time and customer harm are on one scale.

• A review costs 2 cost points. A senior agent spends about 6 minutes on each flagged ticket. This is true for every flag, urgent or not.
• A missed urgent ticket costs 30 cost points. The customer waits too long. Often this means a late refund and a complaint, and sometimes the customer asks the bank to reverse the payment.

The total cost of a threshold is the review cost of all flags plus the cost of all missed urgent tickets:

$$\text{total cost} = 2 \times \text{flagged} + 30 \times \text{missed}$$

“Two cost points for every flagged ticket, plus thirty for every urgent ticket that is not flagged.” A true negative costs nothing, because nobody does any work on it and nothing goes wrong.

Threshold	Flagged	Missed	2 × flagged	30 × missed	Total cost
Flag everything (0.00)	1,000	0	2,000	0	2,000
0.30	442	1	884	30	914
0.50	112	15	224	450	674
0.70	32	53	64	1,590	1,654
Flag nothing (1.00)	0	80	0	2,400	2,400

All costs are per 1,000 tickets. Of these five choices, 0.50 is the cheapest. But there are 101 thresholds from 0.00 to 1.00 in steps of 0.01. The explorer calculates all of them.

Explore: find the cheapest threshold

Explore

Threshold, workload and cost

Solid bars are flagged (score at or above the threshold). Pale bars are not flagged. You can also drag the round handle on the line, or focus it and press the arrow keys (Shift for steps of 0.10). Reviews per week assume 1,200 tickets a week, with the same mix as these 1,000.

At threshold 0.50, the model flags 112: 65 urgent and 47 false alarms. It misses 15 of the 80 urgent tickets. The cheapest threshold for these costs is 0.42 (606).

Confusion matrix at threshold 0.50

Real class	Flagged	Not flagged	Total
Urgent	65true positive	15false negative (missed)	80
Not urgent	47false positive (false alarm)	873true negative	920
Total	112	888	1,000

Flagged

112

of 1,000 tickets

Precision

58.0%

65 of 112 flags

Recall

81.3%

65 of 80

False-positive rate

5.1%

47 of 920

Reviews per week

134

about 19 a day

Total cost

674

cost points per 1,000 tickets

The 1,000 test tickets with the threshold, the review workload at 1,200 tickets a week, and the total cost at every threshold for the costs that you enter.

Text description of the explorer

The upper chart shows the scores of the 920 tickets that are not urgent and the 80 urgent tickets in two panels. A vertical line marks the threshold. Below the chart are the confusion matrix, precision, recall and the false-positive rate. Then come the reviews per week and per day, and the total cost. Two number boxes set the cost of one review (start: 2) and of one missed urgent ticket (start: 30).

The lower chart shows the total cost per 1,000 tickets at every threshold from 0 to 1, for the costs in the boxes. With the starting costs, the curve starts at 2,000 (flag everything). It falls to its lowest point, 606 at threshold 0.42. Then it rises to 2,400 (flag nothing). The curve is low and almost flat from 0.37 to 0.50, where every threshold costs less than 700.

At threshold 0.42, the model flags 198 tickets and misses 7 urgent tickets. That is about 238 reviews a week, or about 34 a day. If the cost of a missed urgent ticket is 60, the cheapest threshold moves down to 0.37. If the cost of a review is 4, it moves up to 0.50. If a box is empty or negative, the explorer asks for a valid cost and does not show a total.

The cost curve

The chart below is the cost curve from the explorer, for the starting costs. Each point is one threshold.

Total cost at each threshold, with a review at 2 cost points and a missed urgent ticket at 30 cost points. The marked dot is the cheapest threshold, 0.42, at 606 cost points.

Text description of the chart

A line chart. The horizontal axis is the threshold from 0 to 1. The vertical axis is the total cost per 1,000 tickets, from 0 to 2,500 cost points (shown in thousands). At threshold 0, the cost is 2,000, because every ticket is reviewed. The cost falls to 914 at 0.30 and reaches its lowest value, 606, at 0.42. Every threshold from 0.37 to 0.50 costs less than 700. After 0.50, the cost rises steeply: 1,220 at 0.60, 1,654 at 0.70 and 2,400 at 1.00. At 1.00, all 80 urgent tickets are missed.

Three things to read from the curve:

1. The left side falls slowly. Each step to the right removes some false alarms, and each false alarm saves only 2 points.
2. The right side rises fast. Each step to the right now misses urgent tickets, and each one costs 30 points.
3. The bottom is wide and a little uneven. Every threshold from 0.37 to 0.50 costs less than 700 points. The small steps in the curve come from single tickets: one more missed urgent ticket adds 30 points at once.

The third point is important. The test set has only 80 urgent tickets. With a different 1,000 tickets, the lowest point could be somewhere else in this range. From Module 5, you know that a small sample gives an uncertain answer. So the honest result is: “the cheapest thresholds are from about 0.37 to 0.50”, not “exactly 0.42”.

Worked example: Priya’s recommendation

Priya prepares a recommendation for Omar. She calculates the cost at the thresholds near the bottom of the curve. She shows every step, so that Omar can check it.

Threshold	Flagged	Missed	2 × flagged	30 × missed	Total cost	Reviews per day
0.40	228	7	456	210	666	228 × 1.2 ÷ 7 ≈ 39
0.42	198	7	396	210	606	198 × 1.2 ÷ 7 ≈ 34
0.45	163	11	326	330	656	163 × 1.2 ÷ 7 ≈ 28
0.50	112	15	224	450	674	112 × 1.2 ÷ 7 ≈ 19

From 0.40 to 0.42, the model stops flagging 30 tickets that are not urgent and still misses only 7 urgent tickets. That saves 60 points. From 0.42 to 0.45, it saves 35 reviews (70 points) but misses 4 more urgent tickets (120 points). That is a bad exchange, so the cost rises.

Priya writes: “On the test set, threshold 0.42 has the lowest cost: 606 points per 1,000 tickets. Every threshold from 0.37 to 0.50 costs less than 700 points, so they are almost equal. At 0.42, senior agents review about 34 tickets a day, and the model misses about 7 of every 80 urgent tickets.”

Grace answers at once: “My two senior agents can review about 30 tickets a day, not 34.” The cheapest threshold is not always a possible threshold. You use this limit in the module practice at the end of the next lesson.

Why the best threshold moves

The cheapest threshold is not a fixed property of the model. It depends on two things that are outside the model.

The costs

The explorer and the curve use 2 points for a review and 30 for a miss. Change them, and the bottom of the curve moves:

Cost of one review	Cost of one missed urgent ticket	Cheapest threshold	Total cost per 1,000 tickets
2	30	0.42	606
2	60	0.37	802
2	15	0.50	449
4	30	0.50	898
1	30	0.37	401

Only the ratio of the two costs matters for where the bottom is. A miss at 60 with a review at 2 gives the same cheapest threshold as a miss at 30 with a review at 1. In both cases, the ratio is 30. When a miss becomes more expensive compared with a review, the cheapest threshold goes down: flag more, miss fewer.

There is a simple way to see why. A review of one ticket costs 2 points. It saves 30 points only if the ticket is urgent. So a review pays for itself when the chance that the ticket is urgent is more than 2 ÷ 30, about 7%. Flag every ticket that is more than about 7% likely to be urgent. If a miss costs 60, the limit falls to 2 ÷ 60, about 3%, and you flag more tickets. You see in the next lesson how to read “the chance that a ticket is urgent” from the scores.

The base rate

The test set has 8% urgent tickets. Suppose a winter storm delays many deliveries, and 16% of tickets become urgent. The model scores each ticket in the same way, but the mix changes. The chart shows the cost curve for three base rates, with the same model and the same costs.

• 4% urgent: cheapest 0.50
• 8% urgent: cheapest 0.42
• 16% urgent: cheapest 0.37

Total cost at each threshold when 4%, 8% or 16% of tickets are urgent, with the same scores and the same costs. The more common urgent tickets are, the lower the cheapest threshold.

Text description of the chart

A line chart with three cost curves, for 4%, 8% and 16% urgent tickets per 1,000. All three start at 2,000 at threshold 0, because every ticket is reviewed. With 4% urgent tickets, the lowest cost is 388 at threshold 0.50, and the curve ends at 1,200. With 8%, the lowest cost is 606 at 0.42, and the curve ends at 2,400. With 16%, the lowest cost is 918 at 0.37, and the curve ends at 4,800. The 4% and 16% curves use the counts of the 8% test set, scaled. Urgent counts are multiplied by 0.5 or 2. The other counts are scaled so that the total stays 1,000.

With more urgent tickets, every threshold misses more of them, so misses become a bigger part of the cost. The bottom of the curve moves left, to a lower threshold. With fewer urgent tickets, it moves right.

This means that a threshold chosen in a calm month can be wrong in a busy month. Larkfield must watch the share of urgent tickets after release and check the threshold again when it changes.

Guided practice: a more expensive miss

Larkfield’s payment provider now charges a fee for every payment that a customer reverses. Omar says a missed urgent ticket now costs 60 cost points. A review still costs 2.

1. In c03-notes.md, add a heading: ### Cheapest threshold.
2. In the explorer, make your prediction, then press the button.
3. Change the cost of one missed urgent ticket to 60.
4. Read the cheapest threshold from the status line or the cost curve.
5. Move the threshold slider to that value.
6. Write down the flagged tickets, the missed urgent tickets, the reviews per day and the total cost.
7. Check the total cost with a calculator: 2 × flagged + 60 × missed.
8. Write one sentence for Grace about her team’s workload at this threshold.

Expected result: the cheapest threshold is 0.37. The model flags 281 tickets and misses 4. The total cost is 2 × 281 + 60 × 4 = 562 + 240 = 802. That is about 337 reviews a week, or about 48 a day.

Show an example answer

• Cheapest threshold: 0.37 (it was 0.42 with a miss at 30).
• Flagged: 281 per 1,000 tickets. Missed: 4 of 80 urgent tickets.
• Total cost: 562 + 240 = 802 points per 1,000 tickets.
• Reviews: 281 × 1.2 = 337 a week, 337 ÷ 7 ≈ 48 a day.

For Grace: “If a missed urgent ticket costs 60 points, the cheapest threshold is 0.37. Your senior agents would review about 48 tickets a day, about 14 more than at 0.42. Fewer than 3 in 10 flags would be urgent.”

The more expensive miss moved the threshold down, as the cost ratio predicts: 60 ÷ 2 = 30 is a bigger ratio than 30 ÷ 2 = 15.

Independent variation: a more expensive review

Set the miss cost back to 30. Now the senior agents are busy with a new task, and Omar values their time higher: a review costs 4 cost points.

Predict first: does the cheapest threshold go up or down? Then find it in the explorer. Write the threshold, the total cost and the reviews per day.

Show an example answer

The ratio is now 30 ÷ 4 = 7.5, smaller than 15, so a review is relatively more expensive. The cheapest threshold goes up, to 0.50.

• Flagged: 112. Missed: 15.
• Total cost: 4 × 112 + 30 × 15 = 448 + 450 = 898 points per 1,000 tickets.
• Reviews: 112 × 1.2 = 134 a week, about 19 a day.

Run Python in your browser

Optional: the cost at several thresholds

This is optional. It calculates the total cost for seven thresholds from the counts in this lesson. Change miss_cost to 60 or review_cost to 4, run it again, and see which row is now the cheapest. In professional work, a team counts at every threshold with a library such as scikit-learn (for example, precision_recall_curve). Then it adds the costs in a short script like this one.

Python code (you can change it)# Counts per 1,000 test tickets at some thresholds: (threshold, flagged, missed) rows = [ (0.30, 442, 1), (0.40, 228, 7), (0.42, 198, 7), (0.45, 163, 11), (0.50, 112, 15), (0.60, 55, 37), (0.70, 32, 53), ] review_cost = 2 miss_cost = 30 for threshold, flagged, missed in rows: total = review_cost * flagged + miss_cost * missed print(threshold, "cost:", total)

Common mistake

Keeping the default threshold of 0.5

Symptom: the model goes live with threshold 0.50. Nobody discussed it. A month later, Omar asks why it was 0.50. The answer is “that is the default”.

Cause: many tools turn a score into a yes or no at 0.50, because it is in the middle of 0 to 1. The tool does not know Larkfield’s costs, the base rate or the size of Grace’s team. For Larkfield, 0.50 costs 674 points per 1,000 tickets, about 11% more than the cheapest threshold. It also misses 8 more urgent tickets than 0.42.

Fix: treat the threshold as a decision that you write down. State the costs, the base rate and the workload limit that you used. Show the cost curve and the range of good thresholds. Plan a date to check it again with new data.

Check your understanding

Why does a lower threshold always give a higher or equal recall?

Show the answer

A lower threshold moves the line to the left. Every ticket that was flagged before is still flagged, and some more tickets are flagged too. So the number of urgent tickets that are flagged cannot go down. Recall is that number divided by the fixed 80 urgent tickets, so it stays the same or goes up.

Check your understanding

Predict: a new contact form makes urgent tickets less common, so only 4% of tickets are urgent. The costs stay the same. Should Larkfield move the threshold up or down? Why?

Show the answer

Up. With fewer urgent tickets, each threshold misses fewer of them, so the misses are a smaller part of the cost. The reviews are now a bigger part, and reviews fall when the threshold goes up. With the same scores, the cheapest threshold moves from 0.42 to 0.50.

Check your understanding

Tomás says: 'The cost curve has its minimum at exactly 0.42, so we use 0.42 and no other number.' What do you tell him?

Show the answer

The curve is almost flat from 0.37 to 0.50 (606 to 696 points), and its small steps come from single urgent tickets. The test set has only 80 urgent tickets, so another test set could put the minimum somewhere else in that range. Report the range, and choose inside it with a second reason, such as the workload of the senior agents.

Recap

• Each threshold has three consequences: missed urgent tickets, false alarms, and the review workload. Multiply counts per 1,000 test tickets by 1.2 for a week at Larkfield.
• The total cost puts both mistakes on one scale: 2 × flagged + 30 × missed. The cost curve shows every threshold, and its lowest point is the cheapest threshold.
• Report a range of good thresholds, not one exact number, because the test set has few urgent tickets.
• The cheapest threshold moves down when a miss costs more compared with a review, or when urgent tickets become more common.

Next, you check whether the scores mean what they seem to mean, and whether the model works equally well for every group of customers.

Next: Calibration and slices