Calibration and slices

What you will be able to do

Before Larkfield trusts one threshold for all tickets, find out two things. Does the score mean what it seems to mean? Does the model work for every group of customers?

NotebookOptional: check the numbers in a notebookColab · Kaggle · GitHub

The notebook is optional: this lesson works with a calculator. It has the scores of the 1,000 test tickets. Build the calibration table from them, calculate recall and precision for each slice, and redo the tasks and the module practice in Python, with checks. It uses only Python’s standard library, so it needs no install.

Open in ColabOpen in KaggleView on GitHub

Colab and Kaggle open your own copy. They need a free Google or Kaggle account. Stuck? See the solution notebook.

Before you startWhat to know and to have readyAbout 50 minutes

What you need to know

• Precision and recall, from Classification trade-offs: Now the names.
• The total cost (2 × flagged + 30 × missed) and how to read the cheapest threshold from a cost curve, from Thresholds and costs: The cost curve.
• The rule “review a ticket when its chance of being urgent is more than 2 ÷ 30”, from Thresholds and costs: The costs. Part 1 checks whether the scores give that chance.
• That a share from a few cases is uncertain, from Estimate uncertainty: Standard error.

What you need to have ready (the notebook needs none of it)

• A calculator. A phone calculator is enough.
• Your c03-notes.md file. You made it in the “Before you start” box of Mathematical reading skills. If you cannot find it, make a new empty file with that name: this lesson does not use earlier answers.
• The explorer in Thresholds and costs: Explore, open in a second tab. Step 5 of the module practice uses it. You do not need your earlier settings: set the costs again in the explorer.
• About 15 minutes at the end for the module practice and the module check.

The notebook was tested in Jupyter with Python 3.14.6 (standard library only).

Where you meet this in AI

A score is easy to read as a chance: “0.8 means 80% likely”. Sometimes that is true, and sometimes it is far from true. And one number for all users can hide a group that the model serves badly.

• A weather app says “70% chance of rain”. If it rains on about 7 of every 10 such days, you can plan with that number.
• A chat assistant says it is “very confident” in an answer. Its confidence and its real accuracy are often different, so teams measure both.
• A risk score in a hospital goes to a doctor as a percent. If the percent is too high, patients get treatment that they do not need.
• Speech recognition that works well on average can make many more mistakes for speakers with some accents. Published studies found such gaps in real systems.

Part 1: can you read the score as a chance?

A small example

A weather app said “70% chance of rain” on 10 different days. It rained on 7 of them. On those days, the forecast was right as a chance: 70% predicted, 7 of 10 observed.

A model is calibrated when its scores match how often things really happen. Of all tickets with a score near 0.3, about 30% should be urgent. Of all tickets with a score near 0.8, about 80% should be urgent.

• If the real share is lower than the score, the model is overconfident: it says “likely” too often.
• If the real share is higher than the score, the model is underconfident: it is too careful.

Check Larkfield’s scores

Priya puts the 1,000 test tickets into ten groups by score: 0.0 to 0.1, 0.1 to 0.2, and so on. A group of values like this is a bin. For each bin, she counts the tickets and the urgent tickets. Then she divides.

Score bin	Tickets	Urgent	Share urgent (observed)	Mean score in the bin
0.0 to 0.1	106	0	0.0%	0.06
0.1 to 0.2	198	0	0.0%	0.15
0.2 to 0.3	254	1	0.4%	0.25
0.3 to 0.4	214	6	2.8%	0.34
0.4 to 0.5	116	8	6.9%	0.44
0.5 to 0.6	57	22	38.6%	0.54
0.6 to 0.7	23	16	69.6%	0.65
0.7 to 0.8	18	13	72.2%	0.74
0.8 to 0.9	11	11	100.0%	0.85
0.9 to 1.0	3	3	100.0%	0.95
All	1,000	80	8.0%

Each bin includes its lower edge, so a score of exactly 0.40 is in the bin “0.4 to 0.5”. For a calibrated model, the last two columns would be about equal. Here they are not. A calibration diagram shows the difference as a picture.

• Larkfield model: one dot per score bin, bigger dot = more tickets
• Perfect calibration (dashed): score = observed share

Calibration diagram of the urgent-ticket model on the 1,000 test tickets. Below the dashed line, the model is overconfident. Above it, the model is underconfident.

Text description of the chart

A square chart. The horizontal axis is the mean score of each bin, from 0 to 1. The vertical axis is the share of really urgent tickets in that bin, from 0% to 100%. A dashed diagonal line shows perfect calibration, where the share equals the score. The area below the diagonal is shaded and labelled overconfident; the area above is labelled underconfident. The model’s ten dots, joined by a line, start on the horizontal axis: 0% urgent at mean scores 0.06 and 0.15. Up to the middle, they stay far below the diagonal: 0.4% at 0.25, 2.8% at 0.34, 6.9% at 0.44. Then the line rises steeply, close to the diagonal: 38.6% at 0.54, 69.6% at 0.65, 72.2% at 0.74. The last two dots, at 0.85 and 0.95, are at 100%, above the diagonal. Those two bins hold only 11 and 3 tickets, so their dots are the smallest. The biggest dot is the bin 0.2 to 0.3, with 254 tickets.

Read the diagram from left to right:

1. From 0.2 to 0.5, the model is overconfident. Tickets with a score near 0.44 are urgent only 7 times in 100, not 44 times. If an agent reads the score as “44% likely urgent”, they expect six times too many urgent tickets.
2. From 0.6 to 0.8, the model is close to calibrated. A score near 0.65 means about 70% urgent.
3. Above 0.8, the model looks underconfident, but those bins have only 11 and 3 tickets. From Module 5, you know that a share from 3 tickets is very uncertain. Do not draw a conclusion from them.

Why calibration matters, and when it does not

Calibration and ranking are different things. The model ranks tickets well: a higher score almost always means a higher share of urgent tickets. The cost curve in the last lesson used only the counts above and below each threshold. That works even when the scores are not calibrated. So the threshold that you found is still valid.

Calibration matters when people or programs use the score as a chance:

• Grace’s dashboard shows “44%” next to a ticket, and an agent gives it priority over a ticket with “30%”. The real chances are about 7% and 3%. Both are low, but the agent does not know that.
• The rule from the last lesson says: flag a ticket when the chance that it is urgent is more than 2 ÷ 30, about 7%. To use that rule, you need real chances. The table gives them: the bin 0.4 to 0.5 is 6.9% urgent, almost exactly at the limit. That is why the cheapest threshold, 0.42, is inside this bin.
• A new model with different scores cannot reuse the old threshold. Its 0.42 can mean something else.

Teams can recalibrate a model: they learn a mapping from score to observed share on held-out data, and they show the mapped value instead. In scikit-learn, CalibratedClassifierCV does this. Whatever the method, check the result with a new calibration diagram.

Part 2: one average hides a weak group

At threshold 0.44, the model catches 70 of the 80 urgent tickets: a recall of 87.5%. That sounds good. Mei, the data engineer, asks a different question: “Is it 87.5% for every customer?”

Most Larkfield customers write in English, but about 15% of tickets are in other languages. Mei splits the test set by language. A part of the data chosen by one property, such as language or team, is a slice.

Slice	Tickets	Urgent	Urgent and flagged	Recall
English	850	64	62	62 ÷ 64 = 96.9%
Other languages	150	16	8	8 ÷ 16 = 50.0%
All tickets	1,000	80	70	70 ÷ 80 = 87.5%

Recall of urgent tickets at threshold 0.44, for all tickets and for each language slice. The overall recall hides a slice where half of the urgent tickets are missed.

Text description of the chart

A column chart of recall at threshold 0.44. All tickets: 87.5%. English: 96.9% (62 of 64 urgent tickets). Other languages: 50.0% (8 of 16 urgent tickets). The overall column is close to the English column, because English tickets are 85% of the test set.

The overall recall, 87.5%, is a mix of the two slices. English tickets are most of the test set, so they decide most of the overall number. A customer who writes in another language has only a 50% chance that their urgent ticket reaches a senior agent.

Is the gap real or chance? The slice has only 16 urgent tickets, so its recall is uncertain: one ticket more or less changes it by about 6 points. But 50% against 97% is far too big a gap to come from a few tickets alone. Mei finds the likely cause: only 1 in 20 training tickets was in another language, so the model learned those tickets badly.

Why one average is not enough: a model is used by groups, not by an average customer. A model can improve its overall recall and become worse for a small group at the same time. Larkfield must report recall for each important slice, and agree a minimum for each, before release.

Guided practice: slices by team

Grace wants the same check for her five teams. Here are the urgent tickets per team at threshold 0.44.

Team	Urgent tickets	Urgent and flagged
Payment	30	29
Delivery	22	20
Returns	12	10
Warranty	8	6
Account	8	5

1. In c03-notes.md, add a heading: ### Slices.
2. Check that the urgent tickets add up to 80 and the flagged ones add up to 70.
3. Calculate the recall for each team.
4. Mark the team with the lowest recall.
5. Calculate how much one more caught ticket would change the recall of that team.
6. Write one sentence for Grace: what you see, and how sure you are.

Expected results: Payment 96.7%, Delivery 90.9%, Returns 83.3%, Warranty 75.0%, Account 62.5%. One ticket changes the Account recall by 12.5 points.

Show an example answer

• Totals: 30 + 22 + 12 + 8 + 8 = 80. 29 + 20 + 10 + 6 + 5 = 70.
• Recall: Payment 29 ÷ 30 = 96.7%. Delivery 20 ÷ 22 = 90.9%. Returns 10 ÷ 12 = 83.3%. Warranty 6 ÷ 8 = 75.0%. Account 5 ÷ 8 = 62.5%.
• Lowest: Account. With one more caught ticket, it would be 6 ÷ 8 = 75.0%, so one ticket moves it by 12.5 points.

For Grace: “The model misses 3 of 8 urgent account tickets, more than any other team. But 8 tickets are too few to be sure. We need more urgent account tickets in the test set before we decide that the model is weak for your account team.”

The difference from the language slice is the size of the gap compared with the sample. The account gap could come from one or two tickets. The language gap could not.

Independent variation: precision by language

Recall is not the only number that can differ between slices. At threshold 0.44, the model flags 170 tickets. 145 are English tickets, and 62 of them are urgent. 25 are in other languages, and 8 of them are urgent.

Calculate the precision for each slice and for all tickets. Then say what a senior agent sees when they open a flagged ticket in another language.

Show an example answer

• English: 62 ÷ 145 = 42.8%.
• Other languages: 8 ÷ 25 = 32.0%.
• All tickets: 70 ÷ 170 = 41.2%.

For tickets in other languages, the model flags fewer urgent tickets (low recall) and its flags are wrong more often (lower precision). A senior agent who opens a flagged ticket in another language finds a false alarm about 2 times in 3. The model is weaker on this slice in both ways, which points to the training data, not to the threshold.

Common mistake

One average hides a weak group

Symptom: the model’s recall on the test set is 87.5%, and the release report shows only that number. After release, customers who write in other languages complain that urgent problems wait for days. The overall recall on new tickets still looks fine.

Cause: the test set and the report used one number for all tickets. Tickets in other languages are 15% of the data, so their 50% recall changed the overall number only a little.

Fix: before release, list the slices that matter: language, team, new or returning customer. Report recall and precision for each, with the count of urgent tickets. Agree a minimum recall per slice with Grace and Omar. If a slice is too small to measure, collect more labelled tickets for it. Do not hide a weak slice under a good average, and do not let one lucky number decide the release.

Check your understanding

For tickets with a score from 0.4 to 0.5, the model's mean score is 0.44, and 6.9% of them are urgent. Is the model overconfident or underconfident here, and what does that mean for an agent who sees the score?

Show the answer

Overconfident: the score (0.44) is much higher than the observed share (6.9%). An agent who reads the score as a chance expects about 44 urgent tickets out of 100 such tickets, but only about 7 are urgent. If the score is shown to agents, it must be recalibrated first, or shown as a rank (“high”, “medium”) and not as a percent.

Check your understanding

Tomás retrains the model. Overall recall rises from 87.5% to 89%. What must you check before you say the new model is better for all customers?

Show the answer

Check the recall (and precision) of each important slice, for example each language and each team, with the number of urgent tickets in it. The overall number can rise while a small slice gets worse. Also check whether 1.5 points is more than chance on 80 urgent tickets, as you learned in Module 5: it is about 1 ticket.

Module practice

Worked example: recommend a threshold with a workload limit

Omar asks Priya for a threshold recommendation. The stated costs are: a review costs 2 cost points and a missed urgent ticket costs 30 cost points. Grace adds a limit: her senior agents can review at most 30 tickets a day.

Step 1: turn the limit into a count per 1,000 test tickets. 30 a day is 30 × 7 = 210 a week. Larkfield gets 1,200 tickets a week, so the limit is 210 ÷ 1.2 = 175 flags per 1,000 tickets.

Step 2: list the thresholds near the bottom of the cost curve. Priya reads the counts from the explorer in the last lesson.

Threshold	Flagged	Reviews per day	Missed	Total cost	Inside the limit?
0.40	228	39.1	7	2 × 228 + 30 × 7 = 666	no
0.42	198	33.9	7	2 × 198 + 30 × 7 = 606	no
0.44	170	29.1	10	2 × 170 + 30 × 10 = 640	yes
0.45	163	27.9	11	2 × 163 + 30 × 11 = 656	yes
0.46	154	26.4	11	2 × 154 + 30 × 11 = 638	yes
0.48	124	21.3	14	2 × 124 + 30 × 14 = 668	yes
0.50	112	19.2	15	2 × 112 + 30 × 15 = 674	yes

Step 3: choose inside the limit. The cheapest threshold, 0.42, needs about 34 reviews a day, so it is not possible. Inside the limit, 0.44 (640) and 0.46 (638) cost almost the same. A difference of 2 points per 1,000 tickets is much smaller than the uncertainty from 80 urgent tickets. Priya uses a second reason: Grace’s main concern is missed urgent tickets, and 0.44 misses one fewer.

Step 4: write the recommendation.

Use threshold 0.44. On the 1,000 test tickets, it flags 170 and catches 70 of 80 urgent tickets (recall 87.5%). It costs 640 points per 1,000 tickets. That is about 29 reviews a day, just inside the limit of 30. If reviews pass 30 a day for a full week, move to 0.46: it costs the same and needs about 26 reviews a day. Before release, the model must improve for tickets in other languages, where recall is only 50%.

Your task: the costs change

The payment provider now charges a fee for every reversed payment. Omar changes the cost of a missed urgent ticket to 60 cost points. A review still costs 2, and the limit is still 30 reviews a day.

1. In c03-notes.md, add a heading: ### Module practice: Model decisions.
2. Write the costs and the limit, and the limit as flags per 1,000 tickets.
3. Calculate the total cost for the thresholds in the table above, with 60 for a miss.
4. Find the cheapest threshold inside the limit.
5. Use the explorer in Thresholds and costs to find the cheapest threshold without the limit.
6. Write a recommendation of three to five sentences for Omar.

Show a model answer

Limit: 30 a day = 210 a week = 175 flags per 1,000 tickets.

Threshold	Flagged	Missed	Total cost (2 × flagged + 60 × missed)	Inside the limit?
0.40	228	7	456 + 420 = 876	no
0.42	198	7	396 + 420 = 816	no
0.44	170	10	340 + 600 = 940	yes
0.45	163	11	326 + 660 = 986	yes
0.46	154	11	308 + 660 = 968	yes
0.48	124	14	248 + 840 = 1,088	yes
0.50	112	15	224 + 900 = 1,124	yes

Inside the limit, 0.44 is the cheapest, at 940 points per 1,000 tickets. Without the limit, the explorer finds 0.37: 281 flags, 4 missed, 802 points, but about 48 reviews a day.

Recommendation: “Use threshold 0.44 now. It is the cheapest choice that the senior agents can handle: about 29 reviews a day, 10 of 80 urgent tickets missed. With a missed ticket now at 60 points, more review capacity pays. At 0.37, the cost falls from 940 to 802 points per 1,000 tickets. That saves about 166 points a week. That needs about 19 more reviews a day, almost 2 hours of senior-agent time. I recommend that we plan that time, then move to about 0.37.”

A good answer can differ in the details. It must use both costs, respect the limit, show the calculation, and say what changes if the limit changes.

Module check

Check your understanding

Retrieval: at one threshold, a model has TP = 40, FN = 10, FP = 60 and TN = 890. What are precision, recall and the false-positive rate?

Show the answer

Precision: 40 ÷ (40 + 60) = 40%. Recall: 40 ÷ (40 + 10) = 80%. False-positive rate: 60 ÷ (60 + 890) = 6.3%. Precision divides by all flags, recall by all real positives, and the false-positive rate by all real negatives.

Check your understanding

Explain or predict: in a busy month, 16% of tickets are urgent instead of 8%. The model and the costs stay the same. Which way does the cheapest threshold move, and what happens to the senior agents' workload?

Show the answer

The cheapest threshold moves down, from 0.42 to about 0.37. With more urgent tickets, each threshold misses more of them. Misses become a bigger part of the cost, so flagging more becomes worth it. The workload grows for two reasons: there are more urgent tickets with high scores, and a lower threshold flags more tickets of both kinds. Grace may need more reviewers in that month, or Larkfield must accept a higher threshold and more missed urgent tickets.

Check your understanding

Completion check: explain your threshold recommendation for Larkfield in a few sentences. Then list the evidence that you still need before the model is used for real tickets.

Show the answer

The recommendation (with the original costs): threshold 0.44. It is inside the limit of 30 reviews a day (about 29). It costs 640 points per 1,000 tickets, almost the same as the cheapest threshold inside the limit. It misses 10 of 80 urgent tickets. Report it as a choice from a range (0.44 to 0.46), not as an exact best value.

More evidence needed before release:

1. Recent data. The test set is from past tickets. Check the counts on tickets from the last few weeks, because the base rate and the kind of tickets change.
2. More urgent tickets. The test set has only 80. The cost curve and every slice recall are uncertain. More labelled urgent tickets make the range of good thresholds smaller.
3. Slices. Recall for tickets in other languages is 50%. Agree a minimum recall per slice, and improve the model for that slice before release.
4. Calibration. If agents see the score, recalibrate it or show it as a rank, because a score of 0.44 means about 7% urgent.
5. The costs. The 2 and 30 cost points are estimates. Ask Omar and Grace how sure they are. Test the recommendation with other values: if a miss costs 60, the cheapest threshold moves to 0.37.
6. A trial without action. Before the model changes any ticket’s queue, run it on new tickets for two weeks. Count the flags, the reviews per day and the missed urgent tickets.

Recap

In this module, you turned model scores into decisions:

• Counts first. A threshold gives four counts. Precision, recall and the false-positive rate each divide by a different total. With a rare class, accuracy misleads: flagging nothing is 92% accurate.
• Costs decide the threshold. The total cost puts reviews and missed urgent tickets on one scale. The cheapest threshold moves down when a miss costs more or when urgent tickets become more common. A workload limit can rule out the cheapest threshold.
• Report a range. With 80 urgent tickets, the bottom of the cost curve is wide. Choose inside the range with a second reason.
• Check the score and the groups. A calibration diagram shows if a score can be read as a chance. Slices show if one average hides a weak group.

Next, in the final assignment, you bring the whole course together in an illustrated reasoning workbook for a small prediction problem.

Next: Final assignment