Is the confidence true?
- Lesson 32
- 35 min
- Module 8: Confidence and people in the loop
Where we are
Our classifiers give a confidence with every label, and the cascade used it to decide. But is a confidence of 0.9 right nine times in ten? In this lesson, we check calibration with reliability charts, for the local models and for the LLM's written confidence.
Lesson 32 of 37 · Text Classification and Routing: Send Each Message to the Right PlaceIs the confidence true?
1. The routing problem and its cost
2. Labels you can trust
3. Baselines: rules and classical models
4. Embeddings and small fine-tuned models
5. LLMs as classifiers
6. Decision models, cascades and distillation
7. Harder label shapes
8. Confidence and people in the loop
9. Run it in production
What you will be able to do
Check whether a router’s confidence means what it says: a confidence of 0.9 should be right about 9 times in 10. Measure it with a reliability chart and one number, the expected calibration error. Do it for a trained model and for a confidence that an LLM writes in its answer.
NotebookDo this lesson in a notebook
The notebook computes the tiny example, the reliability charts of four methods on the test messages, and the LLM’s written confidence, with checks. It installs pandas, scikit-learn and matplotlib and downloads the data, so it needs no local setup. About 20 minutes.
Colab and Kaggle open your own copy. They need a free Google or Kaggle account. Stuck? See the solution notebook.
Before you startWhat to know and to have ready
What you need to know
- The confidence of a router: its top probability for a message, from The cost of each mistake.
- The cascade, which trusts a confidence to decide, from Cascades.
- The LLM’s answer with a written confidence, from Zero-shot classification with an LLM.
- The machine learning course explains calibration with two labels, in Thresholds and calibration. This lesson explains again the parts it needs.
What you need on your computer
- The project route: your
larkrouteproject at the end of Module 6, with its virtual environment, and the labs repository in the folder beside it. If your project is broken, copy the snapshotend-of-m06as in Meet the inbox. - The notebook route: a browser and a free Google account for Colab, or a free Kaggle account.
Tested on macOS (Apple silicon) with Python 3.12.13. The Windows commands were not tested.
The sorter who says how sure she is
Section titled “The sorter who says how sure she is”Back at the post office. A new sorter puts a small note on each letter: how sure she is that it is in the right tray. “9 in 10” on most letters, “6 in 10” on some. Her manager likes the notes. He plans to check by hand only the letters with a low number.
At the end of the week, he takes all the letters marked “9 in 10” and counts. Only 6 in 10 were in the right tray. The sorter was not lying. She just felt more sure than she was right. Her notes still put the letters in a useful order, from sure to unsure. But the number itself was not true.
A router’s confidence is the same kind of note. Before Grace’s team trusts it, someone must count.
The problem: a confident number that is not true
Section titled “The problem: a confident number that is not true”In Module 5, the hosted LLM routed each test message and wrote a confidence in its answer, for example "confidence": 0.99. With route names only (zero-shot), it wrote 0.99 for 693 of the 2,000 test messages. It was right on only 86.6% of them.
Imagine a cascade that trusts that number. It lets the LLM decide every message where it wrote 0.99 or more. About 1 in 7 of those decisions is wrong, and nobody checks them. The number looked like a probability, and it was not one.
So we need a way to check any confidence, from any model, before we use it to decide.
The idea: calibration
Section titled “The idea: calibration”A confidence is calibrated when it matches how often the router is right. Of all the messages with a confidence near 0.9, about 90% should be right. Of those near 0.6, about 60%.
To check it, you group the messages by their confidence and count:
- Put the messages into 10 bins by confidence: 0.0 to 0.1, 0.1 to 0.2, and so on.
- In each bin, compute the mean confidence and the share right (the accuracy in that bin).
- Compare the two. If they are close in every bin, the confidence is true.
A reliability chart draws these pairs: mean confidence across, share right up. A true confidence lies on the diagonal. A point under the diagonal is too sure: the router says 0.9 and is right less often. A point above it is too shy: it is right more often than it says.
The expected calibration error (ECE) puts the chart into one number. It is the gap between confidence and share right in each bin, weighted by the share of messages in that bin. An ECE of 0 means the confidence is true everywhere. An ECE of 0.2 means it is off by 0.2 on average.
A tiny worked example
Section titled “A tiny worked example”Here is a practice problem we made on purpose: 10 messages and two confidence values.
Text description of the diagram
A table with two rows, one per bin. Each row shows five message dots: three blue dots with a tick (right) and two dots with a cross (wrong). Bin 1: the model says 0.9, and 3 of 5 messages are right, which is 0.6. The gap is 0.9 minus 0.6, which is 0.3. The weight is 5 of 10 messages, which is 0.5. Weight times gap is 0.15. Bin 2: the model says 0.6, and 3 of 5 are right, which is 0.6. The gap is 0. Weight times gap is 0. Under the table: ECE equals 0.15 plus 0.00, which is 0.15. A note says that bin 1 is too sure by 0.3 and bin 2 is true.
Both bins have the same accuracy, 0.6. Only the first one is wrong about itself. The ECE, 0.15, comes entirely from it. Section 1 of the notebook computes the same example with the course’s helper and prints ECE 0.15.
Check the router’s confidence
Section titled “Check the router’s confidence”Now check the router that Module 3 built, TF-IDF with logistic regression. The project gets a new module, larkroute/zones.py, with the function reliability(). It does the three steps above with 10 bins. A new command, route calibration, prints the result.
-
Go to your project folder
larkroute. Turn on its virtual environment, as in Meet the inbox. -
Copy this lesson’s three files from the snapshot
end-of-m08. On macOS and Linux:Terminal window S=../nextia-learning-labs/C36/snapshots/end-of-m08cp $S/larkroute/zones.py larkroute/cp $S/larkroute/commands/calibration.py larkroute/commands/cp $S/tests/test_zones.py tests/On Windows (PowerShell):
Terminal window $S = "..\nextia-learning-labs\C36\snapshots\end-of-m08"Copy-Item $S\larkroute\zones.py larkroute\Copy-Item $S\larkroute\commands\calibration.py larkroute\commands\Copy-Item $S\tests\test_zones.py tests\ -
Run the tests. You see
27 passed: the 24 tests of Module 6 and 3 new ones.Terminal window python -m pytest -q -
Run the new command. It trains the model on the 6,000 training messages and checks it on the 2,000 validation messages. It takes a few seconds.
Terminal window route calibration
TF-IDF + logistic regression on the validation messages: expected calibration error 0.046 bin messages mean_confidence accuracy0.2-0.3 4 0.268 0.5000.3-0.4 21 0.355 0.3810.4-0.5 38 0.461 0.6840.5-0.6 61 0.554 0.6230.6-0.7 85 0.649 0.7530.7-0.8 89 0.754 0.9210.8-0.9 169 0.859 0.9700.9-1.0 1533 0.976 0.999The project finds its commands by file, so you did not change cli.py. One of the new tests checks reliability() on a tiny example with two bins, like the one above.
Read the table
Section titled “Read the table”Read it from the bottom, where most messages are:
- 0.9 to 1.0: 1,533 of the 2,000 messages. The mean confidence is 0.976, and 99.9% are right. Close to true, a little shy.
- 0.7 to 0.8: the model says 0.754 and is right 92.1% of the time. It is too shy by 0.17.
- 0.5 to 0.6: it says 0.554 and is right 62.3% of the time. Too shy again.
- 0.2 to 0.3: only 4 messages. “50% right” here means 2 of 4. Do not read much into it.
So on this inbox, TF-IDF with logistic regression is a little too shy in the middle. Its ECE is 0.046 on validation. On the test messages (route calibration --split test), it is 0.050.
Why does it matter? A shy model sends more messages to review than it needs to. You will see in the next lesson that the threshold is chosen by cost, which takes care of part of this. But if Grace’s dashboard shows “55% sure”, her agents will believe the model is guessing, when it is right 6 times in 10.
Reliability charts of three local models
Section titled “Reliability charts of three local models”The chart below draws the test reliability tables of three methods that run on your computer. The dashed diagonal is a true confidence.
- TF-IDF + LR (ECE 0.050)
- Encoder (ECE 0.031)
- NLI (ECE 0.309)
- True (dashed)
Text description of the chart
A line chart. The horizontal axis is the mean confidence in each bin, from 0 to 1. The vertical axis is the share right in that bin, from 0% to 100%. A dashed diagonal shows a true confidence. TF-IDF with logistic regression has 9 points, from 0.19 (0% right, 1 message) to 0.98 (99.9% right, 1,512 messages). From 0.36 to 0.86, its points lie above the diagonal: for example 0.55 is 71% right and 0.75 is 90% right. It is too shy. The fine-tuned encoder has 6 points. From 0.45 to 0.75 they lie under the diagonal: for example, 0.57 is only 27% right (15 messages). Then 0.97 is 99.5% right, with 1,932 messages. The NLI zero-shot model is far under the diagonal everywhere. 0.45 is 21% right, 0.75 is 34% right, and 0.95 is only 58% right.
Three different stories:
- TF-IDF + logistic regression (ECE 0.050): too shy in the middle, as on validation.
- The fine-tuned encoder (ECE 0.031): too sure in the middle bins. But those bins hold only 68 of the 2,000 messages. Almost all messages (1,932) sit at 0.97, and 99.5% of those are right. So its ECE is small.
- NLI zero-shot (ECE 0.309): too sure everywhere. It says 0.95 and is right 58% of the time. This model saw no Larkfield label. Its score says how well a message fits a route name, not how often it is right.
The LLM’s written confidence
Section titled “The LLM’s written confidence”The LLM’s confidence is different from the three above. It is not a probability that the model computed. It is a few characters that the model wrote in its answer, like any other text. The project can check it from the recorded answers, so you need no key.
route calibration --method "hosted LLM chat-small, zero-shot (recorded)" --split testhosted LLM chat-small, zero-shot (recorded) on the test messages: expected calibration error 0.190 bin messages mean_confidence accuracy0.3-0.4 1 0.390 0.0000.4-0.5 1 0.480 0.0000.5-0.6 5 0.568 0.2000.6-0.7 5 0.660 0.4000.7-0.8 40 0.761 0.3000.8-0.9 166 0.859 0.6080.9-1.0 1782 0.971 0.793Use --split test here. The zero-shot answers were recorded for the test messages only, so the command stops on the validation messages.
route calibration --method "hosted LLM chat-small, few-shot (recorded)" --split testhosted LLM chat-small, few-shot (recorded) on the test messages: expected calibration error 0.008 bin messages mean_confidence accuracy0.2-0.3 2 0.250 0.5000.3-0.4 7 0.357 0.2860.4-0.5 14 0.442 0.4290.5-0.6 12 0.558 0.4170.6-0.7 5 0.658 0.8000.7-0.8 6 0.763 0.5000.8-0.9 36 0.858 0.8330.9-1.0 1918 0.990 0.984- Zero-shot (ECE 0.190)
- Few-shot (ECE 0.008)
- True (dashed)
Text description of the chart
A line chart with the mean written confidence across and the share right up, both from 0 to 1, and a dashed diagonal. The zero-shot line lies far under the diagonal. 0.76 is 30% right (40 messages), 0.86 is 61% right (166 messages) and 0.97 is 79% right (1,782 messages). The few-shot line stays near the diagonal at the top, where almost all its answers are. 0.86 is 83% right (36 messages) and 0.99 is 98.4% right (1,918 messages). Its lower bins hold 2 to 14 messages each and jump up and down.
The same model wrote both confidences:
- Zero-shot (route names only): ECE 0.190. It wrote 0.99 for 693 messages and was right on 86.6% of them. It is too sure.
- Few-shot (the route descriptions and the 8 nearest training messages): ECE 0.008. 95.9% of its answers say 0.9 or more, and they are right 98.4% of the time. Here, its number is close to true.
This is a surprise, and it is easy to learn the wrong lesson from it. The few-shot number is true here because we checked it, on this inbox, with this prompt. Change the prompt, the model version or the messages, and it can move, as the zero-shot prompt shows. Also, the written confidence has only a few values (0.99, 1.0, 0.98 …). So it cannot sort the sure messages finely.
A written confidence is not a probability until you check it. Check it on validation messages, for your prompt, each time you change the prompt or the model.
All the methods in one chart
Section titled “All the methods in one chart”Text description of the diagram
A horizontal bar chart with seven bars, from the smallest error to the largest. LLM few-shot, written confidence: 0.008. Embeddings with logistic regression: 0.011. Fine-tuned encoder: 0.031. TF-IDF with logistic regression: 0.050. GLiClass zero-shot: 0.089. LLM zero-shot, written confidence: 0.190, in orange. NLI zero-shot: 0.309. The two zero-shot models without an LLM are grey. A note says: lower is better, and 0 means the confidence is true.
The trained models (blue) all have an ECE of 0.05 or less on this inbox. The methods that saw no Larkfield label are worse calibrated: GLiClass, NLI and the zero-shot LLM. The few-shot LLM sits at the top only because its examples come from the Larkfield labels.
Two more numbers from the course’s runs, without a chart. The small local LLM’s route scores, from Ask for a decision, not for text, had an ECE of 0.390. Temperature scaling brought it to 0.112. That method divides the scores by one number, fitted on the validation messages, so that the confidence becomes true on average. It did not make the model route better, here: it stays at 0.194 accuracy.
Sorting and being true are two different things
Section titled “Sorting and being true are two different things”A confidence can do two jobs:
- Sort the messages, from sure to unsure. A threshold needs only this. The cascade and the next lesson choose a threshold by its cost on the validation messages, so a shy model still gets a good threshold.
- Be true, so that “0.9” means 9 in 10. You need this when people read the number. You also need it to compare two models with one rule, such as “decide at 0.8”, or to forecast the review load.
The sorter at the post office sorted well, but her numbers were not true. Both can happen in a router too.
Your turn: the fine-tuned encoder
Section titled “Your turn: the fine-tuned encoder”Predict first: will the fine-tuned encoder’s ECE on validation be closer to 0.03 or to 0.3? Then run route calibration for it on the validation messages. The method’s name is fine-tuned MiniLM-L12 multilingual encoder. In the notebook, section 3 asks the same question for the LLM’s few-shot confidence on test.
Show the answer
route calibration --method "fine-tuned MiniLM-L12 multilingual encoder"fine-tuned MiniLM-L12 multilingual encoder on the validation messages: expected calibration error 0.028 bin messages mean_confidence accuracy0.2-0.3 1 0.239 1.0000.3-0.4 1 0.366 0.0000.4-0.5 4 0.471 0.5000.5-0.6 13 0.554 0.5380.6-0.7 9 0.653 0.3330.7-0.8 11 0.761 0.4550.8-0.9 24 0.863 0.7080.9-1.0 1937 0.968 0.991It is close to 0.03, not 0.3: 0.028 on validation, and 0.031 on test. The encoder is too sure in the middle bins, but only 63 of the 2,000 messages are there. Almost all of them (1,937) sit above 0.9, and 99.1% of those are right. Look at the two bins with one message each: “100% right” and “0% right” are each one message.
The notebook’s task gives 0.0082 for the few-shot LLM on test. Its check cell prints Check passed: ece_llm is 0.0082.
Common mistake
A reliability chart from too few messages
Symptom: Priya checks the confidence on the needs_person messages only, because their mistakes are expensive. Section 4 of the notebook does it for TF-IDF on the 99 needs_person test messages. The ECE is 0.0986, twice the ECE of all messages. Some bins show “100% right”.
Cause: 99 messages in 10 bins leave some bins with only 2 to 5 messages. “100% right” from 3 messages can happen by chance. One message more or less changes the share by 20 to 50 points.
Fix: always show the number of messages in each bin, as route calibration does. Use fewer bins for a small group, for example 5. Do not compare two ECEs that come from a few dozen messages.
Consequence in AI work: a team can “fix” a calibration problem that was only noise, or miss a real one. For the rare and expensive routes, the next lesson uses the cost on the validation messages instead. Module 7 gives the account takeovers their own threshold.
Knowledge check
Section titled “Knowledge check”Check your understanding
The hosted LLM wrote 0.99 for 693 test messages. Why can you not read this as '99% right', and what does the course measure for it?
Show the answer
A written confidence is text that the model produced, not a probability that anyone checked. With route names only (zero-shot), the 693 answers at 0.99 were right only 86.6% of the time. The ECE was 0.190. With route descriptions and 8 examples (few-shot), the same model’s number was close to true (ECE 0.008). It was true only because we checked it, for that prompt, on this inbox.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
In route calibration, TF-IDF + logistic regression has mean confidence 0.754 and accuracy 0.921 in the bin 0.7 to 0.8. Is it too sure or too shy there, and what can this cost?
Show the answer
Too shy: it is right 92% of the time and says only 75%. The point lies above the diagonal. A shy model can send more messages to review than it needs to. People who read the number also underrate it. The threshold of the next lesson is chosen by cost on validation, so it partly takes care of this. But a number shown to agents should be true.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
Explain the expected calibration error with the 10-message example: 5 messages at 0.9 (3 right) and 5 at 0.6 (3 right).
Show the answer
In each bin, take the gap between the confidence and the share right, and weight it by the share of messages in the bin. Bin 1: |0.9 − 0.6| = 0.3, weight 0.5, so 0.15. Bin 2: |0.6 − 0.6| = 0, weight 0.5, so 0. The ECE is 0.15 + 0 = 0.15. Both bins are 60% right, but only the first one is wrong about itself.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Recap
- A confidence is calibrated when a confidence of 0.9 is right about 9 times in 10. A reliability chart shows it: points under the diagonal are too sure, points above are too shy.
- The expected calibration error (ECE) is the gap in each bin, weighted by the share of messages. In the 10-message example, it is 0.15.
route calibrationprints the reliability table of any method. TF-IDF + logistic regression: ECE 0.046 on validation and 0.050 on test, a little too shy in the middle, on this inbox.- On the test messages: embeddings + LR 0.011, fine-tuned encoder 0.031, GLiClass 0.089, NLI 0.309.
- A written confidence is not a probability until you check it. The same LLM: zero-shot 0.190 (0.99 written 693 times, right 86.6%), few-shot 0.008.
- Show the messages per bin. A bin of 3 messages tells you little.
You now know which confidence numbers you can trust, and how much. Next, you use them to decide. You split the messages into three zones and count the work that each zone gives Grace’s team.