Skip to content
Français

Is the confidence true?

  • Lesson 32
  • 35 min
  • Module 8: Confidence and people in the loop

Where we are

Our classifiers give a confidence with every label, and the cascade used it to decide. But is a confidence of 0.9 right nine times in ten? In this lesson, we check calibration with reliability charts, for the local models and for the LLM's written confidence.

Plays from YouTube. Nothing loads from YouTube until you press play. Privacy
Lesson 32 of 37 · Text Classification and Routing: Send Each Message to the Right PlaceIs the confidence true?

What you will be able to do

Check whether a router’s confidence means what it says: a confidence of 0.9 should be right about 9 times in 10. Measure it with a reliability chart and one number, the expected calibration error. Do it for a trained model and for a confidence that an LLM writes in its answer.

NotebookDo this lesson in a notebookColab · Kaggle · GitHub

The notebook computes the tiny example, the reliability charts of four methods on the test messages, and the LLM’s written confidence, with checks. It installs pandas, scikit-learn and matplotlib and downloads the data, so it needs no local setup. About 20 minutes.

Colab and Kaggle open your own copy. They need a free Google or Kaggle account. Stuck? See the solution notebook.

Before you startWhat to know and to have readyAbout 35 minutes

What you need to know

What you need on your computer

  • The project route: your larkroute project at the end of Module 6, with its virtual environment, and the labs repository in the folder beside it. If your project is broken, copy the snapshot end-of-m06 as in Meet the inbox.
  • The notebook route: a browser and a free Google account for Colab, or a free Kaggle account.

Tested on macOS (Apple silicon) with Python 3.12.13. The Windows commands were not tested.

Back at the post office. A new sorter puts a small note on each letter: how sure she is that it is in the right tray. “9 in 10” on most letters, “6 in 10” on some. Her manager likes the notes. He plans to check by hand only the letters with a low number.

At the end of the week, he takes all the letters marked “9 in 10” and counts. Only 6 in 10 were in the right tray. The sorter was not lying. She just felt more sure than she was right. Her notes still put the letters in a useful order, from sure to unsure. But the number itself was not true.

A router’s confidence is the same kind of note. Before Grace’s team trusts it, someone must count.

The problem: a confident number that is not true

Section titled “The problem: a confident number that is not true”

In Module 5, the hosted LLM routed each test message and wrote a confidence in its answer, for example "confidence": 0.99. With route names only (zero-shot), it wrote 0.99 for 693 of the 2,000 test messages. It was right on only 86.6% of them.

Imagine a cascade that trusts that number. It lets the LLM decide every message where it wrote 0.99 or more. About 1 in 7 of those decisions is wrong, and nobody checks them. The number looked like a probability, and it was not one.

So we need a way to check any confidence, from any model, before we use it to decide.

A confidence is calibrated when it matches how often the router is right. Of all the messages with a confidence near 0.9, about 90% should be right. Of those near 0.6, about 60%.

To check it, you group the messages by their confidence and count:

  1. Put the messages into 10 bins by confidence: 0.0 to 0.1, 0.1 to 0.2, and so on.
  2. In each bin, compute the mean confidence and the share right (the accuracy in that bin).
  3. Compare the two. If they are close in every bin, the confidence is true.

A reliability chart draws these pairs: mean confidence across, share right up. A true confidence lies on the diagonal. A point under the diagonal is too sure: the router says 0.9 and is right less often. A point above it is too shy: it is right more often than it says.

The expected calibration error (ECE) puts the chart into one number. It is the gap between confidence and share right in each bin, weighted by the share of messages in that bin. An ECE of 0 means the confidence is true everywhere. An ECE of 0.2 means it is off by 0.2 on average.

Here is a practice problem we made on purpose: 10 messages and two confidence values.

A worked table for 10 messages. Bin 1: five messages with confidence 0.9, three right and two wrong; right 3 of 5 = 0.6; gap 0.3; weight 5 ÷ 10 = 0.5; weight × gap = 0.15. Bin 2: five messages with confidence 0.6, three right and two wrong; right 0.6; gap 0.0; weight 0.5; weight × gap = 0.00. ECE = 0.15 + 0.00 = 0.15.
The expected calibration error of 10 messages, with every value. The first bin is too sure by 0.3; the second is true.
Text description of the diagram

A table with two rows, one per bin. Each row shows five message dots: three blue dots with a tick (right) and two dots with a cross (wrong). Bin 1: the model says 0.9, and 3 of 5 messages are right, which is 0.6. The gap is 0.9 minus 0.6, which is 0.3. The weight is 5 of 10 messages, which is 0.5. Weight times gap is 0.15. Bin 2: the model says 0.6, and 3 of 5 are right, which is 0.6. The gap is 0. Weight times gap is 0. Under the table: ECE equals 0.15 plus 0.00, which is 0.15. A note says that bin 1 is too sure by 0.3 and bin 2 is true.

Both bins have the same accuracy, 0.6. Only the first one is wrong about itself. The ECE, 0.15, comes entirely from it. Section 1 of the notebook computes the same example with the course’s helper and prints ECE 0.15.

Now check the router that Module 3 built, TF-IDF with logistic regression. The project gets a new module, larkroute/zones.py, with the function reliability(). It does the three steps above with 10 bins. A new command, route calibration, prints the result.

  1. Go to your project folder larkroute. Turn on its virtual environment, as in Meet the inbox.

  2. Copy this lesson’s three files from the snapshot end-of-m08. On macOS and Linux:

    Terminal window
    S=../nextia-learning-labs/C36/snapshots/end-of-m08
    cp $S/larkroute/zones.py larkroute/
    cp $S/larkroute/commands/calibration.py larkroute/commands/
    cp $S/tests/test_zones.py tests/

    On Windows (PowerShell):

    Terminal window
    $S = "..\nextia-learning-labs\C36\snapshots\end-of-m08"
    Copy-Item $S\larkroute\zones.py larkroute\
    Copy-Item $S\larkroute\commands\calibration.py larkroute\commands\
    Copy-Item $S\tests\test_zones.py tests\
  3. Run the tests. You see 27 passed: the 24 tests of Module 6 and 3 new ones.

    Terminal window
    python -m pytest -q
  4. Run the new command. It trains the model on the 6,000 training messages and checks it on the 2,000 validation messages. It takes a few seconds.

    Terminal window
    route calibration
TF-IDF + logistic regression on the validation messages: expected calibration error 0.046
bin messages mean_confidence accuracy
0.2-0.3 4 0.268 0.500
0.3-0.4 21 0.355 0.381
0.4-0.5 38 0.461 0.684
0.5-0.6 61 0.554 0.623
0.6-0.7 85 0.649 0.753
0.7-0.8 89 0.754 0.921
0.8-0.9 169 0.859 0.970
0.9-1.0 1533 0.976 0.999

The project finds its commands by file, so you did not change cli.py. One of the new tests checks reliability() on a tiny example with two bins, like the one above.

Read it from the bottom, where most messages are:

  • 0.9 to 1.0: 1,533 of the 2,000 messages. The mean confidence is 0.976, and 99.9% are right. Close to true, a little shy.
  • 0.7 to 0.8: the model says 0.754 and is right 92.1% of the time. It is too shy by 0.17.
  • 0.5 to 0.6: it says 0.554 and is right 62.3% of the time. Too shy again.
  • 0.2 to 0.3: only 4 messages. “50% right” here means 2 of 4. Do not read much into it.

So on this inbox, TF-IDF with logistic regression is a little too shy in the middle. Its ECE is 0.046 on validation. On the test messages (route calibration --split test), it is 0.050.

Why does it matter? A shy model sends more messages to review than it needs to. You will see in the next lesson that the threshold is chosen by cost, which takes care of part of this. But if Grace’s dashboard shows “55% sure”, her agents will believe the model is guessing, when it is right 6 times in 10.

The chart below draws the test reliability tables of three methods that run on your computer. The dashed diagonal is a true confidence.

  • TF-IDF + LR (ECE 0.050)
  • Encoder (ECE 0.031)
  • NLI (ECE 0.309)
  • True (dashed)
Reliability charts on the 2,000 test messages (measured). Under the dashed line, the method is too sure; above it, too shy.
Text description of the chart

A line chart. The horizontal axis is the mean confidence in each bin, from 0 to 1. The vertical axis is the share right in that bin, from 0% to 100%. A dashed diagonal shows a true confidence. TF-IDF with logistic regression has 9 points, from 0.19 (0% right, 1 message) to 0.98 (99.9% right, 1,512 messages). From 0.36 to 0.86, its points lie above the diagonal: for example 0.55 is 71% right and 0.75 is 90% right. It is too shy. The fine-tuned encoder has 6 points. From 0.45 to 0.75 they lie under the diagonal: for example, 0.57 is only 27% right (15 messages). Then 0.97 is 99.5% right, with 1,932 messages. The NLI zero-shot model is far under the diagonal everywhere. 0.45 is 21% right, 0.75 is 34% right, and 0.95 is only 58% right.

Three different stories:

  • TF-IDF + logistic regression (ECE 0.050): too shy in the middle, as on validation.
  • The fine-tuned encoder (ECE 0.031): too sure in the middle bins. But those bins hold only 68 of the 2,000 messages. Almost all messages (1,932) sit at 0.97, and 99.5% of those are right. So its ECE is small.
  • NLI zero-shot (ECE 0.309): too sure everywhere. It says 0.95 and is right 58% of the time. This model saw no Larkfield label. Its score says how well a message fits a route name, not how often it is right.

The LLM’s confidence is different from the three above. It is not a probability that the model computed. It is a few characters that the model wrote in its answer, like any other text. The project can check it from the recorded answers, so you need no key.

Terminal window
route calibration --method "hosted LLM chat-small, zero-shot (recorded)" --split test
hosted LLM chat-small, zero-shot (recorded) on the test messages: expected calibration error 0.190
bin messages mean_confidence accuracy
0.3-0.4 1 0.390 0.000
0.4-0.5 1 0.480 0.000
0.5-0.6 5 0.568 0.200
0.6-0.7 5 0.660 0.400
0.7-0.8 40 0.761 0.300
0.8-0.9 166 0.859 0.608
0.9-1.0 1782 0.971 0.793

Use --split test here. The zero-shot answers were recorded for the test messages only, so the command stops on the validation messages.

Terminal window
route calibration --method "hosted LLM chat-small, few-shot (recorded)" --split test
hosted LLM chat-small, few-shot (recorded) on the test messages: expected calibration error 0.008
bin messages mean_confidence accuracy
0.2-0.3 2 0.250 0.500
0.3-0.4 7 0.357 0.286
0.4-0.5 14 0.442 0.429
0.5-0.6 12 0.558 0.417
0.6-0.7 5 0.658 0.800
0.7-0.8 6 0.763 0.500
0.8-0.9 36 0.858 0.833
0.9-1.0 1918 0.990 0.984
  • Zero-shot (ECE 0.190)
  • Few-shot (ECE 0.008)
  • True (dashed)
The same hosted model, two prompts: zero-shot (route names only) and few-shot (route descriptions and 8 examples). Reliability of the confidence it wrote, on the 2,000 test messages (recorded answers, measured here).
Text description of the chart

A line chart with the mean written confidence across and the share right up, both from 0 to 1, and a dashed diagonal. The zero-shot line lies far under the diagonal. 0.76 is 30% right (40 messages), 0.86 is 61% right (166 messages) and 0.97 is 79% right (1,782 messages). The few-shot line stays near the diagonal at the top, where almost all its answers are. 0.86 is 83% right (36 messages) and 0.99 is 98.4% right (1,918 messages). Its lower bins hold 2 to 14 messages each and jump up and down.

The same model wrote both confidences:

  • Zero-shot (route names only): ECE 0.190. It wrote 0.99 for 693 messages and was right on 86.6% of them. It is too sure.
  • Few-shot (the route descriptions and the 8 nearest training messages): ECE 0.008. 95.9% of its answers say 0.9 or more, and they are right 98.4% of the time. Here, its number is close to true.

This is a surprise, and it is easy to learn the wrong lesson from it. The few-shot number is true here because we checked it, on this inbox, with this prompt. Change the prompt, the model version or the messages, and it can move, as the zero-shot prompt shows. Also, the written confidence has only a few values (0.99, 1.0, 0.98 …). So it cannot sort the sure messages finely.

A written confidence is not a probability until you check it. Check it on validation messages, for your prompt, each time you change the prompt or the model.

A horizontal bar chart of the expected calibration error on the 2,000 test messages. LLM few-shot (written) 0.008; embeddings + LR 0.011; fine-tuned encoder 0.031; TF-IDF + LR 0.050; GLiClass zero-shot 0.089; LLM zero-shot (written) 0.190; NLI zero-shot 0.309. Lower is better.
Expected calibration error of seven methods on the same 2,000 test messages (measured; the LLM rows from recorded answers). Lower is better.
Text description of the diagram

A horizontal bar chart with seven bars, from the smallest error to the largest. LLM few-shot, written confidence: 0.008. Embeddings with logistic regression: 0.011. Fine-tuned encoder: 0.031. TF-IDF with logistic regression: 0.050. GLiClass zero-shot: 0.089. LLM zero-shot, written confidence: 0.190, in orange. NLI zero-shot: 0.309. The two zero-shot models without an LLM are grey. A note says: lower is better, and 0 means the confidence is true.

The trained models (blue) all have an ECE of 0.05 or less on this inbox. The methods that saw no Larkfield label are worse calibrated: GLiClass, NLI and the zero-shot LLM. The few-shot LLM sits at the top only because its examples come from the Larkfield labels.

Two more numbers from the course’s runs, without a chart. The small local LLM’s route scores, from Ask for a decision, not for text, had an ECE of 0.390. Temperature scaling brought it to 0.112. That method divides the scores by one number, fitted on the validation messages, so that the confidence becomes true on average. It did not make the model route better, here: it stays at 0.194 accuracy.

Sorting and being true are two different things

Section titled “Sorting and being true are two different things”

A confidence can do two jobs:

  • Sort the messages, from sure to unsure. A threshold needs only this. The cascade and the next lesson choose a threshold by its cost on the validation messages, so a shy model still gets a good threshold.
  • Be true, so that “0.9” means 9 in 10. You need this when people read the number. You also need it to compare two models with one rule, such as “decide at 0.8”, or to forecast the review load.

The sorter at the post office sorted well, but her numbers were not true. Both can happen in a router too.

Predict first: will the fine-tuned encoder’s ECE on validation be closer to 0.03 or to 0.3? Then run route calibration for it on the validation messages. The method’s name is fine-tuned MiniLM-L12 multilingual encoder. In the notebook, section 3 asks the same question for the LLM’s few-shot confidence on test.

Show the answer
Terminal window
route calibration --method "fine-tuned MiniLM-L12 multilingual encoder"
fine-tuned MiniLM-L12 multilingual encoder on the validation messages: expected calibration error 0.028
bin messages mean_confidence accuracy
0.2-0.3 1 0.239 1.000
0.3-0.4 1 0.366 0.000
0.4-0.5 4 0.471 0.500
0.5-0.6 13 0.554 0.538
0.6-0.7 9 0.653 0.333
0.7-0.8 11 0.761 0.455
0.8-0.9 24 0.863 0.708
0.9-1.0 1937 0.968 0.991

It is close to 0.03, not 0.3: 0.028 on validation, and 0.031 on test. The encoder is too sure in the middle bins, but only 63 of the 2,000 messages are there. Almost all of them (1,937) sit above 0.9, and 99.1% of those are right. Look at the two bins with one message each: “100% right” and “0% right” are each one message.

The notebook’s task gives 0.0082 for the few-shot LLM on test. Its check cell prints Check passed: ece_llm is 0.0082.

Common mistake

A reliability chart from too few messages

Symptom: Priya checks the confidence on the needs_person messages only, because their mistakes are expensive. Section 4 of the notebook does it for TF-IDF on the 99 needs_person test messages. The ECE is 0.0986, twice the ECE of all messages. Some bins show “100% right”.

Cause: 99 messages in 10 bins leave some bins with only 2 to 5 messages. “100% right” from 3 messages can happen by chance. One message more or less changes the share by 20 to 50 points.

Fix: always show the number of messages in each bin, as route calibration does. Use fewer bins for a small group, for example 5. Do not compare two ECEs that come from a few dozen messages.

Consequence in AI work: a team can “fix” a calibration problem that was only noise, or miss a real one. For the rare and expensive routes, the next lesson uses the cost on the validation messages instead. Module 7 gives the account takeovers their own threshold.

Check your understanding

The hosted LLM wrote 0.99 for 693 test messages. Why can you not read this as '99% right', and what does the course measure for it?

Show the answer

A written confidence is text that the model produced, not a probability that anyone checked. With route names only (zero-shot), the 693 answers at 0.99 were right only 86.6% of the time. The ECE was 0.190. With route descriptions and 8 examples (few-shot), the same model’s number was close to true (ECE 0.008). It was true only because we checked it, for that prompt, on this inbox.

Check your understanding

In route calibration, TF-IDF + logistic regression has mean confidence 0.754 and accuracy 0.921 in the bin 0.7 to 0.8. Is it too sure or too shy there, and what can this cost?

Show the answer

Too shy: it is right 92% of the time and says only 75%. The point lies above the diagonal. A shy model can send more messages to review than it needs to. People who read the number also underrate it. The threshold of the next lesson is chosen by cost on validation, so it partly takes care of this. But a number shown to agents should be true.

Check your understanding

Explain the expected calibration error with the 10-message example: 5 messages at 0.9 (3 right) and 5 at 0.6 (3 right).

Show the answer

In each bin, take the gap between the confidence and the share right, and weight it by the share of messages in the bin. Bin 1: |0.9 − 0.6| = 0.3, weight 0.5, so 0.15. Bin 2: |0.6 − 0.6| = 0, weight 0.5, so 0. The ECE is 0.15 + 0 = 0.15. Both bins are 60% right, but only the first one is wrong about itself.

Recap

  • A confidence is calibrated when a confidence of 0.9 is right about 9 times in 10. A reliability chart shows it: points under the diagonal are too sure, points above are too shy.
  • The expected calibration error (ECE) is the gap in each bin, weighted by the share of messages. In the 10-message example, it is 0.15.
  • route calibration prints the reliability table of any method. TF-IDF + logistic regression: ECE 0.046 on validation and 0.050 on test, a little too shy in the middle, on this inbox.
  • On the test messages: embeddings + LR 0.011, fine-tuned encoder 0.031, GLiClass 0.089, NLI 0.309.
  • A written confidence is not a probability until you check it. The same LLM: zero-shot 0.190 (0.99 written 693 times, right 86.6%), few-shot 0.008.
  • Show the messages per bin. A bin of 3 messages tells you little.

You now know which confidence numbers you can trust, and how much. Next, you use them to decide. You split the messages into three zones and count the work that each zone gives Grace’s team.

Next: Decide, review or send to a person