Skip to content
Français

Ask for a decision, not for text

  • Lesson 24
  • 40 min
  • Module 6: Decision models, cascades and distillation

Where we are

Our LLMs wrote their answers as text, and some wrote a confidence number too. A router needs a label and a real probability. In this lesson, we get both from a small local model: we score each label, we constrain the output, and we check if the probabilities are true.

Plays from YouTube. Nothing loads from YouTube until you press play. Privacy
Lesson 24 of 37 · Text Classification and Routing: Send Each Message to the Right PlaceAsk for a decision, not for text

What you will be able to do

Get a route and a probability from a language model, instead of a sentence that you must read and trust. Score each route, limit what the model may write, or read a trained head, and then check that the probabilities are true.

NotebookDo this lesson in a notebookColab · Kaggle · GitHub

The notebook loads a small local model, Qwen3-0.6B, on your computer’s CPU. It scores the seven route names for one message and for 100 test messages, and reads the reliability table of the course’s full run. Its setup cells install pandas, scikit-learn, Transformers and PyTorch, and the model download is about 1.5 GB, once. It needs about 4 GB of free memory and no key. About 40 minutes.

Colab and Kaggle open your own copy. They need a free Google or Kaggle account. Stuck? See the solution notebook.

Before you startWhat to know and to have readyAbout 40 minutes

What you need to know

What you need on your computer (only for the project step)

  • Your project folder larkroute at the stage end-of-m03, with .venv active: python -m pytest -q prints 18 passed.
  • The practice files nextia-learning-labs next to your project folder, as in Meet the inbox.

Tested on macOS with Python 3.12, pandas 3.1.1, scikit-learn 1.9.1, Transformers 5.22.2 and PyTorch 2.15.0 (CPU). The Windows commands are not tested.

At the post office, a new clerk is unsure about a letter. You ask him: “Where should this go?” He says: “Well, probably the north bag, I think, unless it’s a parcel, then maybe the parcel desk.” You cannot put that answer into the sorting machine. You must read it, guess what he means, and guess how sure he is.

So the manager gives him a form. It has one box for each bag, and he must share 100 points between the boxes. He puts more points where he is more sure. Now every answer has the same shape, and the machine can read it. Over a month, the manager can also check him. When he put 80 points on a bag, was he right about 80 times in 100?

In Module 5, the LLMs answered in text: a small JSON object with a route, a confidence and a reason. A router needs two things from that text: a route from the list, and a probability that it can trust. The text gives neither for sure.

First, the text may not be a route at all. With a short prompt that asked only for the route name, 93% of Qwen3-0.6B’s answers were exactly a route name. The others looked like - delivery: the prompt listed the routes with dashes, and the model copied the dash.

Second, the confidence is a word that the model writes, not a measurement. Here is one test message, with the recorded answer of the hosted LLM without examples:

M-102840 (en, chat): Can I return the storage box if I don’t like the colour?
Label: product
Recorded answer of chat-small (gpt-6-luna-2026-09-22) (zero-shot): returns_refunds, stated confidence 0.99, reason: Asks about returning a product due to its colour.

The question is about the return policy before buying, so Larkfield’s guidelines send it to product. The LLM wrote “0.99” and was wrong. That is not rare: it wrote 0.99 for 693 test messages and was right on 86.6% of them.

Instead of asking the model to write an answer, ask it to choose one from a fixed list, and get a number for each option. There are three ways to do this with a language model:

  1. Score each route. Add each route’s name after the prompt, and ask the model how likely that name is as the answer. Turn the seven scores into probabilities.
  2. Constrain the output. Let the model write, but allow only the words that keep the answer on one of the route names. This is called constrained decoding.
  3. Read a classification head. Train a small layer on top of the model that gives one probability per route. Module 4’s fine-tuned encoder works this way.
One test message, M-100001, in French: «Est-ce que le livreur pourrait laisser mon colis LK-596824 chez ma voisine ?» The right route is delivery. Four boxes follow. Let the model write, the problem: Qwen3-0.6B wrote JSON with route delivery and confidence 0.7; the hosted LLM wrote 1.0; a number in the text, not a measured probability. 1. Score each route, with Qwen3-0.6B: product 0.525, account 0.240, needs_person 0.188, delivery 0.043, orders 0.004, returns_refunds 0.000, billing 0.000; a probability for every route, and here the top one is wrong. 2. Constrain the output with Outlines: the model writes, but each next token must keep the answer on one of the seven route names; always a valid route, not always the right one. 3. Read a classification head, the fine-tuned encoder: delivery 0.979, every other route 0.004 or less; a layer trained on Larkfield's labels, right and sure here.
One message through the ways of getting a decision from a model (measured on a laptop CPU; the written answers are recorded). Each way gives a valid route; only the probabilities say how much to trust it.
Text description of the diagram

The diagram follows one French test message, M-100001: “Est-ce que le livreur pourrait laisser mon colis LK-596824 chez ma voisine ?” (“Could the courier leave my parcel with my neighbour?”). The right route is delivery. The first box shows the problem: when Qwen3-0.6B writes its answer, it gives the route delivery and writes a confidence of 0.7. The hosted LLM wrote 1.0. These are numbers in the text, not measured probabilities. The second box scores each route with Qwen3-0.6B. Product gets 0.525, account 0.240, needs_person 0.188 and delivery 0.043. Orders gets 0.004, and returns_refunds and billing almost 0. There is a probability for every route, but the top one is wrong. The third box constrains the output with Outlines. The model writes, but each next token must keep the answer on one of the seven route names. The answer is always a valid route, but not always the right one. The fourth box reads a classification head: the fine-tuned encoder of Module 4. Delivery gets 0.979, and every other route 0.004 or less. The head was trained on Larkfield’s labels, and here it is right and sure.

A language model works one token at a time: a word or a piece of a word. For each possible next token, it gives a probability. A log-probability is the logarithm of that probability: a negative number, and closer to 0 means more likely. To score a route name, add the log-probabilities of its tokens after the prompt. Then turn the seven scores into probabilities that add up to 1. This last step is the softmax: take e to the power of each score, and divide each result by their sum.

Here is the real calculation for M-100001, with every value:

RouteScoreescore÷ 0.4164
orders-6.360.00170.004
delivery-4.030.01780.043
returns_​refunds-9.220.00010.000
billing-11.500.00000.000
product-1.520.21870.525
account-2.300.09990.240
needs_​person-2.550.07820.188

The last column divides each value of the third column by their sum, 0.4164. So product gets 0.2187 ÷ 0.4164 = 0.525. The model is wrong: the label is delivery, at 0.043. But now we have a number for every route, and nothing to parse.

The course measured this on all 2,000 test messages. It took about 453 ms per message on a laptop CPU. The model reads the prompt once, then makes one short pass for the seven names.

Constrained decoding lets the model write, token by token, but it removes every token that would take the answer off the list. If the answer starts with del, the only allowed next tokens are those that lead to delivery. The Outlines library does this for a list of choices, a regular expression or a JSON schema.

On 500 test messages, Qwen3-0.6B with Outlines gave a valid route name for 100% of them, in about 813 ms per message. It gave the right route for 12.8%. Constrained decoding fixes the shape of the answer; it does not make the model know more.

A classification head is a small layer on top of a model. It takes the model’s numbers for the whole message and gives one probability per route. It must be trained on labelled messages. Module 4’s fine-tuned encoder is exactly this: a small multilingual model with a head trained on Larkfield’s 6,000 training messages. For M-100001, it gives delivery 0.979. The same idea works on a language model, with a head of seven outputs instead of the next-token part. The course did not train one: the encoder already shows the idea.

A probability is calibrated when it matches how often the model is right. Among all messages where it says 0.8, it should be right about 8 times in 10. To check, group the messages by their probability, and compare each group’s mean probability with its accuracy. This is a reliability table, and as a chart a reliability chart. The expected calibration error (ECE) is the mean gap between the two, weighted by the number of messages in each group. The machine learning course explains this in Thresholds and calibration. Module 8 of this course checks every router this way, step by step. Here we need only the idea and one number.

The route scores of Qwen3-0.6B are not calibrated. On the test messages, the ECE is 0.390. The model is often 0.6 or 0.7 sure, and right only about 20% of the time.

Temperature scaling fixes the scale of the probabilities with one number, the temperature. Divide every score by it before the softmax. A temperature above 1 makes the probabilities flatter, closer to “I am not sure”. Choose it on the validation messages: the temperature whose probabilities match the accuracy best. Here it is 5.0, the top of the search grid.

For M-100001, the scores divided by 5 give product 0.250, account 0.214, needs_person 0.203 and delivery 0.151. The order of the routes does not change, so the accuracy does not change either. Only the confidence changes: the model now says it is unsure, which is true.

  • Perfect calibration
  • Route scores (ECE 0.390)
  • After temperature scaling (ECE 0.112)
Reliability of Qwen3-0.6B's route scores on the 2,000 test messages (measured), before and after temperature scaling. Each point is a group of messages with a similar top probability; groups of fewer than 20 messages are left out.
Text description of the chart

The horizontal axis is the mean probability of the top route, and the vertical axis is the share of right answers. Both go from 0 to 1. A dashed diagonal line marks perfect calibration. Before scaling, there are seven points, all far below the diagonal. For example, the messages with a mean probability of 0.65 are right 23% of the time. Those with 0.85 are right 32% of the time. The messages with 0.94 are right 64% of the time. After temperature scaling, almost every message has a probability between 0.2 and 0.4. There are two points: 1,630 messages at 0.26, right 14% of the time, and 350 messages at 0.32, right 41% of the time. Both are close to the diagonal.

Temperature scaling brings the ECE from 0.390 to 0.112 on the test messages. The model is still bad at routing, but now it says so. A router with a threshold, as in The cost of each mistake, would send almost all of its messages to a person. With this model, that is the right thing to do.

Larkfield: a small model fails, a larger one routes

Section titled “Larkfield: a small model fails, a larger one routes”

Small local LLMs showed both local models and their written answers. Here are the three ways of this lesson side by side, for Qwen3-0.6B on the test messages:

Way (Qwen3-0.6B, test) Valid answers Accuracy Cost per 1,000 Time per message
Write a JSON answer 97.2% 0.282 US$7,198.3 2.1 s
Score each route 100% 0.194 US$3,410.0 453 ms
Constrain the output (500 messages) 100% 0.128 813 ms

All three cost more than a person who reads everything (US$1,500 per 1,000). This model is too small for this task. Scoring the routes gives it a probability for every route; it does not make a weak model strong.

So keep the method, and change the model. The course ran Qwen2.5-1.5B-Instruct, a model 2.5 times larger, with the same code, on a sample of 500 test messages (the first 500 by message ID). Small local LLMs shows the sample and its 95% intervals. On it, scoring the routes doubles the accuracy: 0.734 (95% interval 0.694–0.771) against 0.368 for the written answer. Every scored answer is a valid route, where 20.2% of the written answers were not valid JSON. Asking for a decision removed the part of the problem that was writing a clean answer.

One caveat makes this comparison less clean than it looks. The two ways did not use the same prompt. The written answer had the route names only. The route scores had a prompt that also describes each route in one line. So part of the gain may come from the descriptions. Treat “doubles” as the result of both changes together.

It is still not a router for Larkfield. It sends no account takeover to account, takes 7.7 seconds per message, and its scores are not calibrated (ECE 0.161). The method carries over, though: the same code scores any open model whose log-probabilities you can read.

Way Use it when It costs
Score each route You can run the model and read its log-probabilities (open weights, a local model) One pass per message, plus the label names; you must calibrate it
Constrain the output You need a valid answer from a model that writes, for example a hosted model with a schema Nothing extra for accuracy: valid is not right
Classification head You have labelled messages, and you can train Training time and labels; then the fastest and the best calibrated here

Many hosted LLMs give log-probabilities only for the top few tokens, or not at all. Then route scores are not possible, and a schema is the best you can get. That is the problem the next lesson starts from.

  1. Open the notebook from the box Do this lesson in a notebook at the top of this page.
  2. Run the setup cells. They install the packages and download the data. The data cell ends with Ready: 10000 messages (6000 train, 2000 validation, 2000 test), all synthetic.
  3. In section 1, run the cell. It downloads Qwen3-0.6B (about 1.5 GB, once) and prints the end of the prompt.
  4. In section 2, write your prediction. Then run the cell. You see seven probabilities, with product first at 0.525.
  5. In section 3, run the cell. It scores 100 test messages, at about 1 second per message. The accuracy is about 0.27.
  6. In section 4, run the cell. You see the course’s full run, with an ECE of 0.1116 after temperature scaling.

Your project can now replay the answers of every method that the course recorded. These are the embedding models and the fine-tuned encoder of Module 4, the zero-shot models, the hosted LLM and the local model. They are files in data/predictions/ and data/recordings/, which get_data.py downloaded. You add the code that reads them, and the command route llm, which shows one recorded LLM answer.

  1. In your project folder larkroute, with .venv active, copy the files from the snapshot end-of-m06:

    Terminal window
    S=../nextia-learning-labs/C36/snapshots/end-of-m06
    cp "$S/larkroute/llm.py" larkroute/
    cp "$S/larkroute/commands/llm.py" larkroute/commands/
    cp "$S/larkroute/methods/recorded.py" larkroute/methods/
    cp "$S/tests/test_llm.py" "$S/tests/test_recorded.py" tests/
  2. Run the tests. The last line is 23 passed.

    Terminal window
    python -m pytest -q
  3. Show the recorded answer of the LLM without examples for the message of this lesson:

    Terminal window
    route llm M-102840 --mode zero-shot
    M-102840 (en, chat): Can I return the storage box if I don’t like the colour?
    Label: product
    Recorded answer of chat-small (gpt-6-luna-2026-09-22) (zero-shot): returns_refunds, stated confidence 0.99, reason: Asks about returning a product due to its colour.
    Tokens: 160 in, 33 out; cost 0.032 dollars per 1,000 such messages; 2.00 s. (A replayed answer: no call was made.)
  4. Show the answer of the LLM with examples for the same message:

    Terminal window
    route llm M-102840
    M-102840 (en, chat): Can I return the storage box if I don’t like the colour?
    Label: product
    Recorded answer of chat-small (gpt-6-luna-2026-09-22) (few-shot): product, stated confidence 0.99, reason: Asks about return eligibility based on colour preference.
    Tokens: 545 in, 30 out; cost 0.069 dollars per 1,000 such messages; 1.35 s. (A replayed answer: no call was made.)

Both answers say 0.99, and only one is right. “Stated confidence” means the number that the model wrote. route llm replays a recording, so no message leaves your computer.

Change one thing: instead of the local model, check the fine-tuned encoder of Module 4. In section 5 of the notebook, compute its expected calibration error on the test messages, from data/predictions/encoder-finetuned.csv, and put it in ece_encoder. Before you run it, predict: better or worse than the local model after scaling (0.112)?

Show the answer
enc = pd.read_csv("data/predictions/encoder-finetuned.csv").set_index("message_id").loc[test["message_id"]]
_, ece_encoder = rt.reliability(enc["confidence"], enc["prediction"].to_numpy() == test["route"].to_numpy())
print(ece_encoder)
0.0313

Much better: 0.031 without any scaling, and right on 98.0% of the test messages. A head trained on Larkfield’s labels gives both a good route and a true probability.

Common mistake

Valid JSON, and a confidence of 0

Symptom: Tomás runs Qwen3-0.6B with the hosted LLM’s prompt and checks the answers. 97.2% are valid JSON with a route and a confidence. “Good,” he says, “we keep the answers above 0.5.” He finds that most answers have a confidence of 0.

Cause: the model wrote "confidence": 0 in 1,271 of its 1,945 valid answers, and its route was right on 13.8% of those. It also wrote 0.7 for 276 messages and 1.0 for 169. A small model copies the shape of the answer that the prompt asks for; the number in it carries little meaning. A JSON check proves the shape, not the content.

Fix: do not use a written confidence as a probability until you have checked it on labelled messages with a reliability table. Better, ask for a decision: score the routes, or use a trained head, and calibrate the result on validation.

Consequence in AI work: teams build thresholds on a confidence field that only looks like a probability. Without a calibration check, the threshold sends the wrong messages to people.

Check your understanding

Explain the difference between asking a model to write its answer and asking it for a decision, with the numbers of this lesson.

Show the answer

When the model writes, you get text. It may not be a route: only 93% of Qwen3-0.6B’s short answers were exact route names. Its confidence is a written number: the LLM without examples wrote 0.99 and was wrong on M-102840. When you ask for a decision, you always get a valid route. With scores or a head, you also get a probability for each route that you can check with a reliability table.

Check your understanding

Temperature scaling brought the ECE of the route scores from 0.390 to 0.112, but the accuracy stayed at 0.194. Why did the accuracy not change, and why is the scaling still useful?

Show the answer

Dividing every score by the same temperature keeps their order, so the top route and the accuracy stay the same. Only the probabilities change: they become flatter, mostly between 0.2 and 0.4, and now they are true. A router with a threshold then sends these messages to a person, instead of trusting wrong answers that looked 0.6 or 0.7 sure.

Check your understanding

Practical check: after the project step, which command shows a recorded LLM answer, and what does it show for M-102840 without examples?

Show the answer

route llm M-102840 --mode zero-shot. It shows the message, its label product, and the recorded answer: returns_refunds, stated confidence 0.99, with the tokens, the cost and the time. It says that no call was made. python -m pytest -q prints 23 passed.

Recap

  • A router needs a route from the list and a probability it can trust. A written answer gives neither for sure.
  • Three ways to get a decision: score each route (log-probabilities and a softmax), constrain the output (Outlines), or read a classification head.
  • Check every probability with a reliability table and the ECE. Temperature scaling fixes the scale with one number chosen on validation: here 0.390 → 0.112.
  • A valid answer is not a right answer. Qwen3-0.6B is too small for this inbox: 0.282 written, 0.194 by route scores, 0.128 with Outlines.
  • With a larger model, Qwen2.5-1.5B, route scores double the accuracy of the written answer on a 500-message sample (0.734 against 0.368). It still misses every account takeover.
  • Your project now replays every recorded method: route llm and 23 tests.

You can now get a real decision from a model that you run yourself. But you cannot read the log-probabilities of most hosted models. Next, you look at models that companies build to give typed decisions. You also learn how to judge any new model’s claims on your own messages.

Next: Purpose-built decision models