Ask for a decision, not for text
- Lesson 24
- 40 min
- Module 6: Decision models, cascades and distillation
Where we are
Our LLMs wrote their answers as text, and some wrote a confidence number too. A router needs a label and a real probability. In this lesson, we get both from a small local model: we score each label, we constrain the output, and we check if the probabilities are true.
Lesson 24 of 37 · Text Classification and Routing: Send Each Message to the Right PlaceAsk for a decision, not for text
1. The routing problem and its cost
2. Labels you can trust
3. Baselines: rules and classical models
4. Embeddings and small fine-tuned models
5. LLMs as classifiers
6. Decision models, cascades and distillation
7. Harder label shapes
8. Confidence and people in the loop
9. Run it in production
What you will be able to do
Get a route and a probability from a language model, instead of a sentence that you must read and trust. Score each route, limit what the model may write, or read a trained head, and then check that the probabilities are true.
NotebookDo this lesson in a notebook
The notebook loads a small local model, Qwen3-0.6B, on your computer’s CPU. It scores the seven route names for one message and for 100 test messages, and reads the reliability table of the course’s full run. Its setup cells install pandas, scikit-learn, Transformers and PyTorch, and the model download is about 1.5 GB, once. It needs about 4 GB of free memory and no key. About 40 minutes.
Colab and Kaggle open your own copy. They need a free Google or Kaggle account. Stuck? See the solution notebook.
Before you startWhat to know and to have ready
What you need to know
- The LLM as a classifier, its JSON answer and its written confidence, from Zero-shot classification with an LLM.
- Small local LLMs, from Small local LLMs.
- Calibration and the reliability chart, from Thresholds and calibration in the machine learning course (helpful). This lesson explains what it needs.
What you need on your computer (only for the project step)
- Your project folder
larkrouteat the stageend-of-m03, with.venvactive:python -m pytest -qprints18 passed. - The practice files
nextia-learning-labsnext to your project folder, as in Meet the inbox.
Tested on macOS with Python 3.12, pandas 3.1.1, scikit-learn 1.9.1, Transformers 5.22.2 and PyTorch 2.15.0 (CPU). The Windows commands are not tested.
A form with seven boxes
Section titled “A form with seven boxes”At the post office, a new clerk is unsure about a letter. You ask him: “Where should this go?” He says: “Well, probably the north bag, I think, unless it’s a parcel, then maybe the parcel desk.” You cannot put that answer into the sorting machine. You must read it, guess what he means, and guess how sure he is.
So the manager gives him a form. It has one box for each bag, and he must share 100 points between the boxes. He puts more points where he is more sure. Now every answer has the same shape, and the machine can read it. Over a month, the manager can also check him. When he put 80 points on a bag, was he right about 80 times in 100?
The problem: text is not a decision
Section titled “The problem: text is not a decision”In Module 5, the LLMs answered in text: a small JSON object with a route, a confidence and a reason. A router needs two things from that text: a route from the list, and a probability that it can trust. The text gives neither for sure.
First, the text may not be a route at all. With a short prompt that asked only for the route name, 93% of Qwen3-0.6B’s answers were exactly a route name. The others looked like - delivery: the prompt listed the routes with dashes, and the model copied the dash.
Second, the confidence is a word that the model writes, not a measurement. Here is one test message, with the recorded answer of the hosted LLM without examples:
M-102840 (en, chat): Can I return the storage box if I don’t like the colour?Label: productRecorded answer of chat-small (gpt-6-luna-2026-09-22) (zero-shot): returns_refunds, stated confidence 0.99, reason: Asks about returning a product due to its colour.The question is about the return policy before buying, so Larkfield’s guidelines send it to product. The LLM wrote “0.99” and was wrong. That is not rare: it wrote 0.99 for 693 test messages and was right on 86.6% of them.
The idea: ask for a decision
Section titled “The idea: ask for a decision”Instead of asking the model to write an answer, ask it to choose one from a fixed list, and get a number for each option. There are three ways to do this with a language model:
- Score each route. Add each route’s name after the prompt, and ask the model how likely that name is as the answer. Turn the seven scores into probabilities.
- Constrain the output. Let the model write, but allow only the words that keep the answer on one of the route names. This is called constrained decoding.
- Read a classification head. Train a small layer on top of the model that gives one probability per route. Module 4’s fine-tuned encoder works this way.
Text description of the diagram
The diagram follows one French test message, M-100001: “Est-ce que le livreur pourrait laisser mon colis LK-596824 chez ma voisine ?” (“Could the courier leave my parcel with my neighbour?”). The right route is delivery. The first box shows the problem: when Qwen3-0.6B writes its answer, it gives the route delivery and writes a confidence of 0.7. The hosted LLM wrote 1.0. These are numbers in the text, not measured probabilities. The second box scores each route with Qwen3-0.6B. Product gets 0.525, account 0.240, needs_person 0.188 and delivery 0.043. Orders gets 0.004, and returns_refunds and billing almost 0. There is a probability for every route, but the top one is wrong. The third box constrains the output with Outlines. The model writes, but each next token must keep the answer on one of the seven route names. The answer is always a valid route, but not always the right one. The fourth box reads a classification head: the fine-tuned encoder of Module 4. Delivery gets 0.979, and every other route 0.004 or less. The head was trained on Larkfield’s labels, and here it is right and sure.
Way 1: score each route
Section titled “Way 1: score each route”A language model works one token at a time: a word or a piece of a word. For each possible next token, it gives a probability. A log-probability is the logarithm of that probability: a negative number, and closer to 0 means more likely. To score a route name, add the log-probabilities of its tokens after the prompt. Then turn the seven scores into probabilities that add up to 1. This last step is the softmax: take e to the power of each score, and divide each result by their sum.
Here is the real calculation for M-100001, with every value:
| Route | Score | escore | ÷ 0.4164 |
|---|---|---|---|
| orders | -6.36 | 0.0017 | 0.004 |
| delivery | -4.03 | 0.0178 | 0.043 |
| returns_refunds | -9.22 | 0.0001 | 0.000 |
| billing | -11.50 | 0.0000 | 0.000 |
| product | -1.52 | 0.2187 | 0.525 |
| account | -2.30 | 0.0999 | 0.240 |
| needs_person | -2.55 | 0.0782 | 0.188 |
The last column divides each value of the third column by their sum, 0.4164. So product gets 0.2187 ÷ 0.4164 = 0.525. The model is wrong: the label is delivery, at 0.043. But now we have a number for every route, and nothing to parse.
The course measured this on all 2,000 test messages. It took about 453 ms per message on a laptop CPU. The model reads the prompt once, then makes one short pass for the seven names.
Way 2: constrain the output
Section titled “Way 2: constrain the output”Constrained decoding lets the model write, token by token, but it removes every token that would take the answer off the list. If the answer starts with del, the only allowed next tokens are those that lead to delivery. The Outlines library does this for a list of choices, a regular expression or a JSON schema.
On 500 test messages, Qwen3-0.6B with Outlines gave a valid route name for 100% of them, in about 813 ms per message. It gave the right route for 12.8%. Constrained decoding fixes the shape of the answer; it does not make the model know more.
Way 3: read a classification head
Section titled “Way 3: read a classification head”A classification head is a small layer on top of a model. It takes the model’s numbers for the whole message and gives one probability per route. It must be trained on labelled messages. Module 4’s fine-tuned encoder is exactly this: a small multilingual model with a head trained on Larkfield’s 6,000 training messages. For M-100001, it gives delivery 0.979. The same idea works on a language model, with a head of seven outputs instead of the next-token part. The course did not train one: the encoder already shows the idea.
Are the probabilities true?
Section titled “Are the probabilities true?”A probability is calibrated when it matches how often the model is right. Among all messages where it says 0.8, it should be right about 8 times in 10. To check, group the messages by their probability, and compare each group’s mean probability with its accuracy. This is a reliability table, and as a chart a reliability chart. The expected calibration error (ECE) is the mean gap between the two, weighted by the number of messages in each group. The machine learning course explains this in Thresholds and calibration. Module 8 of this course checks every router this way, step by step. Here we need only the idea and one number.
The route scores of Qwen3-0.6B are not calibrated. On the test messages, the ECE is 0.390. The model is often 0.6 or 0.7 sure, and right only about 20% of the time.
Temperature scaling
Section titled “Temperature scaling”Temperature scaling fixes the scale of the probabilities with one number, the temperature. Divide every score by it before the softmax. A temperature above 1 makes the probabilities flatter, closer to “I am not sure”. Choose it on the validation messages: the temperature whose probabilities match the accuracy best. Here it is 5.0, the top of the search grid.
For M-100001, the scores divided by 5 give product 0.250, account 0.214, needs_person 0.203 and delivery 0.151. The order of the routes does not change, so the accuracy does not change either. Only the confidence changes: the model now says it is unsure, which is true.
- Perfect calibration
- Route scores (ECE 0.390)
- After temperature scaling (ECE 0.112)
Text description of the chart
The horizontal axis is the mean probability of the top route, and the vertical axis is the share of right answers. Both go from 0 to 1. A dashed diagonal line marks perfect calibration. Before scaling, there are seven points, all far below the diagonal. For example, the messages with a mean probability of 0.65 are right 23% of the time. Those with 0.85 are right 32% of the time. The messages with 0.94 are right 64% of the time. After temperature scaling, almost every message has a probability between 0.2 and 0.4. There are two points: 1,630 messages at 0.26, right 14% of the time, and 350 messages at 0.32, right 41% of the time. Both are close to the diagonal.
Temperature scaling brings the ECE from 0.390 to 0.112 on the test messages. The model is still bad at routing, but now it says so. A router with a threshold, as in The cost of each mistake, would send almost all of its messages to a person. With this model, that is the right thing to do.
Larkfield: a small model fails, a larger one routes
Section titled “Larkfield: a small model fails, a larger one routes”Small local LLMs showed both local models and their written answers. Here are the three ways of this lesson side by side, for Qwen3-0.6B on the test messages:
| Way (Qwen3-0.6B, test) | Valid answers | Accuracy | Cost per 1,000 | Time per message |
|---|---|---|---|---|
| Write a JSON answer | 97.2% | 0.282 | US$7,198.3 | 2.1 s |
| Score each route | 100% | 0.194 | US$3,410.0 | 453 ms |
| Constrain the output (500 messages) | 100% | 0.128 | 813 ms |
All three cost more than a person who reads everything (US$1,500 per 1,000). This model is too small for this task. Scoring the routes gives it a probability for every route; it does not make a weak model strong.
A larger model: Qwen2.5-1.5B
Section titled “A larger model: Qwen2.5-1.5B”So keep the method, and change the model. The course ran Qwen2.5-1.5B-Instruct, a model 2.5 times larger, with the same code, on a sample of 500 test messages (the first 500 by message ID). Small local LLMs shows the sample and its 95% intervals. On it, scoring the routes doubles the accuracy: 0.734 (95% interval 0.694–0.771) against 0.368 for the written answer. Every scored answer is a valid route, where 20.2% of the written answers were not valid JSON. Asking for a decision removed the part of the problem that was writing a clean answer.
One caveat makes this comparison less clean than it looks. The two ways did not use the same prompt. The written answer had the route names only. The route scores had a prompt that also describes each route in one line. So part of the gain may come from the descriptions. Treat “doubles” as the result of both changes together.
It is still not a router for Larkfield. It sends no account takeover to account, takes 7.7 seconds per message, and its scores are not calibrated (ECE 0.161). The method carries over, though: the same code scores any open model whose log-probabilities you can read.
When to use each way
Section titled “When to use each way”| Way | Use it when | It costs |
|---|---|---|
| Score each route | You can run the model and read its log-probabilities (open weights, a local model) | One pass per message, plus the label names; you must calibrate it |
| Constrain the output | You need a valid answer from a model that writes, for example a hosted model with a schema | Nothing extra for accuracy: valid is not right |
| Classification head | You have labelled messages, and you can train | Training time and labels; then the fastest and the best calibrated here |
Many hosted LLMs give log-probabilities only for the top few tokens, or not at all. Then route scores are not possible, and a schema is the best you can get. That is the problem the next lesson starts from.
Guided practice: score the routes
Section titled “Guided practice: score the routes”- Open the notebook from the box Do this lesson in a notebook at the top of this page.
- Run the setup cells. They install the packages and download the data. The data cell ends with
Ready: 10000 messages (6000 train, 2000 validation, 2000 test), all synthetic. - In section 1, run the cell. It downloads Qwen3-0.6B (about 1.5 GB, once) and prints the end of the prompt.
- In section 2, write your prediction. Then run the cell. You see seven probabilities, with
productfirst at0.525. - In section 3, run the cell. It scores 100 test messages, at about 1 second per message. The accuracy is about
0.27. - In section 4, run the cell. You see the course’s full run, with an ECE of
0.1116after temperature scaling.
Add the recorded methods to the project
Section titled “Add the recorded methods to the project”Your project can now replay the answers of every method that the course recorded. These are the embedding models and the fine-tuned encoder of Module 4, the zero-shot models, the hosted LLM and the local model. They are files in data/predictions/ and data/recordings/, which get_data.py downloaded. You add the code that reads them, and the command route llm, which shows one recorded LLM answer.
-
In your project folder
larkroute, with.venvactive, copy the files from the snapshotend-of-m06:Terminal window S=../nextia-learning-labs/C36/snapshots/end-of-m06cp "$S/larkroute/llm.py" larkroute/cp "$S/larkroute/commands/llm.py" larkroute/commands/cp "$S/larkroute/methods/recorded.py" larkroute/methods/cp "$S/tests/test_llm.py" "$S/tests/test_recorded.py" tests/Terminal window $S = "..\nextia-learning-labs\C36\snapshots\end-of-m06"Copy-Item "$S\larkroute\llm.py" larkroute\Copy-Item "$S\larkroute\commands\llm.py" larkroute\commands\Copy-Item "$S\larkroute\methods\recorded.py" larkroute\methods\Copy-Item "$S\tests\test_llm.py", "$S\tests\test_recorded.py" tests\ -
Run the tests. The last line is
23 passed.Terminal window python -m pytest -q -
Show the recorded answer of the LLM without examples for the message of this lesson:
Terminal window route llm M-102840 --mode zero-shotM-102840 (en, chat): Can I return the storage box if I don’t like the colour?Label: productRecorded answer of chat-small (gpt-6-luna-2026-09-22) (zero-shot): returns_refunds, stated confidence 0.99, reason: Asks about returning a product due to its colour.Tokens: 160 in, 33 out; cost 0.032 dollars per 1,000 such messages; 2.00 s. (A replayed answer: no call was made.) -
Show the answer of the LLM with examples for the same message:
Terminal window route llm M-102840M-102840 (en, chat): Can I return the storage box if I don’t like the colour?Label: productRecorded answer of chat-small (gpt-6-luna-2026-09-22) (few-shot): product, stated confidence 0.99, reason: Asks about return eligibility based on colour preference.Tokens: 545 in, 30 out; cost 0.069 dollars per 1,000 such messages; 1.35 s. (A replayed answer: no call was made.)
Both answers say 0.99, and only one is right. “Stated confidence” means the number that the model wrote. route llm replays a recording, so no message leaves your computer.
Your turn: the encoder’s calibration
Section titled “Your turn: the encoder’s calibration”Change one thing: instead of the local model, check the fine-tuned encoder of Module 4. In section 5 of the notebook, compute its expected calibration error on the test messages, from data/predictions/encoder-finetuned.csv, and put it in ece_encoder. Before you run it, predict: better or worse than the local model after scaling (0.112)?
Show the answer
enc = pd.read_csv("data/predictions/encoder-finetuned.csv").set_index("message_id").loc[test["message_id"]]_, ece_encoder = rt.reliability(enc["confidence"], enc["prediction"].to_numpy() == test["route"].to_numpy())print(ece_encoder)0.0313Much better: 0.031 without any scaling, and right on 98.0% of the test messages. A head trained on Larkfield’s labels gives both a good route and a true probability.
Common mistake
Valid JSON, and a confidence of 0
Symptom: Tomás runs Qwen3-0.6B with the hosted LLM’s prompt and checks the answers. 97.2% are valid JSON with a route and a confidence. “Good,” he says, “we keep the answers above 0.5.” He finds that most answers have a confidence of 0.
Cause: the model wrote "confidence": 0 in 1,271 of its 1,945 valid answers, and its route was right on 13.8% of those. It also wrote 0.7 for 276 messages and 1.0 for 169. A small model copies the shape of the answer that the prompt asks for; the number in it carries little meaning. A JSON check proves the shape, not the content.
Fix: do not use a written confidence as a probability until you have checked it on labelled messages with a reliability table. Better, ask for a decision: score the routes, or use a trained head, and calibrate the result on validation.
Consequence in AI work: teams build thresholds on a confidence field that only looks like a probability. Without a calibration check, the threshold sends the wrong messages to people.
Knowledge check
Section titled “Knowledge check”Check your understanding
Explain the difference between asking a model to write its answer and asking it for a decision, with the numbers of this lesson.
Show the answer
When the model writes, you get text. It may not be a route: only 93% of Qwen3-0.6B’s short answers were exact route names. Its confidence is a written number: the LLM without examples wrote 0.99 and was wrong on M-102840. When you ask for a decision, you always get a valid route. With scores or a head, you also get a probability for each route that you can check with a reliability table.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
Temperature scaling brought the ECE of the route scores from 0.390 to 0.112, but the accuracy stayed at 0.194. Why did the accuracy not change, and why is the scaling still useful?
Show the answer
Dividing every score by the same temperature keeps their order, so the top route and the accuracy stay the same. Only the probabilities change: they become flatter, mostly between 0.2 and 0.4, and now they are true. A router with a threshold then sends these messages to a person, instead of trusting wrong answers that looked 0.6 or 0.7 sure.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
Practical check: after the project step, which command shows a recorded LLM answer, and what does it show for M-102840 without examples?
Show the answer
route llm M-102840 --mode zero-shot. It shows the message, its label product, and the recorded answer: returns_refunds, stated confidence 0.99, with the tokens, the cost and the time. It says that no call was made. python -m pytest -q prints 23 passed.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Recap
- A router needs a route from the list and a probability it can trust. A written answer gives neither for sure.
- Three ways to get a decision: score each route (log-probabilities and a softmax), constrain the output (Outlines), or read a classification head.
- Check every probability with a reliability table and the ECE. Temperature scaling fixes the scale with one number chosen on validation: here 0.390 → 0.112.
- A valid answer is not a right answer. Qwen3-0.6B is too small for this inbox: 0.282 written, 0.194 by route scores, 0.128 with Outlines.
- With a larger model, Qwen2.5-1.5B, route scores double the accuracy of the written answer on a 500-message sample (0.734 against 0.368). It still misses every account takeover.
- Your project now replays every recorded method:
route llmand 23 tests.
You can now get a real decision from a model that you run yourself. But you cannot read the log-probabilities of most hosted models. Next, you look at models that companies build to give typed decisions. You also learn how to judge any new model’s claims on your own messages.