Purpose-built decision models
- Lesson 25
- 30 min
- Module 6: Decision models, cascades and distillation
Where we are
We turned a text model into a decision model by hand. Some companies now build models that only make typed decisions. In this lesson, we read one vendor's claims as an example, and we learn how to test any new model's claims on our own messages.
Lesson 25 of 37 · Text Classification and Routing: Send Each Message to the Right PlacePurpose-built decision models
1. The routing problem and its cost
2. Labels you can trust
3. Baselines: rules and classical models
4. Embeddings and small fine-tuned models
5. LLMs as classifiers
6. Decision models, cascades and distillation
7. Harder label shapes
8. Confidence and people in the loop
9. Run it in production
What you will be able to do
Judge a new model by what it does on your messages, not by what its maker says. Use a five-question checklist, and run the model as one more row of the comparison table before you trust it.
Before you startWhat to know and to have ready
What you need to know
- Route scores, constrained output and calibration, from Ask for a decision, not for text.
- The comparison table and its columns, from Meet the inbox.
What you need on your computer (only for the project step)
- Your project folder
larkroutefrom the last lesson, with.venvactive:python -m pytest -qprints23 passed.
This lesson does not change the project. Tested on macOS with Python 3.12. The Windows commands are not tested.
A sorting machine and its brochure
Section titled “A sorting machine and its brochure”A company visits the post office with a new sorting machine. The brochure says: “200 times faster than a person. Never puts a letter in a bag that does not exist. Knows when it is unsure.” The manager is interested. A faster machine would change her whole morning.
She does not buy it from the brochure. She asks for one thing: “Run it on one week of our letters, next to our clerks.” The brochure’s letters were typed and clean. Hers are handwritten, in two languages, with old addresses. She also asks what happens when the company changes the machine, and what it costs to stop using it.
The problem: claims are not evidence
Section titled “The problem: claims are not evidence”Every model in this course came with a good story. On this inbox, several of the stories did not hold. Here are four real results from the earlier lessons:
| What you might expect | What Larkfield’s test messages showed |
|---|---|
| A newer embedding model is better | The 2026 model: 0.906, 26.9 ms per message (7.0 ms in float32). The 2021 model: 0.920, 6.8 ms |
| A model built for zero-shot classification works without labels | GLiClass: 0.516. The NLI model: 0.329 |
| A valid answer is a good answer | Qwen3-0.6B with Outlines: 100% valid route names, 12.8% right |
| “Confidence 0.99” means 99% sure | The LLM without examples wrote 0.99 693 times, and was right 86.6% of those times |
None of these models is bad in general. They were built and tested on other data, and they did not fit this inbox. Claims describe the maker’s data. Only your data describes yours.
The idea: models built for decisions
Section titled “The idea: models built for decisions”In the last lesson, we took a text model and made it decide by hand. We scored each route’s name, or we limited what it could write. A purpose-built decision model does this by design. You give it the list of possible answers, the schema. It returns one of them, with a probability for each, and it does not write free text. “Typed” means that the answer always has the type that the schema asks for: here, one of seven route names.
Why build a model like this? Three reasons, if the claims hold:
- No parsing. The answer is always a valid route. Nothing can come back as “- delivery” or a cut-off JSON.
- A real probability. The model can be trained so that “0.9” is right about 9 times in 10. Then a threshold on it means something.
- Speed. It does not write token by token, so it can answer in one short pass.
One example: Jev
Section titled “One example: Jev”At the time of writing, one vendor sells this kind of model. TypeSafe AI announced its System One models in September 2026, with Jev as the first. Here is what the vendor’s own article says. Every line is a vendor claim: the course team did not sign up, did not call the model and measured nothing.
| The vendor says | What the article gives as evidence |
|---|---|
| Typed decisions only: the possible outputs and their structure are fixed in advance; no free text | A description of the design |
| Every answer comes with calibrated probabilities | No calibration figures |
| 70 to 500 ms per request, end to end; 40 to 200 times faster than frontier LLMs | Measured by the vendor from laptops on the US West Coast |
| 0% type errors | True by design: the answer always matches the schema. It is not an accuracy |
| Input US$0.042 per million tokens; output free | The vendor says that it cannot prove that the price is not subsidised |
| Accuracy “off the charts” on its own workflow tests | No accuracy figures; no public benchmark |
When the article was read, Jev was in early access for developers from a waitlist. The article does not say that the weights are open, so you cannot run it on your own computer. The course team did not join the waitlist.
Look at the right-hand column. The speed has a source, but from the vendor’s own computers. “0% type errors” is true, and it is the same promise that Outlines gave us: a valid answer, not a right answer. Calibration and accuracy, the two things that a router needs most, have no numbers at all.
A checklist for any new model’s claims
Section titled “A checklist for any new model’s claims”Priya writes a checklist for Grace and Omar. It works for Jev, for the next LLM, and for any model that a vendor shows them:
Text description of the diagram
The diagram is a column of boxes, from a vendor’s claim at the top to a trusted row at the bottom. The claim is, for example, “calibrated” or “200 times faster”. Five questions follow, one per box. First, your own test set: score the model on your 2,000 test messages, as one more row of route compare. Second, calibration on your data: is “0.9 sure” right 9 times in 10 on your messages? Make a reliability table and compute the expected calibration error. Third, time from where you are: measure the median and the slowest 10% of requests from your own servers, not from the vendor’s. Fourth, price and lock-in: work out the cost per 1,000 of your own messages. Ask whether you can leave, and who else can run the model. Fifth, when the vendor changes the model: pin a version, and run your test set again before you accept a new one. The last box says: measured on your messages, the model is a row that you can trust. Until then, it is a vendor claim.
1. Your own test set
Section titled “1. Your own test set”Run the model on the same 2,000 test messages as every other row, with the same cost table. Only then can you compare it with the other rows. TF-IDF costs US$296.0 and the fine-tuned encoder US$103.0. A vendor’s benchmark uses other messages, other labels and another cost of a mistake.
2. Calibration on your data
Section titled “2. Calibration on your data”A probability is useful only if it is true on your messages. Make the reliability table of Ask for a decision, not for text, and compute the expected calibration error. A model can be well calibrated on the vendor’s tests and badly calibrated on French chat messages about garden furniture.
3. Time from where you are
Section titled “3. Time from where you are”The vendor’s 70 to 500 ms were measured from its own laptops. Your requests travel from your servers, through your network, at your busy hours. Measure the median and the slowest 10% of requests, as the course does for every hosted row. The chart shows where the claim would sit among the times that the course measured:
Text description of the diagram
The chart shows one dot per method on a log scale of milliseconds per message, from 0.01 to 10,000. Each step on the scale is 10 times slower than the one before. TF-IDF with Naive Bayes takes 0.02 ms and TF-IDF with logistic regression 0.16 ms. The fine-tuned encoder and the 2021 embeddings take about 7 ms. GLiClass takes 21.3 ms, and the 2026 embeddings 26.9 ms. The local Qwen3-0.6B takes 453 ms with route scores and 2,128 ms when it writes JSON. The hosted LLM takes 1,200 ms without examples and 1,708 ms with examples, and the reasoning model 2,040 ms. An orange band from 70 to 500 ms marks the vendor’s claim for Jev. It sits between the local models and the hosted LLMs, and it is not a measurement.
If the claim holds from your servers, Jev would be about 3 to 24 times faster than the hosted LLM. It would still be 10 to 3,000 times slower than the trained models that run on Larkfield’s own computer.
4. Price and lock-in
Section titled “4. Price and lock-in”Turn the price into your cost per 1,000 messages. Our LLM prompt with examples is about 717 input tokens. At the vendor’s input price, 1,000 such messages would cost about US$0.03, with free output. That is our arithmetic on a vendor claim. On this inbox, it changes little: the hosted LLM already costs only US$0.088 per 1,000 messages to run. The cost of its mistakes is US$116.0. A cheaper model that makes more mistakes costs more.
Then ask about lock-in: how hard it is to leave. A closed model with no open weights runs only at its vendor. If the vendor changes the price, stops the model or closes, can you move? The trained models of this course are files on Larkfield’s own disk.
5. When the vendor changes it
Section titled “5. When the vendor changes it”A hosted model can change behind the same name. Ask if you can pin a version: call one fixed version until you choose to move. Before you move, run your test set again, and compare the two rows. The course’s hosted rows were recorded once, so they cannot change. A live model can.
Larkfield: the comparison table with one more row
Section titled “Larkfield: the comparison table with one more row”Priya adds the vendor’s model to the comparison table as an optional row, clearly marked:
| Method | Accuracy | Total $/1k | ms/message | Label |
|---|---|---|---|---|
| Fine-tuned MiniLM-L12 encoder | 0.980 | 103.0 | 6.7 | measured |
| Hosted LLM chat-small, few-shot | 0.970 | 116.1 | 1,708 | recorded |
| Jev (TypeSafe AI) | not measured | not measured | 70–500 | vendor claim, from its article (September 2026) |
The row tells the truth: nobody knows where Jev sits on this inbox. It cannot be the “best” row, or the “worst”. To earn real numbers, it needs answers for the validation and test messages, and then it goes through the same columns as everything else.
Add a new model as one more row
Section titled “Add a new model as one more row”Your project can already score a model that it cannot run. The recorded methods of Modules 4 and 5 are only files: one CSV and one JSON file per method, in data/predictions/. route compare reads every pair that it finds there. So a new model needs no new code. It needs its answers.
-
In your project folder
larkroute, with.venvactive, score every method on the test messages. It takes about 10 seconds:Terminal window route compareIt prints one line per method while it scores, then a table of 15 rows, cheapest first. The table starts like this (the
ms/msgcolumn changes a little from run to run):method accuracy macro-F1 takeover needs person EN FR mistakes $/1k run $/1k total $/1k ms/msgfine-tuned MiniLM-L12 multilingual encoder 0.980 0.980 1.000 1.000 0.980 0.978 103.0 0.000 103.0 6.73hosted LLM chat-small, few-shot (recorded) 0.970 0.962 1.000 1.000 0.973 0.961 116.0 0.088 116.1 1708.00TF-IDF + linear SVM 0.966 0.969 1.000 1.000 0.971 0.954 175.0 0.000 175.0 0.12 -
Look at how one recorded method is stored: its description, then its first two answers.
Terminal window cat data/predictions/llm-few-shot.jsonhead -3 data/predictions/llm-few-shot.csvTerminal window Get-Content data\predictions\llm-few-shot.jsonGet-Content data\predictions\llm-few-shot.csv -TotalCount 3{"method": "hosted LLM chat-small, few-shot (recorded)","label": "recorded","model": "chat-small (gpt-6-luna-2026-09-22)","ms_per_message": 1708.0,"run_per_1000": 0.0876,"messages": 4000,"notes": "Recorded once from the Azure deployment chat-small (reasoning effort none); replayed. The confidence is the number the model wrote, not a probability."}message_id,prediction,confidenceM-100006,delivery,0.99M-100010,product,0.99
To add a new model, you would write the same two files. They hold one answer per validation and test message, the time per message that you measured, and the running cost per 1,000 messages. A decision model also gives a probability per route: add the columns p_orders to p_needs_person, as in data/predictions/local-llm-scores.csv. Then route compare scores it like every other row. This lesson adds no new model: the course has no answers from Jev.
When a purpose-built decision model makes sense
Section titled “When a purpose-built decision model makes sense”Consider one when you need typed decisions at scale, with a probability, and the hosted LLM is too slow or too costly to call. Consider one also when you cannot train your own model: no labels, or no people to run a model.
Be careful when you already have labels and a trained model that works. On this inbox, the fine-tuned encoder decides in 6.7 ms on a laptop, costs nothing to run, and makes few mistakes. A closed model must beat that on your test set, and you also take on its lock-in. And be careful with sensitive messages: a hosted model sees every message that you send it.
Your turn: apply the checklist to GLiClass
Section titled “Your turn: apply the checklist to GLiClass”GLiClass is also a model built only for classification. Like a decision model, it scores each label’s name with a probability and writes no text. Its makers describe it as a zero-shot classifier: it needs no training labels. Use the checklist and the numbers of this course to answer: should Larkfield use it as its router?
Show the answer
No, not on this inbox. Go through the checklist:
- Your own test set: 0.516 accuracy and US$4,431 per 1,000 messages. That is about three times the cost of a person who reads everything (US$1,500).
- Calibration: its expected calibration error on test is 0.089, against 0.050 for TF-IDF + logistic regression. Its probabilities are less true, and its route is wrong half the time.
- Time: 21 ms per message on a laptop CPU: fast enough.
- Price and lock-in: open weights (Apache-2.0). It runs on Larkfield’s computer at no cost per call, and nobody can take it away.
- Changes: a pinned revision on Hugging Face never changes.
It passes questions 3 to 5 and fails the first two, which matter most. Its claim of “zero-shot classification” is true; it is not enough here. It is still useful for one job: finding a new route without retraining. In Zero-shot without an LLM, it found 31 of 80 messages about the new device.
Common mistake
Newer, so better
Symptom: Tomás reads that a new embedding model, from 2026, beats the 2021 one on public benchmarks. He swaps it in without a test. A week later, Grace reports more messages in the wrong queues.
Cause: on Larkfield’s test messages, the newer model is worse. It gets 0.906 against 0.920, and costs US$895 per 1,000 messages against US$779. Loaded as it is stored (bfloat16), it took 26.9 ms against 6.8 ms; loaded in 32-bit numbers, it took 7.0 ms. The public benchmarks measured other tasks, on other text. A newer model is a candidate, not an upgrade.
Fix: treat every new model, a newer version of the same model too, as a new row. Score it on the same test set and cost table. Check its calibration and its time. Keep the old row until the new one wins.
Consequence in AI work: model swaps that skip the test are a common cause of quiet quality drops in production. The comparison table is cheap to rerun. A week of wrong queues is not.
Knowledge check
Section titled “Knowledge check”Check your understanding
Jev's vendor claims 0% type errors. Explain why that is not the same as accuracy, with an example from this course.
Show the answer
“0% type errors” means that every answer matches the schema: it is always one of the allowed values. That is true by design, and it says nothing about whether the value is right. In this course, Qwen3-0.6B with Outlines gave a valid route name for 100% of 500 test messages. It gave the right route for only 12.8% of them. A valid answer is not a right answer. Only a test on your own labelled messages measures accuracy.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
A vendor says its model is calibrated and answers in 70 to 500 ms. Which two checks of the checklist test these claims, and what exactly do you measure?
Show the answer
Calibration on your data. Run the model on your validation or test messages, and group the answers by their probability. In each group, compare the mean probability with the share that is right: a reliability table. Then compute the expected calibration error. Time from where you are. Send your own messages from your own servers. Measure the median time and the slowest 10% of requests, at normal and busy hours. The vendor measured from its own laptops.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
Practical check: your project scores recorded methods that it cannot run. Where are they, and what do you need to add a new model as one more row of route compare?
Show the answer
They are in data/predictions/, as one CSV and one JSON file per method. The CSV has one row per validation and test message: message_id, prediction, confidence, and for a model with probabilities the columns p_orders to p_needs_person. The JSON file gives the method’s name and how its answers were made. It also gives the measured milliseconds per message and the running cost per 1,000 messages. With those two files, route compare scores the new model with the same test set and cost table as every other row.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Recap
- A purpose-built decision model returns one value from a fixed list (the schema), with a probability, and writes no text.
- Jev (TypeSafe AI, announced in September 2026, early access) is one example. Its speed, price and calibration are vendor claims: the course did not measure it, so its row says “not measured”.
- “0% type errors” is not accuracy. Outlines gave 100% valid answers and 12.8% right ones.
- The checklist: your own test set, calibration on your data, time from where you are, price and lock-in, and what happens when the vendor changes the model.
- A new model joins the project as two files in
data/predictions/, androute comparescores it like every other row.
You can now judge any model by what it does on your messages. But the comparison table still asks you to choose one router. Next, you combine them: the cheap model decides when it is sure, the LLM gets the unsure messages, and a person gets the rest.
Next: Cascades