Skip to content
Français

Purpose-built decision models

  • Lesson 25
  • 30 min
  • Module 6: Decision models, cascades and distillation

Where we are

We turned a text model into a decision model by hand. Some companies now build models that only make typed decisions. In this lesson, we read one vendor's claims as an example, and we learn how to test any new model's claims on our own messages.

Lesson 25 of 37 · Text Classification and Routing: Send Each Message to the Right PlacePurpose-built decision models

What you will be able to do

Judge a new model by what it does on your messages, not by what its maker says. Use a five-question checklist, and run the model as one more row of the comparison table before you trust it.

Before you startWhat to know and to have readyAbout 30 minutes

What you need to know

What you need on your computer (only for the project step)

  • Your project folder larkroute from the last lesson, with .venv active: python -m pytest -q prints 23 passed.

This lesson does not change the project. Tested on macOS with Python 3.12. The Windows commands are not tested.

A company visits the post office with a new sorting machine. The brochure says: “200 times faster than a person. Never puts a letter in a bag that does not exist. Knows when it is unsure.” The manager is interested. A faster machine would change her whole morning.

She does not buy it from the brochure. She asks for one thing: “Run it on one week of our letters, next to our clerks.” The brochure’s letters were typed and clean. Hers are handwritten, in two languages, with old addresses. She also asks what happens when the company changes the machine, and what it costs to stop using it.

Every model in this course came with a good story. On this inbox, several of the stories did not hold. Here are four real results from the earlier lessons:

What you might expect What Larkfield’s test messages showed
A newer embedding model is better The 2026 model: 0.906, 26.9 ms per message (7.0 ms in float32). The 2021 model: 0.920, 6.8 ms
A model built for zero-shot classification works without labels GLiClass: 0.516. The NLI model: 0.329
A valid answer is a good answer Qwen3-0.6B with Outlines: 100% valid route names, 12.8% right
“Confidence 0.99” means 99% sure The LLM without examples wrote 0.99 693 times, and was right 86.6% of those times

None of these models is bad in general. They were built and tested on other data, and they did not fit this inbox. Claims describe the maker’s data. Only your data describes yours.

In the last lesson, we took a text model and made it decide by hand. We scored each route’s name, or we limited what it could write. A purpose-built decision model does this by design. You give it the list of possible answers, the schema. It returns one of them, with a probability for each, and it does not write free text. “Typed” means that the answer always has the type that the schema asks for: here, one of seven route names.

Why build a model like this? Three reasons, if the claims hold:

  • No parsing. The answer is always a valid route. Nothing can come back as “- delivery” or a cut-off JSON.
  • A real probability. The model can be trained so that “0.9” is right about 9 times in 10. Then a threshold on it means something.
  • Speed. It does not write token by token, so it can answer in one short pass.

At the time of writing, one vendor sells this kind of model. TypeSafe AI announced its System One models in September 2026, with Jev as the first. Here is what the vendor’s own article says. Every line is a vendor claim: the course team did not sign up, did not call the model and measured nothing.

The vendor says What the article gives as evidence
Typed decisions only: the possible outputs and their structure are fixed in advance; no free text A description of the design
Every answer comes with calibrated probabilities No calibration figures
70 to 500 ms per request, end to end; 40 to 200 times faster than frontier LLMs Measured by the vendor from laptops on the US West Coast
0% type errors True by design: the answer always matches the schema. It is not an accuracy
Input US$0.042 per million tokens; output free The vendor says that it cannot prove that the price is not subsidised
Accuracy “off the charts” on its own workflow tests No accuracy figures; no public benchmark

When the article was read, Jev was in early access for developers from a waitlist. The article does not say that the weights are open, so you cannot run it on your own computer. The course team did not join the waitlist.

Look at the right-hand column. The speed has a source, but from the vendor’s own computers. “0% type errors” is true, and it is the same promise that Outlines gave us: a valid answer, not a right answer. Calibration and accuracy, the two things that a router needs most, have no numbers at all.

Priya writes a checklist for Grace and Omar. It works for Jev, for the next LLM, and for any model that a vendor shows them:

A column of boxes from top to bottom. At the top: a vendor's claim, such as calibrated or 200 times faster. Then five questions. 1. Your own test set: score it on your 2,000 test messages, as one more row of route compare. 2. Calibration on your data: is 0.9 sure right 9 times in 10 here? Make a reliability table; compute the ECE. 3. Time from where you are: measure the median and the slow 10% from your own servers, not the vendor's. 4. Price and lock-in: the cost per 1,000 of your messages; can you leave, and who else can run it? 5. When the vendor changes it: pin a version, and run your test set again before you accept a new one. At the bottom: measured on your messages, a row you can trust; until then, a vendor claim.
Five questions that turn a vendor's claim into a measurement on your own messages. Until you have the answers, the claim stays a claim.
Text description of the diagram

The diagram is a column of boxes, from a vendor’s claim at the top to a trusted row at the bottom. The claim is, for example, “calibrated” or “200 times faster”. Five questions follow, one per box. First, your own test set: score the model on your 2,000 test messages, as one more row of route compare. Second, calibration on your data: is “0.9 sure” right 9 times in 10 on your messages? Make a reliability table and compute the expected calibration error. Third, time from where you are: measure the median and the slowest 10% of requests from your own servers, not from the vendor’s. Fourth, price and lock-in: work out the cost per 1,000 of your own messages. Ask whether you can leave, and who else can run the model. Fifth, when the vendor changes the model: pin a version, and run your test set again before you accept a new one. The last box says: measured on your messages, the model is a row that you can trust. Until then, it is a vendor claim.

Run the model on the same 2,000 test messages as every other row, with the same cost table. Only then can you compare it with the other rows. TF-IDF costs US$296.0 and the fine-tuned encoder US$103.0. A vendor’s benchmark uses other messages, other labels and another cost of a mistake.

A probability is useful only if it is true on your messages. Make the reliability table of Ask for a decision, not for text, and compute the expected calibration error. A model can be well calibrated on the vendor’s tests and badly calibrated on French chat messages about garden furniture.

The vendor’s 70 to 500 ms were measured from its own laptops. Your requests travel from your servers, through your network, at your busy hours. Measure the median and the slowest 10% of requests, as the course does for every hosted row. The chart shows where the claim would sit among the times that the course measured:

A dot chart of milliseconds per message on a log scale, from 0.01 to 10,000, where each step is 10 times slower. From the fastest: TF-IDF + Naive Bayes 0.02 ms; TF-IDF + logistic regression 0.16 ms; the fine-tuned encoder 6.7 ms; embeddings from 2021 with LR 6.8 ms; GLiClass zero-shot 21.3 ms; embeddings from 2026 with LR 26.9 ms; local Qwen3-0.6B route scores 453 ms; the hosted LLM zero-shot 1,200 ms; the hosted LLM few-shot 1,708 ms; the reasoning model on the hard cases 2,040 ms; local Qwen3-0.6B writing JSON 2,128 ms. An orange band from 70 to 500 ms marks the time that the vendor of Jev claims, not measured here.
Milliseconds per message of the course's methods (measured on a laptop CPU, or recorded from the hosted models), on a log scale. The orange band is the vendor's claim for Jev, not a measurement.
Text description of the diagram

The chart shows one dot per method on a log scale of milliseconds per message, from 0.01 to 10,000. Each step on the scale is 10 times slower than the one before. TF-IDF with Naive Bayes takes 0.02 ms and TF-IDF with logistic regression 0.16 ms. The fine-tuned encoder and the 2021 embeddings take about 7 ms. GLiClass takes 21.3 ms, and the 2026 embeddings 26.9 ms. The local Qwen3-0.6B takes 453 ms with route scores and 2,128 ms when it writes JSON. The hosted LLM takes 1,200 ms without examples and 1,708 ms with examples, and the reasoning model 2,040 ms. An orange band from 70 to 500 ms marks the vendor’s claim for Jev. It sits between the local models and the hosted LLMs, and it is not a measurement.

If the claim holds from your servers, Jev would be about 3 to 24 times faster than the hosted LLM. It would still be 10 to 3,000 times slower than the trained models that run on Larkfield’s own computer.

Turn the price into your cost per 1,000 messages. Our LLM prompt with examples is about 717 input tokens. At the vendor’s input price, 1,000 such messages would cost about US$0.03, with free output. That is our arithmetic on a vendor claim. On this inbox, it changes little: the hosted LLM already costs only US$0.088 per 1,000 messages to run. The cost of its mistakes is US$116.0. A cheaper model that makes more mistakes costs more.

Then ask about lock-in: how hard it is to leave. A closed model with no open weights runs only at its vendor. If the vendor changes the price, stops the model or closes, can you move? The trained models of this course are files on Larkfield’s own disk.

A hosted model can change behind the same name. Ask if you can pin a version: call one fixed version until you choose to move. Before you move, run your test set again, and compare the two rows. The course’s hosted rows were recorded once, so they cannot change. A live model can.

Larkfield: the comparison table with one more row

Section titled “Larkfield: the comparison table with one more row”

Priya adds the vendor’s model to the comparison table as an optional row, clearly marked:

Method Accuracy Total $/1k ms/message Label
Fine-tuned MiniLM-L12 encoder 0.980 103.0 6.7 measured
Hosted LLM chat-small, few-shot 0.970 116.1 1,708 recorded
Jev (TypeSafe AI) not measured not measured 70–500 vendor claim, from its article (September 2026)

The row tells the truth: nobody knows where Jev sits on this inbox. It cannot be the “best” row, or the “worst”. To earn real numbers, it needs answers for the validation and test messages, and then it goes through the same columns as everything else.

Your project can already score a model that it cannot run. The recorded methods of Modules 4 and 5 are only files: one CSV and one JSON file per method, in data/predictions/. route compare reads every pair that it finds there. So a new model needs no new code. It needs its answers.

  1. In your project folder larkroute, with .venv active, score every method on the test messages. It takes about 10 seconds:

    Terminal window
    route compare

    It prints one line per method while it scores, then a table of 15 rows, cheapest first. The table starts like this (the ms/msg column changes a little from run to run):

    method accuracy macro-F1 takeover needs person EN FR mistakes $/1k run $/1k total $/1k ms/msg
    fine-tuned MiniLM-L12 multilingual encoder 0.980 0.980 1.000 1.000 0.980 0.978 103.0 0.000 103.0 6.73
    hosted LLM chat-small, few-shot (recorded) 0.970 0.962 1.000 1.000 0.973 0.961 116.0 0.088 116.1 1708.00
    TF-IDF + linear SVM 0.966 0.969 1.000 1.000 0.971 0.954 175.0 0.000 175.0 0.12
  2. Look at how one recorded method is stored: its description, then its first two answers.

    Terminal window
    cat data/predictions/llm-few-shot.json
    head -3 data/predictions/llm-few-shot.csv
    {
    "method": "hosted LLM chat-small, few-shot (recorded)",
    "label": "recorded",
    "model": "chat-small (gpt-6-luna-2026-09-22)",
    "ms_per_message": 1708.0,
    "run_per_1000": 0.0876,
    "messages": 4000,
    "notes": "Recorded once from the Azure deployment chat-small (reasoning effort none); replayed. The confidence is the number the model wrote, not a probability."
    }
    message_id,prediction,confidence
    M-100006,delivery,0.99
    M-100010,product,0.99

To add a new model, you would write the same two files. They hold one answer per validation and test message, the time per message that you measured, and the running cost per 1,000 messages. A decision model also gives a probability per route: add the columns p_orders to p_needs_person, as in data/predictions/local-llm-scores.csv. Then route compare scores it like every other row. This lesson adds no new model: the course has no answers from Jev.

When a purpose-built decision model makes sense

Section titled “When a purpose-built decision model makes sense”

Consider one when you need typed decisions at scale, with a probability, and the hosted LLM is too slow or too costly to call. Consider one also when you cannot train your own model: no labels, or no people to run a model.

Be careful when you already have labels and a trained model that works. On this inbox, the fine-tuned encoder decides in 6.7 ms on a laptop, costs nothing to run, and makes few mistakes. A closed model must beat that on your test set, and you also take on its lock-in. And be careful with sensitive messages: a hosted model sees every message that you send it.

Your turn: apply the checklist to GLiClass

Section titled “Your turn: apply the checklist to GLiClass”

GLiClass is also a model built only for classification. Like a decision model, it scores each label’s name with a probability and writes no text. Its makers describe it as a zero-shot classifier: it needs no training labels. Use the checklist and the numbers of this course to answer: should Larkfield use it as its router?

Show the answer

No, not on this inbox. Go through the checklist:

  1. Your own test set: 0.516 accuracy and US$4,431 per 1,000 messages. That is about three times the cost of a person who reads everything (US$1,500).
  2. Calibration: its expected calibration error on test is 0.089, against 0.050 for TF-IDF + logistic regression. Its probabilities are less true, and its route is wrong half the time.
  3. Time: 21 ms per message on a laptop CPU: fast enough.
  4. Price and lock-in: open weights (Apache-2.0). It runs on Larkfield’s computer at no cost per call, and nobody can take it away.
  5. Changes: a pinned revision on Hugging Face never changes.

It passes questions 3 to 5 and fails the first two, which matter most. Its claim of “zero-shot classification” is true; it is not enough here. It is still useful for one job: finding a new route without retraining. In Zero-shot without an LLM, it found 31 of 80 messages about the new device.

Common mistake

Newer, so better

Symptom: Tomás reads that a new embedding model, from 2026, beats the 2021 one on public benchmarks. He swaps it in without a test. A week later, Grace reports more messages in the wrong queues.

Cause: on Larkfield’s test messages, the newer model is worse. It gets 0.906 against 0.920, and costs US$895 per 1,000 messages against US$779. Loaded as it is stored (bfloat16), it took 26.9 ms against 6.8 ms; loaded in 32-bit numbers, it took 7.0 ms. The public benchmarks measured other tasks, on other text. A newer model is a candidate, not an upgrade.

Fix: treat every new model, a newer version of the same model too, as a new row. Score it on the same test set and cost table. Check its calibration and its time. Keep the old row until the new one wins.

Consequence in AI work: model swaps that skip the test are a common cause of quiet quality drops in production. The comparison table is cheap to rerun. A week of wrong queues is not.

Check your understanding

Jev's vendor claims 0% type errors. Explain why that is not the same as accuracy, with an example from this course.

Show the answer

“0% type errors” means that every answer matches the schema: it is always one of the allowed values. That is true by design, and it says nothing about whether the value is right. In this course, Qwen3-0.6B with Outlines gave a valid route name for 100% of 500 test messages. It gave the right route for only 12.8% of them. A valid answer is not a right answer. Only a test on your own labelled messages measures accuracy.

Check your understanding

A vendor says its model is calibrated and answers in 70 to 500 ms. Which two checks of the checklist test these claims, and what exactly do you measure?

Show the answer

Calibration on your data. Run the model on your validation or test messages, and group the answers by their probability. In each group, compare the mean probability with the share that is right: a reliability table. Then compute the expected calibration error. Time from where you are. Send your own messages from your own servers. Measure the median time and the slowest 10% of requests, at normal and busy hours. The vendor measured from its own laptops.

Check your understanding

Practical check: your project scores recorded methods that it cannot run. Where are they, and what do you need to add a new model as one more row of route compare?

Show the answer

They are in data/predictions/, as one CSV and one JSON file per method. The CSV has one row per validation and test message: message_id, prediction, confidence, and for a model with probabilities the columns p_orders to p_needs_person. The JSON file gives the method’s name and how its answers were made. It also gives the measured milliseconds per message and the running cost per 1,000 messages. With those two files, route compare scores the new model with the same test set and cost table as every other row.

Recap

  • A purpose-built decision model returns one value from a fixed list (the schema), with a probability, and writes no text.
  • Jev (TypeSafe AI, announced in September 2026, early access) is one example. Its speed, price and calibration are vendor claims: the course did not measure it, so its row says “not measured”.
  • “0% type errors” is not accuracy. Outlines gave 100% valid answers and 12.8% right ones.
  • The checklist: your own test set, calibration on your data, time from where you are, price and lock-in, and what happens when the vendor changes the model.
  • A new model joins the project as two files in data/predictions/, and route compare scores it like every other row.

You can now judge any model by what it does on your messages. But the comparison table still asks you to choose one router. Next, you combine them: the cheap model decides when it is sure, the LLM gets the unsure messages, and a person gets the rest.

Next: Cascades