Cascades
- Lesson 26
- 40 min
- Module 6: Decision models, cascades and distillation
Where we are
We have cheap models that are sure about most messages, and an LLM that costs more. In this lesson, we build a cascade: the cheap model decides when it is sure, the LLM gets only the unsure messages, and a person gets the rest. We choose the two thresholds on validation data.
Lesson 26 of 37 · Text Classification and Routing: Send Each Message to the Right PlaceCascades
1. The routing problem and its cost
2. Labels you can trust
3. Baselines: rules and classical models
4. Embeddings and small fine-tuned models
5. LLMs as classifiers
6. Decision models, cascades and distillation
7. Harder label shapes
8. Confidence and people in the loop
9. Run it in production
What you will be able to do
Use the cheap model for the messages it is sure about, the LLM only for the unsure ones, and a person for the rest. Choose the two thresholds on the validation messages, so the cost per 1,000 messages is as low as you can honestly get.
NotebookDo this lesson in a notebook
The notebook has every code cell, task and check of this lesson. It trains TF-IDF + logistic regression, replays the LLM’s recorded answers and builds the cascade. Then it chooses the two thresholds on the validation messages and scores the choice once on test. Its setup cells install pandas and scikit-learn and download the practice data, so it needs no local setup and no key. About 30 minutes.
Colab and Kaggle open your own copy. They need a free Google or Kaggle account. Stuck? See the solution notebook.
Before you startWhat to know and to have ready
What you need to know
- TF-IDF + logistic regression and its probabilities, from Naive Bayes, logistic regression and linear SVM.
- The LLM with examples and its written confidence, from Few-shot examples and label descriptions.
- A threshold that sends unsure messages to review, from The cost of each mistake.
What you need on your computer (only for the project step)
- Your project folder
larkroutefrom Ask for a decision, not for text, with.venvactive:python -m pytest -qprints23 passed. - The practice files
nextia-learning-labsnext to your project folder, as in Meet the inbox.
Tested on macOS with Python 3.12, pandas 3.1.1 and scikit-learn 1.9.1. The Windows commands are not tested.
A fast clerk, an expert and the supervisor’s desk
Section titled “A fast clerk, an expert and the supervisor’s desk”Back at the post office. Most letters have a clear address, and a fast clerk sorts them in a second. Some letters are hard to read. The fast clerk puts them in a tray for the expert, who is slower and more expensive, but reads bad handwriting well. A few letters puzzle even the expert. They go to the supervisor’s desk, and she decides herself.
Nobody asks the expert to sort every letter. That would be slow, and the expert would also make some mistakes on easy letters that the fast clerk gets right. Nobody asks the fast clerk to guess on every letter either. The trick is that each person knows when they are not sure, and passes the letter on.
The problem: one router for every message
Section titled “The problem: one router for every message”Larkfield has a fast clerk and an expert. The fast clerk is TF-IDF + logistic regression: about 0.1 milliseconds per message, on any laptop. The expert is the hosted LLM with examples: about 1.7 seconds per message, and each message leaves Larkfield’s computers. On this inbox, each one alone has a weak point.
| One router for everything | Accuracy | Cost per 1,000 test messages |
|---|---|---|
| TF-IDF + logistic regression | 0.962 | US$296.0 |
| The LLM with examples (recorded) | 0.970 | US$116.1 |
TF-IDF is cheap, but its mistakes are expensive. Where do they come from? Look at the messages that it is not sure about. With a “sure” threshold of 0.80, TF-IDF is sure about 1,718 of the 2,000 test messages and unsure about 282. Compare the two routers on each group:
| Test messages | TF-IDF wrong | LLM wrong | Cost of TF-IDF’s mistakes | Cost of the LLM’s mistakes |
|---|---|---|---|---|
| TF-IDF is sure (1,718) | 8 | 24 | US$32 | US$64 |
| TF-IDF is unsure (282) | 68 | 37 | US$560 | US$168 |
This is the whole idea in one table. Where TF-IDF is sure, it is right more often than the LLM. Where it is unsure, the LLM is right more often. Neither router is best everywhere.
The LLM for every message has other costs that the table does not show. It makes 1,000 calls per 1,000 messages. A busy morning can hit the provider’s rate limit, the most calls that it accepts per minute. Every customer waits about 1.7 seconds or more. And every message leaves the company, which some customers and some laws do not accept.
The idea: a cascade
Section titled “The idea: a cascade”A cascade is a chain of routers, from the cheapest to the most expensive. Each router decides only the messages that it is sure about, and passes the rest down to the next one. The last step is a person. The name comes from a waterfall that falls in steps: the messages flow down, and each step keeps some of them.
Larkfield’s cascade has three steps and two thresholds:
- TF-IDF decides when its top probability is at least
sure. - Otherwise, the LLM decides when its written confidence is at least
llm_sure. - Otherwise, a person reads the message (a review, US$1.50).
The written confidence of the LLM is a number in its text, not a measured probability. In Module 5, we checked it on this inbox: with examples, it was close to the truth, with an expected calibration error of 0.008. Without that check, a threshold on it would mean nothing.
A tiny worked example: three messages
Section titled “A tiny worked example: three messages”Here are three real test messages through the cascade, with sure = 0.80 and llm_sure = 0.95:
| Message | Label | TF-IDF | LLM | Who decides |
|---|---|---|---|---|
M-100001 “Est-ce que le livreur pourrait laisser mon colis … chez ma voisine ?” |
delivery | delivery, 1.000 | delivery, 1.0 | TF-IDF: 1.000 ≥ 0.80 |
M-100561 “Please cancel order LK-472452 … I also need an invoice, and SPRING15 did not work …” |
orders | orders, 0.593 | orders, 0.99 | the LLM: 0.593 < 0.80, and 0.99 ≥ 0.95 |
M-100005 “Hello, just following up on my last email about LK-300801 …” |
delivery | orders, 0.416 | delivery, 0.62 | a person: both are unsure |
The first message is easy: TF-IDF decides, and the LLM is never called. The second asks for three things, and TF-IDF is torn between them. The LLM follows the guideline (the first request decides) and is sure. The third gives no topic at all: “that arrangement” is in an earlier email. TF-IDF guesses orders and the LLM guesses delivery, both without confidence. A person can open the earlier email; a model cannot.
The cascade on all 2,000 test messages
Section titled “The cascade on all 2,000 test messages”Text description of the diagram
The diagram is a flow from top to bottom. 2,000 test messages arrive at the first question: is the top probability of TF-IDF + logistic regression 0.80 or more? If yes, TF-IDF decides. That happens for 1,718 messages, 85.9%; 1,710 of them are right, and the 8 mistakes cost US$32. If no, the 282 other messages go to the second question: did the LLM with examples write a confidence of 0.95 or more? If yes, the LLM decides: 199 messages, 10.0%, of which 189 are right; the mistakes cost US$32. If no, the 83 last messages (4.2%) go to a person, at US$1.50 each. That costs US$124.50. At the bottom, the total for the 2,000 messages: 32 + 32 + 124.50 = US$188.50 of mistakes and reviews. That is US$94.3 per 1,000 messages. The LLM calls add only US$0.012 per 1,000.
The cascade costs US$94.3 per 1,000 test messages: the lowest row of the course’s comparison table so far. It calls the LLM for 141 messages in 1,000, not 1,000. The decided messages are 99.1% right.
Larkfield’s cascade: choose the thresholds on validation
Section titled “Larkfield’s cascade: choose the thresholds on validation”Where do 0.80 and 0.95 come from? Priya does not guess them, and she does not take them from the test messages. She tries every pair on a grid, on the validation messages, and keeps the cheapest pair. On the grid, sure goes from 0.30 to 1.00 in steps of 0.05. llm_sure is either 0 (the LLM decides everything it gets) or a value on the same grid.
Here is a small part of the search, in US$ per 1,000 validation messages:
| sure ↓ / llm_sure → | 0 (the LLM decides all it gets) | 0.80 | 0.90 | 0.95 |
|---|---|---|---|---|
| 0.50 | 433.0 | 400.8 | 400.8 | 385.3 |
| 0.70 | 300.0 | 233.3 | 237.5 | 224.0 |
| 0.80 | 176.0 | 108.3 | 117.8 | 103.8 |
| 0.90 | 179.0 | 114.3 | 116.0 | 109.5 |
The cheapest pair is sure = 0.80 and llm_sure = 0.95, at US$103.8 on validation. Only then does Priya score it on the test messages, once. The result is US$94.3. Validation costs more because its messages happen to hold more expensive mistakes. Decide, review or send to a person comes back to this noise.
Where the US$94.3 comes from
Section titled “Where the US$94.3 comes from”Tomás smiles: “So the cascade saves money because it calls the LLM less.” The data says no. At these prices, the LLM’s running cost is tiny. It is US$0.088 per 1,000 messages for the LLM alone, and US$0.012 in the cascade. The saving is US$0.076 per 1,000 messages. The real gain is in fewer expensive mistakes. Add the steps one at a time:
- Validation
- Test
Text description of the chart
A column chart with three pairs of columns, one for validation and one for test. The LLM for every message: 189.1 on validation and 116.1 on test. TF-IDF first, then the LLM for the rest, with no person: 176.0 on validation and 100.0 on test. The full cascade, with a person for the messages that the LLM is not sure about: 103.8 on validation and 94.3 on test.
- TF-IDF first removes mistakes, because TF-IDF is right more often than the LLM on the messages it is sure about. On test, this step brings 116.1 down to 100.0; on validation, 189.1 to 176.0.
- The person step sends the 4.2% that both models are unsure about to a review. On validation, where Priya chose, this is most of the gain: 176.0 to 103.8. On test, it brings 100.0 to 94.3.
So say it honestly. The cascade beats the LLM alone because of where it sends each message. Above all, a person reads the few messages that nobody is sure about. It does not win by saving tokens.
Explore the cascade
Section titled “Explore the cascade”The explorer uses the same recordings on the 2,000 test messages. Move the “sure” threshold of TF-IDF. Then change what the LLM does with the messages it gets. It can decide them all, or decide only at 0.95 and send the rest to a person.
Explore
A cheap model, an LLM and a person
Predict first: The threshold starts at 0.30, where TF-IDF decides almost every message, and the LLM decides all it gets. As you raise the threshold to 1.00, more messages go to the LLM. Does the cost fall all the way to the LLM-for-everything line, or is there a lowest point before it?
Where the 2,000 test messages go
- LLM decides all it gets
- LLM decides at ≥ 0.95, else a person
- TF-IDF decides
- 99.7%
- The LLM decides
- 0.3%
- A person reads it
- 0.0%
- Accuracy of the decided
- 96.3%
- Mistakes and reviews per 1,000
- US$293.0
- Running cost per 1,000
- US$0.000
- Total per 1,000 messages
- US$293.0
At sure 0.30, TF-IDF decides 99.7% of the messages, the LLM decides 0.3%, and 0.0% go to a person. The decided messages are 96.3% right. Mistakes and reviews cost US$293.0 and running costs US$0.000: US$293.0 per 1,000 messages. On this curve, the lowest total is US$97.0 per 1,000, at 0.85.
Text description of the explorer
The explorer has a slider for the “sure” threshold of TF-IDF, from 0.30 to 1.00 in steps of 0.01. Two radio buttons choose what the LLM does. It decides everything it gets, or it decides only when it wrote 0.95 or more and sends the rest to a person. Three bars show where the 2,000 test messages go: decided by TF-IDF, decided by the LLM, and read by a person. A line chart shows the total cost per 1,000 messages against the threshold, with one line for each choice of the LLM. Two dashed reference lines show the LLM for everything at US$116.1 and TF-IDF alone at US$296.0. A dashed vertical line marks 0.80, the threshold chosen on validation. Tiles give the exact values. They show the three shares, the accuracy of the decided messages, the cost of mistakes and reviews, the running cost and the total. At 0.30, with the LLM deciding all it gets, TF-IDF decides 99.7% and the total is US$293.0. The total drops by US$102 between 0.45 and 0.46, when one account takeover moves from TF-IDF to the LLM. From 0.73 to 0.90, it stays between US$97 and US$102, with its lowest value, US$97.0, at 0.85. It rises to US$116.1 at 1.00, where the LLM decides everything. With the person step, the line is lower almost everywhere. It is US$94.3 at 0.80. At 1.00, the LLM gets every message, 6.35% go to a person, and the total is US$133.3.
What the explorer shows
Section titled “What the explorer shows”Start at 0.30, with “LLM decides all it gets”. TF-IDF decides 99.7% of the messages, and the total is US$293.0, almost TF-IDF alone. Now raise the threshold. Between 0.45 and 0.46, the total falls by US$102. One account takeover leaves TF-IDF’s decisions and goes to the LLM, which routes it right. One message is worth US$200 here.
The answer to the prediction: there is a lowest point before the end. From 0.73 to 0.90, the total stays between US$97 and US$102. Its lowest value is US$97.0, at 0.85. At 1.00, the LLM decides everything, and the total rises to US$116.1. Above 0.90, more messages that TF-IDF gets right go to the LLM. On those messages, the LLM is wrong more often than TF-IDF.
Now switch to “LLM decides at ≥ 0.95, else a person”. At 0.80, the bars show 85.9%, 10.0% and 4.2%. The total is US$94.3. Move to 1.00: TF-IDF decides nothing, and 6.35% go to a person, at US$133.3. Without TF-IDF in front, the LLM is unsure about more messages, and the reviews cost more.
One more thing to see. On this test chart, the lowest point of the second line is US$89.0, at 0.73. It is not at 0.80. That is the test set’s own noise, and we do not move the choice for it. The failure case below shows why.
Guided practice: build the cascade
Section titled “Guided practice: build the cascade”- Open the notebook from the box Do this lesson in a notebook at the top of this page.
- Run the setup cells. The data cell ends with
Ready: 10000 messages (6000 train, 2000 validation, 2000 test), all synthetic. - In section 1, run the cell. You see
hosted LLM chat-small, few-shot (recorded) - 0.0876 dollars per 1,000 messages. - In section 2, run the cell. You see the three messages of the worked example, with the step that decided each one.
- In section 3, write your prediction. Then run the cell. You see
{'tfidf': 1702, 'llm': 213, 'person': 85}and103.8. - In section 4, run the search. The best pair is
sure0.8 andllm_sure0.95, at103.8. - In section 5, run the test cell once. You see
94.3dollars per 1,000 and an accuracy of0.991.
Add the cascade to the project
Section titled “Add the cascade to the project”Now add the cascade to your project. It is two files: larkroute/cascade.py (the cascade and its search) and the command larkroute/commands/cascade.py. The project finds every command by its file, so no other file changes.
-
In your project folder
larkroute, with.venvactive, copy the cascade, its command, its test and the new README from the snapshotend-of-m06:Terminal window S=../nextia-learning-labs/C36/snapshots/end-of-m06cp "$S/larkroute/cascade.py" larkroute/cp "$S/larkroute/commands/cascade.py" larkroute/commands/cp "$S/tests/test_cascade.py" tests/cp "$S/README.md" .Terminal window $S = "..\nextia-learning-labs\C36\snapshots\end-of-m06"Copy-Item "$S\larkroute\cascade.py" larkroute\Copy-Item "$S\larkroute\commands\cascade.py" larkroute\commands\Copy-Item "$S\tests\test_cascade.py" tests\Copy-Item "$S\README.md" . -
Run the tests. The last line is
24 passed. Your project is now the same as the snapshotend-of-m06.Terminal window python -m pytest -q -
Choose the two thresholds on the validation messages, and score the choice on test. It takes about 5 seconds:
Terminal window route cascade --searchChosen on the validation messages: sure 0.80, LLM sure 0.95Test messages: 85.9% decided by TF-IDF, 10.0% by the LLM (few-shot, recorded), 4.2% to a person. Accuracy of the decided messages 99.1%.Cost per 1,000 messages: mistakes and reviews 94.2 dollars, running 0.012 dollars, total 94.3 dollars. -
Score the same first step with no person: the LLM decides every message that it gets.
Terminal window route cascade --sure 0.8 --llm-sure 0Test messages: 85.9% decided by TF-IDF, 14.1% by the LLM (few-shot, recorded), 0.0% to a person. Accuracy of the decided messages 97.8%.Cost per 1,000 messages: mistakes and reviews 100.0 dollars, running 0.012 dollars, total 100.0 dollars.
route cascade replays the recorded answers of the LLM, so it needs no key, and no message leaves your computer. Use the default mode, few-shot. The recordings of the zero-shot mode cover only the test messages. So --mode zero-shot works with thresholds that you give (--sure and --llm-sure), but not with --search, which needs the validation messages: it then says so and stops.
When to use a cascade, and what it costs
Section titled “When to use a cascade, and what it costs”Use a cascade when one cheap model is sure about most messages and is right when it is sure. On this inbox, TF-IDF is sure about 85.9% of the messages and wrong on only 8 of them. Then an expensive step is worth it for the rest.
Do not use one when the cheap model is rarely sure, or wrong when it is sure. Then the first step only adds a delay and a place for mistakes. Check this on validation before you build anything: count the messages above the threshold, and the cheap model’s mistakes among them. Also, if one model is already the cheapest row everywhere, you may not need the others. On this inbox, the fine-tuned encoder alone costs US$103.0, close to the cascade.
What a cascade costs you:
- Two thresholds to choose, and to choose again. A new model version, a new LLM or a new product moves both. Keep the search as code, and run it on the validation messages of each version.
- More parts to run. Two models, an LLM client with retries, and a review queue. Each part can fail. Decide what happens when the LLM is down: here, its messages could go to a person.
- People’s time. The person step sends 42 messages in 1,000 to Grace’s team, about 3 a day at Larkfield’s 65 messages a day. Check the load with her before you promise it.
- Uneven waits. 85.9% of the customers wait a fraction of a millisecond. The others wait for the LLM, about 1.7 seconds, or for a person, a few hours. The mean LLM wait per message falls from 2.68 seconds (the LLM for all) to 0.39 seconds.
Your turn: the same cascade without the LLM
Section titled “Your turn: the same cascade without the LLM”Change one thing: keep TF-IDF at sure = 0.80, but send every message it is not sure about straight to a person. There is no LLM. In section 6 of the notebook, compute the cost on the validation messages and put it in no_llm_cost. Before you run it, predict: is the LLM step worth its calls?
Show the answer
r, s = cascade(vr, vp, lv_r, lv_c, 0.80, 2.0)no_llm_cost = cascade_cost(validation, r, s)print(no_llm_cost)235.5llm_sure = 2.0 is above any written confidence, so every LLM answer goes to a person. Without the LLM, 298 validation messages go to a person, and the cost is US$235.5, against US$103.8 with the LLM. On test, it is US$227.5 against US$94.3. The LLM is worth its calls. It is right on most of the messages that TF-IDF is unsure about, and a call costs far less than a review.
Common mistake
A better threshold, found on the test messages
Symptom: Tomás tries every “sure” threshold from 0.30 to 1.00, in steps of 0.01, on the test messages. “Look: at 0.73, the cascade costs US$89.0, not US$94.3. Let’s use 0.73.”
Cause: 0.73 is the lowest point of the test messages, so it fits their noise. On the validation messages, the same threshold costs US$225.8, against US$103.8 at 0.80. Near the bottom of the curve, a few expensive messages decide where the lowest point falls. The US$89.0 is a number that the next month will not show.
Fix: choose on the validation messages. Report the test number once, even when a lower one is visible. For a better choice, get more evidence. Use more labelled messages, or the same search with cross-validation on the training messages, as in Train, validation and test sets.
Consequence in AI work: a threshold tuned on the test set looks like a free gain in the report. In production, it turns into a surprise. Two thresholds give twice as many ways to fit the noise.
Knowledge check
Section titled “Knowledge check”Check your understanding
The cascade costs US$94.3 per 1,000 test messages, and the LLM for everything US$116.1. Tomás says the gain comes from fewer LLM calls. Use the numbers to explain where it really comes from.
Show the answer
Not from the calls. The LLM costs US$0.088 per 1,000 messages alone and US$0.012 in the cascade, a saving of US$0.076. The gain comes from fewer expensive mistakes. On the messages that TF-IDF is sure about, it makes 8 mistakes where the LLM makes 24. Putting TF-IDF first brings 116.1 down to 100.0 on test. Then a person reads the 4.2% that both models are unsure about. That brings it to 94.3. On validation, where the thresholds were chosen, the person step is most of the gain: 176.0 to 103.8.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
Predict: in the explorer, with the LLM deciding everything it gets, what happens to the total cost as you raise the sure threshold from 0.85 to 1.00, and why?
Show the answer
It rises, from US$97.0 to US$116.1. Above 0.85, more of the messages that TF-IDF would get right go to the LLM instead. On those messages, the LLM is wrong more often than TF-IDF, so the mistakes cost more. The running cost also rises, but by less than 0.1 dollar per 1,000 messages. The lowest point is before the end, not at “the LLM for everything”.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
Practical check: your project now has the end-of-m06 stage. Which command chooses the two thresholds, which thresholds does it choose, and what share of the test messages goes to a person?
Show the answer
route cascade --search. It chooses the thresholds on the validation messages: sure 0.80 for TF-IDF and 0.95 for the LLM’s written confidence. On the test messages, TF-IDF decides 85.9%, the LLM decides 10.0%, and 4.2% go to a person. The total is US$94.3 per 1,000 messages. python -m pytest -q prints 24 passed.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Recap
- A cascade is a chain of routers from cheap to expensive. Each one decides what it is sure about and passes the rest down. A person is the last step.
- On this inbox, TF-IDF is sure about 85.9% of the test messages and wrong on 8 of them. The LLM decides 10.0%, and a person reads 4.2%.
- Choose the two thresholds on validation: 0.80 and 0.95 here (US$103.8). Score them once on test: US$94.3 per 1,000, the lowest row so far. The LLM for everything costs US$116.1, and TF-IDF alone US$296.0.
- The gain is not from saved tokens (US$0.076 per 1,000). It comes from fewer expensive mistakes, and above all from the few messages that a person reads.
- Your project is at the stage
end-of-m06: 24 tests,route llmandroute cascade.
You can now give the LLM only the messages that need it. But the cascade still calls the LLM, and sends 14.1% of the messages out of Larkfield. Next, you let the LLM teach a cheap model once, and see what the student learns, the mistakes included.