Skip to content
Français

Cascades

  • Lesson 26
  • 40 min
  • Module 6: Decision models, cascades and distillation

Where we are

We have cheap models that are sure about most messages, and an LLM that costs more. In this lesson, we build a cascade: the cheap model decides when it is sure, the LLM gets only the unsure messages, and a person gets the rest. We choose the two thresholds on validation data.

Plays from YouTube. Nothing loads from YouTube until you press play. Privacy
Lesson 26 of 37 · Text Classification and Routing: Send Each Message to the Right PlaceCascades

What you will be able to do

Use the cheap model for the messages it is sure about, the LLM only for the unsure ones, and a person for the rest. Choose the two thresholds on the validation messages, so the cost per 1,000 messages is as low as you can honestly get.

NotebookDo this lesson in a notebookColab · Kaggle · GitHub

The notebook has every code cell, task and check of this lesson. It trains TF-IDF + logistic regression, replays the LLM’s recorded answers and builds the cascade. Then it chooses the two thresholds on the validation messages and scores the choice once on test. Its setup cells install pandas and scikit-learn and download the practice data, so it needs no local setup and no key. About 30 minutes.

Colab and Kaggle open your own copy. They need a free Google or Kaggle account. Stuck? See the solution notebook.

Before you startWhat to know and to have readyAbout 40 minutes

What you need to know

What you need on your computer (only for the project step)

Tested on macOS with Python 3.12, pandas 3.1.1 and scikit-learn 1.9.1. The Windows commands are not tested.

A fast clerk, an expert and the supervisor’s desk

Section titled “A fast clerk, an expert and the supervisor’s desk”

Back at the post office. Most letters have a clear address, and a fast clerk sorts them in a second. Some letters are hard to read. The fast clerk puts them in a tray for the expert, who is slower and more expensive, but reads bad handwriting well. A few letters puzzle even the expert. They go to the supervisor’s desk, and she decides herself.

Nobody asks the expert to sort every letter. That would be slow, and the expert would also make some mistakes on easy letters that the fast clerk gets right. Nobody asks the fast clerk to guess on every letter either. The trick is that each person knows when they are not sure, and passes the letter on.

Larkfield has a fast clerk and an expert. The fast clerk is TF-IDF + logistic regression: about 0.1 milliseconds per message, on any laptop. The expert is the hosted LLM with examples: about 1.7 seconds per message, and each message leaves Larkfield’s computers. On this inbox, each one alone has a weak point.

One router for everything Accuracy Cost per 1,000 test messages
TF-IDF + logistic regression 0.962 US$296.0
The LLM with examples (recorded) 0.970 US$116.1

TF-IDF is cheap, but its mistakes are expensive. Where do they come from? Look at the messages that it is not sure about. With a “sure” threshold of 0.80, TF-IDF is sure about 1,718 of the 2,000 test messages and unsure about 282. Compare the two routers on each group:

Test messages TF-IDF wrong LLM wrong Cost of TF-IDF’s mistakes Cost of the LLM’s mistakes
TF-IDF is sure (1,718) 8 24 US$32 US$64
TF-IDF is unsure (282) 68 37 US$560 US$168

This is the whole idea in one table. Where TF-IDF is sure, it is right more often than the LLM. Where it is unsure, the LLM is right more often. Neither router is best everywhere.

The LLM for every message has other costs that the table does not show. It makes 1,000 calls per 1,000 messages. A busy morning can hit the provider’s rate limit, the most calls that it accepts per minute. Every customer waits about 1.7 seconds or more. And every message leaves the company, which some customers and some laws do not accept.

A cascade is a chain of routers, from the cheapest to the most expensive. Each router decides only the messages that it is sure about, and passes the rest down to the next one. The last step is a person. The name comes from a waterfall that falls in steps: the messages flow down, and each step keeps some of them.

Larkfield’s cascade has three steps and two thresholds:

  1. TF-IDF decides when its top probability is at least sure.
  2. Otherwise, the LLM decides when its written confidence is at least llm_sure.
  3. Otherwise, a person reads the message (a review, US$1.50).

The written confidence of the LLM is a number in its text, not a measured probability. In Module 5, we checked it on this inbox: with examples, it was close to the truth, with an expected calibration error of 0.008. Without that check, a threshold on it would mean nothing.

Here are three real test messages through the cascade, with sure = 0.80 and llm_sure = 0.95:

Message Label TF-IDF LLM Who decides
M-100001 “Est-ce que le livreur pourrait laisser mon colis … chez ma voisine ?” delivery delivery, 1.000 delivery, 1.0 TF-IDF: 1.000 ≥ 0.80
M-100561 “Please cancel order LK-472452 … I also need an invoice, and SPRING15 did not work …” orders orders, 0.593 orders, 0.99 the LLM: 0.593 < 0.80, and 0.99 ≥ 0.95
M-100005 “Hello, just following up on my last email about LK-300801 …” delivery orders, 0.416 delivery, 0.62 a person: both are unsure

The first message is easy: TF-IDF decides, and the LLM is never called. The second asks for three things, and TF-IDF is torn between them. The LLM follows the guideline (the first request decides) and is sure. The third gives no topic at all: “that arrangement” is in an earlier email. TF-IDF guesses orders and the LLM guesses delivery, both without confidence. A person can open the earlier email; a model cannot.

A flow from top to bottom. 2,000 test messages go to the question: TF-IDF + logistic regression, top probability 0.80 or more? Yes: TF-IDF decides 1,718 messages (85.9%), 1,710 right, mistakes US$32. No: 282 messages go to the question: the LLM with examples, written confidence 0.95 or more? Yes: the LLM decides 199 (10.0%), 189 right, mistakes US$32. No: 83 messages: a person reads them, 4.2%, at US$1.50 each, US$124.50. The total for 2,000 messages is 32 + 32 + 124.50 = US$188.50 of mistakes and reviews, which is US$94.3 per 1,000 messages; the LLM calls add US$0.012.
The cascade chosen on the validation messages, on Larkfield's 2,000 test messages (measured on recordings). Each step decides what it is sure about and passes the rest down.
Text description of the diagram

The diagram is a flow from top to bottom. 2,000 test messages arrive at the first question: is the top probability of TF-IDF + logistic regression 0.80 or more? If yes, TF-IDF decides. That happens for 1,718 messages, 85.9%; 1,710 of them are right, and the 8 mistakes cost US$32. If no, the 282 other messages go to the second question: did the LLM with examples write a confidence of 0.95 or more? If yes, the LLM decides: 199 messages, 10.0%, of which 189 are right; the mistakes cost US$32. If no, the 83 last messages (4.2%) go to a person, at US$1.50 each. That costs US$124.50. At the bottom, the total for the 2,000 messages: 32 + 32 + 124.50 = US$188.50 of mistakes and reviews. That is US$94.3 per 1,000 messages. The LLM calls add only US$0.012 per 1,000.

The cascade costs US$94.3 per 1,000 test messages: the lowest row of the course’s comparison table so far. It calls the LLM for 141 messages in 1,000, not 1,000. The decided messages are 99.1% right.

Larkfield’s cascade: choose the thresholds on validation

Section titled “Larkfield’s cascade: choose the thresholds on validation”

Where do 0.80 and 0.95 come from? Priya does not guess them, and she does not take them from the test messages. She tries every pair on a grid, on the validation messages, and keeps the cheapest pair. On the grid, sure goes from 0.30 to 1.00 in steps of 0.05. llm_sure is either 0 (the LLM decides everything it gets) or a value on the same grid.

Here is a small part of the search, in US$ per 1,000 validation messages:

sure ↓ / llm_sure → 0 (the LLM decides all it gets) 0.80 0.90 0.95
0.50 433.0 400.8 400.8 385.3
0.70 300.0 233.3 237.5 224.0
0.80 176.0 108.3 117.8 103.8
0.90 179.0 114.3 116.0 109.5

The cheapest pair is sure = 0.80 and llm_sure = 0.95, at US$103.8 on validation. Only then does Priya score it on the test messages, once. The result is US$94.3. Validation costs more because its messages happen to hold more expensive mistakes. Decide, review or send to a person comes back to this noise.

Tomás smiles: “So the cascade saves money because it calls the LLM less.” The data says no. At these prices, the LLM’s running cost is tiny. It is US$0.088 per 1,000 messages for the LLM alone, and US$0.012 in the cascade. The saving is US$0.076 per 1,000 messages. The real gain is in fewer expensive mistakes. Add the steps one at a time:

  • Validation
  • Test
The cascade built one step at a time (measured on recordings). Most of the gain over the LLM alone comes from the person step on validation, and from TF-IDF's sure decisions on test. Almost none comes from saved LLM calls.
Text description of the chart

A column chart with three pairs of columns, one for validation and one for test. The LLM for every message: 189.1 on validation and 116.1 on test. TF-IDF first, then the LLM for the rest, with no person: 176.0 on validation and 100.0 on test. The full cascade, with a person for the messages that the LLM is not sure about: 103.8 on validation and 94.3 on test.

  • TF-IDF first removes mistakes, because TF-IDF is right more often than the LLM on the messages it is sure about. On test, this step brings 116.1 down to 100.0; on validation, 189.1 to 176.0.
  • The person step sends the 4.2% that both models are unsure about to a review. On validation, where Priya chose, this is most of the gain: 176.0 to 103.8. On test, it brings 100.0 to 94.3.

So say it honestly. The cascade beats the LLM alone because of where it sends each message. Above all, a person reads the few messages that nobody is sure about. It does not win by saving tokens.

The explorer uses the same recordings on the 2,000 test messages. Move the “sure” threshold of TF-IDF. Then change what the LLM does with the messages it gets. It can decide them all, or decide only at 0.95 and send the rest to a person.

Explore

A cheap model, an LLM and a person

Where the 2,000 test messages go

TF-IDF decidesTF-IDF decides: 99.7%99.7%The LLM decidesThe LLM decides: 0.3%0.3%A person reads itA person reads it: 0.0%0.0%
TF-IDF decidesTF-IDF decides: 99.7%99.7%The LLM decidesThe LLM decides: 0.3%0.3%A person reads itA person reads it: 0.0%0.0%
  • LLM decides all it gets
  • LLM decides at ≥ 0.95, else a person
0100200300Total dollars per 1,000 messages0.30.40.50.60.70.80.91TF-IDF "sure" threshold (top probability)LLM for everything: US$116.1TF-IDF alone: US$296.0chosen on validation: 0.80sure 0.30: US$293.0
0100200300Total dollars per 1,000 messages0.40.60.81TF-IDF "sure" threshold (top probability)LLM for everything: US$116.1TF-IDF alone: US$296.0chosen on validation: 0.80sure 0.30: US$293.0
TF-IDF decides
99.7%
The LLM decides
0.3%
A person reads it
0.0%
Accuracy of the decided
96.3%
Mistakes and reviews per 1,000
US$293.0
Running cost per 1,000
US$0.000
Total per 1,000 messages
US$293.0

At sure 0.30, TF-IDF decides 99.7% of the messages, the LLM decides 0.3%, and 0.0% go to a person. The decided messages are 96.3% right. Mistakes and reviews cost US$293.0 and running costs US$0.000: US$293.0 per 1,000 messages. On this curve, the lowest total is US$97.0 per 1,000, at 0.85.

TF-IDF + logistic regression first, the recorded answers of the LLM with examples second, a person last, on Larkfield's 2,000 test messages (measured on recordings). The costs are the course's assumptions.
Text description of the explorer

The explorer has a slider for the “sure” threshold of TF-IDF, from 0.30 to 1.00 in steps of 0.01. Two radio buttons choose what the LLM does. It decides everything it gets, or it decides only when it wrote 0.95 or more and sends the rest to a person. Three bars show where the 2,000 test messages go: decided by TF-IDF, decided by the LLM, and read by a person. A line chart shows the total cost per 1,000 messages against the threshold, with one line for each choice of the LLM. Two dashed reference lines show the LLM for everything at US$116.1 and TF-IDF alone at US$296.0. A dashed vertical line marks 0.80, the threshold chosen on validation. Tiles give the exact values. They show the three shares, the accuracy of the decided messages, the cost of mistakes and reviews, the running cost and the total. At 0.30, with the LLM deciding all it gets, TF-IDF decides 99.7% and the total is US$293.0. The total drops by US$102 between 0.45 and 0.46, when one account takeover moves from TF-IDF to the LLM. From 0.73 to 0.90, it stays between US$97 and US$102, with its lowest value, US$97.0, at 0.85. It rises to US$116.1 at 1.00, where the LLM decides everything. With the person step, the line is lower almost everywhere. It is US$94.3 at 0.80. At 1.00, the LLM gets every message, 6.35% go to a person, and the total is US$133.3.

Start at 0.30, with “LLM decides all it gets”. TF-IDF decides 99.7% of the messages, and the total is US$293.0, almost TF-IDF alone. Now raise the threshold. Between 0.45 and 0.46, the total falls by US$102. One account takeover leaves TF-IDF’s decisions and goes to the LLM, which routes it right. One message is worth US$200 here.

The answer to the prediction: there is a lowest point before the end. From 0.73 to 0.90, the total stays between US$97 and US$102. Its lowest value is US$97.0, at 0.85. At 1.00, the LLM decides everything, and the total rises to US$116.1. Above 0.90, more messages that TF-IDF gets right go to the LLM. On those messages, the LLM is wrong more often than TF-IDF.

Now switch to “LLM decides at ≥ 0.95, else a person”. At 0.80, the bars show 85.9%, 10.0% and 4.2%. The total is US$94.3. Move to 1.00: TF-IDF decides nothing, and 6.35% go to a person, at US$133.3. Without TF-IDF in front, the LLM is unsure about more messages, and the reviews cost more.

One more thing to see. On this test chart, the lowest point of the second line is US$89.0, at 0.73. It is not at 0.80. That is the test set’s own noise, and we do not move the choice for it. The failure case below shows why.

  1. Open the notebook from the box Do this lesson in a notebook at the top of this page.
  2. Run the setup cells. The data cell ends with Ready: 10000 messages (6000 train, 2000 validation, 2000 test), all synthetic.
  3. In section 1, run the cell. You see hosted LLM chat-small, few-shot (recorded) - 0.0876 dollars per 1,000 messages.
  4. In section 2, run the cell. You see the three messages of the worked example, with the step that decided each one.
  5. In section 3, write your prediction. Then run the cell. You see {'tfidf': 1702, 'llm': 213, 'person': 85} and 103.8.
  6. In section 4, run the search. The best pair is sure 0.8 and llm_sure 0.95, at 103.8.
  7. In section 5, run the test cell once. You see 94.3 dollars per 1,000 and an accuracy of 0.991.

Now add the cascade to your project. It is two files: larkroute/cascade.py (the cascade and its search) and the command larkroute/commands/cascade.py. The project finds every command by its file, so no other file changes.

  1. In your project folder larkroute, with .venv active, copy the cascade, its command, its test and the new README from the snapshot end-of-m06:

    Terminal window
    S=../nextia-learning-labs/C36/snapshots/end-of-m06
    cp "$S/larkroute/cascade.py" larkroute/
    cp "$S/larkroute/commands/cascade.py" larkroute/commands/
    cp "$S/tests/test_cascade.py" tests/
    cp "$S/README.md" .
  2. Run the tests. The last line is 24 passed. Your project is now the same as the snapshot end-of-m06.

    Terminal window
    python -m pytest -q
  3. Choose the two thresholds on the validation messages, and score the choice on test. It takes about 5 seconds:

    Terminal window
    route cascade --search
    Chosen on the validation messages: sure 0.80, LLM sure 0.95
    Test messages: 85.9% decided by TF-IDF, 10.0% by the LLM (few-shot, recorded), 4.2% to a person. Accuracy of the decided messages 99.1%.
    Cost per 1,000 messages: mistakes and reviews 94.2 dollars, running 0.012 dollars, total 94.3 dollars.
  4. Score the same first step with no person: the LLM decides every message that it gets.

    Terminal window
    route cascade --sure 0.8 --llm-sure 0
    Test messages: 85.9% decided by TF-IDF, 14.1% by the LLM (few-shot, recorded), 0.0% to a person. Accuracy of the decided messages 97.8%.
    Cost per 1,000 messages: mistakes and reviews 100.0 dollars, running 0.012 dollars, total 100.0 dollars.

route cascade replays the recorded answers of the LLM, so it needs no key, and no message leaves your computer. Use the default mode, few-shot. The recordings of the zero-shot mode cover only the test messages. So --mode zero-shot works with thresholds that you give (--sure and --llm-sure), but not with --search, which needs the validation messages: it then says so and stops.

Use a cascade when one cheap model is sure about most messages and is right when it is sure. On this inbox, TF-IDF is sure about 85.9% of the messages and wrong on only 8 of them. Then an expensive step is worth it for the rest.

Do not use one when the cheap model is rarely sure, or wrong when it is sure. Then the first step only adds a delay and a place for mistakes. Check this on validation before you build anything: count the messages above the threshold, and the cheap model’s mistakes among them. Also, if one model is already the cheapest row everywhere, you may not need the others. On this inbox, the fine-tuned encoder alone costs US$103.0, close to the cascade.

What a cascade costs you:

  • Two thresholds to choose, and to choose again. A new model version, a new LLM or a new product moves both. Keep the search as code, and run it on the validation messages of each version.
  • More parts to run. Two models, an LLM client with retries, and a review queue. Each part can fail. Decide what happens when the LLM is down: here, its messages could go to a person.
  • People’s time. The person step sends 42 messages in 1,000 to Grace’s team, about 3 a day at Larkfield’s 65 messages a day. Check the load with her before you promise it.
  • Uneven waits. 85.9% of the customers wait a fraction of a millisecond. The others wait for the LLM, about 1.7 seconds, or for a person, a few hours. The mean LLM wait per message falls from 2.68 seconds (the LLM for all) to 0.39 seconds.

Your turn: the same cascade without the LLM

Section titled “Your turn: the same cascade without the LLM”

Change one thing: keep TF-IDF at sure = 0.80, but send every message it is not sure about straight to a person. There is no LLM. In section 6 of the notebook, compute the cost on the validation messages and put it in no_llm_cost. Before you run it, predict: is the LLM step worth its calls?

Show the answer
r, s = cascade(vr, vp, lv_r, lv_c, 0.80, 2.0)
no_llm_cost = cascade_cost(validation, r, s)
print(no_llm_cost)
235.5

llm_sure = 2.0 is above any written confidence, so every LLM answer goes to a person. Without the LLM, 298 validation messages go to a person, and the cost is US$235.5, against US$103.8 with the LLM. On test, it is US$227.5 against US$94.3. The LLM is worth its calls. It is right on most of the messages that TF-IDF is unsure about, and a call costs far less than a review.

Common mistake

A better threshold, found on the test messages

Symptom: Tomás tries every “sure” threshold from 0.30 to 1.00, in steps of 0.01, on the test messages. “Look: at 0.73, the cascade costs US$89.0, not US$94.3. Let’s use 0.73.”

Cause: 0.73 is the lowest point of the test messages, so it fits their noise. On the validation messages, the same threshold costs US$225.8, against US$103.8 at 0.80. Near the bottom of the curve, a few expensive messages decide where the lowest point falls. The US$89.0 is a number that the next month will not show.

Fix: choose on the validation messages. Report the test number once, even when a lower one is visible. For a better choice, get more evidence. Use more labelled messages, or the same search with cross-validation on the training messages, as in Train, validation and test sets.

Consequence in AI work: a threshold tuned on the test set looks like a free gain in the report. In production, it turns into a surprise. Two thresholds give twice as many ways to fit the noise.

Check your understanding

The cascade costs US$94.3 per 1,000 test messages, and the LLM for everything US$116.1. Tomás says the gain comes from fewer LLM calls. Use the numbers to explain where it really comes from.

Show the answer

Not from the calls. The LLM costs US$0.088 per 1,000 messages alone and US$0.012 in the cascade, a saving of US$0.076. The gain comes from fewer expensive mistakes. On the messages that TF-IDF is sure about, it makes 8 mistakes where the LLM makes 24. Putting TF-IDF first brings 116.1 down to 100.0 on test. Then a person reads the 4.2% that both models are unsure about. That brings it to 94.3. On validation, where the thresholds were chosen, the person step is most of the gain: 176.0 to 103.8.

Check your understanding

Predict: in the explorer, with the LLM deciding everything it gets, what happens to the total cost as you raise the sure threshold from 0.85 to 1.00, and why?

Show the answer

It rises, from US$97.0 to US$116.1. Above 0.85, more of the messages that TF-IDF would get right go to the LLM instead. On those messages, the LLM is wrong more often than TF-IDF, so the mistakes cost more. The running cost also rises, but by less than 0.1 dollar per 1,000 messages. The lowest point is before the end, not at “the LLM for everything”.

Check your understanding

Practical check: your project now has the end-of-m06 stage. Which command chooses the two thresholds, which thresholds does it choose, and what share of the test messages goes to a person?

Show the answer

route cascade --search. It chooses the thresholds on the validation messages: sure 0.80 for TF-IDF and 0.95 for the LLM’s written confidence. On the test messages, TF-IDF decides 85.9%, the LLM decides 10.0%, and 4.2% go to a person. The total is US$94.3 per 1,000 messages. python -m pytest -q prints 24 passed.

Recap

  • A cascade is a chain of routers from cheap to expensive. Each one decides what it is sure about and passes the rest down. A person is the last step.
  • On this inbox, TF-IDF is sure about 85.9% of the test messages and wrong on 8 of them. The LLM decides 10.0%, and a person reads 4.2%.
  • Choose the two thresholds on validation: 0.80 and 0.95 here (US$103.8). Score them once on test: US$94.3 per 1,000, the lowest row so far. The LLM for everything costs US$116.1, and TF-IDF alone US$296.0.
  • The gain is not from saved tokens (US$0.076 per 1,000). It comes from fewer expensive mistakes, and above all from the few messages that a person reads.
  • Your project is at the stage end-of-m06: 24 tests, route llm and route cascade.

You can now give the LLM only the messages that need it. But the cascade still calls the LLM, and sends 14.1% of the messages out of Larkfield. Next, you let the LLM teach a cheap model once, and see what the student learns, the mistakes included.

Next: Distillation: let the LLM teach a cheap model