Distillation: let the LLM teach a cheap model
- Lesson 27
- 35 min
- Module 6: Decision models, cascades and distillation
Where we are
Our cascade still calls the LLM for some messages. Another way to use an LLM is to let it teach a cheap model once. In this lesson, the LLM labels 5,000 messages, we check a sample by hand, and we train a cheap model on those labels.
Lesson 27 of 37 · Text Classification and Routing: Send Each Message to the Right PlaceDistillation: let the LLM teach a cheap model
1. The routing problem and its cost
2. Labels you can trust
3. Baselines: rules and classical models
4. Embeddings and small fine-tuned models
5. LLMs as classifiers
6. Decision models, cascades and distillation
7. Harder label shapes
8. Confidence and people in the loop
9. Run it in production
What you will be able to do
Let an LLM label your messages once, and train a cheap model on those labels. The cheap model then routes fast, for almost nothing, without sending messages out. You must also check what it copied from its teacher.
NotebookDo this lesson in a notebook
The notebook has every code cell, task and check of this lesson. It replays the LLM’s 5,000 recorded labels and checks a sample of 200. Then it trains the student and a model on people’s labels, and finds the mistakes that the student copied. Its setup cells install pandas and scikit-learn and download the practice data, so it needs no local setup and no key. About 30 minutes.
Colab and Kaggle open your own copy. They need a free Google or Kaggle account. Stuck? See the solution notebook.
Before you startWhat to know and to have ready
What you need to know
- TF-IDF + logistic regression, from Naive Bayes, logistic regression and linear SVM.
- Noisy labels, and what they do to a model, from Find noisy labels.
- The LLM as a classifier, from Zero-shot classification with an LLM.
What you need on your computer
Nothing for the lesson; the notebook runs in Colab or Kaggle. This lesson does not change the project.
A new clerk learns from the expert
Section titled “A new clerk learns from the expert”Back at the post office. The expert sorter is good, but slow, and the office cannot give every letter to her. So a new clerk sits next to her for a week. He watches her sort thousands of letters, and writes down where each one went. After that week, he sorts alone, and he is fast.
He learned her good habits. He also learned her bad ones. The expert always puts gardening magazines in the wrong bag, so now the new clerk does too. Nobody notices, because he does exactly what the expert did.
The problem: the LLM is good, but expensive to keep calling
Section titled “The problem: the LLM is good, but expensive to keep calling”In the last lesson, the cascade still sends 14.1% of Larkfield’s messages to the LLM. For Larkfield, the money is small. Other costs are not:
- Speed. The LLM takes about 1.7 seconds per message (the median); TF-IDF takes about 0.1 milliseconds.
- Privacy. Each message that goes to the LLM leaves Larkfield’s computers. Some customers and some contracts do not allow that.
- Limits and outages. A provider accepts only so many calls per minute, and it is sometimes down.
A cheap model avoids all three, but it needs labels to learn from. People’s labels are slow to get. With the course’s cost table, a person needs about one minute per message. So 5,000 labels take about 83 hours of Chloé’s time: two weeks of work, at about US$2,300. That is an estimate from the cost table’s assumption, not a measurement.
The LLM labelled the same 5,000 messages in 250 calls, for US$0.081, in minutes. Can we use its labels to train the cheap model?
The idea: distillation
Section titled “The idea: distillation”Distillation means training a cheap model, the student, on the answers of an expensive model, the teacher. The name comes from distilling a drink: you keep the strong part and leave the bulk behind. The student is much smaller and faster than the teacher, and it runs where you want.
There are two kinds. In the classic kind, the student learns the teacher’s probabilities for every label. Here, the hosted LLM gives only one label per message, so the student learns those labels as if a person had written them. This is the simple kind, and the commonest with hosted LLMs.
The student can be any model that learns from labels. Here it is TF-IDF + logistic regression, the project’s model. The fine-tuned encoder of Module 4 would work the same way.
A tiny worked example: five labels from the teacher
Section titled “A tiny worked example: five labels from the teacher”Here are five of the 5,000 training messages, with the label that people gave and the label that the LLM gave:
| Message | People | LLM (its confidence) |
|---|---|---|
M-100026 “Please cancel my order LK-817564. I don’t need it anymore.” |
orders | orders (1.0) |
M-100037 “svp, fermez mon compte et arrêtez les courriels. merci.” |
account | account (1.0) |
M-100070 “My son cut his hand on the swing seat. The edge is way too sharp.” |
needs_person | needs_person (1.0) |
M-101118 “Can I return the rug if I don’t like the colour? Just checking the refund policy.” |
product | returns_refunds (0.98) |
M-101701 “@Larkfield, we’re a café interested in ordering 20 coffee maker. Is there a business order form we can use?” |
orders | product (0.89) |
The first three are easy, and the LLM is right. The last two follow Larkfield’s guidelines, which the LLM did not fully learn. A question about the return policy before buying goes to product. A business order goes to orders. The LLM’s labels look right, and its confidence is high. A student trained on them will learn that “return” before buying means returns_refunds.
The whole process
Section titled “The whole process”Text description of the diagram
The diagram is a flow from top to bottom. At the top are 5,000 training messages, without labels. The teacher is the hosted LLM. It labels them in 250 calls of 20 messages, for US$0.081. Its 5,000 labels agree with people’s labels on 91.4% of the messages. Next, a person checks a sample of 200 labels by hand: 90.5% are right. The commonest error of all 5,000 is product to returns_refunds, 64 times. Then there are two boxes side by side. On the left, the student: TF-IDF + logistic regression trained on the LLM’s labels. On the test messages, it is 0.904 accurate and costs US$414 per 1,000 messages. It sends 23 product questions to returns. On the right, a dashed box shows, for comparison, the same model trained on people’s labels for the same messages. It gets 0.958 and US$383 per 1,000. It sends 5 product questions to returns. At the bottom: 137 of the student’s 193 test errors, 71%, are also errors of the LLM’s own answers. The student copied its teacher’s mistakes.
Larkfield: the LLM labels 5,000 messages
Section titled “Larkfield: the LLM labels 5,000 messages”Priya gives the LLM the route descriptions and no examples. It labels the first 5,000 training messages, 20 per call. The course recorded these labels once, so you replay them in the notebook.
Check a sample by hand
Section titled “Check a sample by hand”In real work, you would not have people’s labels for these messages. That is why you asked the LLM. So you check a random sample. Chloé reviews 200 of the LLM’s labels: 90.5% are right. On this inbox we can also check all 5,000, because people labelled them earlier: 91.4%. The sample gave a good estimate.
The overall number hides big differences between routes:
Text description of the chart
A column chart shows, for each of the seven routes, the share of the LLM’s labels that agree with people’s labels. The routes go from the lowest share to the highest. Orders: 83.0%, for 718 messages. Product: 86.3%, for 722. Billing: 93.4%. Returns: 93.5%. Delivery: 93.7%. Account: 95.5%. Needs a person: 98.8%, for 241 messages. The orders and product columns are highlighted.
The weakest routes are orders (83.0%) and product (86.3%). These are exactly the cases of the worked example: business orders, and policy questions before buying. The LLM is very good on needs_person (98.8%). Safety and legal words are clear, whatever the guidelines say.
Train the student, and compare
Section titled “Train the student, and compare”Now train TF-IDF + logistic regression on the LLM’s 5,000 labels: that is the student. It takes about 2 seconds. To see what the label errors cost, also train the same model on people’s labels for the same 5,000 messages. Score both on the 2,000 test messages, which have people’s labels.
| Model (TF-IDF + LR, 5,000 messages) | Accuracy | Takeover (12) | Needs a person (99) | Cost per 1,000 |
|---|---|---|---|---|
| On people’s labels | 0.958 | 0.917 | 0.980 | US$383 |
| On the LLM’s labels (the student) | 0.904 | 1.000 | 1.000 | US$414 |
The student loses 5.4 points of accuracy. Its cost per 1,000 rises less: by US$31. Why so little? The student found every account takeover and every needs_person message. The model on people’s labels missed one takeover (US$200) and two needs_person messages (US$40 each). Most of the student’s extra mistakes are cheap ones: a normal message in the wrong team’s queue, at US$4.
Notice one more thing. The labels were 91.4% right, and the student is 90.4% right. It did not clean up its teacher’s errors. When label errors are random, a model often learns past them. These errors are systematic: the same kind of message gets the same wrong label each time. So the student learns the error as a rule.
The mistakes the student copied
Section titled “The mistakes the student copied”Take the teacher’s three commonest label errors. How often does each model make the same mistake on the test messages?
- Student (LLM labels)
- Same model on people's labels
Text description of the chart
A column chart with two columns for each of the teacher’s three commonest label errors. Product questions sent to returns: the student 23 test messages, the model on people’s labels 5. Orders sent to product: the student 25, the model on people’s labels 2. Delivery sent to product: the student 11, the model on people’s labels 2. In the training labels, these errors appear 64, 55 and 29 times.
The student sends 23 product questions to returns_refunds, against 5 for the model on people’s labels. It sends 25 orders to product, against 2. These are its teacher’s habits.
One more check: of the student’s 193 test errors, 137 (71%) are also errors of the LLM’s own zero-shot answers on the same messages. These answers are not exactly the teacher’s labels: the teacher also had the route descriptions. But they are the closest recorded answers of the same LLM on the test messages. For the model on people’s labels, the overlap is 64 of 85. The student did not invent many new mistakes. It learned its teacher’s.
What distillation buys, and what it loses
Section titled “What distillation buys, and what it loses”| The LLM with examples | The student (TF-IDF on LLM labels) | |
|---|---|---|
| Accuracy on test | 0.970 | 0.904 |
| Cost per 1,000 messages | US$116.1 | US$414 |
| Time per message | about 1.7 s | about 0.1 ms |
| Messages that leave Larkfield | every message, always | 5,000, once, to make the labels |
| Calls per 1,000 messages | 1,000 | 0 |
| One-time cost | none | US$0.081 for the labels, and a person’s check of 200 |
Distillation buys speed, a running cost near zero, no rate limits, and privacy after the first step. The student runs on Larkfield’s own computer. It loses accuracy. Here, the student costs US$414 per 1,000 messages. TF-IDF + logistic regression costs US$296, and the same model on people’s labels for the same 5,000 messages US$383. It also inherits the teacher’s blind spots, and nobody sees them unless someone looks by route and by pair.
So use distillation when you have many messages and no labels. It fits when speed, privacy or the number of calls matter more than the last points of accuracy. Then spend people’s time where the teacher is weak. Here, Chloé could relabel the 64 product questions that the LLM sent to returns. Or Priya could add a line to the LLM’s route descriptions and label again. Do not use it when good people’s labels already exist, as on this inbox. The model on people’s labels is better, and costs nothing more to run.
Guided practice: train the student
Section titled “Guided practice: train the student”- Open the notebook from the box Do this lesson in a notebook at the top of this page.
- Run the setup cells. The data cell ends with
Ready: 10000 messages (6000 train, 2000 validation, 2000 test), all synthetic. - In section 1, run the cell. You see
5000 labels, 20 messages per call, cost 0.081 dollarsand an agreement of0.9142. - In section 2, run the two cells. The sample of 200 gives
0.905, and the commonest pair isproduct -> returns_refunds(64). - In section 3, write your prediction. Then run the cell. The student gets
0.904, and the model on people’s labels0.958. - In section 4, run the cell. You see
student 23 - model on people's labels 5, then193errors and137shared.
Your turn: keep only the labels the teacher is sure about
Section titled “Your turn: keep only the labels the teacher is sure about”The LLM wrote a confidence for each label. Tomás has an idea: “Keep only the labels with a confidence of at least 0.9. The student then learns only from good labels.” In section 5 of the notebook, train the student on those messages only, and put its test accuracy in sure_accuracy. Before you run it, predict: better or worse than the student on all 5,000?
Show the answer
keep = five[five["llm_conf"] >= 0.9]m = tfidf_lr().fit(keep["text"], keep["llm_route"])row = rt.score("student, labels >= 0.9", test, m.predict(test["text"]))sure_accuracy = row["accuracy"]print(len(keep), sure_accuracy, row["total_per_1000"])4331 0.902 559.0Worse: 0.902 and US$559.0 per 1,000, against 0.904 and US$414 on all 5,000. The kept labels are more often right: 94.6%, against 91.4%. But the filter removes 669 messages, and they are the hard ones, where the student most needs an example. A confident teacher is not a complete teacher.
Common mistake
The sample said 90.5%, and missed the commonest error
Symptom: Chloé checks 200 random LLM labels and finds 19 errors: 90.5% right. Priya trains the student, and it sends many product questions to returns_refunds in production. “But the sample was fine,” she says.
Cause: the sample measured the rate of errors well: 90.5% against 91.4% for all 5,000. It did not show where they are. Its 19 errors are spread over 13 different pairs, with at most 4 of one kind. Not one of the 64 product questions labelled returns_refunds is in the sample. A random sample of 200 is a good thermometer, and a poor map.
Fix: after the random sample, check by route and by pair. Look at a few labels of each route, and more of the routes where the teacher is weak (here orders and product). Then compare the student’s mistakes on the test messages with its teacher’s habits, as in section 4 of the notebook.
Consequence in AI work: a student model inherits the systematic errors of its teacher, and an overall accuracy number hides them. The errors then show up in production as one kind of message that always goes to the wrong team.
Knowledge check
Section titled “Knowledge check”Check your understanding
The LLM's labels agree with people's labels on 91.4% of the messages, yet the student is only 90.4% accurate on test. Why did the student not do better than its labels?
Show the answer
The label errors are systematic, not random. The same kind of message gets the same wrong label each time: for example, 64 product questions about the return policy are labelled returns_refunds. A model can often learn past random noise, but a systematic error looks like a rule, so the student learns it. On test, the student sends 23 product questions to returns_refunds, against 5 for the same model on people’s labels. Of its 193 errors, 137 are also errors of the LLM’s own answers.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
Larkfield must stop sending messages to an outside provider. Name what distillation gives it, and what it costs, with the numbers of this lesson.
Show the answer
It gives a router that runs on Larkfield’s own computer. It takes about 0.1 milliseconds per message, with no calls and no rate limits. After the 5,000 messages labelled once (US$0.081), no message leaves. It costs accuracy. The student is 0.904 accurate (US$414 per 1,000). The same model on people’s labels gets 0.958 (US$383), and the LLM with examples 0.970 (US$116.1). It also copies the LLM’s blind spots, such as product questions sent to returns, so a person must check the labels by route.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
Practical check: in the notebook, how many product questions does the student send to returns_refunds on the test messages, and how many does the model trained on people's labels send?
Show the answer
The student sends 23, and the model on people’s labels sends 5. The teacher’s labels had 64 such errors among the 5,000 training messages. The student learned the error from them.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Recap
- Distillation trains a cheap student on the labels of an expensive teacher. Here, the hosted LLM labelled 5,000 messages in 250 calls, for US$0.081.
- Check a random sample for the error rate (200 labels: 90.5%), then check by route and by pair for where the errors are. The sample of 200 missed the commonest error.
- On this inbox, the student is 0.904 accurate (US$414 per 1,000). The same model on people’s labels gets 0.958 (US$383).
- The student copies its teacher’s systematic errors: 23 product questions to returns, against 5. 71% of its errors are errors of the LLM too.
- Distillation buys speed, privacy and no calls; it loses accuracy. Keeping only the teacher’s sure labels made the student worse (0.902).
You can now choose between an LLM, a cascade and a distilled student, and say what each costs. But all of them treated every route as equally common, and every message as having one route. Next, you work on the rare route that costs the most: the account takeover, with only 44 training examples.