Few-shot examples and label descriptions
- Lesson 20
- 35 min
- Module 5: LLMs as classifiers
Where we are
The LLM classified our messages from the label names alone. People who label well read the guidelines and look at examples first. In this lesson, we give the LLM the label descriptions and a few similar labelled examples, and we measure what changes.
Lesson 20 of 37 · Text Classification and Routing: Send Each Message to the Right PlaceFew-shot examples and label descriptions
1. The routing problem and its cost
2. Labels you can trust
3. Baselines: rules and classical models
4. Embeddings and small fine-tuned models
5. LLMs as classifiers
6. Decision models, cascades and distillation
7. Harder label shapes
8. Confidence and people in the loop
9. Run it in production
What you will be able to do
Give the LLM what a good labeller reads first: a description of each route and a few similar messages that are already labelled. Then measure what changes, message by message, and check whether the confidence that the model writes can be trusted.
NotebookDo this lesson in a notebook
The notebook has every code cell, task and check of this lesson. It finds the 8 most similar training messages for one message and replays the LLM’s recorded answers with and without the extra context. Then it counts the messages fixed and broken, and checks the written confidence. You need no account and no key. About 35 minutes.
Colab and Kaggle open your own copy. They need a free Google or Kaggle account. Stuck? See the solution notebook.
Before you startWhat to know and to have ready
What you need to know
- The LLM with the route names only, its schema and its answer check, from Zero-shot classification with an LLM.
- The labelling guidelines, from Labelling guidelines.
- Similarity between two texts, from Text as numbers: TF-IDF and Few labels with embeddings.
What you need on your computer
- Nothing for the notebook route. This module changes no file of your
larkrouteproject.
Tested on macOS (Apple silicon) with Python 3.12, pandas 3.1.1 and scikit-learn 1.9.1, in Jupyter. Colab, Kaggle and Windows were not tested.
The clerk’s second day
Section titled “The clerk’s second day”On his first day, the clever new clerk sorted letters by common sense. On the second day, the manager gives him two things. The first is the rule book, with one line per tray: “Parcels: late or lost parcels, even when the customer is angry.” The second is a small pile of letters like the one in his hand, already sorted by an experienced clerk.
Now he does not guess the house rules. He reads them, and he sees how they were used on letters like this one.
The problem: the LLM did not know the rules
Section titled “The problem: the LLM did not know the rules”In the last lesson, the LLM had only the route names. It sent 287 team messages and 10 of the 12 account takeovers to needs_person. It scored 0.765 on this inbox and cost US$1,049.0 per 1,000 messages. It did not know Larkfield’s rules, because nobody gave them to it.
There is a second problem, and it is about us. Tomás changes the prompt, reads five answers that were wrong, and sees that they are now right. “It’s much better,” he says. That is a feeling. A change of prompt fixes some messages and breaks others. Five answers cannot show both. We must count them on the whole test set.
The idea: describe each route, and show similar examples
Section titled “The idea: describe each route, and show similar examples”We change two parts of the prompt.
1. Route descriptions from the guidelines
Section titled “1. Route descriptions from the guidelines”Each route gets one line, written from Chloé’s labelling guidelines: what belongs to it, and the hard cases at its border. The system prompt also states the rule for two requests:
{l2.prompt}2. Few-shot examples, chosen by similarity
Section titled “2. Few-shot examples, chosen by similarity”Few-shot means “with a few examples”: labelled messages in the prompt, each with its route. Which examples? Not random ones. For each message, we choose the 8 training messages that are most similar to it. We turn every message into TF-IDF numbers of letter groups (3 to 5 characters). Then we keep the 8 training messages whose numbers are closest. Text as numbers: TF-IDF explains these numbers. Similar messages usually show the border that matters for this message.
So each message gets its own prompt: the same descriptions, its own 8 examples, then the message.
A tiny worked example: one message, before and after
Section titled “A tiny worked example: one message, before and after”Here is a test message from the hard cases of the guidelines. The diagram shows the LLM’s answer with the route names only, and with the descriptions and 8 examples:
Text description of the diagram
The top row has three boxes. The first box is the route descriptions, one line per route, written from the guidelines. The second box is the 8 most similar training messages. Four are shown: “Can I return the rug if I don’t like the colour? Just checking the refund policy”, “Can I return the storage box if I don’t like the colour? Just need a yes or no”, “Can I return the cushion set if I don’t like the colour?” and “Can I return the grill cover if I don’t like the colour?”. All 8 are labelled product. The third box is the message, M-105011: “Can I return the rug if I don’t like the colour?” Its label is product: the guidelines send a question about the return policy before buying to the product team. Below, two answers. With the route names only, the LLM answered returns_refunds with a written confidence of 0.99. Its reason was “Asks about returning a rug due to its colour”. That is wrong. With the descriptions and the 8 examples, it answered product with a written confidence of 0.99. Its reason was “Asks about the return policy before initiating a return”. That is right.
The word “return” pulled the first answer to returns_refunds. The examples show that Larkfield sends this question to product, and the second answer follows them. Both answers wrote 0.99. The written number did not change; the route did.
The extra context has a price. The average prompt grows from 186 to 717 tokens, almost four times longer. The cost per 1,000 messages grows from US$0.035 to US$0.088. The median time grows from 1.2 to 1.7 seconds.
Larkfield: measure the change
Section titled “Larkfield: measure the change”Priya recorded the LLM’s answers with descriptions and examples for the 2,000 test messages, and for the 2,000 validation messages. Here are both prompts on the test messages:
| Prompt (2,000 test messages) | Accuracy | English | French | Takeovers found | needs_person found |
Cost per 1,000 | Time |
|---|---|---|---|---|---|---|---|
| Route names only | 0.765 | 0.757 | 0.784 | 2 of 12 | 85 of 99 | US$1,049.0 | 1.2 s |
| Descriptions + 8 examples | 0.970 | 0.973 | 0.961 | 12 of 12 | 99 of 99 | US$116.1 | 1.7 s |
On this inbox, the descriptions and examples take the LLM from 0.765 to 0.970. It finds all 12 takeovers and all 99 needs_person messages. The cost per 1,000 falls from US$1,049.0 to US$116.1. That is the second lowest of all rows so far, after the fine-tuned encoder (US$103.0). On the validation messages, it scores 0.971: the gain is not an accident of the test set.
Fixed and broken, not only “better”
Section titled “Fixed and broken, not only “better””An overall score hides two movements. Count them message by message:
| Few-shot right | Few-shot wrong | |
|---|---|---|
| Zero-shot right | 1,519 | 11 (broken) |
| Zero-shot wrong | 420 (fixed) | 50 |
The new prompt fixes 420 messages and breaks 11. Here, the change is clearly good. But you only know that because you counted both. A change that fixes 30 and breaks 40 also “fixes the five answers I looked at”.
Look at the recall of each route: the share of its messages that get the right route.
| Route | Route names only | Descriptions + 8 examples |
|---|---|---|
orders | 0.712 | 0.975 |
delivery | 0.796 | 0.976 |
returns_refunds | 0.787 | 0.965 |
billing | 0.789 | 0.970 |
product | 0.661 | 0.962 |
account | 0.796 | 0.949 |
needs_person | 0.859 | 1.000 |
Every route improves. The smallest gain is needs_person, which was already good at finding its own messages. The problem there was the 287 team messages that it took, and they fall to 22.
Is the LLM only copying its examples?
Section titled “Is the LLM only copying its examples?”The 8 examples are close to the message. Perhaps the LLM just takes the most common route among them. We can test that without any LLM: give each test message the route that most of its 8 examples have. This is a majority vote of the nearest examples.
Text description of the chart
A column chart of accuracy on the test messages. The LLM with the route names only: 76.5%. The majority vote of the 8 most similar training messages, with no LLM: 90.2%. The LLM with descriptions and 8 examples: 97.0%. The fine-tuned encoder of Module 4: 98.0%.
The vote alone reaches 0.902 (US$747.0 per 1,000). Similar messages carry a lot of the answer. But the LLM does better: it is right on 160 messages where the vote is wrong, and wrong on only 25 where the vote is right. It reads the message, the rules and the examples together. For example, the two-request message M-101535 (“Still waiting on duvet … Also can’t log in …”) has 5 account examples and 3 delivery examples. The vote says account; the LLM applies the “main request” rule and answers delivery, the label.
Is the written confidence true now?
Section titled “Is the written confidence true now?”Both prompts asked for a confidence from 0 to 1. A confidence is calibrated when it matches how often the model is right. Among the answers where it writes 0.9, about 9 in 10 should be right. The expected calibration error (ECE) puts the gap into one number, and 0 means the confidence is true everywhere. The LLM’s written confidence in Module 8 draws the chart and explains the number in full.
On Larkfield’s test messages, the ECE is 0.190 with the route names only and 0.008 with descriptions and examples. With the extra context, 95.9% of the answers have a written confidence of 0.9 or more, and 98.4% of those are right. The other 4.1% are right only 62.2% of the time. So a low written confidence really marks an unsure answer here.
A written confidence is not a probability until you check it. Here, it is close to the truth because we checked it on 2,000 labelled messages, for this prompt, this model and this inbox. Change any of the three, and you must check again. Cascades uses this confidence, once checked, to decide which messages go to a person.
When to use descriptions and few-shot examples
Section titled “When to use descriptions and few-shot examples”- Use descriptions always: they cost a few tokens and carry the rules. Write them from your labelling guidelines, with the hard cases.
- Use similar examples when you have even a few hundred labelled messages. Similar beats random: the examples show the border that matters for this message.
- Measure every prompt change on the same labelled set: accuracy, the cost per 1,000, messages fixed and broken, and each route and language.
- What it costs: about 4 times more tokens (US$0.088 per 1,000 here) and 1.7 seconds per message. Each call needs a search for similar messages first, and every message still leaves Larkfield. On this inbox, the fine-tuned encoder is still a little more accurate and cheaper (0.980, US$103.0).
Guided practice: replay both prompts
Section titled “Guided practice: replay both prompts”- Open the notebook from the box Do this lesson in a notebook at the top of this page.
- Run the setup cells. The data cell ends with
Ready: 10000 messages (6000 train, 2000 validation, 2000 test), all synthetic. - In section 1, read the system prompt. Predict the routes of the 8 similar messages, then run the cell. All 8 are
product. - In section 2, replay both recordings. The few-shot row shows
0.970and116.1. - In section 3, count the messages. You see
420fixed and11broken. - In section 5, run the reliability tables. The ECE is
0.1901and0.0082.
Your turn: the vote of the examples
Section titled “Your turn: the vote of the examples”In section 4 of the notebook, the cell finds the 8 nearest training messages of every test message. Complete vote: for each test message, the most common route among its 8 neighbours (on a tie, the first route in rt.ROUTES). Predict first: will the vote beat TF-IDF with logistic regression (0.962)?
Show the answer
vote = [max(rt.ROUTES, key=list(row).count) for row in neigh_routes]Check passed: the accuracy of the vote is 0.902.Messages that the LLM gets right and the vote gets wrong: 160The vote does not beat TF-IDF: 0.902 against 0.962. Eight neighbours are a strong signal, but a vote cannot read the rules of the message. The LLM with the same 8 examples reaches 0.970.
Common mistake
The prompt follows an old rule, and the labels follow the new one
Symptom: the new prompt breaks 11 messages. Tomás reads them and finds a pattern. In 6 of the 11, the customer mentions a lawyer, for example: “my card was charged $92.42, more than the order total. Please correct it. I may need to speak with a lawyer if this isn’t resolved.” The label is billing. The LLM with descriptions answers needs_person and writes 0.99.
Cause: the route descriptions in the recorded prompt come from the first version of the guidelines. That version sent every legal threat to needs_person. The team then changed the rule, as Labelling guidelines shows: the route follows the main request, and a threat mentioned in passing raises the priority and gets a flag. The labels follow the new rule. The prompt still says “a legal or press threat” with no word about the main request. So the model’s “mistake” is the prompt following the old rule.
Fix: read the broken messages before you blame the model. Here, write the current rule into the needs_person description: “only when the main request is a safety problem, a legal or press threat, or a manager; a passing mention keeps the team’s route”. Then score again. Keep the flag as a separate step, with the priority, so a person still reads these messages. When the rule and the labels truly disagree, Chloé and Grace decide, and fix one of them, as in Find noisy labels.
Consequence in AI work: a model that reads the rules exposes every place where the prompt, the guidelines and the labels disagree. Its score then measures all three. Keep the prompt’s descriptions in step with the guidelines, or you will tune the prompt to an old rule.
Knowledge check
Section titled “Knowledge check”Check your understanding
The few-shot prompt reaches 0.970 against 0.765. Why should you also count the messages that it fixes and breaks, and what are the counts here?
Show the answer
An overall score is a net result. A prompt change fixes some messages and breaks others, and a few examples that look better cannot show both. Here, on the 2,000 test messages, the new prompt fixes 420 messages and breaks 11, so the change is clearly good. Reading the 11 broken ones also shows a pattern: 6 mention a lawyer only in passing. The labels follow the current guideline (the main request decides), but the prompt’s description still follows the old rule.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
The majority vote of the 8 similar examples scores 0.902, and the LLM with the same examples 0.970. What does that tell you about where the LLM's gain comes from?
Show the answer
The similar examples carry much of the answer: a vote with no LLM already reaches 0.902. But the LLM adds more. It is right on 160 messages where the vote is wrong, and wrong on only 25 where the vote is right. It reads the descriptions and the message itself. For example, it applies the rule for two requests when the examples point to the second request.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
With descriptions and examples, the ECE of the written confidence is 0.008. Can Larkfield now use the written confidence as a probability everywhere? Explain.
Show the answer
Only where it was checked. An ECE of 0.008 means that the written confidence matched the accuracy on these 2,000 test messages, with this prompt and this model. Answers at 0.99 were right 98.4% of the time. With the route names only, the same kind of number had an ECE of 0.190. A new prompt, a new model version or a new product can change it. So Larkfield must check it again after each change, on labelled messages, before any threshold uses it.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Recap
- Route descriptions from the guidelines carry the house rules into the prompt.
- Few-shot examples chosen by similarity: the 8 most similar training messages, each with its route.
- On this inbox, the LLM goes from 0.765 to 0.970, and from US$1,049.0 to US$116.1 per 1,000. It finds 12 of 12 takeovers. The prompt is about 4 times longer: US$0.088 per 1,000 in tokens, 1.7 s per message.
- Measure the change: 420 messages fixed, 11 broken. Read the broken ones: 6 mention a lawyer in passing, where the prompt still follows an old rule.
- The vote of the 8 examples alone reaches 0.902: the LLM adds more than copying.
- A written confidence is not a probability until you check it: ECE 0.190 without the context, 0.008 with it, on this inbox.
Next, we leave the provider’s computers. Can a small LLM on Larkfield’s own laptop do the same job?
Next: Small local LLMs