Skip to content
Français

Few-shot examples and label descriptions

  • Lesson 20
  • 35 min
  • Module 5: LLMs as classifiers

Where we are

The LLM classified our messages from the label names alone. People who label well read the guidelines and look at examples first. In this lesson, we give the LLM the label descriptions and a few similar labelled examples, and we measure what changes.

Lesson 20 of 37 · Text Classification and Routing: Send Each Message to the Right PlaceFew-shot examples and label descriptions

What you will be able to do

Give the LLM what a good labeller reads first: a description of each route and a few similar messages that are already labelled. Then measure what changes, message by message, and check whether the confidence that the model writes can be trusted.

NotebookDo this lesson in a notebookColab · Kaggle · GitHub

The notebook has every code cell, task and check of this lesson. It finds the 8 most similar training messages for one message and replays the LLM’s recorded answers with and without the extra context. Then it counts the messages fixed and broken, and checks the written confidence. You need no account and no key. About 35 minutes.

Colab and Kaggle open your own copy. They need a free Google or Kaggle account. Stuck? See the solution notebook.

Before you startWhat to know and to have readyAbout 35 minutes

What you need to know

What you need on your computer

  • Nothing for the notebook route. This module changes no file of your larkroute project.

Tested on macOS (Apple silicon) with Python 3.12, pandas 3.1.1 and scikit-learn 1.9.1, in Jupyter. Colab, Kaggle and Windows were not tested.

On his first day, the clever new clerk sorted letters by common sense. On the second day, the manager gives him two things. The first is the rule book, with one line per tray: “Parcels: late or lost parcels, even when the customer is angry.” The second is a small pile of letters like the one in his hand, already sorted by an experienced clerk.

Now he does not guess the house rules. He reads them, and he sees how they were used on letters like this one.

The problem: the LLM did not know the rules

Section titled “The problem: the LLM did not know the rules”

In the last lesson, the LLM had only the route names. It sent 287 team messages and 10 of the 12 account takeovers to needs_person. It scored 0.765 on this inbox and cost US$1,049.0 per 1,000 messages. It did not know Larkfield’s rules, because nobody gave them to it.

There is a second problem, and it is about us. Tomás changes the prompt, reads five answers that were wrong, and sees that they are now right. “It’s much better,” he says. That is a feeling. A change of prompt fixes some messages and breaks others. Five answers cannot show both. We must count them on the whole test set.

The idea: describe each route, and show similar examples

Section titled “The idea: describe each route, and show similar examples”

We change two parts of the prompt.

Each route gets one line, written from Chloé’s labelling guidelines: what belongs to it, and the hard cases at its border. The system prompt also states the rule for two requests:

{l2.prompt}

2. Few-shot examples, chosen by similarity

Section titled “2. Few-shot examples, chosen by similarity”

Few-shot means “with a few examples”: labelled messages in the prompt, each with its route. Which examples? Not random ones. For each message, we choose the 8 training messages that are most similar to it. We turn every message into TF-IDF numbers of letter groups (3 to 5 characters). Then we keep the 8 training messages whose numbers are closest. Text as numbers: TF-IDF explains these numbers. Similar messages usually show the border that matters for this message.

So each message gets its own prompt: the same descriptions, its own 8 examples, then the message.

A tiny worked example: one message, before and after

Section titled “A tiny worked example: one message, before and after”

Here is a test message from the hard cases of the guidelines. The diagram shows the LLM’s answer with the route names only, and with the descriptions and 8 examples:

Three input boxes, then two answer boxes. Route descriptions from the guidelines, one line per route; product says questions about products, before or after buying. 8 similar labelled messages: 'Can I return the rug if I don't like the colour? Just checking the refund policy' → product, '@Larkfield, can I return the storage box if I don't like the colour? Just need a yes or no.' → product, 'Can I return the cushion set if I don't like the colour?' → product, 'Can I return the grill cover if I don't like the colour?' → product, and 4 more, all product. The message, M-105011: 'Can I return the rug if I don't like the colour?' Label: product, because the guidelines send a return question before buying to product. The answer with route names only: returns_refunds, written confidence 0.99, reason 'Asks about returning a rug due to its colour', wrong. The answer with descriptions and 8 examples: product, written confidence 0.99, reason 'Asks about the return policy before initiating a return', right.
One test message, M-105011, with the route names only and with the route descriptions and its 8 most similar training messages (recorded answers). All 8 examples show the house rule: a question about the return policy goes to product.
Text description of the diagram

The top row has three boxes. The first box is the route descriptions, one line per route, written from the guidelines. The second box is the 8 most similar training messages. Four are shown: “Can I return the rug if I don’t like the colour? Just checking the refund policy”, “Can I return the storage box if I don’t like the colour? Just need a yes or no”, “Can I return the cushion set if I don’t like the colour?” and “Can I return the grill cover if I don’t like the colour?”. All 8 are labelled product. The third box is the message, M-105011: “Can I return the rug if I don’t like the colour?” Its label is product: the guidelines send a question about the return policy before buying to the product team. Below, two answers. With the route names only, the LLM answered returns_refunds with a written confidence of 0.99. Its reason was “Asks about returning a rug due to its colour”. That is wrong. With the descriptions and the 8 examples, it answered product with a written confidence of 0.99. Its reason was “Asks about the return policy before initiating a return”. That is right.

The word “return” pulled the first answer to returns_refunds. The examples show that Larkfield sends this question to product, and the second answer follows them. Both answers wrote 0.99. The written number did not change; the route did.

The extra context has a price. The average prompt grows from 186 to 717 tokens, almost four times longer. The cost per 1,000 messages grows from US$0.035 to US$0.088. The median time grows from 1.2 to 1.7 seconds.

Priya recorded the LLM’s answers with descriptions and examples for the 2,000 test messages, and for the 2,000 validation messages. Here are both prompts on the test messages:

Prompt (2,000 test messages) Accuracy English French Takeovers found needs_person found Cost per 1,000 Time
Route names only 0.765 0.757 0.784 2 of 12 85 of 99 US$1,049.0 1.2 s
Descriptions + 8 examples 0.970 0.973 0.961 12 of 12 99 of 99 US$116.1 1.7 s

On this inbox, the descriptions and examples take the LLM from 0.765 to 0.970. It finds all 12 takeovers and all 99 needs_person messages. The cost per 1,000 falls from US$1,049.0 to US$116.1. That is the second lowest of all rows so far, after the fine-tuned encoder (US$103.0). On the validation messages, it scores 0.971: the gain is not an accident of the test set.

An overall score hides two movements. Count them message by message:

Few-shot right Few-shot wrong
Zero-shot right 1,519 11 (broken)
Zero-shot wrong 420 (fixed) 50

The new prompt fixes 420 messages and breaks 11. Here, the change is clearly good. But you only know that because you counted both. A change that fixes 30 and breaks 40 also “fixes the five answers I looked at”.

Look at the recall of each route: the share of its messages that get the right route.

RouteRoute names onlyDescriptions + 8 examples
orders0.7120.975
delivery0.7960.976
returns_refunds0.7870.965
billing0.7890.970
product0.6610.962
account0.7960.949
needs_person0.8591.000

Every route improves. The smallest gain is needs_person, which was already good at finding its own messages. The problem there was the 287 team messages that it took, and they fall to 22.

The 8 examples are close to the message. Perhaps the LLM just takes the most common route among them. We can test that without any LLM: give each test message the route that most of its 8 examples have. This is a majority vote of the nearest examples.

Accuracy on the 2,000 test messages: the LLM with route names only and with descriptions + 8 examples (recorded), the majority vote of the same 8 examples, and the fine-tuned encoder (measured).
Text description of the chart

A column chart of accuracy on the test messages. The LLM with the route names only: 76.5%. The majority vote of the 8 most similar training messages, with no LLM: 90.2%. The LLM with descriptions and 8 examples: 97.0%. The fine-tuned encoder of Module 4: 98.0%.

The vote alone reaches 0.902 (US$747.0 per 1,000). Similar messages carry a lot of the answer. But the LLM does better: it is right on 160 messages where the vote is wrong, and wrong on only 25 where the vote is right. It reads the message, the rules and the examples together. For example, the two-request message M-101535 (“Still waiting on duvet … Also can’t log in …”) has 5 account examples and 3 delivery examples. The vote says account; the LLM applies the “main request” rule and answers delivery, the label.

Both prompts asked for a confidence from 0 to 1. A confidence is calibrated when it matches how often the model is right. Among the answers where it writes 0.9, about 9 in 10 should be right. The expected calibration error (ECE) puts the gap into one number, and 0 means the confidence is true everywhere. The LLM’s written confidence in Module 8 draws the chart and explains the number in full.

On Larkfield’s test messages, the ECE is 0.190 with the route names only and 0.008 with descriptions and examples. With the extra context, 95.9% of the answers have a written confidence of 0.9 or more, and 98.4% of those are right. The other 4.1% are right only 62.2% of the time. So a low written confidence really marks an unsure answer here.

A written confidence is not a probability until you check it. Here, it is close to the truth because we checked it on 2,000 labelled messages, for this prompt, this model and this inbox. Change any of the three, and you must check again. Cascades uses this confidence, once checked, to decide which messages go to a person.

When to use descriptions and few-shot examples

Section titled “When to use descriptions and few-shot examples”
  • Use descriptions always: they cost a few tokens and carry the rules. Write them from your labelling guidelines, with the hard cases.
  • Use similar examples when you have even a few hundred labelled messages. Similar beats random: the examples show the border that matters for this message.
  • Measure every prompt change on the same labelled set: accuracy, the cost per 1,000, messages fixed and broken, and each route and language.
  • What it costs: about 4 times more tokens (US$0.088 per 1,000 here) and 1.7 seconds per message. Each call needs a search for similar messages first, and every message still leaves Larkfield. On this inbox, the fine-tuned encoder is still a little more accurate and cheaper (0.980, US$103.0).
  1. Open the notebook from the box Do this lesson in a notebook at the top of this page.
  2. Run the setup cells. The data cell ends with Ready: 10000 messages (6000 train, 2000 validation, 2000 test), all synthetic.
  3. In section 1, read the system prompt. Predict the routes of the 8 similar messages, then run the cell. All 8 are product.
  4. In section 2, replay both recordings. The few-shot row shows 0.970 and 116.1.
  5. In section 3, count the messages. You see 420 fixed and 11 broken.
  6. In section 5, run the reliability tables. The ECE is 0.1901 and 0.0082.

In section 4 of the notebook, the cell finds the 8 nearest training messages of every test message. Complete vote: for each test message, the most common route among its 8 neighbours (on a tie, the first route in rt.ROUTES). Predict first: will the vote beat TF-IDF with logistic regression (0.962)?

Show the answer
vote = [max(rt.ROUTES, key=list(row).count) for row in neigh_routes]
Check passed: the accuracy of the vote is 0.902.
Messages that the LLM gets right and the vote gets wrong: 160

The vote does not beat TF-IDF: 0.902 against 0.962. Eight neighbours are a strong signal, but a vote cannot read the rules of the message. The LLM with the same 8 examples reaches 0.970.

Common mistake

The prompt follows an old rule, and the labels follow the new one

Symptom: the new prompt breaks 11 messages. Tomás reads them and finds a pattern. In 6 of the 11, the customer mentions a lawyer, for example: “my card was charged $92.42, more than the order total. Please correct it. I may need to speak with a lawyer if this isn’t resolved.” The label is billing. The LLM with descriptions answers needs_person and writes 0.99.

Cause: the route descriptions in the recorded prompt come from the first version of the guidelines. That version sent every legal threat to needs_person. The team then changed the rule, as Labelling guidelines shows: the route follows the main request, and a threat mentioned in passing raises the priority and gets a flag. The labels follow the new rule. The prompt still says “a legal or press threat” with no word about the main request. So the model’s “mistake” is the prompt following the old rule.

Fix: read the broken messages before you blame the model. Here, write the current rule into the needs_person description: “only when the main request is a safety problem, a legal or press threat, or a manager; a passing mention keeps the team’s route”. Then score again. Keep the flag as a separate step, with the priority, so a person still reads these messages. When the rule and the labels truly disagree, Chloé and Grace decide, and fix one of them, as in Find noisy labels.

Consequence in AI work: a model that reads the rules exposes every place where the prompt, the guidelines and the labels disagree. Its score then measures all three. Keep the prompt’s descriptions in step with the guidelines, or you will tune the prompt to an old rule.

Check your understanding

The few-shot prompt reaches 0.970 against 0.765. Why should you also count the messages that it fixes and breaks, and what are the counts here?

Show the answer

An overall score is a net result. A prompt change fixes some messages and breaks others, and a few examples that look better cannot show both. Here, on the 2,000 test messages, the new prompt fixes 420 messages and breaks 11, so the change is clearly good. Reading the 11 broken ones also shows a pattern: 6 mention a lawyer only in passing. The labels follow the current guideline (the main request decides), but the prompt’s description still follows the old rule.

Check your understanding

The majority vote of the 8 similar examples scores 0.902, and the LLM with the same examples 0.970. What does that tell you about where the LLM's gain comes from?

Show the answer

The similar examples carry much of the answer: a vote with no LLM already reaches 0.902. But the LLM adds more. It is right on 160 messages where the vote is wrong, and wrong on only 25 where the vote is right. It reads the descriptions and the message itself. For example, it applies the rule for two requests when the examples point to the second request.

Check your understanding

With descriptions and examples, the ECE of the written confidence is 0.008. Can Larkfield now use the written confidence as a probability everywhere? Explain.

Show the answer

Only where it was checked. An ECE of 0.008 means that the written confidence matched the accuracy on these 2,000 test messages, with this prompt and this model. Answers at 0.99 were right 98.4% of the time. With the route names only, the same kind of number had an ECE of 0.190. A new prompt, a new model version or a new product can change it. So Larkfield must check it again after each change, on labelled messages, before any threshold uses it.

Recap

  • Route descriptions from the guidelines carry the house rules into the prompt.
  • Few-shot examples chosen by similarity: the 8 most similar training messages, each with its route.
  • On this inbox, the LLM goes from 0.765 to 0.970, and from US$1,049.0 to US$116.1 per 1,000. It finds 12 of 12 takeovers. The prompt is about 4 times longer: US$0.088 per 1,000 in tokens, 1.7 s per message.
  • Measure the change: 420 messages fixed, 11 broken. Read the broken ones: 6 mention a lawyer in passing, where the prompt still follows an old rule.
  • The vote of the 8 examples alone reaches 0.902: the LLM adds more than copying.
  • A written confidence is not a probability until you check it: ECE 0.190 without the context, 0.008 with it, on this inbox.

Next, we leave the provider’s computers. Can a small LLM on Larkfield’s own laptop do the same job?

Next: Small local LLMs