Zero-shot classification with an LLM
- Lesson 19
- 35 min
- Module 5: LLMs as classifiers
Where we are
The zero-shot models of the last module needed no training data. A large language model can do the same, and Tomás wants to use one for everything. In this lesson, we classify with an LLM from the label list alone, with structured output, and we check every answer.
Lesson 19 of 37 · Text Classification and Routing: Send Each Message to the Right PlaceZero-shot classification with an LLM
1. The routing problem and its cost
2. Labels you can trust
3. Baselines: rules and classical models
4. Embeddings and small fine-tuned models
5. LLMs as classifiers
6. Decision models, cascades and distillation
7. Harder label shapes
8. Confidence and people in the loop
9. Run it in production
What you will be able to do
Classify messages with a large language model and the route names alone. Get the answer in a fixed shape, and check every answer before the router uses it. Then measure it on the same test set as every other method. On this inbox, it is weaker than you may expect.
NotebookDo this lesson in a notebook
The notebook has every code cell, task and check of this lesson. It shows the prompt and the answer schema, checks four example answers, and replays the LLM’s recorded answers for the 2,000 test messages. You need no account and no key. An optional last cell sends one synthetic message to a model of your own. About 35 minutes.
Colab and Kaggle open your own copy. They need a free Google or Kaggle account. Stuck? See the solution notebook.
Before you startWhat to know and to have ready
What you need to know
- The seven routes and the labelling guidelines, from Design the labels and Labelling guidelines.
- The cost table and the comparison table, from The cost of each mistake and What the extra accuracy costs.
- Structured output, from Define a schema in the LLM API course (helpful). This lesson explains what it needs.
What you need on your computer
- Nothing for the notebook route. This module changes no file of your
larkrouteproject.
Tested on macOS (Apple silicon) with Python 3.12, pandas 3.1.1 and scikit-learn 1.9.1, in Jupyter. Colab, Kaggle and Windows were not tested.
A clever new clerk on the first day
Section titled “A clever new clerk on the first day”A new clerk joins the post office. He has read a lot. He knows what a parcel is, what a refund is and what an angry letter looks like. Nobody has told him the post office’s own rules. You give him only the names of the trays: “Parcels”, “Returns”, “Payments”, “Manager”.
He sorts quickly, and many letters land in the right tray. But some of his choices follow his common sense, not the house rules. An angry letter about a late parcel goes to “Manager”, because it sounds serious. The house rule says: late parcels go to “Parcels”, even when the customer is angry.
The problem: does an LLM know Larkfield’s rules?
Section titled “The problem: does an LLM know Larkfield’s rules?”Tomás, Larkfield’s AI engineer, wants to send every message to a large language model (LLM). An LLM is a model trained on a huge amount of text: it reads a request and writes an answer. “An LLM is always the most accurate,” he says. “And we need no labels.”
That is a claim, not a measurement. Two problems hide behind it:
- An LLM writes text. The router needs one of seven route names, nothing else. “Probably delivery, I think” breaks the router.
- An LLM knows the world, not Larkfield’s guidelines. Larkfield sends a message with two requests to the route of the main request, usually the first one. It sends an account takeover to
account, not to a person. The route names alone do not say that.
We measure both on the same 2,000 test messages as every other method.
The idea: zero-shot classification with a schema
Section titled “The idea: zero-shot classification with a schema”Zero-shot means “with zero examples”, as in Zero-shot without an LLM. The LLM gets the list of route names in its instructions, and the message. It gets no labelled example. Three parts make this usable by a router.
1. The route names in the prompt
Section titled “1. The route names in the prompt”The prompt is the text that we send to the model. The system prompt is the part that sets the task, the same for every message:
{l1.prompt}The message follows, as the second part of the prompt.
2. Structured output with a schema
Section titled “2. Structured output with a schema”A schema describes the exact shape of the answer: which fields, of which type, with which allowed values. Here, the answer is JSON with three fields:
route: a text, and only one of the seven route names (a list of allowed values, called an enum);confidence: a number;reason: a short text.
Structured output means that the provider forces the model’s answer to match the schema. The LLM API course teaches this in Define a schema.
3. A check of every answer
Section titled “3. A check of every answer”A schema checks the shape. Your code must still check every answer before the router uses it: valid JSON, a known route, and a confidence from 0 to 1. A bad answer goes to review. The LLM API course calls this validate the meaning.
Here is the whole path of one test message, with the answer that the LLM really gave:
Text description of the diagram
The diagram has four steps. First, the message M-101926, from social media, in English: “@Larkfield, my bath mat says delivered, but nothing’s at my door. Order LK-169866. I was also charged twice.” Second, the prompt, with the route names only, which asks for JSON with the route, a confidence from 0 to 1 and a reason. Third, the answer that the LLM gave, recorded: route needs_person, confidence 0.96, reason “Reports both a missing delivery and a duplicate charge.” Fourth, your check: the JSON is valid, the route is known, and the confidence is from 0 to 1. So the router sends the message to needs_person. Below, a red box says: valid, but wrong. The guidelines send a message with two requests to the route of the first one, here delivery. A senior person now reads a message that the delivery team could answer, which costs US$2 in the cost table. The check tests the shape of the answer, not its truth.
A tiny worked example: four answers, one check
Section titled “A tiny worked example: four answers, one check”Before we trust any answer, we test the check itself. Here are four answers that a model could write, and what the check does with each:
| Answer | Check result | Why |
|---|---|---|
{"route": "billing", "confidence": 0.9, "reason": "double charge"} |
billing, 0.9 | valid JSON, known route, 0.9 is from 0 to 1 |
Billing, I think. |
review | not JSON |
{"route": "refunds", "confidence": 0.8, "reason": "refund"} |
review | “refunds” is not a route: the route is returns_refunds |
{"route": "delivery", "confidence": 95, "reason": "late"} |
review | 95 is not a confidence from 0 to 1 |
With structured output, the last three should not happen. But a schema can be turned off by mistake, a provider can change, and a local model may not support schemas at all. In Small local LLMs, one model writes invalid answers for one message in five. The check costs one function, and it protects the router.
Larkfield: the LLM on the 2,000 test messages
Section titled “Larkfield: the LLM on the 2,000 test messages”Priya sent every test message once to a small hosted LLM, the Azure deployment chat-small (model gpt-6-luna), and recorded the answers. You replay them: no key, no account, no cost. Before you look, predict: will the LLM beat TF-IDF with logistic regression (0.962 on this inbox)?
| Method (2,000 test messages) | Accuracy | English | French | Takeovers found | needs_person found |
Cost per 1,000 |
|---|---|---|---|---|---|---|
| TF-IDF + logistic regression | 0.962 | 0.966 | 0.952 | 11 of 12 | 99 of 99 | US$296.0 |
| Frozen embeddings (older model) + LR | 0.920 | 0.931 | 0.896 | 10 of 12 | 92 of 99 | US$779.0 |
| LLM, route names only (recorded) | 0.765 | 0.757 | 0.784 | 2 of 12 | 85 of 99 | US$1,049.0 |
| Everything to a person | US$1,500.0 |
Every answer was valid JSON with a known route: structured output worked. But on this inbox, the LLM with the route names only is less accurate than every trained model of Modules 3 and 4. It costs US$1,049.0 per 1,000 messages, more than three times TF-IDF. Tomás’s claim fails here.
Where the mistakes go
Section titled “Where the mistakes go”Count the LLM’s choices. It chose needs_person for 372 test messages, but only 99 messages need a person.
Text description of the chart
A column chart of the test messages that the LLM with route names only sent to needs_person, by their real route. Orders 44, delivery 76, returns_refunds 55, billing 34, product 45, account 33. These six columns are team messages: 287 in all. The last column, needs_person, is 85: the messages that really need a person.
287 team messages went to a senior person. Read the LLM’s own reasons, and a pattern appears. “Multiple unrelated account and product questions require assistance.” “Two separate requests: return for wrong colour and an overdue, potentially lost delivery.” The model reads two requests, or an upset customer, as “a person must read this first”. That is common sense. It is not Larkfield’s rule.
The account takeovers show the same thing. Here is one, with the LLM’s answer:
Hello, I think someone got into my account, and I’m not sure what to do next. The address was changed to Halifax and there’s a new order I didn’t place, so please look into this right away. RileyLLM:
needs_person, written confidence 0.99, “Suspected account compromise and unauthorized address change and order require urgent investigation.”
The reason makes sense: a takeover is urgent. But Larkfield’s guidelines say that the account team locks the account first, so a takeover goes to account. The model sent 10 of the 12 takeovers to needs_person. With the route names only, it cannot know where Larkfield wants them. In the cost table, each of these costs US$2, like any team message sent to a person. A senior person reads it and passes it on. The real risk is the delay before the account is locked.
Where the US$1,049.0 comes from
Section titled “Where the US$1,049.0 comes from”The cost table prices each mistake. Here are the LLM’s mistakes on the 2,000 test messages, per 1,000 messages:
| Kind of mistake | Messages (of 2,000) | Cost each | Per 1,000 messages |
|---|---|---|---|
| Team message sent to a person | 287 | US$2 | US$287.0 |
| Wrong queue | 151 | US$4 | US$302.0 |
| Urgent message in the wrong queue | 18 | US$20 | US$180.0 |
needs_person message sent to a team |
14 | US$40 | US$280.0 |
| All mistakes | 470 | US$1,049.0 |
The 14 messages that needed a person and went to a team cost almost as much as the 287 that went the other way. A safety problem or a legal threat answered by a team is the expensive mistake.
What a call costs
Section titled “What a call costs”Each call sent about 186 tokens and got back about 33 tokens. A token is a piece of a word: about 4 letters of English text. The provider charges per million tokens: US$0.10 for input and US$0.50 for output, at the price of the recording. For one message:
- input: 186 × 0.10 ÷ 1,000,000 = US$0.0000186;
- output: 33 × 0.50 ÷ 1,000,000 = US$0.0000165;
- one call: US$0.0000351, so US$0.035 per 1,000 messages.
That is about 3.5 cents for 1,000 messages. Prices change: check the provider’s current price page before you use these numbers. The mistakes cost US$1,049.0 per 1,000. At these prices, the running cost is about 30,000 times smaller than the cost of the mistakes. Time matters more: each call took 1.2 seconds (median), and 2.1 seconds for the slowest 10%. Cost per 1,000 messages puts all these costs together.
The written confidence
Section titled “The written confidence”The schema asked for a confidence, and the model wrote one for every message. But a number that a model writes is part of its text. Nobody measured it. Here are the confidences that it wrote most often, and how often it was right at each:
| Written confidence | Messages | Right |
|---|---|---|
| 0.99 | 693 | 86.6% |
| 0.98 | 429 | 83.4% |
| 0.97 | 163 | 76.1% |
| 0.96 | 147 | 72.1% |
| 0.91 | 110 | 55.5% |
| 0.94 | 86 | 67.4% |
It wrote 0.99 for 693 messages, and it was right on 86.6% of them, not 99%. At 0.91, it was right only about half the time. The next lesson measures this gap with one number, and shows when it closes.
When to use an LLM with the route names only
Section titled “When to use an LLM with the route names only”- Use it when you have no labels at all and need a first sort today, for a person to check. Its answers also make a quick first draft of labels for a new route.
- Use it as a baseline for better prompts: the next lesson adds descriptions and examples, and you measure the change against this row.
- Do not use it as the router when you have labelled messages. Here, frozen embeddings with logistic regression beat it, and TF-IDF beats it by far.
- What it costs: about US$0.035 per 1,000 messages in tokens, 1.2 seconds per message and a rate limit. Every message leaves Larkfield for the provider. Its mistakes cost much more than all of that.
Guided practice: replay the LLM
Section titled “Guided practice: replay the LLM”- Open the notebook from the box Do this lesson in a notebook at the top of this page.
- Run the setup cells. The data cell ends with
Ready: 10000 messages (6000 train, 2000 validation, 2000 test), all synthetic. - In section 1, read the system prompt and the answer schema.
- In section 2, predict which of the four answers pass. Then run the check. Only the first one passes.
- In section 3, replay the recorded answers. You see
valid answers: 100.0%and the LLM’s accuracy,0.765. - Read the table of labels against the LLM’s routes. Find the column
needs_person.
Your turn: the cost from the tokens
Section titled “Your turn: the cost from the tokens”In section 4 of the notebook, compute the cost of 1,000 test messages. Use the mean input and output tokens of the recorded answers, and the chat-small prices in data/prices.json. Predict first: more or less than 10 cents?
Show the answer
price = json.load(open("data/prices.json"))["models"]["chat-small"]cost_1000 = round(1000 * (llm["tokens_in"].mean() * price["input"] + llm["tokens_out"].mean() * price["output"]) / 1e6, 4)print(cost_1000)0.0351About 3.5 cents for 1,000 messages: less than 10 cents. The recorded cost of the calls gives the same number. The output tokens are fewer, but each costs five times more, so they make almost half of the bill.
Common mistake
A confident answer is not a right answer
Symptom: the LLM wrote a confidence of 0.99 for 693 test messages. A developer sends every answer at 0.95 or more straight to the teams, and reviews the rest. About 93 of those 693 messages still go to the wrong queue, with no review.
Cause: the confidence is a number in the model’s text. The model writes the number that fits the answer, as a person says “I’m sure” out of habit. Nobody measured how often 0.99 is right on Larkfield’s messages. Here, it is right 86.6% of the time.
Fix: never use a written confidence as a threshold before you check it on labelled messages. Group the answers by their written confidence and compare with the accuracy, as in the table above. The next lesson turns this check into one number. Ask for a decision, not for text shows how to get a real probability from a model.
Consequence in AI work: a threshold on an unchecked confidence looks like a safety net, and it lets the most confident mistakes through. Those are the mistakes that nobody reviews.
Knowledge check
Section titled “Knowledge check”Check your understanding
The LLM's answers were 100% valid JSON with a known route. Why is that not enough for the router? Use one example from this lesson.
Show the answer
Valid means the answer has the right shape: JSON, one of the seven route names, a confidence from 0 to 1. It does not mean the route is right. For example, M-101926 says that a bath mat is missing and that the customer was charged twice. The LLM answered needs_person with confidence 0.96, and the answer passed every check. But Larkfield’s guidelines send it to delivery, the route of the first request. Overall, the LLM was valid 100% of the time and right 76.5% of the time.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
With the route names only, the LLM sends 287 team messages and 10 of the 12 account takeovers to needs_person. What does it lack, and what does that cost here?
Show the answer
It lacks Larkfield’s rules. Its common sense says that two requests, an upset customer or a takeover need a person. Larkfield’s guidelines say: the route of the main request, and takeovers to the account team, which locks the account first. The route names do not carry those rules. In the cost table, each team message sent to a person costs US$2, so the 287 cost US$287.0 per 1,000 messages. For the takeovers, the bigger risk is the delay before the account is locked.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
A call costs about US$0.035 per 1,000 messages, and the LLM's total is US$1,049.0 per 1,000. Where does the rest come from, and what does that tell you about choosing a model here?
Show the answer
Almost all of it comes from the mistakes, priced by the cost table. Wrong queues cost US$302.0, and team messages sent to a person US$287.0. needs_person messages sent to a team cost US$280.0, and urgent messages in the wrong queue US$180.0. The tokens are about 30,000 times cheaper than the mistakes. So on this inbox, a cheaper model or fewer tokens barely changes the total. Fewer mistakes change it a lot. The price per token is not the cost that decides.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Recap
- Zero-shot with an LLM: the route names in the system prompt, the message, and no example.
- A schema with an enum fixes the shape of the answer; structured output makes the provider follow it. Your code still checks every answer, and a bad one goes to review.
- On this inbox, the LLM with the route names only reaches 0.765 (US$1,049.0 per 1,000): worse than every trained model. “An LLM is always the most accurate” is not true here.
- It sends 287 team messages and 10 of 12 takeovers to
needs_person: common sense instead of Larkfield’s rules. - A call costs about US$0.035 per 1,000 messages and takes about 1.2 s. The mistakes cost far more.
- A written confidence is text, not a measurement: 0.99 was right 86.6% of the time.
Next, we give the LLM what a new clerk needs: the description of each route, and a few labelled examples.