Skip to content
Français

Zero-shot classification with an LLM

  • Lesson 19
  • 35 min
  • Module 5: LLMs as classifiers

Where we are

The zero-shot models of the last module needed no training data. A large language model can do the same, and Tomás wants to use one for everything. In this lesson, we classify with an LLM from the label list alone, with structured output, and we check every answer.

Plays from YouTube. Nothing loads from YouTube until you press play. Privacy
Lesson 19 of 37 · Text Classification and Routing: Send Each Message to the Right PlaceZero-shot classification with an LLM

What you will be able to do

Classify messages with a large language model and the route names alone. Get the answer in a fixed shape, and check every answer before the router uses it. Then measure it on the same test set as every other method. On this inbox, it is weaker than you may expect.

NotebookDo this lesson in a notebookColab · Kaggle · GitHub

The notebook has every code cell, task and check of this lesson. It shows the prompt and the answer schema, checks four example answers, and replays the LLM’s recorded answers for the 2,000 test messages. You need no account and no key. An optional last cell sends one synthetic message to a model of your own. About 35 minutes.

Colab and Kaggle open your own copy. They need a free Google or Kaggle account. Stuck? See the solution notebook.

Before you startWhat to know and to have readyAbout 35 minutes

What you need to know

What you need on your computer

  • Nothing for the notebook route. This module changes no file of your larkroute project.

Tested on macOS (Apple silicon) with Python 3.12, pandas 3.1.1 and scikit-learn 1.9.1, in Jupyter. Colab, Kaggle and Windows were not tested.

A new clerk joins the post office. He has read a lot. He knows what a parcel is, what a refund is and what an angry letter looks like. Nobody has told him the post office’s own rules. You give him only the names of the trays: “Parcels”, “Returns”, “Payments”, “Manager”.

He sorts quickly, and many letters land in the right tray. But some of his choices follow his common sense, not the house rules. An angry letter about a late parcel goes to “Manager”, because it sounds serious. The house rule says: late parcels go to “Parcels”, even when the customer is angry.

The problem: does an LLM know Larkfield’s rules?

Section titled “The problem: does an LLM know Larkfield’s rules?”

Tomás, Larkfield’s AI engineer, wants to send every message to a large language model (LLM). An LLM is a model trained on a huge amount of text: it reads a request and writes an answer. “An LLM is always the most accurate,” he says. “And we need no labels.”

That is a claim, not a measurement. Two problems hide behind it:

  1. An LLM writes text. The router needs one of seven route names, nothing else. “Probably delivery, I think” breaks the router.
  2. An LLM knows the world, not Larkfield’s guidelines. Larkfield sends a message with two requests to the route of the main request, usually the first one. It sends an account takeover to account, not to a person. The route names alone do not say that.

We measure both on the same 2,000 test messages as every other method.

The idea: zero-shot classification with a schema

Section titled “The idea: zero-shot classification with a schema”

Zero-shot means “with zero examples”, as in Zero-shot without an LLM. The LLM gets the list of route names in its instructions, and the message. It gets no labelled example. Three parts make this usable by a router.

The prompt is the text that we send to the model. The system prompt is the part that sets the task, the same for every message:

{l1.prompt}

The message follows, as the second part of the prompt.

A schema describes the exact shape of the answer: which fields, of which type, with which allowed values. Here, the answer is JSON with three fields:

  • route: a text, and only one of the seven route names (a list of allowed values, called an enum);
  • confidence: a number;
  • reason: a short text.

Structured output means that the provider forces the model’s answer to match the schema. The LLM API course teaches this in Define a schema.

A schema checks the shape. Your code must still check every answer before the router uses it: valid JSON, a known route, and a confidence from 0 to 1. A bad answer goes to review. The LLM API course calls this validate the meaning.

Here is the whole path of one test message, with the answer that the LLM really gave:

Four boxes joined by arrows, then a red box. 1. The message M-101926 (social, English): '@Larkfield, my bath mat says delivered, but nothing's at my door. Order LK-169866. I was also charged twice.' 2. The prompt: the route names only, and answer in JSON with the route, a confidence from 0 to 1 and a reason. 3. The recorded answer: route needs_person, confidence 0.96, reason 'Reports both a missing delivery and a duplicate charge.' 4. Your check: valid JSON yes, a known route yes, confidence from 0 to 1 yes, so the router sends it to needs_person. The red box: valid, but wrong. The guidelines send a message with two requests to the route of the first one, delivery. A senior person reads a message that the delivery team could answer, which costs US$2. The check tests the shape of the answer, not its truth.
One test message through the LLM, with its recorded answer. The answer passes every check, and it is still wrong: the check tests the shape, not the truth.
Text description of the diagram

The diagram has four steps. First, the message M-101926, from social media, in English: “@Larkfield, my bath mat says delivered, but nothing’s at my door. Order LK-169866. I was also charged twice.” Second, the prompt, with the route names only, which asks for JSON with the route, a confidence from 0 to 1 and a reason. Third, the answer that the LLM gave, recorded: route needs_person, confidence 0.96, reason “Reports both a missing delivery and a duplicate charge.” Fourth, your check: the JSON is valid, the route is known, and the confidence is from 0 to 1. So the router sends the message to needs_person. Below, a red box says: valid, but wrong. The guidelines send a message with two requests to the route of the first one, here delivery. A senior person now reads a message that the delivery team could answer, which costs US$2 in the cost table. The check tests the shape of the answer, not its truth.

A tiny worked example: four answers, one check

Section titled “A tiny worked example: four answers, one check”

Before we trust any answer, we test the check itself. Here are four answers that a model could write, and what the check does with each:

Answer Check result Why
{"route": "billing", "confidence": 0.9, "reason": "double charge"} billing, 0.9 valid JSON, known route, 0.9 is from 0 to 1
Billing, I think. review not JSON
{"route": "refunds", "confidence": 0.8, "reason": "refund"} review “refunds” is not a route: the route is returns_refunds
{"route": "delivery", "confidence": 95, "reason": "late"} review 95 is not a confidence from 0 to 1

With structured output, the last three should not happen. But a schema can be turned off by mistake, a provider can change, and a local model may not support schemas at all. In Small local LLMs, one model writes invalid answers for one message in five. The check costs one function, and it protects the router.

Larkfield: the LLM on the 2,000 test messages

Section titled “Larkfield: the LLM on the 2,000 test messages”

Priya sent every test message once to a small hosted LLM, the Azure deployment chat-small (model gpt-6-luna), and recorded the answers. You replay them: no key, no account, no cost. Before you look, predict: will the LLM beat TF-IDF with logistic regression (0.962 on this inbox)?

Method (2,000 test messages) Accuracy English French Takeovers found needs_person found Cost per 1,000
TF-IDF + logistic regression 0.962 0.966 0.952 11 of 12 99 of 99 US$296.0
Frozen embeddings (older model) + LR 0.920 0.931 0.896 10 of 12 92 of 99 US$779.0
LLM, route names only (recorded) 0.765 0.757 0.784 2 of 12 85 of 99 US$1,049.0
Everything to a person US$1,500.0

Every answer was valid JSON with a known route: structured output worked. But on this inbox, the LLM with the route names only is less accurate than every trained model of Modules 3 and 4. It costs US$1,049.0 per 1,000 messages, more than three times TF-IDF. Tomás’s claim fails here.

Count the LLM’s choices. It chose needs_person for 372 test messages, but only 99 messages need a person.

The test messages that the LLM sent to needs_person, by their real route (recorded answers). Only 85 really need a person; 287 belong to a team.
Text description of the chart

A column chart of the test messages that the LLM with route names only sent to needs_person, by their real route. Orders 44, delivery 76, returns_refunds 55, billing 34, product 45, account 33. These six columns are team messages: 287 in all. The last column, needs_person, is 85: the messages that really need a person.

287 team messages went to a senior person. Read the LLM’s own reasons, and a pattern appears. “Multiple unrelated account and product questions require assistance.” “Two separate requests: return for wrong colour and an overdue, potentially lost delivery.” The model reads two requests, or an upset customer, as “a person must read this first”. That is common sense. It is not Larkfield’s rule.

The account takeovers show the same thing. Here is one, with the LLM’s answer:

Hello, I think someone got into my account, and I’m not sure what to do next. The address was changed to Halifax and there’s a new order I didn’t place, so please look into this right away. Riley

LLM: needs_person, written confidence 0.99, “Suspected account compromise and unauthorized address change and order require urgent investigation.”

The reason makes sense: a takeover is urgent. But Larkfield’s guidelines say that the account team locks the account first, so a takeover goes to account. The model sent 10 of the 12 takeovers to needs_person. With the route names only, it cannot know where Larkfield wants them. In the cost table, each of these costs US$2, like any team message sent to a person. A senior person reads it and passes it on. The real risk is the delay before the account is locked.

The cost table prices each mistake. Here are the LLM’s mistakes on the 2,000 test messages, per 1,000 messages:

Kind of mistake Messages (of 2,000) Cost each Per 1,000 messages
Team message sent to a person 287 US$2 US$287.0
Wrong queue 151 US$4 US$302.0
Urgent message in the wrong queue 18 US$20 US$180.0
needs_person message sent to a team 14 US$40 US$280.0
All mistakes 470 US$1,049.0

The 14 messages that needed a person and went to a team cost almost as much as the 287 that went the other way. A safety problem or a legal threat answered by a team is the expensive mistake.

Each call sent about 186 tokens and got back about 33 tokens. A token is a piece of a word: about 4 letters of English text. The provider charges per million tokens: US$0.10 for input and US$0.50 for output, at the price of the recording. For one message:

  • input: 186 × 0.10 ÷ 1,000,000 = US$0.0000186;
  • output: 33 × 0.50 ÷ 1,000,000 = US$0.0000165;
  • one call: US$0.0000351, so US$0.035 per 1,000 messages.

That is about 3.5 cents for 1,000 messages. Prices change: check the provider’s current price page before you use these numbers. The mistakes cost US$1,049.0 per 1,000. At these prices, the running cost is about 30,000 times smaller than the cost of the mistakes. Time matters more: each call took 1.2 seconds (median), and 2.1 seconds for the slowest 10%. Cost per 1,000 messages puts all these costs together.

The schema asked for a confidence, and the model wrote one for every message. But a number that a model writes is part of its text. Nobody measured it. Here are the confidences that it wrote most often, and how often it was right at each:

Written confidenceMessagesRight
0.9969386.6%
0.9842983.4%
0.9716376.1%
0.9614772.1%
0.9111055.5%
0.948667.4%

It wrote 0.99 for 693 messages, and it was right on 86.6% of them, not 99%. At 0.91, it was right only about half the time. The next lesson measures this gap with one number, and shows when it closes.

When to use an LLM with the route names only

Section titled “When to use an LLM with the route names only”
  • Use it when you have no labels at all and need a first sort today, for a person to check. Its answers also make a quick first draft of labels for a new route.
  • Use it as a baseline for better prompts: the next lesson adds descriptions and examples, and you measure the change against this row.
  • Do not use it as the router when you have labelled messages. Here, frozen embeddings with logistic regression beat it, and TF-IDF beats it by far.
  • What it costs: about US$0.035 per 1,000 messages in tokens, 1.2 seconds per message and a rate limit. Every message leaves Larkfield for the provider. Its mistakes cost much more than all of that.
  1. Open the notebook from the box Do this lesson in a notebook at the top of this page.
  2. Run the setup cells. The data cell ends with Ready: 10000 messages (6000 train, 2000 validation, 2000 test), all synthetic.
  3. In section 1, read the system prompt and the answer schema.
  4. In section 2, predict which of the four answers pass. Then run the check. Only the first one passes.
  5. In section 3, replay the recorded answers. You see valid answers: 100.0% and the LLM’s accuracy, 0.765.
  6. Read the table of labels against the LLM’s routes. Find the column needs_person.

In section 4 of the notebook, compute the cost of 1,000 test messages. Use the mean input and output tokens of the recorded answers, and the chat-small prices in data/prices.json. Predict first: more or less than 10 cents?

Show the answer
price = json.load(open("data/prices.json"))["models"]["chat-small"]
cost_1000 = round(1000 * (llm["tokens_in"].mean() * price["input"] + llm["tokens_out"].mean() * price["output"]) / 1e6, 4)
print(cost_1000)
0.0351

About 3.5 cents for 1,000 messages: less than 10 cents. The recorded cost of the calls gives the same number. The output tokens are fewer, but each costs five times more, so they make almost half of the bill.

Common mistake

A confident answer is not a right answer

Symptom: the LLM wrote a confidence of 0.99 for 693 test messages. A developer sends every answer at 0.95 or more straight to the teams, and reviews the rest. About 93 of those 693 messages still go to the wrong queue, with no review.

Cause: the confidence is a number in the model’s text. The model writes the number that fits the answer, as a person says “I’m sure” out of habit. Nobody measured how often 0.99 is right on Larkfield’s messages. Here, it is right 86.6% of the time.

Fix: never use a written confidence as a threshold before you check it on labelled messages. Group the answers by their written confidence and compare with the accuracy, as in the table above. The next lesson turns this check into one number. Ask for a decision, not for text shows how to get a real probability from a model.

Consequence in AI work: a threshold on an unchecked confidence looks like a safety net, and it lets the most confident mistakes through. Those are the mistakes that nobody reviews.

Check your understanding

The LLM's answers were 100% valid JSON with a known route. Why is that not enough for the router? Use one example from this lesson.

Show the answer

Valid means the answer has the right shape: JSON, one of the seven route names, a confidence from 0 to 1. It does not mean the route is right. For example, M-101926 says that a bath mat is missing and that the customer was charged twice. The LLM answered needs_person with confidence 0.96, and the answer passed every check. But Larkfield’s guidelines send it to delivery, the route of the first request. Overall, the LLM was valid 100% of the time and right 76.5% of the time.

Check your understanding

With the route names only, the LLM sends 287 team messages and 10 of the 12 account takeovers to needs_person. What does it lack, and what does that cost here?

Show the answer

It lacks Larkfield’s rules. Its common sense says that two requests, an upset customer or a takeover need a person. Larkfield’s guidelines say: the route of the main request, and takeovers to the account team, which locks the account first. The route names do not carry those rules. In the cost table, each team message sent to a person costs US$2, so the 287 cost US$287.0 per 1,000 messages. For the takeovers, the bigger risk is the delay before the account is locked.

Check your understanding

A call costs about US$0.035 per 1,000 messages, and the LLM's total is US$1,049.0 per 1,000. Where does the rest come from, and what does that tell you about choosing a model here?

Show the answer

Almost all of it comes from the mistakes, priced by the cost table. Wrong queues cost US$302.0, and team messages sent to a person US$287.0. needs_person messages sent to a team cost US$280.0, and urgent messages in the wrong queue US$180.0. The tokens are about 30,000 times cheaper than the mistakes. So on this inbox, a cheaper model or fewer tokens barely changes the total. Fewer mistakes change it a lot. The price per token is not the cost that decides.

Recap

  • Zero-shot with an LLM: the route names in the system prompt, the message, and no example.
  • A schema with an enum fixes the shape of the answer; structured output makes the provider follow it. Your code still checks every answer, and a bad one goes to review.
  • On this inbox, the LLM with the route names only reaches 0.765 (US$1,049.0 per 1,000): worse than every trained model. “An LLM is always the most accurate” is not true here.
  • It sends 287 team messages and 10 of 12 takeovers to needs_person: common sense instead of Larkfield’s rules.
  • A call costs about US$0.035 per 1,000 messages and takes about 1.2 s. The mistakes cost far more.
  • A written confidence is text, not a measurement: 0.99 was right 86.6% of the time.

Next, we give the LLM what a new clerk needs: the description of each route, and a few labelled examples.

Next: Few-shot examples and label descriptions