Zero-shot without an LLM
- Lesson 17
- 35 min
- Module 4: Embeddings and small fine-tuned models
Where we are
Every model so far learned from labelled examples. What if a new label has no examples yet? In this lesson, we classify with the label names only: an NLI model checks each label one by one, and a GLiClass model compares the message with all labels at once.
Lesson 17 of 37 · Text Classification and Routing: Send Each Message to the Right PlaceZero-shot without an LLM
1. The routing problem and its cost
2. Labels you can trust
3. Baselines: rules and classical models
4. Embeddings and small fine-tuned models
5. LLMs as classifiers
6. Decision models, cascades and distillation
7. Harder label shapes
8. Confidence and people in the loop
9. Run it in production
What you will be able to do
Route messages before anyone has labelled a single one, for example on the first day of a new route. Zero-shot models read a short description of each route instead of examples. They are useful to start and to suggest; on this inbox, they are far too weak to route alone.
NotebookDo this lesson in a notebook
The notebook has every code cell, task and check of this lesson. It runs the NLI model and the GLiClass model on one message, with every score visible. It runs GLiClass on the 2,000 test messages, replays the recorded NLI run, and adds the new Grow Hub route without training. Its setup cells install the packages and download the data and the two models (about 550 MB and 280 MB), so it needs no local setup. About 35 minutes.
Colab and Kaggle open your own copy. They need a free Google or Kaggle account. Stuck? See the solution notebook.
Before you startWhat to know and to have ready
What you need to know
- The seven routes and the labelling guidelines, from Design the labels and Labelling guidelines.
- The cost table and “everything to a person” (US$1,500 per 1,000 messages), from The cost of each mistake.
- The Grow Hub, the new product of the launch month, from the failure case of Text as numbers: bag of words and TF-IDF. This lesson introduces the 250 launch messages that Chloé labelled.
What you need on your computer
- Nothing for the notebook route. This module changes no file of your
larkrouteproject.
Tested on macOS (Apple silicon) with Python 3.12, Transformers 5.22.2, GLiClass 0.1.22 and PyTorch 2.15.0 (CPU), in Jupyter. Colab, Kaggle and Windows were not tested.
A temporary worker on the first day
Section titled “A temporary worker on the first day”A temporary worker arrives at the post office on a busy morning. Nobody has time to train him. He gets no example letters, only the signs above the desks: “Parcels”, “Returns”, “Payments”, “Complaints to the manager”. He reads each letter and puts it under the sign that seems to fit.
He does better than nothing. A letter about a lost parcel goes to “Parcels”. But he makes strange mistakes. A letter that names the product, “my garden chair never came”, may go to the desk for product questions, because the letter mentions a product. The signs are short, and he takes them literally.
The problem: a new route has no labels
Section titled “The problem: a new route has no labels”Every model in this module so far learned from labelled messages: 6,000, or at least a few per route. In the launch month, Larkfield starts to sell the Grow Hub, a smart watering device: an indoor garden with a Wi-Fi controller and soil sensors. Customers write about setting it up, and no route exists for them yet. On the first day of a new route, there are no labelled messages at all.
The models of the earlier lessons cannot help: a classifier can only choose among the routes that it was trained on. We need a model that can read a route’s description and decide, with no example.
The idea: zero-shot classification
Section titled “The idea: zero-shot classification”Zero-shot means “with zero examples”: the model gets only a description of each class. We give each route a short phrase, the labelling guidelines in a few words:
| Route | Phrase |
|---|---|
| orders | changing or cancelling an order |
| delivery | a late or missing delivery |
| returns_refunds | a return, a refund or a damaged item |
| billing | a charge, a payment or an invoice |
| product | a question about a product |
| account | a customer account or login |
| needs_person | a safety problem, a legal threat or a complaint to a manager |
Two kinds of model can use these phrases without an LLM.
NLI: one check per route
Section titled “NLI: one check per route”NLI (natural language inference) is a task: given two sentences, does the first one support the second? An NLI model was trained on many such pairs. For routing, the first sentence is the message. The second is a guess: “This message is about a late or missing delivery.” The model gives a score for “supports”. We ask seven guesses, one per route, and the route with the best score wins.
The model here is mDeBERTa-v3-base-xnli-multilingual-nli-2mil7: multilingual, about 280 million parameters, MIT licence. It must read the message seven times, once with each guess.
GLiClass: all routes in one pass
Section titled “GLiClass: all routes in one pass”A GLiClass model (generalist lightweight classifier) reads the message and all the phrases together, in one pass. It gives each phrase a score at the same time. Adding a route makes the input a little longer, not the work seven times larger. The model here is gliclass-multilang-edge: multilingual, 143 million parameters, Apache-2.0 licence.
A worked example with every score
Section titled “A worked example with every score”Here is one test message, M-100597, labelled delivery:
Great, the tracking for LK-354854 hasn’t moved in 6 days. Have my solar lights been lost?
Text description of the diagram
The left column is the NLI model. The message is at the top. Below it are seven guesses, one per route, each of the form “This message is about …?”, with its score. The scores are: orders 0.06, delivery 0.241, returns_refunds 0.039, billing 0.017, product 0.569, account 0.07, needs_person 0.004. The product guess has the highest score, so NLI chooses product. The right column is the GLiClass model. The message and the seven phrases go in together, in one pass. Its scores, divided by their sum, are: orders 0.039, delivery 0.471, returns_refunds 0.017, billing 0.002, product 0.243, account 0.02, needs_person 0.208. Delivery has the highest score, so GLiClass chooses delivery.
| Route | NLI score | GLiClass share |
|---|---|---|
orders | 0.060 | 0.039 |
delivery | 0.241 | 0.471 |
returns_refunds | 0.039 | 0.017 |
billing | 0.017 | 0.002 |
product | 0.569 | 0.243 |
account | 0.070 | 0.020 |
needs_person | 0.004 | 0.208 |
The NLI pipeline turns the seven “supports” scores into shares that add up to 1. GLiClass gives each phrase its own score from 0 to 1. We divide them by their sum, so that the two columns compare. The NLI model chose product: the message names a product, “solar lights”, and the guess “a question about a product” fitted. GLiClass chose delivery, the right route.
Time matters too. On our laptop, NLI took about 0.7 s for this message (seven passes), and GLiClass about 17 ms. Your times will differ.
Larkfield: zero-shot on the test messages
Section titled “Larkfield: zero-shot on the test messages”Priya runs both models on the 2,000 test messages, with the same seven phrases. Neither model ever saw a Larkfield label. Before you look, predict: will either beat “everything to a person”, which costs US$1,500 per 1,000 messages?
- Zero-shot: no Larkfield labels
- Trained on 6,000 labels
Text description of the chart
A column chart with five columns, on an axis from 0 to 100%. The two zero-shot models, which never saw a Larkfield label: NLI 32.9% and GLiClass 51.6%. The three trained models of this course: frozen embeddings with logistic regression (Frozen) 92.0%, TF-IDF with logistic regression 96.2% and the fine-tuned encoder (Tuned) 98.0%.
| Method (2,000 test messages) | Accuracy | English | French | Takeovers found | needs_person found |
Cost per 1,000 | Time per message |
|---|---|---|---|---|---|---|---|
| GLiClass zero-shot | 0.516 | 0.519 | 0.508 | 9 of 12 | 22 of 99 | US$4,431.0 | 21 ms |
| NLI zero-shot | 0.329 | 0.334 | 0.317 | 8 of 12 | 0 of 99 | US$6,570.0 | 0.8 s |
| Everything to a person | US$1,500.0 |
On this inbox, both zero-shot models cost far more than a person who reads every message. A wrong route costs US$4 or more, and US$40 for a needs_person message that goes to a team. Half the messages wrong is worse than no routing. NLI never chose needs_person at all, so every safety problem and legal threat in the test set went to a team.
Why they fail here
Section titled “Why they fail here”The predicted routes show the pattern. NLI chose product for 1,082 of the 2,000 test messages, although only 289 are product questions. Almost every Larkfield message names a product (“solar lights”, “garden chair”, “herb kit”), so “a question about a product” fits almost everything. NLI chose returns_refunds only 33 times, for 404 real return messages. GLiClass leans the other way: it chose delivery 797 times, for 461 real delivery messages.
A short phrase cannot carry the labelling guidelines. “A return, a refund or a damaged item” does not say “a refund request that names a product is still a return”. The trained models learned these borders from 6,000 examples. A zero-shot model only has the words of the phrase.
Speed is the other problem for NLI. It reads each message seven times, so it needs about 0.8 seconds per message in the table’s run. Times depend on the computer and on what else it is doing: the worked example above took 0.7 s for one message. Either way, NLI is many times slower than GLiClass. With eight routes, it would need eight passes. GLiClass reads all routes in one pass, in about 21 ms.
A new route without training
Section titled “A new route without training”Zero-shot models have one real strength: a new route is one more phrase. Priya adds the Grow Hub route with the phrase “setting up a smart watering device and its app”, and scores the 250 launch messages that Chloé labelled (data/launch_labels_sample.csv). 80 of them are about setting up the device.
| Model | Device messages found (of 80) | Recall | Precision | Accuracy on the other 170 |
|---|---|---|---|---|
| GLiClass | 31 | 0.388 | 0.838 | 0.553 |
| NLI | 0 | 0.000 | 0.341 |
Recall is the share of the 80 device messages that get the new route. Precision is the share of the messages given the new route that really are device messages. GLiClass finds 31 device messages, and when it says devices, it is right 84% of the time. On the first day, before any label exists, that is a useful signal: a list of likely Grow Hub messages for Chloé to check and label first. NLI finds none.
Zero-shot with an LLM, for comparison
Section titled “Zero-shot with an LLM, for comparison”Module 5 asks a hosted LLM the same question: the route names only, no examples. It reaches 0.765 on the test messages, much better than both models here, at about 1.2 seconds per message and a price per token. With descriptions and 8 similar examples in the prompt, it reaches 0.970. The lesson Zero-shot classification with an LLM starts there.
When to use zero-shot models without an LLM
Section titled “When to use zero-shot models without an LLM”- Use them on the first day of a new route or a new project. They can sort a pile of unlabelled messages for a person, suggest which messages to label first, or watch for a new topic.
- Prefer GLiClass over NLI when you have more than a few routes: one pass instead of one pass per route.
- Do not use them as the router when you have labels. On this inbox, even 8 labels per route with embeddings (0.655) beat both.
- What they cost: a model of 280 MB to 550 MB, and on a CPU from tens of milliseconds (GLiClass) to seconds (NLI) per message. Their mistakes cost the most of all.
Guided practice: two zero-shot models
Section titled “Guided practice: two zero-shot models”- Open the notebook from the box Do this lesson in a notebook at the top of this page.
- Run the setup cells, and the GLiClass cell. It prints
gliclass 0.1.22. - In section 1, read the seven phrases and the example message.
- In section 2, write your prediction. Then run NLI on the message. It puts
productfirst. - In section 3, run GLiClass on the message. It puts
deliveryfirst. - In section 4, run GLiClass on the 2,000 test messages and replay NLI. GLiClass gets
0.516, NLI0.329.
Your turn: a new route without training
Section titled “Your turn: a new route without training”In section 5 of the notebook, score Chloé’s 250 launch messages with GLiClass, with the seven phrases plus the new devices phrase. Compute the recall and the precision of devices. Predict first: will GLiClass find more than half of the 80 device messages?
Show the answer
pred = gli_scores(launch["text"], NEW.values()).set_axis(list(NEW), axis=1).idxmax(axis=1).to_numpy()is_dev = launch["route"].to_numpy() == "devices"dev_recall = round(float((pred[is_dev] == "devices").mean()), 3)dev_precision = round(float((is_dev[pred == "devices"]).mean()), 3)print(dev_recall, dev_precision)0.388 0.838GLiClass finds 31 of the 80 device messages, and 31 of its 37 devices answers are right. Try another phrase for the new route. If you keep the phrase that does best on these 250 messages, remember that you chose it on them: test it again on new launch messages.
Common mistake
The phrase that takes everything
Symptom: with the seven phrases, the NLI model sends 1,082 of 2,000 test messages to product. Delivery complaints, return requests and order changes all land in the product team’s queue.
Cause: almost every Larkfield message names a product, and the phrase for the product route is “a question about a product”. For the model, “this message is about a question about a product” is true of a message like “my solar lights have not arrived”. The phrase is too wide: it describes something that most messages contain, not what makes a message belong to the product team.
Fix: write phrases that separate the routes, not phrases that describe them. Test each phrase on a small labelled set, and look at which route takes too many messages. For the product route, a phrase like “how a product works, its size or its materials, before or after buying” is narrower. Then test again on other messages, because a phrase tuned on one set can be tuned to its accidents.
Consequence in AI work: with zero-shot models, the label descriptions are the model’s only training data. A careless description silently moves hundreds of messages to one team, and the team’s queue is the first place where anyone notices.
Knowledge check
Section titled “Knowledge check”Check your understanding
Explain the difference between the NLI model and the GLiClass model for one message and seven routes. Why does it matter for speed?
Show the answer
The NLI model reads the message together with one guess at a time, for example “This message is about a late or missing delivery.” So it reads the message seven times, one pass per route, and the best “supports” score wins. GLiClass reads the message and all seven phrases together, in one pass, and scores every phrase at once. NLI’s work grows with the number of routes: on our laptop it took close to a second per message, against about 21 ms for GLiClass. With more routes, the gap grows.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
GLiClass gets 0.516 on the test messages and costs US$4,431.0 per 1,000. Everything to a person costs US$1,500.0. How can a model that is right half the time cost three times more than doing nothing automatically?
Show the answer
A person costs US$1.50 per message and makes no routing mistake. A model that routes half the messages wrongly pays US$4 for each wrong queue and US$20 for an urgent one. It pays US$40 for each needs_person message sent to a team: GLiClass finds only 22 of 99. And it pays US$200 for each missed account takeover: it misses 3 of 12. Wrong routes cost much more than a review. So a weak router is worse than none.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
The Grow Hub route has no labels yet. How would you use GLiClass on the first day, and what would you not use it for? Use the launch numbers.
Show the answer
Use it to make a list of likely Grow Hub messages for Chloé. On her 250 launch messages, GLiClass finds 31 of the 80 device messages (recall 0.388). And 84% of the messages that it calls devices are right (precision 0.838). She checks and labels them first, and a trained model takes over once enough labels exist. Do not use it as the router: it misses 6 in 10 device messages. On the other 170 launch messages, it is right only 55% of the time.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Recap
- Zero-shot models classify with a short phrase per route and no labelled example.
- An NLI model checks one guess per route (“This message is about …”): seven passes, close to a second per message on a laptop. A GLiClass model reads all phrases in one pass: about 21 ms.
- On this inbox, GLiClass reaches 0.516 and NLI 0.329. Both cost far more than everything to a person (US$4,431.0 and US$6,570.0 against US$1,500.0 per 1,000). NLI found none of the 99
needs_personmessages. - The phrases are the model’s only training data. “A question about a product” took 1,082 of 2,000 messages, because almost every message names a product.
- A new route is one more phrase. GLiClass finds 31 of 80 Grow Hub messages (precision 0.838) with no label: a good start for labelling, not a router.
The comparison table now has rules, TF-IDF, embeddings, a fine-tuned encoder and two zero-shot models. Next, we put every row’s accuracy next to everything it costs, and choose.