Skip to content
Français

Zero-shot without an LLM

  • Lesson 17
  • 35 min
  • Module 4: Embeddings and small fine-tuned models

Where we are

Every model so far learned from labelled examples. What if a new label has no examples yet? In this lesson, we classify with the label names only: an NLI model checks each label one by one, and a GLiClass model compares the message with all labels at once.

Lesson 17 of 37 · Text Classification and Routing: Send Each Message to the Right PlaceZero-shot without an LLM

What you will be able to do

Route messages before anyone has labelled a single one, for example on the first day of a new route. Zero-shot models read a short description of each route instead of examples. They are useful to start and to suggest; on this inbox, they are far too weak to route alone.

NotebookDo this lesson in a notebookColab · Kaggle · GitHub

The notebook has every code cell, task and check of this lesson. It runs the NLI model and the GLiClass model on one message, with every score visible. It runs GLiClass on the 2,000 test messages, replays the recorded NLI run, and adds the new Grow Hub route without training. Its setup cells install the packages and download the data and the two models (about 550 MB and 280 MB), so it needs no local setup. About 35 minutes.

Colab and Kaggle open your own copy. They need a free Google or Kaggle account. Stuck? See the solution notebook.

Before you startWhat to know and to have readyAbout 35 minutes

What you need to know

What you need on your computer

  • Nothing for the notebook route. This module changes no file of your larkroute project.

Tested on macOS (Apple silicon) with Python 3.12, Transformers 5.22.2, GLiClass 0.1.22 and PyTorch 2.15.0 (CPU), in Jupyter. Colab, Kaggle and Windows were not tested.

A temporary worker arrives at the post office on a busy morning. Nobody has time to train him. He gets no example letters, only the signs above the desks: “Parcels”, “Returns”, “Payments”, “Complaints to the manager”. He reads each letter and puts it under the sign that seems to fit.

He does better than nothing. A letter about a lost parcel goes to “Parcels”. But he makes strange mistakes. A letter that names the product, “my garden chair never came”, may go to the desk for product questions, because the letter mentions a product. The signs are short, and he takes them literally.

Every model in this module so far learned from labelled messages: 6,000, or at least a few per route. In the launch month, Larkfield starts to sell the Grow Hub, a smart watering device: an indoor garden with a Wi-Fi controller and soil sensors. Customers write about setting it up, and no route exists for them yet. On the first day of a new route, there are no labelled messages at all.

The models of the earlier lessons cannot help: a classifier can only choose among the routes that it was trained on. We need a model that can read a route’s description and decide, with no example.

Zero-shot means “with zero examples”: the model gets only a description of each class. We give each route a short phrase, the labelling guidelines in a few words:

Route Phrase
orders changing or cancelling an order
delivery a late or missing delivery
returns_refunds a return, a refund or a damaged item
billing a charge, a payment or an invoice
product a question about a product
account a customer account or login
needs_person a safety problem, a legal threat or a complaint to a manager

Two kinds of model can use these phrases without an LLM.

NLI (natural language inference) is a task: given two sentences, does the first one support the second? An NLI model was trained on many such pairs. For routing, the first sentence is the message. The second is a guess: “This message is about a late or missing delivery.” The model gives a score for “supports”. We ask seven guesses, one per route, and the route with the best score wins.

The model here is mDeBERTa-v3-base-xnli-multilingual-nli-2mil7: multilingual, about 280 million parameters, MIT licence. It must read the message seven times, once with each guess.

A GLiClass model (generalist lightweight classifier) reads the message and all the phrases together, in one pass. It gives each phrase a score at the same time. Adding a route makes the input a little longer, not the work seven times larger. The model here is gliclass-multilang-edge: multilingual, 143 million parameters, Apache-2.0 licence.

Here is one test message, M-100597, labelled delivery:

Great, the tracking for LK-354854 hasn’t moved in 6 days. Have my solar lights been lost?
Two columns. Left, NLI: the message, then seven guesses of the form 'This message is about …?', each with its score. The guess 'a question about a product' has the highest score and is highlighted. NLI chooses product. Right, GLiClass: the message plus the seven route phrases read together, then seven route names with their scores. Delivery has the highest score and is highlighted. GLiClass chooses delivery.
One message, two zero-shot models. NLI checks seven guesses, one at a time, and the product guess wins because the message names a product. GLiClass reads all seven phrases at once and chooses delivery, the right route.
Text description of the diagram

The left column is the NLI model. The message is at the top. Below it are seven guesses, one per route, each of the form “This message is about …?”, with its score. The scores are: orders 0.06, delivery 0.241, returns_refunds 0.039, billing 0.017, product 0.569, account 0.07, needs_person 0.004. The product guess has the highest score, so NLI chooses product. The right column is the GLiClass model. The message and the seven phrases go in together, in one pass. Its scores, divided by their sum, are: orders 0.039, delivery 0.471, returns_refunds 0.017, billing 0.002, product 0.243, account 0.02, needs_person 0.208. Delivery has the highest score, so GLiClass chooses delivery.

RouteNLI scoreGLiClass share
orders0.0600.039
delivery0.2410.471
returns_refunds0.0390.017
billing0.0170.002
product0.5690.243
account0.0700.020
needs_person0.0040.208

The NLI pipeline turns the seven “supports” scores into shares that add up to 1. GLiClass gives each phrase its own score from 0 to 1. We divide them by their sum, so that the two columns compare. The NLI model chose product: the message names a product, “solar lights”, and the guess “a question about a product” fitted. GLiClass chose delivery, the right route.

Time matters too. On our laptop, NLI took about 0.7 s for this message (seven passes), and GLiClass about 17 ms. Your times will differ.

Priya runs both models on the 2,000 test messages, with the same seven phrases. Neither model ever saw a Larkfield label. Before you look, predict: will either beat “everything to a person”, which costs US$1,500 per 1,000 messages?

  • Zero-shot: no Larkfield labels
  • Trained on 6,000 labels
Test accuracy of the two zero-shot models next to three trained models: frozen embeddings, TF-IDF and the fine-tuned (tuned) encoder. With route phrases only, GLiClass gets half the messages right and NLI a third.
Text description of the chart

A column chart with five columns, on an axis from 0 to 100%. The two zero-shot models, which never saw a Larkfield label: NLI 32.9% and GLiClass 51.6%. The three trained models of this course: frozen embeddings with logistic regression (Frozen) 92.0%, TF-IDF with logistic regression 96.2% and the fine-tuned encoder (Tuned) 98.0%.

Method (2,000 test messages) Accuracy English French Takeovers found needs_person found Cost per 1,000 Time per message
GLiClass zero-shot 0.516 0.519 0.508 9 of 12 22 of 99 US$4,431.0 21 ms
NLI zero-shot 0.329 0.334 0.317 8 of 12 0 of 99 US$6,570.0 0.8 s
Everything to a person US$1,500.0

On this inbox, both zero-shot models cost far more than a person who reads every message. A wrong route costs US$4 or more, and US$40 for a needs_person message that goes to a team. Half the messages wrong is worse than no routing. NLI never chose needs_person at all, so every safety problem and legal threat in the test set went to a team.

The predicted routes show the pattern. NLI chose product for 1,082 of the 2,000 test messages, although only 289 are product questions. Almost every Larkfield message names a product (“solar lights”, “garden chair”, “herb kit”), so “a question about a product” fits almost everything. NLI chose returns_refunds only 33 times, for 404 real return messages. GLiClass leans the other way: it chose delivery 797 times, for 461 real delivery messages.

A short phrase cannot carry the labelling guidelines. “A return, a refund or a damaged item” does not say “a refund request that names a product is still a return”. The trained models learned these borders from 6,000 examples. A zero-shot model only has the words of the phrase.

Speed is the other problem for NLI. It reads each message seven times, so it needs about 0.8 seconds per message in the table’s run. Times depend on the computer and on what else it is doing: the worked example above took 0.7 s for one message. Either way, NLI is many times slower than GLiClass. With eight routes, it would need eight passes. GLiClass reads all routes in one pass, in about 21 ms.

Zero-shot models have one real strength: a new route is one more phrase. Priya adds the Grow Hub route with the phrase “setting up a smart watering device and its app”, and scores the 250 launch messages that Chloé labelled (data/launch_labels_sample.csv). 80 of them are about setting up the device.

Model Device messages found (of 80) Recall Precision Accuracy on the other 170
GLiClass 31 0.388 0.838 0.553
NLI 0 0.000 0.341

Recall is the share of the 80 device messages that get the new route. Precision is the share of the messages given the new route that really are device messages. GLiClass finds 31 device messages, and when it says devices, it is right 84% of the time. On the first day, before any label exists, that is a useful signal: a list of likely Grow Hub messages for Chloé to check and label first. NLI finds none.

Module 5 asks a hosted LLM the same question: the route names only, no examples. It reaches 0.765 on the test messages, much better than both models here, at about 1.2 seconds per message and a price per token. With descriptions and 8 similar examples in the prompt, it reaches 0.970. The lesson Zero-shot classification with an LLM starts there.

When to use zero-shot models without an LLM

Section titled “When to use zero-shot models without an LLM”
  • Use them on the first day of a new route or a new project. They can sort a pile of unlabelled messages for a person, suggest which messages to label first, or watch for a new topic.
  • Prefer GLiClass over NLI when you have more than a few routes: one pass instead of one pass per route.
  • Do not use them as the router when you have labels. On this inbox, even 8 labels per route with embeddings (0.655) beat both.
  • What they cost: a model of 280 MB to 550 MB, and on a CPU from tens of milliseconds (GLiClass) to seconds (NLI) per message. Their mistakes cost the most of all.
  1. Open the notebook from the box Do this lesson in a notebook at the top of this page.
  2. Run the setup cells, and the GLiClass cell. It prints gliclass 0.1.22.
  3. In section 1, read the seven phrases and the example message.
  4. In section 2, write your prediction. Then run NLI on the message. It puts product first.
  5. In section 3, run GLiClass on the message. It puts delivery first.
  6. In section 4, run GLiClass on the 2,000 test messages and replay NLI. GLiClass gets 0.516, NLI 0.329.

In section 5 of the notebook, score Chloé’s 250 launch messages with GLiClass, with the seven phrases plus the new devices phrase. Compute the recall and the precision of devices. Predict first: will GLiClass find more than half of the 80 device messages?

Show the answer
pred = gli_scores(launch["text"], NEW.values()).set_axis(list(NEW), axis=1).idxmax(axis=1).to_numpy()
is_dev = launch["route"].to_numpy() == "devices"
dev_recall = round(float((pred[is_dev] == "devices").mean()), 3)
dev_precision = round(float((is_dev[pred == "devices"]).mean()), 3)
print(dev_recall, dev_precision)
0.388 0.838

GLiClass finds 31 of the 80 device messages, and 31 of its 37 devices answers are right. Try another phrase for the new route. If you keep the phrase that does best on these 250 messages, remember that you chose it on them: test it again on new launch messages.

Common mistake

The phrase that takes everything

Symptom: with the seven phrases, the NLI model sends 1,082 of 2,000 test messages to product. Delivery complaints, return requests and order changes all land in the product team’s queue.

Cause: almost every Larkfield message names a product, and the phrase for the product route is “a question about a product”. For the model, “this message is about a question about a product” is true of a message like “my solar lights have not arrived”. The phrase is too wide: it describes something that most messages contain, not what makes a message belong to the product team.

Fix: write phrases that separate the routes, not phrases that describe them. Test each phrase on a small labelled set, and look at which route takes too many messages. For the product route, a phrase like “how a product works, its size or its materials, before or after buying” is narrower. Then test again on other messages, because a phrase tuned on one set can be tuned to its accidents.

Consequence in AI work: with zero-shot models, the label descriptions are the model’s only training data. A careless description silently moves hundreds of messages to one team, and the team’s queue is the first place where anyone notices.

Check your understanding

Explain the difference between the NLI model and the GLiClass model for one message and seven routes. Why does it matter for speed?

Show the answer

The NLI model reads the message together with one guess at a time, for example “This message is about a late or missing delivery.” So it reads the message seven times, one pass per route, and the best “supports” score wins. GLiClass reads the message and all seven phrases together, in one pass, and scores every phrase at once. NLI’s work grows with the number of routes: on our laptop it took close to a second per message, against about 21 ms for GLiClass. With more routes, the gap grows.

Check your understanding

GLiClass gets 0.516 on the test messages and costs US$4,431.0 per 1,000. Everything to a person costs US$1,500.0. How can a model that is right half the time cost three times more than doing nothing automatically?

Show the answer

A person costs US$1.50 per message and makes no routing mistake. A model that routes half the messages wrongly pays US$4 for each wrong queue and US$20 for an urgent one. It pays US$40 for each needs_person message sent to a team: GLiClass finds only 22 of 99. And it pays US$200 for each missed account takeover: it misses 3 of 12. Wrong routes cost much more than a review. So a weak router is worse than none.

Check your understanding

The Grow Hub route has no labels yet. How would you use GLiClass on the first day, and what would you not use it for? Use the launch numbers.

Show the answer

Use it to make a list of likely Grow Hub messages for Chloé. On her 250 launch messages, GLiClass finds 31 of the 80 device messages (recall 0.388). And 84% of the messages that it calls devices are right (precision 0.838). She checks and labels them first, and a trained model takes over once enough labels exist. Do not use it as the router: it misses 6 in 10 device messages. On the other 170 launch messages, it is right only 55% of the time.

Recap

  • Zero-shot models classify with a short phrase per route and no labelled example.
  • An NLI model checks one guess per route (“This message is about …”): seven passes, close to a second per message on a laptop. A GLiClass model reads all phrases in one pass: about 21 ms.
  • On this inbox, GLiClass reaches 0.516 and NLI 0.329. Both cost far more than everything to a person (US$4,431.0 and US$6,570.0 against US$1,500.0 per 1,000). NLI found none of the 99 needs_person messages.
  • The phrases are the model’s only training data. “A question about a product” took 1,082 of 2,000 messages, because almost every message names a product.
  • A new route is one more phrase. GLiClass finds 31 of 80 Grow Hub messages (precision 0.838) with no label: a good start for labelling, not a router.

The comparison table now has rules, TF-IDF, embeddings, a fine-tuned encoder and two zero-shot models. Next, we put every row’s accuracy next to everything it costs, and choose.

Next: What the extra accuracy costs