Few labels with embeddings
- Lesson 14
- 30 min
- Module 4: Embeddings and small fine-tuned models
Where we are
Our embedding classifier learned from 6,000 labelled messages. A new team or a new product often has only a few. In this lesson, we classify with nearest examples and with SetFit-style training from a few examples per class, and we see when this beats TF-IDF.
Lesson 14 of 37 · Text Classification and Routing: Send Each Message to the Right PlaceFew labels with embeddings
1. The routing problem and its cost
2. Labels you can trust
3. Baselines: rules and classical models
4. Embeddings and small fine-tuned models
5. LLMs as classifiers
6. Decision models, cascades and distillation
7. Harder label shapes
8. Confidence and people in the loop
9. Run it in production
What you will be able to do
Route messages when each route has only a few labelled examples. This happens with a new team, a new product, or a new client of a routing service. Embeddings carry general meaning, so they need far fewer labels than TF-IDF, up to a point that you can measure.
NotebookDo this lesson in a notebook
The notebook has every code cell, task and check of this lesson. It works the nearest-centre example by hand and trains three methods on 8 and 32 labels per route. It runs SetFit, and shows how much one random draw changes the result. Its setup cells install the packages and download the data and the model, so it needs no local setup. About 30 minutes.
Colab and Kaggle open your own copy. They need a free Google or Kaggle account. Stuck? See the solution notebook.
Before you startWhat to know and to have ready
What you need to know
- Sentence embeddings and the cosine similarity, from Sentence embeddings for classification.
- TF-IDF with logistic regression, from Naive Bayes, logistic regression and linear SVM.
- How many labels Chloé can make, and active learning, from When you have few labels.
What you need on your computer
- Nothing for the notebook route. This module changes no file of your
larkrouteproject.
Tested on macOS (Apple silicon) with Python 3.12, sentence-transformers 6.2.0, SetFit 1.2.0, PyTorch 2.15.0 (CPU) and scikit-learn 1.9.1, in Jupyter. Colab, Kaggle and Windows were not tested.
A new desk with a handful of examples
Section titled “A new desk with a handful of examples”The post office opens a new desk for parcels to the islands. Nobody has sorted island parcels before. The manager gives the clerks a small folder: eight example envelopes for each desk.
The new clerk who matches exact words is lost. Eight envelopes show only a few of the words that customers use. The older clerk, who understands meaning, looks at the eight examples and gets the idea: “these are about boats and islands”. She can sort a new envelope that uses other words, because she compares meanings, not words.
Larkfield meets the same situation. When a new route appears, or when a new client starts with a routing service, there are not 6,000 labelled messages. There are the few that Chloé labelled in an afternoon.
The problem: TF-IDF needs many labels
Section titled “The problem: TF-IDF needs many labels”TF-IDF learns one weight per word, and it can only learn the words that it sees. With 8 labelled messages per route (56 in all), most words of the test messages never appear in training. On Larkfield’s inbox, TF-IDF with logistic regression then routes only about half of the test messages correctly (0.496, the mean of 5 random draws). With all 6,000 labels, it reached 0.962.
The frozen embedding model already knows that “parcel”, “colis” and “delivery” are related. It learned this from general text, before it ever saw a Larkfield message. So it should need fewer labels. How many fewer, and with which method?
Three ways to learn from a few labels
Section titled “Three ways to learn from a few labels”Nearest centre: the simplest idea
Section titled “Nearest centre: the simplest idea”The centre of a route is the average of its examples’ embeddings: one point that summarises the route. A new message goes to the route whose centre is the most similar. This method is also called nearest centroid (“centroid” is the geometric word for a centre). It needs no training at all: only averages.
Here is a tiny example with two routes, two examples each, and two made-up numbers per message:
| Message | Route | Numbers |
|---|---|---|
| D1 | delivery | (0.9, 0.1) |
| D2 | delivery | (0.7, 0.3) |
| B1 | billing | (0.2, 0.9) |
| B2 | billing | (0.0, 0.8) |
The delivery centre is ((0.9 + 0.7) ÷ 2, (0.1 + 0.3) ÷ 2) = (0.8, 0.2). The billing centre is ((0.2 + 0.0) ÷ 2, (0.9 + 0.8) ÷ 2) = (0.1, 0.85).
A new message N has the numbers (0.8, 0.3). Its cosine similarity is 0.994 with the delivery centre and 0.458 with the billing centre, so it goes to delivery.
Text description of the chart
Four example points: D1 at (0.9, 0.1) and D2 at (0.7, 0.3) for delivery, B1 at (0.2, 0.9) and B2 at (0.0, 0.8) for billing. Two arrows from (0, 0) show the centres: delivery at (0.8, 0.2), between D1 and D2, and billing at (0.1, 0.85), between B1 and B2. The new message N is a point at (0.8, 0.3), just above the delivery arrow. The angle between N and the delivery arrow is very small; the angle to the billing arrow is large.
The method is close to how a person learns from a few examples: “it looks like those”. It has one weakness: one average per route is a rough summary. A route with two kinds of message, such as “late” and “lost” deliveries, gets one centre somewhere between them.
Logistic regression on the embeddings
Section titled “Logistic regression on the embeddings”The second method is the one of the last lesson: logistic regression on the 384 numbers, trained on only the few labelled messages. It learns a weight for each of the 384 numbers and each route. With few examples, it has little to learn from, but it can still draw better borders than one average per route.
SetFit-style training
Section titled “SetFit-style training”The third method changes the embedding model itself. SetFit (sentence transformer fine-tuning) makes pairs from the few labelled messages. Two messages of the same route form a “similar” pair, and two of different routes form a “different” pair. With 56 messages, there are 1,540 possible pairs; SetFit used 1,120 of them here. SetFit trains the embedding model to bring similar pairs closer and to push different pairs apart. Then it trains logistic regression on top of the new embeddings.
It costs training time: 24.7 seconds on our laptop for 8 messages per route (22 seconds in the notebook’s run).
Larkfield: from 4 to 64 labels per route
Section titled “Larkfield: from 4 to 64 labels per route”Priya draws 4, 8, 16, 32 and 64 training messages per route at random. For each size, she makes 5 different random draws, trains the three methods on each, and scores all 2,000 test messages. The chart shows the mean of the 5 draws.
Before you look, predict: at which size does TF-IDF catch up with the embeddings?
- Embeddings + logistic regression
- TF-IDF + logistic regression
- Nearest centre (embeddings)
Text description of the chart
Three lines, from 4 to 64 labelled messages per route; each size is twice the one before. Embeddings with logistic regression: 54.4% with 4 per route, 65.5% with 8, 72.1% with 16, 77.5% with 32 and 82.0% with 64. TF-IDF with logistic regression starts much lower: 35.8%, 49.6%, 65.0%, 76.4% and 84.8%. The two lines almost meet at 32 per route, and TF-IDF is higher at 64. The nearest centre (dashed) starts at 52.0%, rises to 59.5% and 60.8%, and stays flat: 66.2% at 32 and 67.4% at 64.
| Labels per route | TF-IDF + LR | Embeddings + LR | Nearest centre |
|---|---|---|---|
| 4 (28 in all) | 0.358 | 0.544 | 0.520 |
| 8 (56) | 0.496 | 0.655 | 0.595 |
| 16 (112) | 0.650 | 0.721 | 0.608 |
| 32 (224) | 0.764 | 0.775 | 0.662 |
| 64 (448) | 0.848 | 0.820 | 0.674 |
Three things stand out on this inbox:
- With few labels, embeddings win by far. With 4 labels per route, logistic regression on embeddings gets 0.544, against 0.358 for TF-IDF: 18.6 points.
- TF-IDF catches up at about 32 per route and passes the embeddings at 64. Once it has seen enough words, the exact words of the inbox are worth more than general meaning. With 6,000 labels, the gap is 0.962 against 0.920.
- The nearest centre is good at the very start (0.520 with 4 per route, close to logistic regression), but it stops improving. One average per route cannot learn finer borders.
And SetFit? On the draw with 8 messages per route, SetFit-style training gets 0.630. Logistic regression on the frozen embeddings gets 0.632 on the same draw, and TF-IDF 0.510. On this inbox, SetFit’s 24.7 seconds of training buy nothing. On other data, it often helps: the case study Sort requests with few labels lets you test it on real requests in English and French.
What a “few labels” result costs
Section titled “What a “few labels” result costs”None of these models is ready to route Larkfield’s messages alone. Even the best one with 8 labels per route, at 0.655, gets a third of the messages wrong. A wrong queue costs US$4, and much more for an urgent message or an account takeover. With so few labels, the model’s job is different: it suggests a route, and a person checks it. Grace’s team still reads every message, but faster, and Chloé gets more labelled messages every day. The lesson When you have few labels showed how to choose which messages she labels next.
When to use which
Section titled “When to use which”- Nearest centre for the first day: no training, a result in seconds, and easy to explain (“this message is closest to these examples”). It is also how a new route can start with a handful of examples.
- Embeddings + logistic regression from about 4 to 32 labels per route: on this inbox, it is the best method in that range.
- TF-IDF once you have more than about 32 to 64 labels per route, as in Module 3. It is cheaper, faster and, on this inbox, more accurate.
- SetFit when you can afford a short training run and it wins on your validation messages. Here it did not.
The size where TF-IDF catches up depends on the data. On an inbox with more varied wording, or in more languages, it comes later. Measure it on your own messages, as Priya did.
Guided practice: three methods, few labels
Section titled “Guided practice: three methods, few labels”- Open the notebook from the box Do this lesson in a notebook at the top of this page.
- Run the setup cells, and the SetFit cell. It prints
setfit 1.2.0. - In section 1, compute the two centres on paper. Then run the cell. The new message goes to
delivery. - In section 2, run the cell that embeds the inbox. It prints
(6000, 384) (2000, 384). - In section 3, run the three methods on 8 labels per route. They print
(0.51, 0.632, 0.525). - In section 5, run SetFit. It prints about
0.63. - In section 6, run the 5 draws with 4 labels per route. Compare the highest and lowest result.
Your turn: 32 labels per route
Section titled “Your turn: 32 labels per route”In section 4 of the notebook, draw 32 messages per route with draw(32) and compute the accuracy of the nearest centre on the test messages. Predict first: will it be closer to 0.6 or to 0.8?
Show the answer
idx32 = draw(32)near32 = round(float((nearest_centre(idx32) == truth).mean()), 3)print(near32, three_methods(idx32))0.665 (0.758, 0.767, 0.665)The nearest centre gets 0.665 with 224 labels (the mean of 5 draws in the chart was 0.662). Logistic regression on the same embeddings does much better, and TF-IDF has almost caught up. One average per route is not enough when the routes overlap, as delivery, returns and orders do.
Common mistake
The lucky draw
Symptom: Tomás tries embeddings with 4 labels per route and gets 0.606. He tells Grace: “With 28 labels, the model gets 61% right.” A week later, Chloé labels 28 other messages. The same method gets 0.514.
Cause: with 4 examples per route, the choice of those 4 decides a lot. A typical example helps; an odd one misleads. Over 5 random draws, logistic regression on embeddings went from 0.514 to 0.606, and TF-IDF from 0.298 to 0.388. Tomás reported his best draw.
Fix: with few labels, run several draws (here 5) and report the mean and the range: “0.544, from 0.514 to 0.606”. Choose methods on the mean, not on one draw. When you choose which messages to label, prefer typical ones that a person agrees on.
Consequence in AI work: a few-label result from one draw can be off by 10 points. A team that plans its review work on the lucky number finds that its new route needs far more checking than it promised.
Knowledge check
Section titled “Knowledge check”Check your understanding
In the tiny example, a new message has the numbers (0.1, 0.7). Which route does the nearest centre choose? Show the two cosine similarities.
Show the answer
The centres are delivery (0.8, 0.2) and billing (0.1, 0.85). With delivery, the dot product is 0.1 × 0.8 + 0.7 × 0.2 = 0.22. The lengths are 0.707 and 0.825, so the cosine is 0.22 ÷ 0.583 = 0.377. With billing, the dot product is 0.1 × 0.1 + 0.7 × 0.85 = 0.605. The lengths are 0.707 and 0.856, so the cosine is 0.605 ÷ 0.605 = 1.000 (rounded). The message goes to billing.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
A new route starts with 8 labelled messages per route. Which method do you choose today, and when do you plan to change it? Use the numbers of this lesson.
Show the answer
Today: logistic regression on frozen embeddings. On this inbox, with 8 per route, it gets 0.655 against 0.496 for TF-IDF and 0.595 for the nearest centre (means of 5 draws). At 0.655, it only suggests routes; a person checks every message. Plan to compare again as the labels grow: TF-IDF catches up at about 32 per route and passes the embeddings at 64 (0.848 against 0.820). Check the crossing on your own validation messages, because it depends on the data.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
SetFit gets 0.630 with 8 labels per route, and logistic regression on the frozen embeddings gets 0.632 on the same draw. Tomás wants to use SetFit “because it fine-tunes the model”. What do you answer?
Show the answer
On this inbox and this draw, SetFit gives nothing: 0.630 against 0.632, and it costs 24.7 seconds of training plus a fine-tuned model to store and version. A more complex method is not better by itself. Compare on several draws and on validation messages; if SetFit wins there on your data, use it. The case study with few labels is a good place to test it on real requests.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Recap
- With few labels, general meaning helps. On this inbox, with 4 labels per route, embeddings + logistic regression get 0.544 and TF-IDF 0.358.
- The nearest centre averages each route’s embeddings and picks the most similar centre. It needs no training, is good at the very start, and stops improving early (0.674 at 64 per route).
- TF-IDF catches up at about 32 labels per route and passes the embeddings at 64 (0.848 against 0.820). Measure this crossing on your own data.
- SetFit fine-tunes the embedding model with pairs of examples. Here, with 8 per route, it gets 0.630, the same as plain logistic regression (0.632).
- With few labels, one random draw can be off by 10 points. Report the mean and the range of several draws.
So far, the embedding model never learned from Larkfield’s labels. Next, we let the whole model learn, on a laptop CPU, in about 6 minutes.