Skip to content
Français

Few labels with embeddings

  • Lesson 14
  • 30 min
  • Module 4: Embeddings and small fine-tuned models

Where we are

Our embedding classifier learned from 6,000 labelled messages. A new team or a new product often has only a few. In this lesson, we classify with nearest examples and with SetFit-style training from a few examples per class, and we see when this beats TF-IDF.

Lesson 14 of 37 · Text Classification and Routing: Send Each Message to the Right PlaceFew labels with embeddings

What you will be able to do

Route messages when each route has only a few labelled examples. This happens with a new team, a new product, or a new client of a routing service. Embeddings carry general meaning, so they need far fewer labels than TF-IDF, up to a point that you can measure.

NotebookDo this lesson in a notebookColab · Kaggle · GitHub

The notebook has every code cell, task and check of this lesson. It works the nearest-centre example by hand and trains three methods on 8 and 32 labels per route. It runs SetFit, and shows how much one random draw changes the result. Its setup cells install the packages and download the data and the model, so it needs no local setup. About 30 minutes.

Colab and Kaggle open your own copy. They need a free Google or Kaggle account. Stuck? See the solution notebook.

Before you startWhat to know and to have readyAbout 30 minutes

What you need to know

What you need on your computer

  • Nothing for the notebook route. This module changes no file of your larkroute project.

Tested on macOS (Apple silicon) with Python 3.12, sentence-transformers 6.2.0, SetFit 1.2.0, PyTorch 2.15.0 (CPU) and scikit-learn 1.9.1, in Jupyter. Colab, Kaggle and Windows were not tested.

The post office opens a new desk for parcels to the islands. Nobody has sorted island parcels before. The manager gives the clerks a small folder: eight example envelopes for each desk.

The new clerk who matches exact words is lost. Eight envelopes show only a few of the words that customers use. The older clerk, who understands meaning, looks at the eight examples and gets the idea: “these are about boats and islands”. She can sort a new envelope that uses other words, because she compares meanings, not words.

Larkfield meets the same situation. When a new route appears, or when a new client starts with a routing service, there are not 6,000 labelled messages. There are the few that Chloé labelled in an afternoon.

TF-IDF learns one weight per word, and it can only learn the words that it sees. With 8 labelled messages per route (56 in all), most words of the test messages never appear in training. On Larkfield’s inbox, TF-IDF with logistic regression then routes only about half of the test messages correctly (0.496, the mean of 5 random draws). With all 6,000 labels, it reached 0.962.

The frozen embedding model already knows that “parcel”, “colis” and “delivery” are related. It learned this from general text, before it ever saw a Larkfield message. So it should need fewer labels. How many fewer, and with which method?

The centre of a route is the average of its examples’ embeddings: one point that summarises the route. A new message goes to the route whose centre is the most similar. This method is also called nearest centroid (“centroid” is the geometric word for a centre). It needs no training at all: only averages.

Here is a tiny example with two routes, two examples each, and two made-up numbers per message:

Message Route Numbers
D1 delivery (0.9, 0.1)
D2 delivery (0.7, 0.3)
B1 billing (0.2, 0.9)
B2 billing (0.0, 0.8)

The delivery centre is ((0.9 + 0.7) ÷ 2, (0.1 + 0.3) ÷ 2) = (0.8, 0.2). The billing centre is ((0.2 + 0.0) ÷ 2, (0.9 + 0.8) ÷ 2) = (0.1, 0.85).

A new message N has the numbers (0.8, 0.3). Its cosine similarity is 0.994 with the delivery centre and 0.458 with the billing centre, so it goes to delivery.

Two examples per route and their centres (arrows). The new message N points almost the same way as the delivery centre, so it goes to delivery. The numbers are made up for the example.
Text description of the chart

Four example points: D1 at (0.9, 0.1) and D2 at (0.7, 0.3) for delivery, B1 at (0.2, 0.9) and B2 at (0.0, 0.8) for billing. Two arrows from (0, 0) show the centres: delivery at (0.8, 0.2), between D1 and D2, and billing at (0.1, 0.85), between B1 and B2. The new message N is a point at (0.8, 0.3), just above the delivery arrow. The angle between N and the delivery arrow is very small; the angle to the billing arrow is large.

The method is close to how a person learns from a few examples: “it looks like those”. It has one weakness: one average per route is a rough summary. A route with two kinds of message, such as “late” and “lost” deliveries, gets one centre somewhere between them.

The second method is the one of the last lesson: logistic regression on the 384 numbers, trained on only the few labelled messages. It learns a weight for each of the 384 numbers and each route. With few examples, it has little to learn from, but it can still draw better borders than one average per route.

The third method changes the embedding model itself. SetFit (sentence transformer fine-tuning) makes pairs from the few labelled messages. Two messages of the same route form a “similar” pair, and two of different routes form a “different” pair. With 56 messages, there are 1,540 possible pairs; SetFit used 1,120 of them here. SetFit trains the embedding model to bring similar pairs closer and to push different pairs apart. Then it trains logistic regression on top of the new embeddings.

It costs training time: 24.7 seconds on our laptop for 8 messages per route (22 seconds in the notebook’s run).

Priya draws 4, 8, 16, 32 and 64 training messages per route at random. For each size, she makes 5 different random draws, trains the three methods on each, and scores all 2,000 test messages. The chart shows the mean of the 5 draws.

Before you look, predict: at which size does TF-IDF catch up with the embeddings?

  • Embeddings + logistic regression
  • TF-IDF + logistic regression
  • Nearest centre (embeddings)
Test accuracy with 4 to 64 labelled messages per route, the mean of 5 random draws. Embeddings win up to about 32 per route; then TF-IDF passes them. The nearest centre stops improving early.
Text description of the chart

Three lines, from 4 to 64 labelled messages per route; each size is twice the one before. Embeddings with logistic regression: 54.4% with 4 per route, 65.5% with 8, 72.1% with 16, 77.5% with 32 and 82.0% with 64. TF-IDF with logistic regression starts much lower: 35.8%, 49.6%, 65.0%, 76.4% and 84.8%. The two lines almost meet at 32 per route, and TF-IDF is higher at 64. The nearest centre (dashed) starts at 52.0%, rises to 59.5% and 60.8%, and stays flat: 66.2% at 32 and 67.4% at 64.

Labels per route TF-IDF + LR Embeddings + LR Nearest centre
4 (28 in all) 0.358 0.544 0.520
8 (56) 0.496 0.655 0.595
16 (112) 0.650 0.721 0.608
32 (224) 0.764 0.775 0.662
64 (448) 0.848 0.820 0.674

Three things stand out on this inbox:

  1. With few labels, embeddings win by far. With 4 labels per route, logistic regression on embeddings gets 0.544, against 0.358 for TF-IDF: 18.6 points.
  2. TF-IDF catches up at about 32 per route and passes the embeddings at 64. Once it has seen enough words, the exact words of the inbox are worth more than general meaning. With 6,000 labels, the gap is 0.962 against 0.920.
  3. The nearest centre is good at the very start (0.520 with 4 per route, close to logistic regression), but it stops improving. One average per route cannot learn finer borders.

And SetFit? On the draw with 8 messages per route, SetFit-style training gets 0.630. Logistic regression on the frozen embeddings gets 0.632 on the same draw, and TF-IDF 0.510. On this inbox, SetFit’s 24.7 seconds of training buy nothing. On other data, it often helps: the case study Sort requests with few labels lets you test it on real requests in English and French.

None of these models is ready to route Larkfield’s messages alone. Even the best one with 8 labels per route, at 0.655, gets a third of the messages wrong. A wrong queue costs US$4, and much more for an urgent message or an account takeover. With so few labels, the model’s job is different: it suggests a route, and a person checks it. Grace’s team still reads every message, but faster, and Chloé gets more labelled messages every day. The lesson When you have few labels showed how to choose which messages she labels next.

  • Nearest centre for the first day: no training, a result in seconds, and easy to explain (“this message is closest to these examples”). It is also how a new route can start with a handful of examples.
  • Embeddings + logistic regression from about 4 to 32 labels per route: on this inbox, it is the best method in that range.
  • TF-IDF once you have more than about 32 to 64 labels per route, as in Module 3. It is cheaper, faster and, on this inbox, more accurate.
  • SetFit when you can afford a short training run and it wins on your validation messages. Here it did not.

The size where TF-IDF catches up depends on the data. On an inbox with more varied wording, or in more languages, it comes later. Measure it on your own messages, as Priya did.

Guided practice: three methods, few labels

Section titled “Guided practice: three methods, few labels”
  1. Open the notebook from the box Do this lesson in a notebook at the top of this page.
  2. Run the setup cells, and the SetFit cell. It prints setfit 1.2.0.
  3. In section 1, compute the two centres on paper. Then run the cell. The new message goes to delivery.
  4. In section 2, run the cell that embeds the inbox. It prints (6000, 384) (2000, 384).
  5. In section 3, run the three methods on 8 labels per route. They print (0.51, 0.632, 0.525).
  6. In section 5, run SetFit. It prints about 0.63.
  7. In section 6, run the 5 draws with 4 labels per route. Compare the highest and lowest result.

In section 4 of the notebook, draw 32 messages per route with draw(32) and compute the accuracy of the nearest centre on the test messages. Predict first: will it be closer to 0.6 or to 0.8?

Show the answer
idx32 = draw(32)
near32 = round(float((nearest_centre(idx32) == truth).mean()), 3)
print(near32, three_methods(idx32))
0.665 (0.758, 0.767, 0.665)

The nearest centre gets 0.665 with 224 labels (the mean of 5 draws in the chart was 0.662). Logistic regression on the same embeddings does much better, and TF-IDF has almost caught up. One average per route is not enough when the routes overlap, as delivery, returns and orders do.

Common mistake

The lucky draw

Symptom: Tomás tries embeddings with 4 labels per route and gets 0.606. He tells Grace: “With 28 labels, the model gets 61% right.” A week later, Chloé labels 28 other messages. The same method gets 0.514.

Cause: with 4 examples per route, the choice of those 4 decides a lot. A typical example helps; an odd one misleads. Over 5 random draws, logistic regression on embeddings went from 0.514 to 0.606, and TF-IDF from 0.298 to 0.388. Tomás reported his best draw.

Fix: with few labels, run several draws (here 5) and report the mean and the range: “0.544, from 0.514 to 0.606”. Choose methods on the mean, not on one draw. When you choose which messages to label, prefer typical ones that a person agrees on.

Consequence in AI work: a few-label result from one draw can be off by 10 points. A team that plans its review work on the lucky number finds that its new route needs far more checking than it promised.

Check your understanding

In the tiny example, a new message has the numbers (0.1, 0.7). Which route does the nearest centre choose? Show the two cosine similarities.

Show the answer

The centres are delivery (0.8, 0.2) and billing (0.1, 0.85). With delivery, the dot product is 0.1 × 0.8 + 0.7 × 0.2 = 0.22. The lengths are 0.707 and 0.825, so the cosine is 0.22 ÷ 0.583 = 0.377. With billing, the dot product is 0.1 × 0.1 + 0.7 × 0.85 = 0.605. The lengths are 0.707 and 0.856, so the cosine is 0.605 ÷ 0.605 = 1.000 (rounded). The message goes to billing.

Check your understanding

A new route starts with 8 labelled messages per route. Which method do you choose today, and when do you plan to change it? Use the numbers of this lesson.

Show the answer

Today: logistic regression on frozen embeddings. On this inbox, with 8 per route, it gets 0.655 against 0.496 for TF-IDF and 0.595 for the nearest centre (means of 5 draws). At 0.655, it only suggests routes; a person checks every message. Plan to compare again as the labels grow: TF-IDF catches up at about 32 per route and passes the embeddings at 64 (0.848 against 0.820). Check the crossing on your own validation messages, because it depends on the data.

Check your understanding

SetFit gets 0.630 with 8 labels per route, and logistic regression on the frozen embeddings gets 0.632 on the same draw. Tomás wants to use SetFit “because it fine-tunes the model”. What do you answer?

Show the answer

On this inbox and this draw, SetFit gives nothing: 0.630 against 0.632, and it costs 24.7 seconds of training plus a fine-tuned model to store and version. A more complex method is not better by itself. Compare on several draws and on validation messages; if SetFit wins there on your data, use it. The case study with few labels is a good place to test it on real requests.

Recap

  • With few labels, general meaning helps. On this inbox, with 4 labels per route, embeddings + logistic regression get 0.544 and TF-IDF 0.358.
  • The nearest centre averages each route’s embeddings and picks the most similar centre. It needs no training, is good at the very start, and stops improving early (0.674 at 64 per route).
  • TF-IDF catches up at about 32 labels per route and passes the embeddings at 64 (0.848 against 0.820). Measure this crossing on your own data.
  • SetFit fine-tunes the embedding model with pairs of examples. Here, with 8 per route, it gets 0.630, the same as plain logistic regression (0.632).
  • With few labels, one random draw can be off by 10 points. Report the mean and the range of several draws.

So far, the embedding model never learned from Larkfield’s labels. Next, we let the whole model learn, on a laptop CPU, in about 6 minutes.

Next: Fine-tune a small encoder on CPU