Skip to content
Français

Sentence embeddings for classification

  • Lesson 13
  • 35 min
  • Module 4: Embeddings and small fine-tuned models

Where we are

In Module 3, we routed messages with TF-IDF, which counts words. So "my parcel never came" and "the delivery did not arrive" look different to it. In this lesson, we use a sentence embedding model that gives similar sentences similar numbers, in English and in French, and we train a simple classifier on top.

Plays from YouTube. Nothing loads from YouTube until you press play. Privacy
Lesson 13 of 37 · Text Classification and Routing: Send Each Message to the Right PlaceSentence embeddings for classification

What you will be able to do

Give the router a sense of meaning, not only of words. A sentence embedding model turns each message into a list of numbers, so that messages with a similar meaning get similar numbers, in English and in French. A simple classifier on top of these numbers routes the messages.

NotebookDo this lesson in a notebookColab · Kaggle · GitHub

The notebook has every code cell, task and check of this lesson. It compares two sentence pairs, embeds the 8,000 messages, trains logistic regression on the embeddings and compares it with TF-IDF in English and in French. Its setup cells install the packages and download the data and the model (about 470 MB, once), so it needs no local setup. About 35 minutes.

Colab and Kaggle open your own copy. They need a free Google or Kaggle account. Stuck? See the solution notebook.

Before you startWhat to know and to have readyAbout 35 minutes

What you need to know

What you need on your computer

  • Nothing for the notebook route (Colab or Kaggle). This module changes no file of your larkroute project.
  • To run the notebook on your own computer instead, open it in Jupyter with Python 3.12. Its first cell installs the packages that it needs.

Tested on macOS (Apple silicon) with Python 3.12, sentence-transformers 6.2.0, PyTorch 2.15.0 (CPU) and scikit-learn 1.9.1, in Jupyter. Colab, Kaggle and Windows were not tested.

At the post office, a new clerk sorts letters by the words on the envelope. “Parcel” goes to the parcel desk. One morning, a letter says “the delivery did not arrive”. It does not say “parcel”, so the new clerk puts it aside. An older clerk takes it and walks straight to the parcel desk. She does not match words. She understands what the customer means.

The next letter is in French: “mon colis n’est jamais arrivé”. The new clerk knows no French. The older clerk reads it and sends it to the same desk. For her, the language does not matter: the meaning is the same.

The TF-IDF model of Module 3 is the new clerk. This lesson gives the router something closer to the older clerk.

TF-IDF gives one number to each word or word pair. Two messages look similar to it only when they share words. Here are three short messages:

Message Route
A my parcel never came delivery
B the delivery did not arrive delivery
C my refund never came returns_refunds

A and B mean the same thing, but they share no word. A and C share three words (“my”, “never”, “came”), but they belong to two different teams. We compare two TF-IDF vectors with the cosine similarity, which the next section explains. It is 1 when two messages point the same way and 0 when they have nothing in common. With the project’s TF-IDF features, A and B get 0.000: nothing in common. A and C get 0.586.

So TF-IDF thinks that the refund message is closer to the parcel message than the other parcel message is. With 6,000 labelled messages, this rarely hurts: the model sees enough ways to say “parcel”. With fewer labels, with new wording, or with a language that has fewer training messages, it hurts more.

A sentence embedding is a list of numbers that a model computes for a whole sentence. The model learned from many pairs of sentences that sentences with a similar meaning must get similar numbers. “Embedding” means that each sentence gets a place, a point, in a space of many dimensions. Similar sentences get places near each other.

The model of this lesson is paraphrase-multilingual-MiniLM-L12-v2. It was published in 2021, it has 118 million parameters and the Apache-2.0 licence, and it gives 384 numbers for each message. Its model card says that it works in more than 50 languages: an English sentence and its French translation get similar numbers. We call it the older model in this module, because a later lesson compares it with a newer one.

384 numbers are too many to check by hand. So here is the same idea with two numbers per message. These numbers are made up for the example; think of the first number as “about delivery” and the second as “about money”.

Message Numbers Length
A: my parcel never came (3, 1) √(3² + 1²) = √10 = 3.162
B: the delivery did not arrive (4, 1) √(4² + 1²) = √17 = 4.123
C: my refund never came (1, 3) √(1² + 3²) = √10 = 3.162
Three made-up embeddings with two numbers each. A and B point almost the same way; C points another way. The angle between two arrows is what the cosine similarity measures.
Text description of the chart

Three arrows start at (0, 0). A goes to (3, 1) and B goes to (4, 1): both lie close to the horizontal axis “about delivery”, with a very small angle between them. C goes to (1, 3), close to the vertical axis “about money”. The angle between A and C is much larger than the angle between A and B.

The dot product multiplies the numbers in the same position and adds the results. Then we divide it by the two lengths, so that a long message with large numbers does not look more similar only because of its size. The result is the cosine similarity: the cosine of the angle between the two arrows. It is 1 when the arrows point the same way.

Pair Dot product Lengths multiplied Cosine similarity
A and B 3 × 4 + 1 × 1 = 13 3.162 × 4.123 = 13.04 13 ÷ 13.04 = 0.997
A and C 3 × 1 + 1 × 3 = 6 3.162 × 3.162 = 10 6 ÷ 10 = 0.600
B and C 4 × 1 + 1 × 3 = 7 4.123 × 3.162 = 13.04 7 ÷ 13.04 = 0.537

A and B are almost the same direction. C is further away from both. This is what we want for routing: the two delivery messages together, the refund message apart. The course Retrieval-augmented generation uses the same measure to find documents.

The same three messages with the real model

Section titled “The same three messages with the real model”

Now the real model, with all 384 numbers. The sentence-transformers library gives each message its numbers. With normalize_embeddings=True, every embedding gets the length 1, so the cosine similarity is simply the dot product:

from sentence_transformers import SentenceTransformer
embedder = SentenceTransformer("sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2",
revision="e8f8c211226b894fcb81acc59f3b34ba3efd5f42", device="cpu")
e = embedder.encode(["my parcel never came", "the delivery did not arrive", "my refund never came"],
normalize_embeddings=True)
print(e.shape, round(float(e[0] @ e[1]), 3), round(float(e[0] @ e[2]), 3), round(float(e[1] @ e[2]), 3))
Pair TF-IDF Older embedding model
A and B (same meaning, no shared word) 0.000 0.697
A and C (shared words, other team) 0.586 0.668
B and C 0.000 0.496

The code prints (3, 384) and the three numbers. The real model puts A closer to B than to C, as we hoped, but only a little: 0.697 against 0.668. TF-IDF had it the wrong way round (0.000 against 0.586). The embedding knows meaning, but not perfectly: sentences with the same shape also get high numbers. The notebook shows a second case. “The delivery is late” and its French translation “la livraison est en retard” get 0.953. “The delivery is late” and “the payment is late”, which share three words, get 0.804.

To route messages, we need more than similarity: we need a decision. The plan has two steps:

  1. The embedding model turns each message into 384 numbers. It stays frozen: its own parameters do not change, and it does not learn from Larkfield’s labels.
  2. A logistic regression, the same kind of model as in Module 3, learns from the 6,000 labelled training messages. It gets 384 numbers per message instead of 38,615 TF-IDF numbers.
from sklearn.linear_model import LogisticRegression
E_train = embedder.encode(train["text"].tolist(), batch_size=64, normalize_embeddings=True)
E_test = embedder.encode(test["text"].tolist(), batch_size=64, normalize_embeddings=True)
clf = LogisticRegression(C=10, max_iter=3000).fit(E_train, train["route"])
rt.show([rt.score("embeddings MiniLM + logistic regression", test, clf.predict(E_test))])

batch_size=64 means that the model reads 64 messages at a time. Embedding the 8,000 messages takes about a minute on a laptop CPU. Logistic regression then trains in a fraction of a second, because 384 numbers per message is very little.

Before you read on, predict: will the embeddings beat TF-IDF (0.962), now that the router understands meaning?

Method (2,000 test messages) Accuracy English French Cost per 1,000
TF-IDF + logistic regression 0.962 0.966 0.952 US$296.0
Embeddings, older model + logistic regression 0.920 0.931 0.896 US$779.0

On this inbox, the frozen embeddings lose, in English and in French. They make about twice as many mistakes, and the mistakes cost more than twice as much. They miss 2 of the 12 account takeovers (TF-IDF misses 1), which explains much of the cost. The notebook’s own run, with a slightly different setting, gives 0.918.

This is an honest surprise, and the reason matters. TF-IDF learns the exact words of Larkfield’s 6,000 messages, including habits of this inbox. The frozen model only knows general meaning. A straight line through its 384 numbers cannot separate seven routes as well as 38,615 word features can. Remember also that this inbox is easier than a real one: its words are regular. On real customer messages, with more ways to say the same thing, the balance can move. The case study Route banking questions compares the methods on real data.

Can we see why? Each message has 384 numbers, and a picture has two dimensions. PCA (principal component analysis) finds the two directions in which the points spread the most, and draws each message at its place along these two directions.

A scatter plot of 2,000 small dots, one per test message, coloured by route. Product messages form a group on the left. Account messages form a group at the bottom. Delivery, returns_refunds and orders messages overlap in one large group in the upper middle, marked with a dashed red ellipse. Billing and needs_person dots sit near this group. The messages with a wrong route have a dark ring; most of them are inside the ellipse.
The 2,000 test messages, placed by the two main directions of their embeddings (older model). Product and account messages stand apart; delivery, returns and orders overlap, and most wrong routes are there.
Text description of the diagram

The chart shows 2,000 dots, one for each test message, in seven colours, one for each route. Product messages form a separate group on the left. Account messages form a separate group at the bottom. Delivery, returns_refunds and orders messages overlap in one large cloud in the upper middle; a dashed red ellipse marks it. Billing and needs_person messages sit close to this cloud. Messages that the embedding classifier routes wrongly are drawn larger, with a dark ring. Most of them are inside the ellipse. The two directions keep 16% of the spread of the 384 numbers, so the picture hides a lot. Two dots that touch here can be far apart in the full space.

The picture shows the problem in one look. A late parcel, a return of a parcel and a change to an order all talk about an order, a parcel and a date. In general meaning, they are neighbours. For Larkfield, they belong to three teams. The frozen model does not know Larkfield’s borders; only the small classifier on top learns them.

Read the picture with care. Two dimensions keep only 16% of the spread, so the real space separates the routes better than the picture suggests. Use such a picture to find where to look, not to measure.

The embeddings still know things that TF-IDF does not. The notebook counts the French test messages that the embedding model routes correctly and TF-IDF routes wrongly: there are 9. The embedding model knows that “colis” is close to “parcel” because it learned from translated sentence pairs, not from Larkfield’s labels.

Embeddings help most in three cases:

  • Few labels. With 8 labelled messages per route, TF-IDF has seen too few words. The next lesson measures this.
  • New wording and other languages. A paraphrase or a language with few training messages still lands near the right messages.
  • A starting point for learning. In Fine-tune a small encoder on CPU, the whole model learns from the labels. That model is the best on this inbox.

An embedding model is much heavier than TF-IDF:

TF-IDF + logistic regression Older embedding model + logistic regression
Size on disk about 2.1 MB (the trained pipeline, compressed) 470.6 MB (the model) + a tiny classifier
Time per message, one at a time about 0.1 ms 6.8 ms
Time per message, 64 at a time 2.9 ms
Training time about 3 seconds about a minute to embed, then under a second

At the course’s price of US$0.05 per CPU-hour, even 6.8 ms per message costs far less than a cent per 1,000 messages. Here, the cost that matters is the cost of the mistakes: US$779.0 per 1,000 against US$296.0.

  • Use them when you have few labels, many languages, or messages whose wording changes often. Use them also as a quick second opinion next to TF-IDF.
  • Do not use them alone when you have thousands of labelled messages in a regular domain, as here: TF-IDF is cheaper, faster and more accurate on this inbox.
  • What they cost: a 470 MB model in memory and about 7 ms per message. Pin the download to a revision, so that the numbers do not change under you.

Guided practice: embeddings against TF-IDF

Section titled “Guided practice: embeddings against TF-IDF”
  1. Open the notebook from the box Do this lesson in a notebook at the top of this page.
  2. Run the setup cells. The data cell ends with Ready: 10000 messages (6000 train, 2000 validation, 2000 test), all synthetic.
  3. In section 1, write your prediction. Then run the cell. You see 0.953 for the French pair and 0.804 for the pair that shares words.
  4. In section 2, run the cell that embeds the inbox. It prints (6000, 384) (2000, 384).
  5. In section 3, run the comparison. The embeddings get 0.918, TF-IDF 0.962.
  6. In section 5, run the speed test. One message at a time is several times slower than in batches.

In section 4 of the notebook, count the French test messages that the embedding model routes correctly and TF-IDF routes wrongly. Put the number in fr_better. Predict first: more or fewer than 20?

Show the answer
fr = (test["language"] == "fr").to_numpy()
emb_right = clf.predict(E_test) == test["route"].to_numpy()
tfidf_right = tfidf_model.predict(test["text"]) == test["route"].to_numpy()
fr_better = int((fr & emb_right & ~tfidf_right).sum())
print(fr_better)
9

Nine of 589 French test messages. The embeddings help on a few messages, and lose on more of them. Count the other direction too (TF-IDF right, embeddings wrong): you get a much larger number.

Common mistake

Fast in the notebook, slow in the router

Symptom: Tomás measures the embedding model in a notebook. He embeds 2,000 messages in about 6 seconds and writes: “3 ms per message”. In the routing service, each message takes more than twice as long.

Cause: the notebook embedded the messages 64 at a time. The model then uses the processor well, and the time is shared by 64 messages. The routing service gets one message at a time, so it pays the full cost for each: 6.8 ms against 2.9 ms on our laptop.

Fix: measure the time per message the way the service will run: one message at a time, on the same kind of computer, after a warm-up. Report both numbers, and say which one the service sees. The notebook’s section 5 does this.

Consequence in AI work: a speed measured in batches makes a model look cheaper and faster than it is in production. Every time and cost per message in this course’s comparison table is measured one message at a time.

Check your understanding

With the made-up numbers A = (3, 1) and C = (1, 3), compute the cosine similarity step by step. What does the result say about the two messages?

Show the answer

The dot product is 3 × 1 + 1 × 3 = 6. The lengths are √10 = 3.162 each, so their product is 10. The cosine similarity is 6 ÷ 10 = 0.600. A and B got 0.997. So A is much closer to B (the other delivery message) than to C (the refund message), even though A and C share three words. That is what an embedding is for: similar meaning gives a similar direction, whatever the words.

Check your understanding

The frozen embeddings + logistic regression reach 0.920 on Larkfield's test messages, and TF-IDF reaches 0.962. Tomás says: “Embeddings understand meaning, so the test must be wrong.” What do you answer?

Show the answer

The test is right; the expectation is wrong. The embedding model is frozen: it knows general meaning, not Larkfield’s borders between teams. In general meaning, a late parcel, a returned parcel and a changed order are neighbours, and the 2-D picture shows these three routes overlapping. Only the small classifier on top learns the borders, from 384 numbers. TF-IDF learns from 38,615 word features of Larkfield’s own 6,000 messages, which are regular on this synthetic inbox. Embeddings help with few labels and new wording; on this inbox with 6,000 labels, TF-IDF wins.

Check your understanding

The service must answer one message at a time. Which time per message do you put in the comparison table for the embedding model, 2.9 ms or 6.8 ms? Why?

Show the answer

6.8 ms, measured one message at a time. 2.9 ms is the time when the model reads 64 messages together and shares the work between them. The service does not get 64 messages at once, so it never sees that speed. A table that mixes batch times and single times compares methods unfairly.

Recap

  • A sentence embedding is a list of numbers for a whole sentence (here 384). Sentences with a similar meaning get similar numbers, also across English and French.
  • The cosine similarity compares two embeddings: the dot product divided by the two lengths. With the made-up numbers, A and B get 0.997 and A and C get 0.600. TF-IDF gives A and B 0 because they share no word.
  • A frozen embedding model with logistic regression on top reaches 0.920 on this inbox (English 0.931, French 0.896, US$779.0 per 1,000). TF-IDF reaches 0.962 (US$296.0). The frozen model knows meaning, not Larkfield’s borders.
  • The 2-D picture (PCA) shows delivery, returns and orders overlapping, with most wrong routes there. It keeps only part of the spread: use it to look, not to measure.
  • Measure speed one message at a time: 6.8 ms, not 2.9 ms in batches.

Embeddings did not win with 6,000 labels. Next, we take most of the labels away, and see which method still works with 8 labelled messages per route.

Next: Few labels with embeddings