Sentence embeddings for classification
- Lesson 13
- 35 min
- Module 4: Embeddings and small fine-tuned models
Where we are
In Module 3, we routed messages with TF-IDF, which counts words. So "my parcel never came" and "the delivery did not arrive" look different to it. In this lesson, we use a sentence embedding model that gives similar sentences similar numbers, in English and in French, and we train a simple classifier on top.
Lesson 13 of 37 · Text Classification and Routing: Send Each Message to the Right PlaceSentence embeddings for classification
1. The routing problem and its cost
2. Labels you can trust
3. Baselines: rules and classical models
4. Embeddings and small fine-tuned models
5. LLMs as classifiers
6. Decision models, cascades and distillation
7. Harder label shapes
8. Confidence and people in the loop
9. Run it in production
What you will be able to do
Give the router a sense of meaning, not only of words. A sentence embedding model turns each message into a list of numbers, so that messages with a similar meaning get similar numbers, in English and in French. A simple classifier on top of these numbers routes the messages.
NotebookDo this lesson in a notebook
The notebook has every code cell, task and check of this lesson. It compares two sentence pairs, embeds the 8,000 messages, trains logistic regression on the embeddings and compares it with TF-IDF in English and in French. Its setup cells install the packages and download the data and the model (about 470 MB, once), so it needs no local setup. About 35 minutes.
Colab and Kaggle open your own copy. They need a free Google or Kaggle account. Stuck? See the solution notebook.
Before you startWhat to know and to have ready
What you need to know
- TF-IDF, from Text as numbers: bag of words and TF-IDF.
- Logistic regression and the comparison table, from Naive Bayes, logistic regression and linear SVM.
- How a model turns words into vectors (helpful, not required): Tokens and embeddings in the deep learning course.
What you need on your computer
- Nothing for the notebook route (Colab or Kaggle). This module changes no file of your
larkrouteproject. - To run the notebook on your own computer instead, open it in Jupyter with Python 3.12. Its first cell installs the packages that it needs.
Tested on macOS (Apple silicon) with Python 3.12, sentence-transformers 6.2.0, PyTorch 2.15.0 (CPU) and scikit-learn 1.9.1, in Jupyter. Colab, Kaggle and Windows were not tested.
Two letters that say the same thing
Section titled “Two letters that say the same thing”At the post office, a new clerk sorts letters by the words on the envelope. “Parcel” goes to the parcel desk. One morning, a letter says “the delivery did not arrive”. It does not say “parcel”, so the new clerk puts it aside. An older clerk takes it and walks straight to the parcel desk. She does not match words. She understands what the customer means.
The next letter is in French: “mon colis n’est jamais arrivé”. The new clerk knows no French. The older clerk reads it and sends it to the same desk. For her, the language does not matter: the meaning is the same.
The TF-IDF model of Module 3 is the new clerk. This lesson gives the router something closer to the older clerk.
The problem: TF-IDF only sees words
Section titled “The problem: TF-IDF only sees words”TF-IDF gives one number to each word or word pair. Two messages look similar to it only when they share words. Here are three short messages:
| Message | Route | |
|---|---|---|
| A | my parcel never came |
delivery |
| B | the delivery did not arrive |
delivery |
| C | my refund never came |
returns_refunds |
A and B mean the same thing, but they share no word. A and C share three words (“my”, “never”, “came”), but they belong to two different teams. We compare two TF-IDF vectors with the cosine similarity, which the next section explains. It is 1 when two messages point the same way and 0 when they have nothing in common. With the project’s TF-IDF features, A and B get 0.000: nothing in common. A and C get 0.586.
So TF-IDF thinks that the refund message is closer to the parcel message than the other parcel message is. With 6,000 labelled messages, this rarely hurts: the model sees enough ways to say “parcel”. With fewer labels, with new wording, or with a language that has fewer training messages, it hurts more.
The idea: numbers for meaning
Section titled “The idea: numbers for meaning”A sentence embedding is a list of numbers that a model computes for a whole sentence. The model learned from many pairs of sentences that sentences with a similar meaning must get similar numbers. “Embedding” means that each sentence gets a place, a point, in a space of many dimensions. Similar sentences get places near each other.
The model of this lesson is paraphrase-multilingual-MiniLM-L12-v2. It was published in 2021, it has 118 million parameters and the Apache-2.0 licence, and it gives 384 numbers for each message. Its model card says that it works in more than 50 languages: an English sentence and its French translation get similar numbers. We call it the older model in this module, because a later lesson compares it with a newer one.
Cosine similarity: a tiny example
Section titled “Cosine similarity: a tiny example”384 numbers are too many to check by hand. So here is the same idea with two numbers per message. These numbers are made up for the example; think of the first number as “about delivery” and the second as “about money”.
| Message | Numbers | Length |
|---|---|---|
| A: my parcel never came | (3, 1) | √(3² + 1²) = √10 = 3.162 |
| B: the delivery did not arrive | (4, 1) | √(4² + 1²) = √17 = 4.123 |
| C: my refund never came | (1, 3) | √(1² + 3²) = √10 = 3.162 |
Text description of the chart
Three arrows start at (0, 0). A goes to (3, 1) and B goes to (4, 1): both lie close to the horizontal axis “about delivery”, with a very small angle between them. C goes to (1, 3), close to the vertical axis “about money”. The angle between A and C is much larger than the angle between A and B.
The dot product multiplies the numbers in the same position and adds the results. Then we divide it by the two lengths, so that a long message with large numbers does not look more similar only because of its size. The result is the cosine similarity: the cosine of the angle between the two arrows. It is 1 when the arrows point the same way.
| Pair | Dot product | Lengths multiplied | Cosine similarity |
|---|---|---|---|
| A and B | 3 × 4 + 1 × 1 = 13 | 3.162 × 4.123 = 13.04 | 13 ÷ 13.04 = 0.997 |
| A and C | 3 × 1 + 1 × 3 = 6 | 3.162 × 3.162 = 10 | 6 ÷ 10 = 0.600 |
| B and C | 4 × 1 + 1 × 3 = 7 | 4.123 × 3.162 = 13.04 | 7 ÷ 13.04 = 0.537 |
A and B are almost the same direction. C is further away from both. This is what we want for routing: the two delivery messages together, the refund message apart. The course Retrieval-augmented generation uses the same measure to find documents.
The same three messages with the real model
Section titled “The same three messages with the real model”Now the real model, with all 384 numbers. The sentence-transformers library gives each message its numbers. With normalize_embeddings=True, every embedding gets the length 1, so the cosine similarity is simply the dot product:
from sentence_transformers import SentenceTransformerembedder = SentenceTransformer("sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2", revision="e8f8c211226b894fcb81acc59f3b34ba3efd5f42", device="cpu")e = embedder.encode(["my parcel never came", "the delivery did not arrive", "my refund never came"], normalize_embeddings=True)print(e.shape, round(float(e[0] @ e[1]), 3), round(float(e[0] @ e[2]), 3), round(float(e[1] @ e[2]), 3))| Pair | TF-IDF | Older embedding model |
|---|---|---|
| A and B (same meaning, no shared word) | 0.000 | 0.697 |
| A and C (shared words, other team) | 0.586 | 0.668 |
| B and C | 0.000 | 0.496 |
The code prints (3, 384) and the three numbers. The real model puts A closer to B than to C, as we hoped, but only a little: 0.697 against 0.668. TF-IDF had it the wrong way round (0.000 against 0.586). The embedding knows meaning, but not perfectly: sentences with the same shape also get high numbers. The notebook shows a second case. “The delivery is late” and its French translation “la livraison est en retard” get 0.953. “The delivery is late” and “the payment is late”, which share three words, get 0.804.
A frozen model with a classifier on top
Section titled “A frozen model with a classifier on top”To route messages, we need more than similarity: we need a decision. The plan has two steps:
- The embedding model turns each message into 384 numbers. It stays frozen: its own parameters do not change, and it does not learn from Larkfield’s labels.
- A logistic regression, the same kind of model as in Module 3, learns from the 6,000 labelled training messages. It gets 384 numbers per message instead of 38,615 TF-IDF numbers.
from sklearn.linear_model import LogisticRegressionE_train = embedder.encode(train["text"].tolist(), batch_size=64, normalize_embeddings=True)E_test = embedder.encode(test["text"].tolist(), batch_size=64, normalize_embeddings=True)clf = LogisticRegression(C=10, max_iter=3000).fit(E_train, train["route"])rt.show([rt.score("embeddings MiniLM + logistic regression", test, clf.predict(E_test))])batch_size=64 means that the model reads 64 messages at a time. Embedding the 8,000 messages takes about a minute on a laptop CPU. Logistic regression then trains in a fraction of a second, because 384 numbers per message is very little.
The result on Larkfield’s inbox
Section titled “The result on Larkfield’s inbox”Before you read on, predict: will the embeddings beat TF-IDF (0.962), now that the router understands meaning?
| Method (2,000 test messages) | Accuracy | English | French | Cost per 1,000 |
|---|---|---|---|---|
| TF-IDF + logistic regression | 0.962 | 0.966 | 0.952 | US$296.0 |
| Embeddings, older model + logistic regression | 0.920 | 0.931 | 0.896 | US$779.0 |
On this inbox, the frozen embeddings lose, in English and in French. They make about twice as many mistakes, and the mistakes cost more than twice as much. They miss 2 of the 12 account takeovers (TF-IDF misses 1), which explains much of the cost. The notebook’s own run, with a slightly different setting, gives 0.918.
This is an honest surprise, and the reason matters. TF-IDF learns the exact words of Larkfield’s 6,000 messages, including habits of this inbox. The frozen model only knows general meaning. A straight line through its 384 numbers cannot separate seven routes as well as 38,615 word features can. Remember also that this inbox is easier than a real one: its words are regular. On real customer messages, with more ways to say the same thing, the balance can move. The case study Route banking questions compares the methods on real data.
The picture of the embeddings
Section titled “The picture of the embeddings”Can we see why? Each message has 384 numbers, and a picture has two dimensions. PCA (principal component analysis) finds the two directions in which the points spread the most, and draws each message at its place along these two directions.
Text description of the diagram
The chart shows 2,000 dots, one for each test message, in seven colours, one for each route. Product messages form a separate group on the left. Account messages form a separate group at the bottom. Delivery, returns_refunds and orders messages overlap in one large cloud in the upper middle; a dashed red ellipse marks it. Billing and needs_person messages sit close to this cloud. Messages that the embedding classifier routes wrongly are drawn larger, with a dark ring. Most of them are inside the ellipse. The two directions keep 16% of the spread of the 384 numbers, so the picture hides a lot. Two dots that touch here can be far apart in the full space.
The picture shows the problem in one look. A late parcel, a return of a parcel and a change to an order all talk about an order, a parcel and a date. In general meaning, they are neighbours. For Larkfield, they belong to three teams. The frozen model does not know Larkfield’s borders; only the small classifier on top learns them.
Read the picture with care. Two dimensions keep only 16% of the spread, so the real space separates the routes better than the picture suggests. Use such a picture to find where to look, not to measure.
Where the embeddings help
Section titled “Where the embeddings help”The embeddings still know things that TF-IDF does not. The notebook counts the French test messages that the embedding model routes correctly and TF-IDF routes wrongly: there are 9. The embedding model knows that “colis” is close to “parcel” because it learned from translated sentence pairs, not from Larkfield’s labels.
Embeddings help most in three cases:
- Few labels. With 8 labelled messages per route, TF-IDF has seen too few words. The next lesson measures this.
- New wording and other languages. A paraphrase or a language with few training messages still lands near the right messages.
- A starting point for learning. In Fine-tune a small encoder on CPU, the whole model learns from the labels. That model is the best on this inbox.
Speed, size and cost
Section titled “Speed, size and cost”An embedding model is much heavier than TF-IDF:
| TF-IDF + logistic regression | Older embedding model + logistic regression | |
|---|---|---|
| Size on disk | about 2.1 MB (the trained pipeline, compressed) | 470.6 MB (the model) + a tiny classifier |
| Time per message, one at a time | about 0.1 ms | 6.8 ms |
| Time per message, 64 at a time | 2.9 ms | |
| Training time | about 3 seconds | about a minute to embed, then under a second |
At the course’s price of US$0.05 per CPU-hour, even 6.8 ms per message costs far less than a cent per 1,000 messages. Here, the cost that matters is the cost of the mistakes: US$779.0 per 1,000 against US$296.0.
When to use frozen embeddings
Section titled “When to use frozen embeddings”- Use them when you have few labels, many languages, or messages whose wording changes often. Use them also as a quick second opinion next to TF-IDF.
- Do not use them alone when you have thousands of labelled messages in a regular domain, as here: TF-IDF is cheaper, faster and more accurate on this inbox.
- What they cost: a 470 MB model in memory and about 7 ms per message. Pin the download to a revision, so that the numbers do not change under you.
Guided practice: embeddings against TF-IDF
Section titled “Guided practice: embeddings against TF-IDF”- Open the notebook from the box Do this lesson in a notebook at the top of this page.
- Run the setup cells. The data cell ends with
Ready: 10000 messages (6000 train, 2000 validation, 2000 test), all synthetic. - In section 1, write your prediction. Then run the cell. You see
0.953for the French pair and0.804for the pair that shares words. - In section 2, run the cell that embeds the inbox. It prints
(6000, 384) (2000, 384). - In section 3, run the comparison. The embeddings get
0.918, TF-IDF0.962. - In section 5, run the speed test. One message at a time is several times slower than in batches.
Your turn: where the embeddings win
Section titled “Your turn: where the embeddings win”In section 4 of the notebook, count the French test messages that the embedding model routes correctly and TF-IDF routes wrongly. Put the number in fr_better. Predict first: more or fewer than 20?
Show the answer
fr = (test["language"] == "fr").to_numpy()emb_right = clf.predict(E_test) == test["route"].to_numpy()tfidf_right = tfidf_model.predict(test["text"]) == test["route"].to_numpy()fr_better = int((fr & emb_right & ~tfidf_right).sum())print(fr_better)9Nine of 589 French test messages. The embeddings help on a few messages, and lose on more of them. Count the other direction too (TF-IDF right, embeddings wrong): you get a much larger number.
Common mistake
Fast in the notebook, slow in the router
Symptom: Tomás measures the embedding model in a notebook. He embeds 2,000 messages in about 6 seconds and writes: “3 ms per message”. In the routing service, each message takes more than twice as long.
Cause: the notebook embedded the messages 64 at a time. The model then uses the processor well, and the time is shared by 64 messages. The routing service gets one message at a time, so it pays the full cost for each: 6.8 ms against 2.9 ms on our laptop.
Fix: measure the time per message the way the service will run: one message at a time, on the same kind of computer, after a warm-up. Report both numbers, and say which one the service sees. The notebook’s section 5 does this.
Consequence in AI work: a speed measured in batches makes a model look cheaper and faster than it is in production. Every time and cost per message in this course’s comparison table is measured one message at a time.
Knowledge check
Section titled “Knowledge check”Check your understanding
With the made-up numbers A = (3, 1) and C = (1, 3), compute the cosine similarity step by step. What does the result say about the two messages?
Show the answer
The dot product is 3 × 1 + 1 × 3 = 6. The lengths are √10 = 3.162 each, so their product is 10. The cosine similarity is 6 ÷ 10 = 0.600. A and B got 0.997. So A is much closer to B (the other delivery message) than to C (the refund message), even though A and C share three words. That is what an embedding is for: similar meaning gives a similar direction, whatever the words.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
The frozen embeddings + logistic regression reach 0.920 on Larkfield's test messages, and TF-IDF reaches 0.962. Tomás says: “Embeddings understand meaning, so the test must be wrong.” What do you answer?
Show the answer
The test is right; the expectation is wrong. The embedding model is frozen: it knows general meaning, not Larkfield’s borders between teams. In general meaning, a late parcel, a returned parcel and a changed order are neighbours, and the 2-D picture shows these three routes overlapping. Only the small classifier on top learns the borders, from 384 numbers. TF-IDF learns from 38,615 word features of Larkfield’s own 6,000 messages, which are regular on this synthetic inbox. Embeddings help with few labels and new wording; on this inbox with 6,000 labels, TF-IDF wins.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
The service must answer one message at a time. Which time per message do you put in the comparison table for the embedding model, 2.9 ms or 6.8 ms? Why?
Show the answer
6.8 ms, measured one message at a time. 2.9 ms is the time when the model reads 64 messages together and shares the work between them. The service does not get 64 messages at once, so it never sees that speed. A table that mixes batch times and single times compares methods unfairly.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Recap
- A sentence embedding is a list of numbers for a whole sentence (here 384). Sentences with a similar meaning get similar numbers, also across English and French.
- The cosine similarity compares two embeddings: the dot product divided by the two lengths. With the made-up numbers, A and B get 0.997 and A and C get 0.600. TF-IDF gives A and B 0 because they share no word.
- A frozen embedding model with logistic regression on top reaches 0.920 on this inbox (English 0.931, French 0.896, US$779.0 per 1,000). TF-IDF reaches 0.962 (US$296.0). The frozen model knows meaning, not Larkfield’s borders.
- The 2-D picture (PCA) shows delivery, returns and orders overlapping, with most wrong routes there. It keeps only part of the spread: use it to look, not to measure.
- Measure speed one message at a time: 6.8 ms, not 2.9 ms in batches.
Embeddings did not win with 6,000 labels. Next, we take most of the labels away, and see which method still works with 8 labelled messages per route.