Newer embedding models
- Lesson 16
- 30 min
- Module 4: Embeddings and small fine-tuned models
Where we are
We used one embedding model and fine-tuned one encoder. New embedding models appear every few months, each with a better leaderboard score. In this lesson, we compare an older and a newer small multilingual model on our own English and French messages.
Lesson 16 of 37 · Text Classification and Routing: Send Each Message to the Right PlaceNewer embedding models
1. The routing problem and its cost
2. Labels you can trust
3. Baselines: rules and classical models
4. Embeddings and small fine-tuned models
5. LLMs as classifiers
6. Decision models, cascades and distillation
7. Harder label shapes
8. Confidence and people in the loop
9. Run it in production
What you will be able to do
Decide whether a new model is better for your messages, before you change anything. A newer, smaller model with a higher leaderboard score can route your inbox worse. A small test on your own data answers the question in minutes.
NotebookDo this lesson in a notebook
The notebook has every code cell, task and check of this lesson. It loads both models and measures their speed one message at a time. It replays their scores on the test messages, finds the messages behind the gap, and runs a small test of your own. Its setup cells install the packages and download the data and the two models (about 470 MB and 200 MB), so it needs no local setup. About 30 minutes.
Colab and Kaggle open your own copy. They need a free Google or Kaggle account. Stuck? See the solution notebook.
Before you startWhat to know and to have ready
What you need to know
- Sentence embeddings with logistic regression, from Sentence embeddings for classification.
- Slices: a score on a part of the data, here English and French. The evaluation course explains slices in Evaluating and Testing AI Systems; this lesson uses only the two languages.
What you need on your computer
- Nothing for the notebook route. This module changes no file of your
larkrouteproject.
Tested on macOS (Apple silicon) with Python 3.12, sentence-transformers 6.2.0 and PyTorch 2.15.0 (CPU), in Jupyter. Colab, Kaggle and Windows were not tested.
The clerk with the best exam results
Section titled “The clerk with the best exam results”The post office needs a new clerk. One candidate has the best results in the national postal exam. The exam has hundreds of questions about many kinds of mail: letters, parcels, forms, foreign addresses. The other candidate is older, with average results.
The manager does a simple thing. He gives both candidates one morning of his office’s real letters. The candidate with the best exam results is slower, and makes more mistakes with the local addresses. The exam measured something real, but not the job of this office.
Embedding models have such exams too. They are called benchmarks, and their results are collected on public leaderboards.
The problem: a new model every few months
Section titled “The problem: a new model every few months”Since the older model of this module was published in 2021, many newer embedding models have appeared. Each new model card shows a table where the new model beats the old ones. Tomás reads one of these tables and says: “Let’s switch. The new model is newer, smaller and better.”
Is it better for Larkfield? The leaderboard cannot say. A leaderboard averages a model’s results over many public tasks: finding documents for a question, comparing sentences, grouping texts, in many languages. Your inbox is one task, with your routes, your wording and your mix of languages. A model can lead the average and still be worse at your task.
The idea: test on your own messages
Section titled “The idea: test on your own messages”The rule is short: a leaderboard tells you which models to try; your own test set tells you which model to use.
The most used leaderboard for embeddings is MTEB (Massive Text Embedding Benchmark). It groups tasks into families such as retrieval (finding documents), classification, clustering and similarity. Read three things before you trust a score:
- Which task family? A retrieval score says little about routing messages.
- Which languages? An average over 50 languages says little about French support messages.
- Measured on what hardware? Speeds on a large GPU say little about a CPU that answers one message at a time.
Then run the same test that you use for every method. Use the same training messages, the same 2,000 test messages and the same cost table, with English and French as separate slices.
A tiny example: same messages, different numbers
Section titled “A tiny example: same messages, different numbers”The two models of this lesson give different numbers to the same messages. Here are the three messages of the first lesson:
| Pair | Older model (2021) | Newer model (2026) |
|---|---|---|
A and B: my parcel never came / the delivery did not arrive |
0.697 | 0.909 |
A and C: my parcel never came / my refund never came |
0.668 | 0.875 |
B and C: the delivery did not arrive / my refund never came |
0.496 | 0.828 |
Both models put A closest to B. But the newer model gives every pair a high number (0.828 to 0.909), and the older one spreads them more (0.496 to 0.697). Is 0.875 “similar”? For the newer model, it is a fairly low value; for the older one, it would be very high. A number from one model cannot be compared with a number from another model: each model has its own space. Only the result on your task can be compared.
Two small multilingual models
Section titled “Two small multilingual models”| Older model | Newer model | |
|---|---|---|
| Name | paraphrase-multilingual-MiniLM-L12-v2 |
granite-embedding-97m-multilingual-r2 |
| Published | 2021 | 2026 |
| Parameters | 118 million | 97 million |
| Numbers per message | 384 | 384 |
| Layers | 12 | 12 |
| Word table | 250,037 tokens | 180,000 tokens |
| Longest input | 512 tokens | 32,768 tokens |
| Size on disk | 470.6 MB | 194.9 MB |
| Licence | Apache-2.0 | Apache-2.0 |
Both licences allow commercial use. The newer model is smaller mainly because its word table is smaller: 180,000 tokens instead of 250,037. It reads much longer texts. That helps for long documents, and it does not matter for support messages of about 28 words.
Its model card makes a vendor claim, which we did not measure. It says that the model scores 60.3 on the multilingual MTEB retrieval tasks, “the highest retrieval score of any open multilingual embedding model under 100M parameters”. The card measured its speed on a large GPU, in documents per second. Larkfield’s task is not retrieval, and Larkfield’s router runs on a CPU, one message at a time.
Larkfield: the newer model on our messages
Section titled “Larkfield: the newer model on our messages”Priya runs the same pipeline with both models. She embeds the 6,000 training messages and trains logistic regression on them, with the same setting for both. Then she scores the 2,000 test messages.
Before you look, predict: will the newer model win on accuracy, on French, or on speed?
- Older model (2021)
- Newer model (2026)
Text description of the chart
A column chart with four columns, on an axis from 80% to 100%. The older model gets 93.1% on the 1,411 English test messages and 89.6% on the 589 French ones. The newer model gets 92.1% in English and 86.8% in French. The newer model is 1.0 point behind in English and 2.8 points behind in French.
- TF-IDF + logistic regression
- Older model (2021)
- Newer model (2026), loaded in 16-bit or 32-bit numbers
Text description of the chart
A column chart with four columns. TF-IDF with logistic regression takes about 0.1 ms per message. The older embedding model takes 6.8 ms. The newer embedding model, loaded in its stored 16-bit numbers (the default), takes 26.9 ms. The same model loaded in 32-bit numbers takes 7.0 ms, about the same as the older model, with the same accuracy.
| Method (2,000 test messages) | Accuracy | Macro-F1 | Takeovers found | needs_person found |
Cost per 1,000 | ms per message |
|---|---|---|---|---|---|---|
| Embeddings, older model + LR | 0.920 | 0.924 | 10 of 12 | 92 of 99 | US$779.0 | 6.8 |
| Embeddings, newer model + LR | 0.906 | 0.904 | 11 of 12 | 84 of 99 | US$895.0 | 26.9 (7.0 in 32-bit) |
On this inbox, the newer model is worse on almost every column. It makes 189 mistakes against 159, and its French accuracy falls from 0.896 to 0.868. It misses 15 of the 99 messages that need a person; the older model misses 7. It finds one more account takeover, 11 of 12. That is good, but not enough to make up for the rest: the total cost is US$895.0 per 1,000 against US$779.0.
Why? We do not know for sure, and we did not measure the reasons. The newer model was trained mostly for retrieval and long documents. The older one was trained to give paraphrases similar numbers, which is close to what routing needs. But a plausible story is not a measurement. The measurement is the table.
The speed was a setting, not the model
Section titled “The speed was a setting, not the model”When the library loaded the newer model, it printed a warning: “This model was loaded in torch.bfloat16 on CPU. Depending on your CPU, inference may be significantly slower than with torch.float32.” bfloat16 is a way to store each number in 16 bits instead of 32. It halves the size of the model file, and many CPUs compute with it slowly.
The comparison table’s 26.9 ms used this default. We loaded the model again in 32-bit numbers, with model_kwargs={"dtype": torch.float32}, and measured again: 7.0 ms per message, about the same as the older model. The accuracy did not change: 0.906, and the cost stayed US$895.0 per 1,000. So “the newer model is four times slower” was wrong. A loading setting was slow, not the model.
Read the warnings that a library prints, and measure the way you will run. The newer model is still the less accurate one on this inbox; it is not the slower one.
The messages behind the gap
Section titled “The messages behind the gap”The two models do not simply make the same mistakes plus a few more. The notebook counts 72 test messages that only the newer model routes correctly, and 102 that only the older one does. The models disagree in both directions. If a new model is “a bit worse” on average, it can still be better on a slice that matters to you, for example account takeovers. Look at the slices before you decide.
A small test can point the other way
Section titled “A small test can point the other way”A full test with 6,000 labels takes a while, so a quick test is tempting. The notebook trains both models with only 100 labelled messages per route (700 in all) and scores the same 2,000 test messages:
| Labels for training | Older model | Newer model |
|---|---|---|
| 100 per route (700) | 0.836 | 0.859 |
| all 6,000 | 0.920 | 0.906 |
With 700 labels, the newer model wins, in English (0.878 against 0.851) and in French (0.813 against 0.798). With 6,000 labels, the older one wins. Both results are real. We did not measure why the order changes; what matters is that it changes with the amount of training data.
Test with the amount of labelled data that you will really use. If you will have 6,000 labels, a test with 700 can choose the wrong model.
What it would take to switch
Section titled “What it would take to switch”Changing an embedding model is never only a new name in the code:
- Embed again. Every stored embedding, for training and for search, comes from the old model and must be computed again.
- Train again. The classifier on top learned from the old model’s numbers; it means nothing on the new numbers. The failure case below shows what happens if you forget.
- Test again. On your test set, by slice, with the cost table, and with the time per message measured one at a time.
- Pin the version. Use a fixed revision, as this course does for every model, so that the model cannot change under you.
When to try a newer model
Section titled “When to try a newer model”- Try it when a new model is strong on your task family (here classification, not retrieval) and in your languages. Its size must fit your hardware, and its licence must fit your use.
- Switch only when it wins on your validation messages and your slices, and the gain pays for embedding, training and testing again.
- Do not switch because it is newer, smaller, or first on a leaderboard. On this inbox, the 2021 model is better than the 2026 one.
New models keep coming. Keep your test set and your comparison script ready, so that each new candidate takes an hour to judge, not a project.
Guided practice: test a newer model
Section titled “Guided practice: test a newer model”- Open the notebook from the box Do this lesson in a notebook at the top of this page.
- Run the setup cells.
- In section 1, write your prediction. Then load both models. The older has
117,653,760parameters and the newer97,441,152. - In section 2, measure the speed. The newer model is several times slower as loaded. Then run the next cell, which loads it in 32-bit numbers: it is about as fast as the older one.
- In section 3, replay the recorded scores. The older gets
0.92, the newer0.906. - In section 5, run your own small test with 100 labels per route. It takes about two minutes. The newer model wins this one:
0.859against0.836.
Your turn: the messages behind the gap
Section titled “Your turn: the messages behind the gap”In section 4 of the notebook, count the test messages that only the newer model routes correctly (new_only) and those that only the older one does (old_only). Predict first: is new_only close to zero?
Show the answer
truth = test["route"].to_numpy()old_right = old["prediction"].to_numpy() == truthnew_right = new["prediction"].to_numpy() == truthnew_only = int((new_right & ~old_right).sum())old_only = int((old_right & ~new_right).sum())print(new_only, old_only)72 102The difference, 102 − 72 = 30, is the gap between 189 and 159 mistakes. But there are 174 messages where exactly one of the two models is right. “Worse on average” hides many messages where the newer model is right.
Common mistake
A new model under an old classifier
Symptom: Tomás changes the model name in the code from the older model to the newer one. The service starts, every message gets a route, and no error appears. A week later, Grace’s team says that most messages arrive in the wrong queue.
Cause: the logistic regression on top was trained on the older model’s numbers. Both models give 384 numbers, so the shapes match and nothing fails. But number 17 of the newer model means something different from number 17 of the older one. We planted this mistake and measured it: the same classifier gets 0.920 on the older model’s embeddings and 0.178 on the newer model’s embeddings. It sends 1,550 of the 2,000 test messages to product.
Fix: treat the embedding model and the classifier as one unit with one version. When the embedding model changes, embed again, train again and test again. Save the embedding model’s name and revision next to the classifier, and refuse to load a classifier with another embedding model.
Consequence in AI work: “same shape” errors are silent. A model, an index or a cache built from one embedding model and used with another gives nonsense with full confidence. Version them together.
Knowledge check
Section titled “Knowledge check”Check your understanding
The newer model's card reports the best multilingual retrieval score in its size class. Name three reasons why this does not tell you its score on Larkfield's inbox.
Show the answer
First, the task family: the score is for retrieval (finding documents for a question), and routing is classification. Second, the data: an average over many public datasets and languages, not Larkfield’s routes, wording and its mix of English (1,411 test messages) and French (589). Third, the conditions: the card measured speed on a large GPU, in documents per second. Larkfield’s router runs on a CPU, one message at a time (6.8 ms for the older model; 7.0 ms for the newer one in 32-bit numbers, 26.9 ms in its stored 16-bit numbers). A vendor claim is a reason to test, not a result.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
On this inbox, the newer model gets 0.906 and the older 0.920. The newer model finds 11 of 12 account takeovers, the older 10. Would you switch? Use the cost table.
Show the answer
No. The newer model’s total cost is higher, US$895.0 per 1,000 against US$779.0. The extra takeover that it finds is worth US$200. It does not make up for 30 more mistakes overall, a worse French slice (0.868 against 0.896) and 15 missed needs_person messages against 7, each worth US$40. If takeovers matter most, a better answer is a model that finds all 12, such as the fine-tuned encoder (12 of 12, US$103.0).
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
Both models give 384 numbers per message. Tomás wants to keep the trained classifier and only change the embedding model. What happens, and what must he do instead?
Show the answer
Nothing fails, because the shapes match, but the classifier’s weights were learned for the older model’s numbers. Each number means something different in the newer model’s space, so the routes become close to random. In our planted test, the classifier fell from 0.920 to 0.178. He must embed the training messages again with the new model and train a new classifier. Then he tests it on the test set and its slices, and versions the embedding model and the classifier together.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Recap
- A leaderboard such as MTEB averages many public tasks. It tells you which models to try, not which one to use. Check the task family, the languages and the hardware of a score.
- On this inbox, the newer model (2026, 97 million parameters) is worse than the older one (2021, 118 million): 0.906 against 0.920, French 0.868 against 0.896, and US$895.0 against US$779.0 per 1,000.
- Newer is not better. Measure accuracy by slice, cost and time per message one at a time. Read the library’s warnings: in its stored 16-bit numbers, the newer model took 26.9 ms per message; in 32-bit numbers, 7.0 ms.
- A quick test can choose the wrong model: with 100 labels per route, the newer model wins (0.859 against 0.836). Test with the amount of data that you will really use.
- Switching models means embedding, training and testing again. A classifier trained on one model’s numbers gives nonsense on another model’s numbers, without an error.
- Pin every model to a revision, and keep the model card’s numbers labelled as vendor claims.
Every model so far learned from labelled examples. Next, we classify with the route names only, without a single labelled message.
Next: Zero-shot without an LLM