Skip to content
Français

Newer embedding models

  • Lesson 16
  • 30 min
  • Module 4: Embeddings and small fine-tuned models

Where we are

We used one embedding model and fine-tuned one encoder. New embedding models appear every few months, each with a better leaderboard score. In this lesson, we compare an older and a newer small multilingual model on our own English and French messages.

Lesson 16 of 37 · Text Classification and Routing: Send Each Message to the Right PlaceNewer embedding models

What you will be able to do

Decide whether a new model is better for your messages, before you change anything. A newer, smaller model with a higher leaderboard score can route your inbox worse. A small test on your own data answers the question in minutes.

NotebookDo this lesson in a notebookColab · Kaggle · GitHub

The notebook has every code cell, task and check of this lesson. It loads both models and measures their speed one message at a time. It replays their scores on the test messages, finds the messages behind the gap, and runs a small test of your own. Its setup cells install the packages and download the data and the two models (about 470 MB and 200 MB), so it needs no local setup. About 30 minutes.

Colab and Kaggle open your own copy. They need a free Google or Kaggle account. Stuck? See the solution notebook.

Before you startWhat to know and to have readyAbout 30 minutes

What you need to know

What you need on your computer

  • Nothing for the notebook route. This module changes no file of your larkroute project.

Tested on macOS (Apple silicon) with Python 3.12, sentence-transformers 6.2.0 and PyTorch 2.15.0 (CPU), in Jupyter. Colab, Kaggle and Windows were not tested.

The post office needs a new clerk. One candidate has the best results in the national postal exam. The exam has hundreds of questions about many kinds of mail: letters, parcels, forms, foreign addresses. The other candidate is older, with average results.

The manager does a simple thing. He gives both candidates one morning of his office’s real letters. The candidate with the best exam results is slower, and makes more mistakes with the local addresses. The exam measured something real, but not the job of this office.

Embedding models have such exams too. They are called benchmarks, and their results are collected on public leaderboards.

Since the older model of this module was published in 2021, many newer embedding models have appeared. Each new model card shows a table where the new model beats the old ones. Tomás reads one of these tables and says: “Let’s switch. The new model is newer, smaller and better.”

Is it better for Larkfield? The leaderboard cannot say. A leaderboard averages a model’s results over many public tasks: finding documents for a question, comparing sentences, grouping texts, in many languages. Your inbox is one task, with your routes, your wording and your mix of languages. A model can lead the average and still be worse at your task.

The rule is short: a leaderboard tells you which models to try; your own test set tells you which model to use.

The most used leaderboard for embeddings is MTEB (Massive Text Embedding Benchmark). It groups tasks into families such as retrieval (finding documents), classification, clustering and similarity. Read three things before you trust a score:

  1. Which task family? A retrieval score says little about routing messages.
  2. Which languages? An average over 50 languages says little about French support messages.
  3. Measured on what hardware? Speeds on a large GPU say little about a CPU that answers one message at a time.

Then run the same test that you use for every method. Use the same training messages, the same 2,000 test messages and the same cost table, with English and French as separate slices.

A tiny example: same messages, different numbers

Section titled “A tiny example: same messages, different numbers”

The two models of this lesson give different numbers to the same messages. Here are the three messages of the first lesson:

Pair Older model (2021) Newer model (2026)
A and B: my parcel never came / the delivery did not arrive 0.697 0.909
A and C: my parcel never came / my refund never came 0.668 0.875
B and C: the delivery did not arrive / my refund never came 0.496 0.828

Both models put A closest to B. But the newer model gives every pair a high number (0.828 to 0.909), and the older one spreads them more (0.496 to 0.697). Is 0.875 “similar”? For the newer model, it is a fairly low value; for the older one, it would be very high. A number from one model cannot be compared with a number from another model: each model has its own space. Only the result on your task can be compared.

Older model Newer model
Name paraphrase-multilingual-MiniLM-L12-v2 granite-embedding-97m-multilingual-r2
Published 2021 2026
Parameters 118 million 97 million
Numbers per message 384 384
Layers 12 12
Word table 250,037 tokens 180,000 tokens
Longest input 512 tokens 32,768 tokens
Size on disk 470.6 MB 194.9 MB
Licence Apache-2.0 Apache-2.0

Both licences allow commercial use. The newer model is smaller mainly because its word table is smaller: 180,000 tokens instead of 250,037. It reads much longer texts. That helps for long documents, and it does not matter for support messages of about 28 words.

Its model card makes a vendor claim, which we did not measure. It says that the model scores 60.3 on the multilingual MTEB retrieval tasks, “the highest retrieval score of any open multilingual embedding model under 100M parameters”. The card measured its speed on a large GPU, in documents per second. Larkfield’s task is not retrieval, and Larkfield’s router runs on a CPU, one message at a time.

Larkfield: the newer model on our messages

Section titled “Larkfield: the newer model on our messages”

Priya runs the same pipeline with both models. She embeds the 6,000 training messages and trains logistic regression on them, with the same setting for both. Then she scores the 2,000 test messages.

Before you look, predict: will the newer model win on accuracy, on French, or on speed?

  • Older model (2021)
  • Newer model (2026)
Test accuracy by language, frozen embeddings + logistic regression. The newer model is behind in both languages, and further behind in French. The axis starts at 80%.
Text description of the chart

A column chart with four columns, on an axis from 80% to 100%. The older model gets 93.1% on the 1,411 English test messages and 89.6% on the 589 French ones. The newer model gets 92.1% in English and 86.8% in French. The newer model is 1.0 point behind in English and 2.8 points behind in French.

  • TF-IDF + logistic regression
  • Older model (2021)
  • Newer model (2026), loaded in 16-bit or 32-bit numbers
Time per message, one at a time, on a laptop CPU (4 threads). Loaded as it is stored, in 16-bit numbers, the newer model takes about four times as long as the older one. Loaded in 32-bit numbers, it takes about the same time.
Text description of the chart

A column chart with four columns. TF-IDF with logistic regression takes about 0.1 ms per message. The older embedding model takes 6.8 ms. The newer embedding model, loaded in its stored 16-bit numbers (the default), takes 26.9 ms. The same model loaded in 32-bit numbers takes 7.0 ms, about the same as the older model, with the same accuracy.

Method (2,000 test messages) Accuracy Macro-F1 Takeovers found needs_person found Cost per 1,000 ms per message
Embeddings, older model + LR 0.920 0.924 10 of 12 92 of 99 US$779.0 6.8
Embeddings, newer model + LR 0.906 0.904 11 of 12 84 of 99 US$895.0 26.9 (7.0 in 32-bit)

On this inbox, the newer model is worse on almost every column. It makes 189 mistakes against 159, and its French accuracy falls from 0.896 to 0.868. It misses 15 of the 99 messages that need a person; the older model misses 7. It finds one more account takeover, 11 of 12. That is good, but not enough to make up for the rest: the total cost is US$895.0 per 1,000 against US$779.0.

Why? We do not know for sure, and we did not measure the reasons. The newer model was trained mostly for retrieval and long documents. The older one was trained to give paraphrases similar numbers, which is close to what routing needs. But a plausible story is not a measurement. The measurement is the table.

When the library loaded the newer model, it printed a warning: “This model was loaded in torch.bfloat16 on CPU. Depending on your CPU, inference may be significantly slower than with torch.float32.” bfloat16 is a way to store each number in 16 bits instead of 32. It halves the size of the model file, and many CPUs compute with it slowly.

The comparison table’s 26.9 ms used this default. We loaded the model again in 32-bit numbers, with model_kwargs={"dtype": torch.float32}, and measured again: 7.0 ms per message, about the same as the older model. The accuracy did not change: 0.906, and the cost stayed US$895.0 per 1,000. So “the newer model is four times slower” was wrong. A loading setting was slow, not the model.

Read the warnings that a library prints, and measure the way you will run. The newer model is still the less accurate one on this inbox; it is not the slower one.

The two models do not simply make the same mistakes plus a few more. The notebook counts 72 test messages that only the newer model routes correctly, and 102 that only the older one does. The models disagree in both directions. If a new model is “a bit worse” on average, it can still be better on a slice that matters to you, for example account takeovers. Look at the slices before you decide.

A full test with 6,000 labels takes a while, so a quick test is tempting. The notebook trains both models with only 100 labelled messages per route (700 in all) and scores the same 2,000 test messages:

Labels for training Older model Newer model
100 per route (700) 0.836 0.859
all 6,000 0.920 0.906

With 700 labels, the newer model wins, in English (0.878 against 0.851) and in French (0.813 against 0.798). With 6,000 labels, the older one wins. Both results are real. We did not measure why the order changes; what matters is that it changes with the amount of training data.

Test with the amount of labelled data that you will really use. If you will have 6,000 labels, a test with 700 can choose the wrong model.

Changing an embedding model is never only a new name in the code:

  • Embed again. Every stored embedding, for training and for search, comes from the old model and must be computed again.
  • Train again. The classifier on top learned from the old model’s numbers; it means nothing on the new numbers. The failure case below shows what happens if you forget.
  • Test again. On your test set, by slice, with the cost table, and with the time per message measured one at a time.
  • Pin the version. Use a fixed revision, as this course does for every model, so that the model cannot change under you.
  • Try it when a new model is strong on your task family (here classification, not retrieval) and in your languages. Its size must fit your hardware, and its licence must fit your use.
  • Switch only when it wins on your validation messages and your slices, and the gain pays for embedding, training and testing again.
  • Do not switch because it is newer, smaller, or first on a leaderboard. On this inbox, the 2021 model is better than the 2026 one.

New models keep coming. Keep your test set and your comparison script ready, so that each new candidate takes an hour to judge, not a project.

  1. Open the notebook from the box Do this lesson in a notebook at the top of this page.
  2. Run the setup cells.
  3. In section 1, write your prediction. Then load both models. The older has 117,653,760 parameters and the newer 97,441,152.
  4. In section 2, measure the speed. The newer model is several times slower as loaded. Then run the next cell, which loads it in 32-bit numbers: it is about as fast as the older one.
  5. In section 3, replay the recorded scores. The older gets 0.92, the newer 0.906.
  6. In section 5, run your own small test with 100 labels per route. It takes about two minutes. The newer model wins this one: 0.859 against 0.836.

In section 4 of the notebook, count the test messages that only the newer model routes correctly (new_only) and those that only the older one does (old_only). Predict first: is new_only close to zero?

Show the answer
truth = test["route"].to_numpy()
old_right = old["prediction"].to_numpy() == truth
new_right = new["prediction"].to_numpy() == truth
new_only = int((new_right & ~old_right).sum())
old_only = int((old_right & ~new_right).sum())
print(new_only, old_only)
72 102

The difference, 102 − 72 = 30, is the gap between 189 and 159 mistakes. But there are 174 messages where exactly one of the two models is right. “Worse on average” hides many messages where the newer model is right.

Common mistake

A new model under an old classifier

Symptom: Tomás changes the model name in the code from the older model to the newer one. The service starts, every message gets a route, and no error appears. A week later, Grace’s team says that most messages arrive in the wrong queue.

Cause: the logistic regression on top was trained on the older model’s numbers. Both models give 384 numbers, so the shapes match and nothing fails. But number 17 of the newer model means something different from number 17 of the older one. We planted this mistake and measured it: the same classifier gets 0.920 on the older model’s embeddings and 0.178 on the newer model’s embeddings. It sends 1,550 of the 2,000 test messages to product.

Fix: treat the embedding model and the classifier as one unit with one version. When the embedding model changes, embed again, train again and test again. Save the embedding model’s name and revision next to the classifier, and refuse to load a classifier with another embedding model.

Consequence in AI work: “same shape” errors are silent. A model, an index or a cache built from one embedding model and used with another gives nonsense with full confidence. Version them together.

Check your understanding

The newer model's card reports the best multilingual retrieval score in its size class. Name three reasons why this does not tell you its score on Larkfield's inbox.

Show the answer

First, the task family: the score is for retrieval (finding documents for a question), and routing is classification. Second, the data: an average over many public datasets and languages, not Larkfield’s routes, wording and its mix of English (1,411 test messages) and French (589). Third, the conditions: the card measured speed on a large GPU, in documents per second. Larkfield’s router runs on a CPU, one message at a time (6.8 ms for the older model; 7.0 ms for the newer one in 32-bit numbers, 26.9 ms in its stored 16-bit numbers). A vendor claim is a reason to test, not a result.

Check your understanding

On this inbox, the newer model gets 0.906 and the older 0.920. The newer model finds 11 of 12 account takeovers, the older 10. Would you switch? Use the cost table.

Show the answer

No. The newer model’s total cost is higher, US$895.0 per 1,000 against US$779.0. The extra takeover that it finds is worth US$200. It does not make up for 30 more mistakes overall, a worse French slice (0.868 against 0.896) and 15 missed needs_person messages against 7, each worth US$40. If takeovers matter most, a better answer is a model that finds all 12, such as the fine-tuned encoder (12 of 12, US$103.0).

Check your understanding

Both models give 384 numbers per message. Tomás wants to keep the trained classifier and only change the embedding model. What happens, and what must he do instead?

Show the answer

Nothing fails, because the shapes match, but the classifier’s weights were learned for the older model’s numbers. Each number means something different in the newer model’s space, so the routes become close to random. In our planted test, the classifier fell from 0.920 to 0.178. He must embed the training messages again with the new model and train a new classifier. Then he tests it on the test set and its slices, and versions the embedding model and the classifier together.

Recap

  • A leaderboard such as MTEB averages many public tasks. It tells you which models to try, not which one to use. Check the task family, the languages and the hardware of a score.
  • On this inbox, the newer model (2026, 97 million parameters) is worse than the older one (2021, 118 million): 0.906 against 0.920, French 0.868 against 0.896, and US$895.0 against US$779.0 per 1,000.
  • Newer is not better. Measure accuracy by slice, cost and time per message one at a time. Read the library’s warnings: in its stored 16-bit numbers, the newer model took 26.9 ms per message; in 32-bit numbers, 7.0 ms.
  • A quick test can choose the wrong model: with 100 labels per route, the newer model wins (0.859 against 0.836). Test with the amount of data that you will really use.
  • Switching models means embedding, training and testing again. A classifier trained on one model’s numbers gives nonsense on another model’s numbers, without an error.
  • Pin every model to a revision, and keep the model card’s numbers labelled as vendor claims.

Every model so far learned from labelled examples. Next, we classify with the route names only, without a single labelled message.

Next: Zero-shot without an LLM