Skip to content
Français

Fine-tune a small encoder on CPU

  • Lesson 15
  • 45 min
  • Module 4: Embeddings and small fine-tuned models

Where we are

So far, the embedding model stayed frozen, and only the classifier on top learned. In this lesson, we fine-tune a small multilingual encoder on a laptop CPU: the whole model learns from the Larkfield labels. We watch the validation loss, and we save and load the checkpoint.

Lesson 15 of 37 · Text Classification and Routing: Send Each Message to the Right PlaceFine-tune a small encoder on CPU

What you will be able to do

Let the whole model learn Larkfield’s routes, not only a small classifier on top. We fine-tune a small multilingual encoder for about 6 minutes on a laptop CPU. On this inbox, it becomes the most accurate and the cheapest method in the comparison table so far.

NotebookDo this lesson in a notebookColab · Kaggle · GitHub

The notebook has every code cell, task and check of this lesson. It adds a classification head, freezes the word table, and trains for 32 short steps while you watch the loss. Then it loads the course’s ready checkpoint and scores the test messages. The full 6-minute training is in an optional cell. Its setup cells install the packages and download the data, the base model (about 470 MB) and the checkpoint (43 MB), so it needs no local setup. About 45 minutes.

Colab and Kaggle open your own copy. They need a free Google or Kaggle account. Stuck? See the solution notebook.

Before you startWhat to know and to have readyAbout 45 minutes

What you need to know

What you need on your computer

  • Nothing for the notebook route. This module changes no file of your larkroute project; the fine-tuned model’s recorded predictions join the project in Module 6.
  • For the optional full training: about 6 minutes of a laptop CPU. On free Colab, expect longer.

Tested on macOS (Apple silicon, 4 threads) with Python 3.12, Transformers 5.22.2 and PyTorch 2.15.0 (CPU), in Jupyter. Colab, Kaggle, GPUs and Windows were not tested.

The clerk who learns the post office’s own rules

Section titled “The clerk who learns the post office’s own rules”

Remember the older clerk who understands meaning. She came from another post office. She knows what letters mean, but she does not know the rules of this office. Here, a returned parcel goes to the returns desk, not to the parcel desk. A change of address on an order goes to orders, even when the letter says “parcel”.

In her first weeks, a supervisor watches her work. After each letter, he says “right” or “no, that one goes to returns”. Slowly, she changes how she reads letters. She starts to notice the words “send back” and “change my order”, which she ignored before. After a few weeks, she sorts better than anyone.

Until now, our embedding model was the clerk who never changes her way of reading. Only a small classifier behind her learned the office’s rules. In this lesson, the clerk herself learns.

The problem: a frozen model cannot learn Larkfield’s borders

Section titled “The problem: a frozen model cannot learn Larkfield’s borders”

The picture of the embeddings in the first lesson showed it. Delivery, returns and orders messages overlap in the frozen model’s space, because in general meaning they are neighbours. A classifier on top can only draw borders through that space; it cannot move the messages. On this inbox, the frozen embeddings with logistic regression reach 0.920 and cost US$779.0 per 1,000 messages, worse than TF-IDF (0.962, US$296.0).

What if the model could move the messages, so that each route gets its own region?

Fine-tuning means that we take a model that already learned from general text and train it a little more on our own labelled data. All, or part, of its parameters change. “Fine” means small changes: we start from a good model and adjust it.

The model here is an encoder: a transformer that reads a whole message and gives numbers for it, as the embedding model does. A transformer is a kind of neural network that reads each word together with the words around it. Transformer blocks in the deep learning course explains it. It does not write text. We use the older embedding model of this module as the starting point, because it is small, multilingual and already good with meaning.

To classify, the encoder gets a classification head. That is a small new layer on top: it turns the encoder’s 384 numbers into 7 scores, one per route. A last step, called softmax, turns the scores into 7 probabilities that add up to 1. The head starts with random weights, so at first it guesses.

Two columns of boxes, the same model drawn twice. Left, frozen embeddings plus logistic regression: the message goes through the word table (frozen, grey), 12 layers (frozen, grey), 384 numbers, then logistic regression (learns, orange). Result: test accuracy 0.920, US$779.0 per 1,000. Parameters that learn: 2,695. Right, fine-tuned encoder: the word table is frozen (grey), the 12 layers learn (orange), then a classification head (learns, orange). Result: 0.980, US$103.0 per 1,000. Parameters that learn: 21.6 million of 117.7 million.
The same model, used two ways. Frozen: only a small classifier learns. Fine-tuned: the 12 layers learn too, so the model changes how it reads Larkfield's messages. In both, the large word table stays frozen.
Text description of the diagram

The diagram has two columns. Both start with the message “My parcel never came.” and go down through the same parts. First, the word table: 250,037 tokens with 384 numbers each, 96 million numbers, frozen in both columns (grey). Then the 12 layers. In the left column, they are frozen (grey) and give general meaning only. In the right column, they learn from Larkfield’s labels (orange). Then 384 numbers for the message. Then, on the left, a logistic regression with 7 × 384 weights + 7, which learns; on the right, a classification head with 7 scores, which learns from the start. Results: on the left, test accuracy 0.920 and US$779.0 per 1,000; on the right, 0.980 and US$103.0 per 1,000. On the left, 2,695 parameters learn; on the right, 21.6 million of 117.7 million.

The model has 117.7 million parameters. Most of them, 96 million, are the word table: one row of 384 numbers for each of its 250,037 tokens (pieces of words) in many languages. Larkfield’s messages use only a small part of them, and their meaning does not need to change for routing. So we freeze the word table: it does not learn.

from transformers import AutoModelForSequenceClassification, AutoTokenizer
BASE = "sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2"
REV = "e8f8c211226b894fcb81acc59f3b34ba3efd5f42"
tok = AutoTokenizer.from_pretrained(BASE, revision=REV)
model = AutoModelForSequenceClassification.from_pretrained(BASE, revision=REV, num_labels=7)
model.base_model.embeddings.word_embeddings.weight.requires_grad = False # freeze the word table

The notebook counts the parameters:

117,656,455 parameters, 21,642,247 learn (18%)

21.6 million of 117.7 million parameters learn: the 12 layers, the head and a few small parts. Freezing makes each step faster and uses less memory, and the saved changes are much smaller. When it loads, Transformers reports that classifier.weight and classifier.bias are missing and “newly initialized”: that is the head, and it is expected.

The loss measures how wrong the model’s probabilities are. For one message, it is small when the model gives a high probability to the right route, and large when it gives a low one. Training changes the parameters, step by step, to make the loss smaller on the training messages. One step uses a small group of messages (here 32), and one epoch is one pass over all 6,000 training messages: 188 steps.

The training loss alone can fool you: a model that memorises its training messages gets a tiny training loss. So we also measure the loss on validation messages that the model never learns from. That number tells you if the model gets better on new messages.

Priya trains the model for 3 epochs on the 6,000 training messages, on a laptop CPU (Apple M4, 4 threads). Every half epoch, she measures the loss on 600 validation messages.

  • Validation loss (600 messages)
  • Training loss (last batch of 32)
The loss during the full training: 3 epochs, 327 seconds on a laptop CPU. The validation loss falls from 1.951 to 0.138 and stops falling after about 2 epochs.
Text description of the chart

The validation loss starts at 1.951 before training. It falls to 0.658 after half an epoch and 0.254 after one epoch. Then it is 0.162, 0.141, 0.145 and, after three epochs, 0.138. The training loss is measured on the last batch of 32 messages. It falls from 0.604 to 0.198, 0.084 and 0.045, jumps to 0.199 at two and a half epochs, and ends at 0.034. The validation curve is flat in the last epoch.

Read the chart from left to right:

  • Before training, the validation loss is 1.951. A model that gives each of the 7 routes the same probability has a loss of about 1.95 (the natural logarithm of 7). The random head knows nothing yet.
  • After half an epoch (57 seconds), it is already 0.658. The model learns fast, because it starts from a model that knows meaning.
  • After about 2 epochs, the validation loss stops falling: 0.141, 0.145, 0.138. More epochs would cost time and give little. If it started to rise while the training loss kept falling, the model would be memorising: that is the sign to stop.
  • The training loss jumps once (0.199 at 2.5 epochs). It is measured on one batch of 32 messages, so one hard batch moves it. That is why we judge on the validation loss, measured on 600 messages.

The run took 327 seconds, about 5.5 minutes, plus about 20 seconds to load, save and score the model.

Method (2,000 test messages) Accuracy English French Takeovers found Cost per 1,000
Embeddings, older model + logistic regression (frozen) 0.920 0.931 0.896 10 of 12 US$779.0
TF-IDF + logistic regression 0.962 0.966 0.952 11 of 12 US$296.0
TF-IDF + linear SVM 0.966 0.971 0.954 12 of 12 US$175.0
Fine-tuned encoder 0.980 0.980 0.978 12 of 12 US$103.0

On this inbox, the fine-tuned encoder is the best method so far: 41 mistakes in 2,000 messages, against 76 for TF-IDF with logistic regression. It finds all 12 account takeovers and all 99 needs_person messages, and it is almost as good in French as in English. The same model, frozen, made 159 mistakes. Fine-tuning made the difference.

A message in English and the same message in French show what changed. The probabilities of delivery:

Message Frozen + logistic regression Fine-tuned encoder
My parcel never came. 0.060 0.857
Mon colis n'est jamais arrivé. 0.021 0.853

These two short messages are not from the inbox; we wrote them for this example. The frozen model with its classifier sends both to account, with 0.932 and 0.720. These very short messages are unlike most Larkfield messages (the median is 28 words). The classifier on top of the frozen numbers gets them badly wrong, and we did not measure why. The fine-tuned encoder gives delivery 0.857 in English and 0.853 in French. It learned what makes a message a delivery message, in both languages.

A checkpoint is the saved state of a trained model: its parameters, so that you can load it later without training again. A full copy of this model would be about 470 MB. But the word table did not change, so we save only the parameters that changed, in 16-bit numbers. The result is 43.3 MB.

The course’s ready checkpoint is in the labs repository, in C36/models/finetuned-encoder/: changed_weights.safetensors and routes.json (the base model, its revision, the order of the routes and the longest input in tokens). To load it, build the base model again, then put the changed weights on top:

import json
from safetensors.torch import load_file
meta = json.load(open("models/finetuned-encoder/routes.json"))
weights = {k: v.float() for k, v in load_file("models/finetuned-encoder/changed_weights.safetensors").items()}
missing, unexpected = model.load_state_dict(weights, strict=False) # the word table comes from the base model
model.eval()

strict=False allows weights to be missing. Check what is missing: it must be only the word table (...word_embeddings.weight), and unexpected must be empty. If other names are missing, the checkpoint and the base model do not match, and the model will guess.

The ready checkpoint pins the base model’s revision. If the base model changed on Hugging Face, the saved weights would sit on top of other weights, and the scores would fall without any error.

Fine-tuned encoder
Training 327 s on 4 CPU threads (once per new model version)
Time per message, one at a time 6.7 ms
On disk the base model (470.6 MB) + the changed weights (43.3 MB)
Hardware a laptop CPU; a GPU is not needed
Running cost per 1,000 messages far less than a cent at US$0.05 per CPU-hour

At 65 messages a day, Larkfield’s inbox needs less than a second of CPU time per day for this model. The cost that matters is still the cost of mistakes: US$103.0 per 1,000, the lowest so far.

  • Fine-tune when you have a few thousand labelled messages, the classic models plateau, and you can run a short training job when the labels change.
  • Do not fine-tune with a few dozen labels: there is too little to learn from, and the previous lesson’s methods are better. Do not fine-tune either when TF-IDF is already good enough for the business, because a neural model is more work to run and to explain.
  • What it costs: a training job of a few minutes for each new version, and a 470 MB base model plus your weights to store and version. It needs about 7 ms per message, and it is harder to explain than TF-IDF’s top words.
  1. Open the notebook from the box Do this lesson in a notebook at the top of this page.
  2. Run the setup cells. The checkpoint cell prints Ready checkpoint: 2 files.
  3. In section 1, write your prediction. Then run the cell. It prints 117,656,455 parameters, 21,642,247 learn (18%).
  4. In section 2, run the short training. The validation loss falls from about 1.95 to about 1.3.
  5. Skip section 3, or set TRAIN_FULL = True if you have 6 minutes or more.
  6. In section 4, load the ready checkpoint. The table shows accuracy 0.98 and cost 103.0.
  7. In section 6, measure the time per message, one at a time.

In section 5 of the notebook, compare the fine-tuned encoder with the frozen embeddings of the first lesson (recorded in the course data). Count the test messages that only the fine-tuned encoder routes correctly (enc_only), and those that only the frozen embeddings route correctly (emb_only). Predict first: is emb_only zero?

Show the answer
frozen, _ = rt.predictions("embeddings-older", test)
truth = test["route"].to_numpy()
enc_right = pred == truth
emb_right = frozen["prediction"].to_numpy() == truth
enc_only = int((enc_right & ~emb_right).sum())
emb_only = int((emb_right & ~enc_right).sum())
print(enc_only, emb_only)
132 14

Fine-tuning fixes 132 messages and breaks 14. It is not zero: a better model is not better on every message. Read a few of the 14 messages to see what the fine-tuned model got wrong.

Common mistake

The checkpoint that did not load

Symptom: Amira loads the ready checkpoint in her own script. There is no error, and the model routes every message. But on the 2,000 test messages, its accuracy is 0.164, not 0.980.

Cause: she loaded the weights into the encoder part, model.base_model, instead of the whole model. The names in the checkpoint are the whole model’s names. Inside model.base_model, the names are different, so none of them matched, and with strict=False nothing complained. The model kept the base encoder and a random head. We planted this mistake on purpose and measured it: load_state_dict returned all 200 names in unexpected, a clear sign that nobody read.

Fix: always read the result of load_state_dict. missing must contain only the word table (...word_embeddings.weight), and unexpected must be empty. Then score a known set, for example the 2,000 test messages, and compare with the recorded score (0.980) before you use the model. Load the base model at the revision in routes.json, so that the weights sit on the model they were trained on.

Consequence in AI work: a model that loads without an error is not a model that works. A small check of the loaded weights and one known score catch a silent fault that would otherwise send customers to the wrong teams for weeks.

Check your understanding

Before training, the validation loss is 1.951. Why that number? What would a loss near 0 on the training messages and rising on the validation messages tell you?

Show the answer

The new classification head starts with random weights. So the model gives each of the 7 routes about the same probability, 1/7. The loss of a probability of 1/7 for the right route is the natural logarithm of 7, about 1.95. A training loss near 0 with a rising validation loss means that the model memorises the training messages and gets worse on new ones: overfitting. Stop at the epoch with the lowest validation loss, or train less.

Check your understanding

Why does the course freeze the word table? Give two reasons, with the numbers of this lesson.

Show the answer

The word table is most of the model: 96 million of 117.7 million parameters, one row of 384 numbers for each of 250,037 tokens. Larkfield’s routes do not need new meanings for words, so it can stay as it is. When it is frozen, only 21.6 million parameters learn. Each step is faster and needs less memory on a laptop CPU. It also makes the checkpoint small: only the changed weights, 43.3 MB, instead of a full copy of about 470 MB.

Check your understanding

Grace asks whether the fine-tuned encoder is worth it compared with TF-IDF + logistic regression. Answer with accuracy, cost and what it takes to run.

Show the answer

On this inbox, yes. It gets 0.980 against 0.962: 41 mistakes instead of 76 in 2,000. It finds all 12 account takeovers against 11, and costs US$103.0 per 1,000 messages against US$296.0. It needs a training job of about 6 minutes on a laptop CPU for each new version. It needs a 470 MB base model plus 43.3 MB of weights, and about 7 ms per message instead of 0.1 ms. At 65 messages a day, that is nothing. Remember that this inbox is easier than a real one; the gain must be checked on real messages too.

Recap

  • Fine-tuning trains a model that already knows general meaning a little more, on your labels. A classification head turns the encoder’s 384 numbers into 7 route scores.
  • We freeze the word table (96 million parameters): 21.6 million of 117.7 million parameters learn. 3 epochs on 6,000 messages took 327 seconds on a laptop CPU.
  • Watch the validation loss: it fell from 1.951 to 0.138 and stopped falling after about 2 epochs. A falling training loss with a rising validation loss means memorising.
  • On this inbox, the fine-tuned encoder reaches 0.980 (French 0.978), finds all 12 takeovers, and costs US$103.0 per 1,000 messages: the best row so far.
  • The ready checkpoint keeps only the changed weights (43.3 MB). Load it on the pinned base model, and check what is missing before you trust it.

We fine-tuned an embedding model from 2021. Newer models appear every few months, each with a better leaderboard score. Next, we test one on our own messages.

Next: Newer embedding models