Fine-tune a small encoder on CPU
- Lesson 15
- 45 min
- Module 4: Embeddings and small fine-tuned models
Where we are
So far, the embedding model stayed frozen, and only the classifier on top learned. In this lesson, we fine-tune a small multilingual encoder on a laptop CPU: the whole model learns from the Larkfield labels. We watch the validation loss, and we save and load the checkpoint.
Lesson 15 of 37 · Text Classification and Routing: Send Each Message to the Right PlaceFine-tune a small encoder on CPU
1. The routing problem and its cost
2. Labels you can trust
3. Baselines: rules and classical models
4. Embeddings and small fine-tuned models
5. LLMs as classifiers
6. Decision models, cascades and distillation
7. Harder label shapes
8. Confidence and people in the loop
9. Run it in production
What you will be able to do
Let the whole model learn Larkfield’s routes, not only a small classifier on top. We fine-tune a small multilingual encoder for about 6 minutes on a laptop CPU. On this inbox, it becomes the most accurate and the cheapest method in the comparison table so far.
NotebookDo this lesson in a notebook
The notebook has every code cell, task and check of this lesson. It adds a classification head, freezes the word table, and trains for 32 short steps while you watch the loss. Then it loads the course’s ready checkpoint and scores the test messages. The full 6-minute training is in an optional cell. Its setup cells install the packages and download the data, the base model (about 470 MB) and the checkpoint (43 MB), so it needs no local setup. About 45 minutes.
Colab and Kaggle open your own copy. They need a free Google or Kaggle account. Stuck? See the solution notebook.
Before you startWhat to know and to have ready
What you need to know
- Sentence embeddings and the frozen model with logistic regression, from Sentence embeddings for classification.
- Why we watch a validation score while a model learns, from Train, validation and test sets in the machine learning course.
- Helpful, not required: the training loop, loss and epochs in the deep learning course, from The training loop and Batches, epochs and validation. This lesson explains what it needs.
What you need on your computer
- Nothing for the notebook route. This module changes no file of your
larkrouteproject; the fine-tuned model’s recorded predictions join the project in Module 6. - For the optional full training: about 6 minutes of a laptop CPU. On free Colab, expect longer.
Tested on macOS (Apple silicon, 4 threads) with Python 3.12, Transformers 5.22.2 and PyTorch 2.15.0 (CPU), in Jupyter. Colab, Kaggle, GPUs and Windows were not tested.
The clerk who learns the post office’s own rules
Section titled “The clerk who learns the post office’s own rules”Remember the older clerk who understands meaning. She came from another post office. She knows what letters mean, but she does not know the rules of this office. Here, a returned parcel goes to the returns desk, not to the parcel desk. A change of address on an order goes to orders, even when the letter says “parcel”.
In her first weeks, a supervisor watches her work. After each letter, he says “right” or “no, that one goes to returns”. Slowly, she changes how she reads letters. She starts to notice the words “send back” and “change my order”, which she ignored before. After a few weeks, she sorts better than anyone.
Until now, our embedding model was the clerk who never changes her way of reading. Only a small classifier behind her learned the office’s rules. In this lesson, the clerk herself learns.
The problem: a frozen model cannot learn Larkfield’s borders
Section titled “The problem: a frozen model cannot learn Larkfield’s borders”The picture of the embeddings in the first lesson showed it. Delivery, returns and orders messages overlap in the frozen model’s space, because in general meaning they are neighbours. A classifier on top can only draw borders through that space; it cannot move the messages. On this inbox, the frozen embeddings with logistic regression reach 0.920 and cost US$779.0 per 1,000 messages, worse than TF-IDF (0.962, US$296.0).
What if the model could move the messages, so that each route gets its own region?
The idea: fine-tuning
Section titled “The idea: fine-tuning”Fine-tuning means that we take a model that already learned from general text and train it a little more on our own labelled data. All, or part, of its parameters change. “Fine” means small changes: we start from a good model and adjust it.
The model here is an encoder: a transformer that reads a whole message and gives numbers for it, as the embedding model does. A transformer is a kind of neural network that reads each word together with the words around it. Transformer blocks in the deep learning course explains it. It does not write text. We use the older embedding model of this module as the starting point, because it is small, multilingual and already good with meaning.
To classify, the encoder gets a classification head. That is a small new layer on top: it turns the encoder’s 384 numbers into 7 scores, one per route. A last step, called softmax, turns the scores into 7 probabilities that add up to 1. The head starts with random weights, so at first it guesses.
Text description of the diagram
The diagram has two columns. Both start with the message “My parcel never came.” and go down through the same parts. First, the word table: 250,037 tokens with 384 numbers each, 96 million numbers, frozen in both columns (grey). Then the 12 layers. In the left column, they are frozen (grey) and give general meaning only. In the right column, they learn from Larkfield’s labels (orange). Then 384 numbers for the message. Then, on the left, a logistic regression with 7 × 384 weights + 7, which learns; on the right, a classification head with 7 scores, which learns from the start. Results: on the left, test accuracy 0.920 and US$779.0 per 1,000; on the right, 0.980 and US$103.0 per 1,000. On the left, 2,695 parameters learn; on the right, 21.6 million of 117.7 million.
Freeze the word table
Section titled “Freeze the word table”The model has 117.7 million parameters. Most of them, 96 million, are the word table: one row of 384 numbers for each of its 250,037 tokens (pieces of words) in many languages. Larkfield’s messages use only a small part of them, and their meaning does not need to change for routing. So we freeze the word table: it does not learn.
from transformers import AutoModelForSequenceClassification, AutoTokenizerBASE = "sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2"REV = "e8f8c211226b894fcb81acc59f3b34ba3efd5f42"tok = AutoTokenizer.from_pretrained(BASE, revision=REV)model = AutoModelForSequenceClassification.from_pretrained(BASE, revision=REV, num_labels=7)model.base_model.embeddings.word_embeddings.weight.requires_grad = False # freeze the word tableThe notebook counts the parameters:
117,656,455 parameters, 21,642,247 learn (18%)21.6 million of 117.7 million parameters learn: the 12 layers, the head and a few small parts. Freezing makes each step faster and uses less memory, and the saved changes are much smaller. When it loads, Transformers reports that classifier.weight and classifier.bias are missing and “newly initialized”: that is the head, and it is expected.
Watch the loss
Section titled “Watch the loss”The loss measures how wrong the model’s probabilities are. For one message, it is small when the model gives a high probability to the right route, and large when it gives a low one. Training changes the parameters, step by step, to make the loss smaller on the training messages. One step uses a small group of messages (here 32), and one epoch is one pass over all 6,000 training messages: 188 steps.
The training loss alone can fool you: a model that memorises its training messages gets a tiny training loss. So we also measure the loss on validation messages that the model never learns from. That number tells you if the model gets better on new messages.
Larkfield: the full training run
Section titled “Larkfield: the full training run”Priya trains the model for 3 epochs on the 6,000 training messages, on a laptop CPU (Apple M4, 4 threads). Every half epoch, she measures the loss on 600 validation messages.
- Validation loss (600 messages)
- Training loss (last batch of 32)
Text description of the chart
The validation loss starts at 1.951 before training. It falls to 0.658 after half an epoch and 0.254 after one epoch. Then it is 0.162, 0.141, 0.145 and, after three epochs, 0.138. The training loss is measured on the last batch of 32 messages. It falls from 0.604 to 0.198, 0.084 and 0.045, jumps to 0.199 at two and a half epochs, and ends at 0.034. The validation curve is flat in the last epoch.
Read the chart from left to right:
- Before training, the validation loss is 1.951. A model that gives each of the 7 routes the same probability has a loss of about 1.95 (the natural logarithm of 7). The random head knows nothing yet.
- After half an epoch (57 seconds), it is already 0.658. The model learns fast, because it starts from a model that knows meaning.
- After about 2 epochs, the validation loss stops falling: 0.141, 0.145, 0.138. More epochs would cost time and give little. If it started to rise while the training loss kept falling, the model would be memorising: that is the sign to stop.
- The training loss jumps once (0.199 at 2.5 epochs). It is measured on one batch of 32 messages, so one hard batch moves it. That is why we judge on the validation loss, measured on 600 messages.
The run took 327 seconds, about 5.5 minutes, plus about 20 seconds to load, save and score the model.
The result
Section titled “The result”| Method (2,000 test messages) | Accuracy | English | French | Takeovers found | Cost per 1,000 |
|---|---|---|---|---|---|
| Embeddings, older model + logistic regression (frozen) | 0.920 | 0.931 | 0.896 | 10 of 12 | US$779.0 |
| TF-IDF + logistic regression | 0.962 | 0.966 | 0.952 | 11 of 12 | US$296.0 |
| TF-IDF + linear SVM | 0.966 | 0.971 | 0.954 | 12 of 12 | US$175.0 |
| Fine-tuned encoder | 0.980 | 0.980 | 0.978 | 12 of 12 | US$103.0 |
On this inbox, the fine-tuned encoder is the best method so far: 41 mistakes in 2,000 messages, against 76 for TF-IDF with logistic regression. It finds all 12 account takeovers and all 99 needs_person messages, and it is almost as good in French as in English. The same model, frozen, made 159 mistakes. Fine-tuning made the difference.
A message in English and the same message in French show what changed. The probabilities of delivery:
| Message | Frozen + logistic regression | Fine-tuned encoder |
|---|---|---|
| My parcel never came. | 0.060 | 0.857 |
| Mon colis n'est jamais arrivé. | 0.021 | 0.853 |
These two short messages are not from the inbox; we wrote them for this example. The frozen model with its classifier sends both to account, with 0.932 and 0.720. These very short messages are unlike most Larkfield messages (the median is 28 words). The classifier on top of the frozen numbers gets them badly wrong, and we did not measure why. The fine-tuned encoder gives delivery 0.857 in English and 0.853 in French. It learned what makes a message a delivery message, in both languages.
Save and load the checkpoint
Section titled “Save and load the checkpoint”A checkpoint is the saved state of a trained model: its parameters, so that you can load it later without training again. A full copy of this model would be about 470 MB. But the word table did not change, so we save only the parameters that changed, in 16-bit numbers. The result is 43.3 MB.
The course’s ready checkpoint is in the labs repository, in C36/models/finetuned-encoder/: changed_weights.safetensors and routes.json (the base model, its revision, the order of the routes and the longest input in tokens). To load it, build the base model again, then put the changed weights on top:
import jsonfrom safetensors.torch import load_filemeta = json.load(open("models/finetuned-encoder/routes.json"))weights = {k: v.float() for k, v in load_file("models/finetuned-encoder/changed_weights.safetensors").items()}missing, unexpected = model.load_state_dict(weights, strict=False) # the word table comes from the base modelmodel.eval()strict=False allows weights to be missing. Check what is missing: it must be only the word table (...word_embeddings.weight), and unexpected must be empty. If other names are missing, the checkpoint and the base model do not match, and the model will guess.
The ready checkpoint pins the base model’s revision. If the base model changed on Hugging Face, the saved weights would sit on top of other weights, and the scores would fall without any error.
Speed and cost
Section titled “Speed and cost”| Fine-tuned encoder | |
|---|---|
| Training | 327 s on 4 CPU threads (once per new model version) |
| Time per message, one at a time | 6.7 ms |
| On disk | the base model (470.6 MB) + the changed weights (43.3 MB) |
| Hardware | a laptop CPU; a GPU is not needed |
| Running cost per 1,000 messages | far less than a cent at US$0.05 per CPU-hour |
At 65 messages a day, Larkfield’s inbox needs less than a second of CPU time per day for this model. The cost that matters is still the cost of mistakes: US$103.0 per 1,000, the lowest so far.
When to fine-tune
Section titled “When to fine-tune”- Fine-tune when you have a few thousand labelled messages, the classic models plateau, and you can run a short training job when the labels change.
- Do not fine-tune with a few dozen labels: there is too little to learn from, and the previous lesson’s methods are better. Do not fine-tune either when TF-IDF is already good enough for the business, because a neural model is more work to run and to explain.
- What it costs: a training job of a few minutes for each new version, and a 470 MB base model plus your weights to store and version. It needs about 7 ms per message, and it is harder to explain than TF-IDF’s top words.
Guided practice: fine-tune and load
Section titled “Guided practice: fine-tune and load”- Open the notebook from the box Do this lesson in a notebook at the top of this page.
- Run the setup cells. The checkpoint cell prints
Ready checkpoint: 2 files. - In section 1, write your prediction. Then run the cell. It prints
117,656,455 parameters, 21,642,247 learn (18%). - In section 2, run the short training. The validation loss falls from about
1.95to about1.3. - Skip section 3, or set
TRAIN_FULL = Trueif you have 6 minutes or more. - In section 4, load the ready checkpoint. The table shows accuracy
0.98and cost103.0. - In section 6, measure the time per message, one at a time.
Your turn: where fine-tuning helps
Section titled “Your turn: where fine-tuning helps”In section 5 of the notebook, compare the fine-tuned encoder with the frozen embeddings of the first lesson (recorded in the course data). Count the test messages that only the fine-tuned encoder routes correctly (enc_only), and those that only the frozen embeddings route correctly (emb_only). Predict first: is emb_only zero?
Show the answer
frozen, _ = rt.predictions("embeddings-older", test)truth = test["route"].to_numpy()enc_right = pred == truthemb_right = frozen["prediction"].to_numpy() == truthenc_only = int((enc_right & ~emb_right).sum())emb_only = int((emb_right & ~enc_right).sum())print(enc_only, emb_only)132 14Fine-tuning fixes 132 messages and breaks 14. It is not zero: a better model is not better on every message. Read a few of the 14 messages to see what the fine-tuned model got wrong.
Common mistake
The checkpoint that did not load
Symptom: Amira loads the ready checkpoint in her own script. There is no error, and the model routes every message. But on the 2,000 test messages, its accuracy is 0.164, not 0.980.
Cause: she loaded the weights into the encoder part, model.base_model, instead of the whole model. The names in the checkpoint are the whole model’s names. Inside model.base_model, the names are different, so none of them matched, and with strict=False nothing complained. The model kept the base encoder and a random head. We planted this mistake on purpose and measured it: load_state_dict returned all 200 names in unexpected, a clear sign that nobody read.
Fix: always read the result of load_state_dict. missing must contain only the word table (...word_embeddings.weight), and unexpected must be empty. Then score a known set, for example the 2,000 test messages, and compare with the recorded score (0.980) before you use the model. Load the base model at the revision in routes.json, so that the weights sit on the model they were trained on.
Consequence in AI work: a model that loads without an error is not a model that works. A small check of the loaded weights and one known score catch a silent fault that would otherwise send customers to the wrong teams for weeks.
Knowledge check
Section titled “Knowledge check”Check your understanding
Before training, the validation loss is 1.951. Why that number? What would a loss near 0 on the training messages and rising on the validation messages tell you?
Show the answer
The new classification head starts with random weights. So the model gives each of the 7 routes about the same probability, 1/7. The loss of a probability of 1/7 for the right route is the natural logarithm of 7, about 1.95. A training loss near 0 with a rising validation loss means that the model memorises the training messages and gets worse on new ones: overfitting. Stop at the epoch with the lowest validation loss, or train less.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
Why does the course freeze the word table? Give two reasons, with the numbers of this lesson.
Show the answer
The word table is most of the model: 96 million of 117.7 million parameters, one row of 384 numbers for each of 250,037 tokens. Larkfield’s routes do not need new meanings for words, so it can stay as it is. When it is frozen, only 21.6 million parameters learn. Each step is faster and needs less memory on a laptop CPU. It also makes the checkpoint small: only the changed weights, 43.3 MB, instead of a full copy of about 470 MB.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
Grace asks whether the fine-tuned encoder is worth it compared with TF-IDF + logistic regression. Answer with accuracy, cost and what it takes to run.
Show the answer
On this inbox, yes. It gets 0.980 against 0.962: 41 mistakes instead of 76 in 2,000. It finds all 12 account takeovers against 11, and costs US$103.0 per 1,000 messages against US$296.0. It needs a training job of about 6 minutes on a laptop CPU for each new version. It needs a 470 MB base model plus 43.3 MB of weights, and about 7 ms per message instead of 0.1 ms. At 65 messages a day, that is nothing. Remember that this inbox is easier than a real one; the gain must be checked on real messages too.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Recap
- Fine-tuning trains a model that already knows general meaning a little more, on your labels. A classification head turns the encoder’s 384 numbers into 7 route scores.
- We freeze the word table (96 million parameters): 21.6 million of 117.7 million parameters learn. 3 epochs on 6,000 messages took 327 seconds on a laptop CPU.
- Watch the validation loss: it fell from 1.951 to 0.138 and stopped falling after about 2 epochs. A falling training loss with a rising validation loss means memorising.
- On this inbox, the fine-tuned encoder reaches 0.980 (French 0.978), finds all 12 takeovers, and costs US$103.0 per 1,000 messages: the best row so far.
- The ready checkpoint keeps only the changed weights (43.3 MB). Load it on the pinned base model, and check what is missing before you trust it.
We fine-tuned an embedding model from 2021. Newer models appear every few months, each with a better leaderboard score. Next, we test one on our own messages.
Next: Newer embedding models