Transfer learning
- Lesson 8
- 45 min
- Module 2: Baselines and pretrained classifiers
Where we are
We now know what a CNN and a vision transformer look at. Training one from the start needs millions of images. In this lesson, we take a model that already learned from millions of photos and train only its last layer on our catalogue. First, Hamid hires a worker who already knows warehouses.
Lesson 8 of 30 · Vision AI: See, Count and InspectTransfer learning
1. Seeing is a business problem
2. Baselines and pretrained classifiers
3. Embeddings, CLIP and few labels
4. Detect and count
6. Inspect for defects
7. Match, search and describe
8. Run vision in production
What you will be able to do
Adapt a model that learned from ImageNet’s photos to Larkfield’s eight categories, with 453 labelled photos and a laptop CPU. It is the method that most image work starts with, and it fills the comparison table with its first strong rows.
NotebookDo this lesson in a notebook
The notebook installs PyTorch and timm, about 200 MB the first time. It downloads the catalogue photos (about 9 MB) and two models (47 MB and 15 MB). It runs in about 2 minutes on a laptop CPU, and longer on a free Colab or Kaggle computer. No account and no key. About 30 minutes.
Colab and Kaggle open your own copy. They need a free Google or Kaggle account. Stuck? See the solution notebook.
Before you startWhat to know and to have ready
What you need to know
- What a CNN’s layers and its last layer do, and how to read a model card, from Convolution and patches, in one lesson.
- The comparison table’s columns, from Meet the Larkfield images.
- Helpful, not required: epochs and overfitting, from Batches, epochs and validation in Deep Learning and Transformers Explained.
What you need on your computer
- The notebook. This lesson adds nothing to the
larkvisionproject. The project gets pretrained models in the next module, through saved embeddings.
Tested with Python 3.12.13, PyTorch 2.15.0 and timm 1.0.31 on macOS, on the CPU.
A worker who already knows warehouses
Section titled “A worker who already knows warehouses”Hamid needs a new worker. He can hire someone who has never seen a warehouse. Then he must teach everything: how to lift a box, how to read a label, what “fragile” means. That takes months. Or he can hire a worker from a supermarket’s warehouse. She already knows boxes, labels and scanners. Hamid teaches her only Larkfield’s eight categories. That takes a morning.
The second worker transfers what she learned elsewhere. She does not forget it, and she does not need it all again. She learns only the part that is new.
The problem: 453 photos are not enough to start from zero
Section titled “The problem: 453 photos are not enough to start from zero”The CNN of the last lesson learned its filters from about 1.3 million ImageNet photos. Larkfield has 453 labelled catalogue photos, and only 15 candles. A network with 11.7 million parameters that starts from random numbers would learn these 453 photos by heart and fail on new ones.
The pretrained ResNet-18 does not have this problem. It already sees edges, textures and parts. But ask it about Larkfield’s photos, and it answers with its own 1,000 classes. Section 1 of the notebook asks it about three test photos:
test/mug/P-ab240c16.jpg -> [('espresso', 0.69), ('cup', 0.23)]test/lamp/P-b87b9773.jpg -> [("jack-o'-lantern", 1.0), ('partridge', 0.0)]test/cushion/P-a4e99b4e.jpg -> [('studio couch, day bed', 0.92), ('paper towel', 0.02)]It sees the photos well. A carved pumpkin with a light inside is a “jack-o’-lantern” for ImageNet. In Larkfield’s catalogue, it is a “lamp”: some catalogue labels are like this. The model’s eyes are right. Only its last step, the names it gives, does not fit.
The idea: keep the eyes, replace the last layer
Section titled “The idea: keep the eyes, replace the last layer”Transfer learning keeps what a model learned on one task and adapts it to another. For a CNN, that means keeping the filters and replacing the last layer. Recall the end of ResNet-18: every photo becomes 512 numbers, and the last layer turns them into 1,000 scores. We remove that last layer, and put a new one with 8 outputs, one per Larkfield category.
A tiny worked example of a last layer
Section titled “A tiny worked example of a last layer”A last layer gives each class a weight for each number, and adds them up. Take only three of the 512 numbers, and two classes. The numbers are made up for this example:
| Round rim | Leaves | Flame | |
|---|---|---|---|
| A photo’s numbers | 0.9 | 0.1 | 0.0 |
| Weights for mug | 2.0 | −1.0 | −1.0 |
| Weights for candle | 0.5 | −1.0 | 3.0 |
- Score for mug: 0.9 × 2.0 + 0.1 × (−1.0) + 0.0 × (−1.0) = 1.7
- Score for candle: 0.9 × 0.5 + 0.1 × (−1.0) + 0.0 × 3.0 = 0.35
The mug wins. Training the new last layer means finding good weights from the 453 labelled photos. With 512 numbers and 8 classes, there are 512 × 8 weights, plus one number per class: 4,104 numbers to learn. The whole ResNet-18 has 11,689,512.
Two ways to adapt
Section titled “Two ways to adapt”There are two common settings. They differ in what learns:
Text description of the diagram
Two rows of four boxes, joined by arrows: first layers (edges, textures), middle blocks, last block (layer4), and a new last layer from 512 numbers to 8 classes. In the first row, frozen plus last layer, the first three boxes are dashed grey: they are frozen. Only the new last layer is thick blue: it learns 4,104 numbers, and the photos go through the model only once. In the second row, fine-tune, the first two boxes are frozen. The last block and the new last layer are thick blue: together they learn 8,397,832 numbers, and every epoch runs the photos through the model again.
- Frozen + last layer. The model is frozen: its weights do not change. The pretrained network without its last layer is often called the backbone. Each photo goes through it once and becomes 512 numbers. A logistic regression on those numbers is the new last layer. This is cheap: the slow part, the network, runs once per photo.
- Fine-tuning. The last block of the model learns too, for a few passes over the training photos, called epochs. It can adapt its deep filters to Larkfield’s photos. It costs more: every epoch runs the photos through the network again, and backwards to update the weights.
Larkfield: frozen + last layer
Section titled “Larkfield: frozen + last layer”Section 2 of the notebook removes the last layer (num_classes=0), turns every photo into 512 numbers, and trains a logistic regression on them:
backbone = load("resnet18.a1_in1k", num_classes=0).eval()Xtr = embed(backbone, list("data/" + train["file"]))Xte = embed(backbone, list("data/" + test["file"]))head = LogisticRegression(C=1.0, max_iter=5000).fit(Xtr, train["label"])catalogue test accuracy 0.812 worst candle 0.45new studio accuracy 0.794 worst candle 0.513.7 ms per photo on this computer0.812, from 0.356 with HOG, on the same photos and the same 453 labels. The model’s ImageNet filters describe Larkfield’s products far better than any hand-made feature. The warmer studio costs 2 points, less than the histogram lost. Candle is still the worst class: 9 of 20 test candles found.
The notebook’s embed function prepares each photo the way the model card says: timm builds the resize, the crop and the normalising step from the card. The time per photo is the course team’s run; yours will differ.
Does a bigger model see better?
Section titled “Does a bigger model see better?”Tomás, the AI engineer, has an opinion: “Take ResNet-50. A bigger model always sees better.” The course team measured four models of timm in the same way, frozen + last layer, on the same photos. Predict first: which of the four is best, and by how much?
Text description of the chart
Four points. ConvNeXt-Atto: 15 MB, accuracy 0.887. EfficientNet-B0: 21 MB, 0.844. ResNet-18: 47 MB, 0.812. ResNet-50: 103 MB, 0.894. The vertical axis runs from 0.7 to 1. The smallest model is almost as good as the largest and much better than the two in the middle.
The largest model, ResNet-50 (103 MB), reaches 0.894. The smallest, ConvNeXt-Atto (15 MB), reaches 0.887: one photo less, out of 160. With 160 test photos, one photo is 0.6 points, and a difference under about 3 points is noise. So the two are equal here. ConvNeXt-Atto is 7 times smaller and took 29.92 ms a photo in the measurement, against 68.86 for ResNet-50. Its worst class, candle, reaches 0.80 (16 of 20 found). That is as good as ResNet-50’s worst class, plant pot, and much better than ResNet-18’s 0.45 for candle.
Why? Size is one property of a model, and it is not the most important one. ConvNeXt is a newer design, trained with a newer recipe. What a model learned, and how, matters more than how many parameters it has. Tomás’s rule fails here. The only way to know is to measure on your own photos.
Fine-tune a little more
Section titled “Fine-tune a little more”Section 4 of the notebook lets the last block (layer4) learn too, with a new last layer, for 4 epochs:
epoch 1: loss 1.793epoch 2: loss 0.996epoch 3: loss 0.511epoch 4: loss 0.246training: 44 sfine-tuned, no augmentation: 0.85 | worst candle 0.4The loss measures how wrong the model is on the training photos: it falls in every epoch. The test accuracy rises from 0.812 to 0.850. The training took about 31 seconds in the course team’s measurement and 44 seconds in the notebook run, on a laptop CPU. That is the price: 4 epochs of fine-tuning cost as much time as embedding the catalogue several times.
Fine-tuning helped ResNet-18 by 4 points. It did not help the candle: 8 of 20, against 9 when frozen. And the frozen ConvNeXt-Atto (0.887) is still ahead of the fine-tuned ResNet-18. Choosing the right pretrained model mattered more than training it longer.
Augmentation, and when it hurts
Section titled “Augmentation, and when it hurts”The notebook trains on the same 453 photos in every epoch. Augmentation changes each training photo a little in every epoch, by code: a mirror image, a smaller crop, other colours. The model sees more kinds of photos, and memorising them gets harder. These are the three changes the course team tested:

Text description of the diagram
Four versions of one photo: a plant with green leaves in an orange pot. The original. Flipped: a mirror image, left and right swapped. Cropped: a closer view that shows a larger part of the plant and cuts its edges. Colour changed: the same view with other brightness, contrast and colours. Each one is still clearly a plant pot.
The course team fine-tuned ResNet-18 three times, with the same seed, epochs and settings:
- no augmentation
- flips and crops
- flips, crops and colour
Text description of the chart
Three lines over 4 epochs. No augmentation: 1.79, 1.00, 0.51, 0.25. Flips and crops: 1.80, 1.12, 0.67, 0.40. Flips, crops and colour: 1.83, 1.18, 0.77, 0.49. The runs with augmentation end with a higher loss, because their training photos are harder. The test accuracies are 0.850, 0.844 and 0.863.
With augmentation, the training loss stays higher: the photos are harder to learn by heart. That is the point of it. But the test accuracies are 0.850, 0.844 and 0.863: a difference of 2 photos at most, which is noise. Augmentation did not help here. Most of the model was frozen, and only the last block learned. So in 4 epochs, the model had little room to memorise.
Augmentation can also hurt. It is safe only when the change keeps the label true. The Overfitting controls lesson of the deep learning course explains this rule. At Larkfield:
- A colour change is wrong when colour is the answer. If Omar’s team asks “what colour is this mug?”, a colour-changed red mug teaches the model that red is blue.
- A crop is wrong when the important part is small. Crop a returns photo, and the damaged corner of a candle may fall outside the crop, while the label still says “damaged”.
- A flip is wrong when left and right matter, for example for a printed text on a product.
Common mistake
A winner chosen from two photos
Symptom: you compare the three fine-tuning runs on the test photos. “Flips, crops and colour” has the highest accuracy, 0.863 against 0.850. You write “colour augmentation improves the catalogue model” in your report.
Cause: 0.863 − 0.850 is 0.013: two test photos out of 160. Another seed, or another order of the training photos, can change two answers. You also chose the winner by looking at the test set, so its score is now a little too good.
Fix: treat differences under about 3 points as noise with 160 test photos. Choose between settings on the validation photos, not the test photos. Repeat a run with two or three seeds when the choice matters. Keep the simplest setting when the results are equal: here, no augmentation.
Consequence in AI work: small differences on a small test set create false progress. Report the size of the test set next to every number, as the comparison table does.
The comparison table so far
Section titled “The comparison table so far”Here is every catalogue method of this module, on the same 160 test photos:
- Classical (no model file)
- Pretrained, frozen + last layer
- ResNet-18, fine-tuned 4 epochs
Text description of the chart
One row per method, with its accuracy on an axis from 0 to 1. Classical methods: rules 0.269, colour histogram with 5 neighbours 0.263, HOG with logistic regression 0.356. Pretrained, frozen with a new last layer: ResNet-18 0.812, EfficientNet-B0 0.844, ConvNeXt-Atto 0.887, ResNet-50 0.894. ResNet-18 fine-tuned for 4 epochs: no augmentation 0.850, flips and crops 0.844, flips, crops and colour 0.863. There is a large gap between the classical methods, all under 0.4, and the pretrained ones, all over 0.8.
The gap between the two groups is the main result of this module. The classical methods stop below 0.4. Every pretrained model is above 0.8, with the same 453 labels. Within the pretrained group, the differences are small, and the smallest model is among the best.
When to use transfer learning
Section titled “When to use transfer learning”- Use frozen + last layer first. It is cheap, it runs on a CPU in seconds, and here it was as good as fine-tuning. Try two or three models, and keep the one that the table supports.
- Fine-tune when frozen features are not enough, and you have more labelled photos than here. Your photos may also look very different from ImageNet’s, like X-rays or pallets drawn by code. Keep a validation set to choose the number of epochs.
- Do not trust the size or the model card’s score alone. Measure on your own photos.
- What it costs: a model file of 15 to 103 MB here, and 13 to 69 ms a photo on the course team’s CPU. The histogram needs about 1 ms. Also the model’s licence: these weights are Apache-2.0, but they learned from ImageNet, whose own terms When not to build it discusses.
Your turn: a smaller model
Section titled “Your turn: a smaller model”Section 6 of the notebook has one task and a check cell. This is the guided practice of the module: train a last-layer classifier, and add it to the table with its CPU time.
- Load
"convnext_atto.d2_in1k"withnum_classes=0, and call.eval(). - Start a timer with
time.perf_counter(). - Embed the training photos and the test photos with
embed. - Put the milliseconds per photo in
ms_convnext. - Train
LogisticRegression(C=1.0, max_iter=5000)on the training embeddings. - Put the test accuracy in
acc_convnext, and run the check cell. - Add your row to
minewithcp.classification_row, and show the table.
Show the answer
small = load("convnext_atto.d2_in1k", num_classes=0).eval()start = time.perf_counter()Ctr = embed(small, list("data/" + train["file"]))Cte = embed(small, list("data/" + test["file"]))ms_convnext = 1000 * (time.perf_counter() - start) / (len(train) + len(test))pred = LogisticRegression(C=1.0, max_iter=5000).fit(Ctr, train["label"]).predict(Cte)acc_convnext = cp.classification_scores(list(test["label"]), list(pred), CLASSES)["accuracy"]mine.add(cp.classification_row("ConvNeXt-Atto, frozen + last layer (yours)", list(test["label"]), list(pred), "catalogue test", round(ms_convnext, 1), 15, len(train), "Apache-2.0 (weights)"))mine.show("catalogue", columns=("method", "value", "worst", "worst_value", "ms_per_image", "model_mb"))catalogue: accuracymethod value worst worst_value ms_per_image model_mbResNet-18, frozen + last layer (yours) 0.812 candle 0.450 13.7 47ConvNeXt-Atto, frozen + last layer (yours) 0.887 candle 0.800 40.6 15The check prints Check passed: acc_convnext is 0.887. The time per photo depends on the computer and on what else runs on it. The course team’s measurement gave 29.9 ms, and this notebook run 40.6 ms.
Knowledge check
Section titled “Knowledge check”Check your understanding
Why does training a new last layer on 453 photos work, when training ResNet-18 from random numbers on the same photos would not?
Show the answer
A frozen ResNet-18 already turns each photo into 512 numbers that describe edges, textures and parts. It learned them from about 1.3 million ImageNet photos. The new last layer only learns how to weigh those numbers for 8 classes: 4,104 numbers. 453 photos are enough for that. Training all 11.7 million parameters from random numbers on 453 photos would memorise the training photos and fail on new ones.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
Tomás says: 'ResNet-50 is the biggest model, so we should use it.' Answer with the numbers of this lesson.
Show the answer
ResNet-50 (103 MB) reaches 0.894 and ConvNeXt-Atto (15 MB) reaches 0.887: one test photo apart, which is noise with 160 photos. ConvNeXt-Atto is 7 times smaller and took about 30 ms a photo against about 69 ms. Its worst class (candle, 0.80) is as good as ResNet-50’s worst class (plant pot, 0.80). A bigger model did not see better here. Choose the smaller one, unless a larger test set shows a real difference.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
Hamid asks: 'The histogram gets 0.263 and ConvNeXt-Atto 0.887. Is there any case where Larkfield should use the histogram?' Answer with numbers from your comparison table.
Show the answer
Yes, where colour is the answer and the camera and the light are fixed. The histogram needs no model file and about 1 ms a photo, against 15 MB and about 30 ms for ConvNeXt-Atto. On the returns, with one camera and one light, a colour rule costs 1,779 dollars a day against 2,229 with no check. It needs no labels at all. For the catalogue, no: photos come from many cameras and lights, and the histogram is right on only about one photo in four. It also loses about 4 points under the warmer studio light.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Recap
- Transfer learning keeps a pretrained model’s filters and replaces its last layer. ResNet-18 with a new last layer learns 4,104 numbers and reaches 0.812, against 0.356 for HOG.
- Bigger is not better here: ConvNeXt-Atto (15 MB) reaches 0.887. That is as good as ResNet-50 (103 MB, 0.894) and better than ResNet-18 (0.812).
- Fine-tuning the last block for 4 epochs raised ResNet-18 to 0.850, in about 30 seconds on a CPU. Augmentation did not help on 453 photos: 0.844 and 0.863 are within noise.
- Augmentation must keep the label true: no colour change when colour is the answer, no crop that can remove the damage.
- The comparison table now has two groups: classical methods under 0.4, pretrained models over 0.8. The candle stays the worst class.
You can now adapt a pretrained model with a few hundred labels. But each method so far needed a model trained for this one task. Next, in Module 3, you use the 512 numbers themselves. You compare photos by these numbers, draw the catalogue as a map, and find the classes that the model confuses.
Next: Image embeddings