Skip to content
Français

Transfer learning

  • Lesson 8
  • 45 min
  • Module 2: Baselines and pretrained classifiers

Where we are

We now know what a CNN and a vision transformer look at. Training one from the start needs millions of images. In this lesson, we take a model that already learned from millions of photos and train only its last layer on our catalogue. First, Hamid hires a worker who already knows warehouses.

Plays from YouTube. Nothing loads from YouTube until you press play. Privacy

What you will be able to do

Adapt a model that learned from ImageNet’s photos to Larkfield’s eight categories, with 453 labelled photos and a laptop CPU. It is the method that most image work starts with, and it fills the comparison table with its first strong rows.

NotebookDo this lesson in a notebookColab · Kaggle · GitHub

The notebook installs PyTorch and timm, about 200 MB the first time. It downloads the catalogue photos (about 9 MB) and two models (47 MB and 15 MB). It runs in about 2 minutes on a laptop CPU, and longer on a free Colab or Kaggle computer. No account and no key. About 30 minutes.

Colab and Kaggle open your own copy. They need a free Google or Kaggle account. Stuck? See the solution notebook.

Before you startWhat to know and to have readyAbout 45 minutes

What you need to know

What you need on your computer

  • The notebook. This lesson adds nothing to the larkvision project. The project gets pretrained models in the next module, through saved embeddings.

Tested with Python 3.12.13, PyTorch 2.15.0 and timm 1.0.31 on macOS, on the CPU.

Hamid needs a new worker. He can hire someone who has never seen a warehouse. Then he must teach everything: how to lift a box, how to read a label, what “fragile” means. That takes months. Or he can hire a worker from a supermarket’s warehouse. She already knows boxes, labels and scanners. Hamid teaches her only Larkfield’s eight categories. That takes a morning.

The second worker transfers what she learned elsewhere. She does not forget it, and she does not need it all again. She learns only the part that is new.

The problem: 453 photos are not enough to start from zero

Section titled “The problem: 453 photos are not enough to start from zero”

The CNN of the last lesson learned its filters from about 1.3 million ImageNet photos. Larkfield has 453 labelled catalogue photos, and only 15 candles. A network with 11.7 million parameters that starts from random numbers would learn these 453 photos by heart and fail on new ones.

The pretrained ResNet-18 does not have this problem. It already sees edges, textures and parts. But ask it about Larkfield’s photos, and it answers with its own 1,000 classes. Section 1 of the notebook asks it about three test photos:

test/mug/P-ab240c16.jpg -> [('espresso', 0.69), ('cup', 0.23)]
test/lamp/P-b87b9773.jpg -> [("jack-o'-lantern", 1.0), ('partridge', 0.0)]
test/cushion/P-a4e99b4e.jpg -> [('studio couch, day bed', 0.92), ('paper towel', 0.02)]

It sees the photos well. A carved pumpkin with a light inside is a “jack-o’-lantern” for ImageNet. In Larkfield’s catalogue, it is a “lamp”: some catalogue labels are like this. The model’s eyes are right. Only its last step, the names it gives, does not fit.

The idea: keep the eyes, replace the last layer

Section titled “The idea: keep the eyes, replace the last layer”

Transfer learning keeps what a model learned on one task and adapts it to another. For a CNN, that means keeping the filters and replacing the last layer. Recall the end of ResNet-18: every photo becomes 512 numbers, and the last layer turns them into 1,000 scores. We remove that last layer, and put a new one with 8 outputs, one per Larkfield category.

A last layer gives each class a weight for each number, and adds them up. Take only three of the 512 numbers, and two classes. The numbers are made up for this example:

Round rim Leaves Flame
A photo’s numbers 0.9 0.1 0.0
Weights for mug 2.0 −1.0 −1.0
Weights for candle 0.5 −1.0 3.0
  • Score for mug: 0.9 × 2.0 + 0.1 × (−1.0) + 0.0 × (−1.0) = 1.7
  • Score for candle: 0.9 × 0.5 + 0.1 × (−1.0) + 0.0 × 3.0 = 0.35

The mug wins. Training the new last layer means finding good weights from the 453 labelled photos. With 512 numbers and 8 classes, there are 512 × 8 weights, plus one number per class: 4,104 numbers to learn. The whole ResNet-18 has 11,689,512.

There are two common settings. They differ in what learns:

Two rows of four boxes. Row 1, frozen plus last layer: first layers, middle blocks and last block are dashed grey (frozen); only the new last layer, 512 to 8 classes, is thick blue. It learns 4,104 numbers, and the photos go through the model once. Row 2, fine-tune: first layers and middle blocks are frozen; the last block (layer4) and the new last layer are thick blue. It learns 8,397,832 numbers, and every epoch runs the model again.
What learns in each setting of ResNet-18. Dashed grey: frozen, the ImageNet weights do not change. Thick blue: learns from the 453 catalogue photos.
Text description of the diagram

Two rows of four boxes, joined by arrows: first layers (edges, textures), middle blocks, last block (layer4), and a new last layer from 512 numbers to 8 classes. In the first row, frozen plus last layer, the first three boxes are dashed grey: they are frozen. Only the new last layer is thick blue: it learns 4,104 numbers, and the photos go through the model only once. In the second row, fine-tune, the first two boxes are frozen. The last block and the new last layer are thick blue: together they learn 8,397,832 numbers, and every epoch runs the photos through the model again.

  • Frozen + last layer. The model is frozen: its weights do not change. The pretrained network without its last layer is often called the backbone. Each photo goes through it once and becomes 512 numbers. A logistic regression on those numbers is the new last layer. This is cheap: the slow part, the network, runs once per photo.
  • Fine-tuning. The last block of the model learns too, for a few passes over the training photos, called epochs. It can adapt its deep filters to Larkfield’s photos. It costs more: every epoch runs the photos through the network again, and backwards to update the weights.

Section 2 of the notebook removes the last layer (num_classes=0), turns every photo into 512 numbers, and trains a logistic regression on them:

backbone = load("resnet18.a1_in1k", num_classes=0).eval()
Xtr = embed(backbone, list("data/" + train["file"]))
Xte = embed(backbone, list("data/" + test["file"]))
head = LogisticRegression(C=1.0, max_iter=5000).fit(Xtr, train["label"])
catalogue test accuracy 0.812 worst candle 0.45
new studio accuracy 0.794 worst candle 0.5
13.7 ms per photo on this computer

0.812, from 0.356 with HOG, on the same photos and the same 453 labels. The model’s ImageNet filters describe Larkfield’s products far better than any hand-made feature. The warmer studio costs 2 points, less than the histogram lost. Candle is still the worst class: 9 of 20 test candles found.

The notebook’s embed function prepares each photo the way the model card says: timm builds the resize, the crop and the normalising step from the card. The time per photo is the course team’s run; yours will differ.

Tomás, the AI engineer, has an opinion: “Take ResNet-50. A bigger model always sees better.” The course team measured four models of timm in the same way, frozen + last layer, on the same photos. Predict first: which of the four is best, and by how much?

Accuracy against model file size, for four pretrained models with a new last layer, on the 160 catalogue test photos. The axis starts at 0.7. The smallest model, ConvNeXt-Atto (15 MB), is within one photo of the largest.
Text description of the chart

Four points. ConvNeXt-Atto: 15 MB, accuracy 0.887. EfficientNet-B0: 21 MB, 0.844. ResNet-18: 47 MB, 0.812. ResNet-50: 103 MB, 0.894. The vertical axis runs from 0.7 to 1. The smallest model is almost as good as the largest and much better than the two in the middle.

The largest model, ResNet-50 (103 MB), reaches 0.894. The smallest, ConvNeXt-Atto (15 MB), reaches 0.887: one photo less, out of 160. With 160 test photos, one photo is 0.6 points, and a difference under about 3 points is noise. So the two are equal here. ConvNeXt-Atto is 7 times smaller and took 29.92 ms a photo in the measurement, against 68.86 for ResNet-50. Its worst class, candle, reaches 0.80 (16 of 20 found). That is as good as ResNet-50’s worst class, plant pot, and much better than ResNet-18’s 0.45 for candle.

Why? Size is one property of a model, and it is not the most important one. ConvNeXt is a newer design, trained with a newer recipe. What a model learned, and how, matters more than how many parameters it has. Tomás’s rule fails here. The only way to know is to measure on your own photos.

Section 4 of the notebook lets the last block (layer4) learn too, with a new last layer, for 4 epochs:

epoch 1: loss 1.793
epoch 2: loss 0.996
epoch 3: loss 0.511
epoch 4: loss 0.246
training: 44 s
fine-tuned, no augmentation: 0.85 | worst candle 0.4

The loss measures how wrong the model is on the training photos: it falls in every epoch. The test accuracy rises from 0.812 to 0.850. The training took about 31 seconds in the course team’s measurement and 44 seconds in the notebook run, on a laptop CPU. That is the price: 4 epochs of fine-tuning cost as much time as embedding the catalogue several times.

Fine-tuning helped ResNet-18 by 4 points. It did not help the candle: 8 of 20, against 9 when frozen. And the frozen ConvNeXt-Atto (0.887) is still ahead of the fine-tuned ResNet-18. Choosing the right pretrained model mattered more than training it longer.

The notebook trains on the same 453 photos in every epoch. Augmentation changes each training photo a little in every epoch, by code: a mirror image, a smaller crop, other colours. The model sees more kinds of photos, and memorising them gets harder. These are the three changes the course team tested:

Four versions of one photo of a green plant in an orange pot: the original; flipped left to right; a crop that shows a closer, larger part of the plant; and with changed brightness, contrast and colour.
One training photo and three augmented copies, made by code (torchvision): flipped, cropped, colour changed. Photo: Sara K, CC BY 2.0, from Open Images.
Text description of the diagram

Four versions of one photo: a plant with green leaves in an orange pot. The original. Flipped: a mirror image, left and right swapped. Cropped: a closer view that shows a larger part of the plant and cuts its edges. Colour changed: the same view with other brightness, contrast and colours. Each one is still clearly a plant pot.

The course team fine-tuned ResNet-18 three times, with the same seed, epochs and settings:

  • no augmentation
  • flips and crops
  • flips, crops and colour
Training loss per epoch, for three fine-tuning runs of ResNet-18. Test accuracy: no augmentation 0.850, flips and crops 0.844, flips, crops and colour 0.863.
Text description of the chart

Three lines over 4 epochs. No augmentation: 1.79, 1.00, 0.51, 0.25. Flips and crops: 1.80, 1.12, 0.67, 0.40. Flips, crops and colour: 1.83, 1.18, 0.77, 0.49. The runs with augmentation end with a higher loss, because their training photos are harder. The test accuracies are 0.850, 0.844 and 0.863.

With augmentation, the training loss stays higher: the photos are harder to learn by heart. That is the point of it. But the test accuracies are 0.850, 0.844 and 0.863: a difference of 2 photos at most, which is noise. Augmentation did not help here. Most of the model was frozen, and only the last block learned. So in 4 epochs, the model had little room to memorise.

Augmentation can also hurt. It is safe only when the change keeps the label true. The Overfitting controls lesson of the deep learning course explains this rule. At Larkfield:

  • A colour change is wrong when colour is the answer. If Omar’s team asks “what colour is this mug?”, a colour-changed red mug teaches the model that red is blue.
  • A crop is wrong when the important part is small. Crop a returns photo, and the damaged corner of a candle may fall outside the crop, while the label still says “damaged”.
  • A flip is wrong when left and right matter, for example for a printed text on a product.

Common mistake

A winner chosen from two photos

Symptom: you compare the three fine-tuning runs on the test photos. “Flips, crops and colour” has the highest accuracy, 0.863 against 0.850. You write “colour augmentation improves the catalogue model” in your report.

Cause: 0.863 − 0.850 is 0.013: two test photos out of 160. Another seed, or another order of the training photos, can change two answers. You also chose the winner by looking at the test set, so its score is now a little too good.

Fix: treat differences under about 3 points as noise with 160 test photos. Choose between settings on the validation photos, not the test photos. Repeat a run with two or three seeds when the choice matters. Keep the simplest setting when the results are equal: here, no augmentation.

Consequence in AI work: small differences on a small test set create false progress. Report the size of the test set next to every number, as the comparison table does.

Here is every catalogue method of this module, on the same 160 test photos:

  • Classical (no model file)
  • Pretrained, frozen + last layer
  • ResNet-18, fine-tuned 4 epochs
The catalogue rows of the comparison table after this module (accuracy on 160 test photos; 0.6 points is one photo). Classical methods stop below 0.4; every pretrained model is above 0.8.
Text description of the chart

One row per method, with its accuracy on an axis from 0 to 1. Classical methods: rules 0.269, colour histogram with 5 neighbours 0.263, HOG with logistic regression 0.356. Pretrained, frozen with a new last layer: ResNet-18 0.812, EfficientNet-B0 0.844, ConvNeXt-Atto 0.887, ResNet-50 0.894. ResNet-18 fine-tuned for 4 epochs: no augmentation 0.850, flips and crops 0.844, flips, crops and colour 0.863. There is a large gap between the classical methods, all under 0.4, and the pretrained ones, all over 0.8.

The gap between the two groups is the main result of this module. The classical methods stop below 0.4. Every pretrained model is above 0.8, with the same 453 labels. Within the pretrained group, the differences are small, and the smallest model is among the best.

  • Use frozen + last layer first. It is cheap, it runs on a CPU in seconds, and here it was as good as fine-tuning. Try two or three models, and keep the one that the table supports.
  • Fine-tune when frozen features are not enough, and you have more labelled photos than here. Your photos may also look very different from ImageNet’s, like X-rays or pallets drawn by code. Keep a validation set to choose the number of epochs.
  • Do not trust the size or the model card’s score alone. Measure on your own photos.
  • What it costs: a model file of 15 to 103 MB here, and 13 to 69 ms a photo on the course team’s CPU. The histogram needs about 1 ms. Also the model’s licence: these weights are Apache-2.0, but they learned from ImageNet, whose own terms When not to build it discusses.

Section 6 of the notebook has one task and a check cell. This is the guided practice of the module: train a last-layer classifier, and add it to the table with its CPU time.

  1. Load "convnext_atto.d2_in1k" with num_classes=0, and call .eval().
  2. Start a timer with time.perf_counter().
  3. Embed the training photos and the test photos with embed.
  4. Put the milliseconds per photo in ms_convnext.
  5. Train LogisticRegression(C=1.0, max_iter=5000) on the training embeddings.
  6. Put the test accuracy in acc_convnext, and run the check cell.
  7. Add your row to mine with cp.classification_row, and show the table.
Show the answer
small = load("convnext_atto.d2_in1k", num_classes=0).eval()
start = time.perf_counter()
Ctr = embed(small, list("data/" + train["file"]))
Cte = embed(small, list("data/" + test["file"]))
ms_convnext = 1000 * (time.perf_counter() - start) / (len(train) + len(test))
pred = LogisticRegression(C=1.0, max_iter=5000).fit(Ctr, train["label"]).predict(Cte)
acc_convnext = cp.classification_scores(list(test["label"]), list(pred), CLASSES)["accuracy"]
mine.add(cp.classification_row("ConvNeXt-Atto, frozen + last layer (yours)", list(test["label"]), list(pred),
"catalogue test", round(ms_convnext, 1), 15, len(train), "Apache-2.0 (weights)"))
mine.show("catalogue", columns=("method", "value", "worst", "worst_value", "ms_per_image", "model_mb"))
catalogue: accuracy
method value worst worst_value ms_per_image model_mb
ResNet-18, frozen + last layer (yours) 0.812 candle 0.450 13.7 47
ConvNeXt-Atto, frozen + last layer (yours) 0.887 candle 0.800 40.6 15

The check prints Check passed: acc_convnext is 0.887. The time per photo depends on the computer and on what else runs on it. The course team’s measurement gave 29.9 ms, and this notebook run 40.6 ms.

Check your understanding

Why does training a new last layer on 453 photos work, when training ResNet-18 from random numbers on the same photos would not?

Show the answer

A frozen ResNet-18 already turns each photo into 512 numbers that describe edges, textures and parts. It learned them from about 1.3 million ImageNet photos. The new last layer only learns how to weigh those numbers for 8 classes: 4,104 numbers. 453 photos are enough for that. Training all 11.7 million parameters from random numbers on 453 photos would memorise the training photos and fail on new ones.

Check your understanding

Tomás says: 'ResNet-50 is the biggest model, so we should use it.' Answer with the numbers of this lesson.

Show the answer

ResNet-50 (103 MB) reaches 0.894 and ConvNeXt-Atto (15 MB) reaches 0.887: one test photo apart, which is noise with 160 photos. ConvNeXt-Atto is 7 times smaller and took about 30 ms a photo against about 69 ms. Its worst class (candle, 0.80) is as good as ResNet-50’s worst class (plant pot, 0.80). A bigger model did not see better here. Choose the smaller one, unless a larger test set shows a real difference.

Check your understanding

Hamid asks: 'The histogram gets 0.263 and ConvNeXt-Atto 0.887. Is there any case where Larkfield should use the histogram?' Answer with numbers from your comparison table.

Show the answer

Yes, where colour is the answer and the camera and the light are fixed. The histogram needs no model file and about 1 ms a photo, against 15 MB and about 30 ms for ConvNeXt-Atto. On the returns, with one camera and one light, a colour rule costs 1,779 dollars a day against 2,229 with no check. It needs no labels at all. For the catalogue, no: photos come from many cameras and lights, and the histogram is right on only about one photo in four. It also loses about 4 points under the warmer studio light.

Recap

  • Transfer learning keeps a pretrained model’s filters and replaces its last layer. ResNet-18 with a new last layer learns 4,104 numbers and reaches 0.812, against 0.356 for HOG.
  • Bigger is not better here: ConvNeXt-Atto (15 MB) reaches 0.887. That is as good as ResNet-50 (103 MB, 0.894) and better than ResNet-18 (0.812).
  • Fine-tuning the last block for 4 epochs raised ResNet-18 to 0.850, in about 30 seconds on a CPU. Augmentation did not help on 453 photos: 0.844 and 0.863 are within noise.
  • Augmentation must keep the label true: no colour change when colour is the answer, no crop that can remove the damage.
  • The comparison table now has two groups: classical methods under 0.4, pretrained models over 0.8. The candle stays the worst class.

You can now adapt a pretrained model with a few hundred labels. But each method so far needed a model trained for this one task. Next, in Module 3, you use the 512 numbers themselves. You compare photos by these numbers, draw the catalogue as a map, and find the classes that the model confuses.

Next: Image embeddings