Skip to content
Français

Fine-tune a detector on a few images

  • Lesson 15
  • 45 min
  • Module 4: Detect and count

Where we are

The pretrained detectors know the objects of their training set, and a Larkfield box is not one of them. In this lesson, we fine-tune a small detector on a few labelled pallet photos on a CPU, and compare it with a detector that reads a text prompt.

What you will be able to do

Teach a pretrained detector Larkfield’s boxes with 40 labelled photos and about two minutes on a laptop CPU. Measure it with AP and the count error, and compare it with an open-vocabulary detector that finds boxes from a text, with no labels at all.

NotebookDo this lesson in a notebookColab · Kaggle · GitHub

The notebook fine-tunes Faster R-CNN on the 40 training pallets, measures it on the 30 test pallets, and replays the recorded boxes of the open-vocabulary detector OWLv2. Its setup cells install PyTorch and download the pallet photos and the detector’s weights (74 MB). About 45 minutes, with about 2 minutes of training on a laptop CPU.

Colab and Kaggle open your own copy. They need a free Google or Kaggle account. Stuck? See the solution notebook.

Before you startWhat to know and to have readyAbout 45 minutes

What you need to know

What you need on your computer

  • For the notebook: nothing. It runs in Colab or Kaggle, on a CPU.
  • For the project steps: your larkvision project with the vision count command.

Tested with Python 3.12, PyTorch 2.15.0 and torchvision 0.29.2 on macOS (Apple M4 CPU). The Windows commands are the same; they were not run.

Remember the new worker from the supermarket. They already know edges, corners, cardboard and shadows. They do not need to learn to see again. They only need to learn one new thing: what one Larkfield box looks like, and where it ends.

So Hamid shows them 40 pallets, and points at every box. After a few rounds, they find the boxes alone.

In the last lesson, two COCO detectors gave AP50 0.000 and 0.013 on the pallets. They know 80 classes, and “box” is not one of them.

Training a new detector from nothing needs thousands of labelled photos. Larkfield has 40 labelled training pallets, with 817 boxes.

The idea: fine-tune the top, freeze the bottom

Section titled “The idea: fine-tune the top, freeze the bottom”

Fine-tuning means: start from a trained model and train it a little more on your own labelled data. We do not change all of it.

A photo goes into three blocks in a row: the backbone (grey, dashed, frozen), the region proposals (blue, trained) and the box head (blue, trained, 2 classes: background and box).
Fine-tuning Faster R-CNN on 40 pallet photos: the backbone keeps what it learned and does not change (frozen); the region proposals and the box head learn the class 'box'.
Text description of the diagram

A diagram with a small “photo” square and three blocks connected by arrows. The first block, the backbone (MobileNetV3, learned on ImageNet and COCO), is grey with a dashed border and the words “frozen: not changed”. The second block, the region proposals (where an object may be), is blue and “trained”. The third block, the box head, now has 2 classes, background and box, and is blue and “trained”.

  • The backbone turns the photo into features. It already knows edges, corners and textures, so we freeze it: its numbers do not change.
  • The region proposals suggest places where an object may be. They learn where Larkfield’s boxes usually are.
  • The box head names each region and fixes its box. We replace it with a new one that has two classes: background and box.

Freezing the backbone keeps what it learned from millions of photos: 40 pallet photos could not teach it better features, and could make it forget good ones. It also means fewer numbers to learn. In this model the frozen part is smaller than you may think: the notebook prints 15,982,677 of 18,930,229 numbers will learn. A part between the backbone and the proposals, the feature pyramid, still learns too.

One epoch is one pass over all training photos. With 40 photos and 12 epochs, the detector sees 40 × 12 = 480 photos in all. For each photo, it computes a loss: one number that says how wrong its boxes and classes are. Each step changes the trained numbers a little, so that the loss goes down.

Half of the time, a photo is flipped left to right first. A flipped pallet is still a pallet, so the detector gets more variety for free. This is augmentation, as in Transfer learning.

Larkfield: fine-tune Faster R-CNN on a CPU

Section titled “Larkfield: fine-tune Faster R-CNN on a CPU”

In the notebook, Priya fine-tunes torchvision’s Faster R-CNN MobileNetV3 with the course’s settings: backbone frozen, 12 epochs, flips. The cell that trains prints one line per epoch:

epoch 1: loss 1.490
epoch 2: loss 1.002
epoch 3: loss 0.931
epoch 4: loss 0.930
epoch 5: loss 0.824
epoch 6: loss 0.771
epoch 7: loss 0.775
epoch 8: loss 0.769
epoch 9: loss 0.700
epoch 10: loss 0.695
epoch 11: loss 0.681
epoch 12: loss 0.701
trained in 98 s

The course’s measurement run took 107 seconds, about 1.8 minutes, on the same laptop; this notebook run took 98. Your time will differ. The next cell measures the detector on the 30 test pallets:

AP50 0.867 | count error at score 0.6: 2.5 | by crowding: {'loose': 0.2, 'medium': 0.4, 'tight': 6.9}

The same numbers as the course’s run: the code, the seed and the photos are the same.

The loss falls quickly in the first epochs, then slowly. It does not fall in every epoch: a small set of photos makes the loss jump a little.

  • 40 photos, 12 epochs
  • 10 photos, 30 epochs
The training loss of the course's two runs. With 40 photos, the loss falls from 1.49 to about 0.70 in 12 epochs (107 seconds). With 10 photos, it falls from 1.94 to about 0.8, with more noise, in 30 epochs (60 seconds).
Text description of the chart

A line chart with epochs from 1 to 30 across and the mean training loss from 0 to 2 up. The blue line, 40 photos, has 12 points: 1.49, 1.00, 0.93, 0.93, 0.82, 0.77, 0.77, 0.77, 0.70, 0.70, 0.68 and 0.70. The orange line, 10 photos, has 30 points: it starts at 1.94, falls to about 0.9 by epoch 6, then moves between 0.71 and 0.99 until epoch 30, where it is 0.83.

A low training loss is not the goal. The goal is good boxes on photos that the detector has never seen. So we measure on the 30 test pallets.

The course’s measurements:

Detector Labelled photos Training AP50 mAP50–95 Count error (score line)
RT-DETR, COCO, no fine-tuning 0 none 0.013 0.011 19.73 (0.5)
Faster R-CNN, fine-tuned 10 (197 boxes) 30 epochs, 60 s 0.705 0.339 5.0 (0.1)
Faster R-CNN, fine-tuned 40 (817 boxes) 12 epochs, 107 s 0.867 0.451 2.5 (0.6)

The score line of each detector was chosen on the 20 validation pallets, where it gave the lowest count error. The 10-photo detector is less sure of itself, so its best line is much lower: 0.1.

Forty labelled photos and two minutes on a CPU take AP50 from 0.013 to 0.867. mAP50–95 is lower, 0.451, because it also asks for boxes that fit tightly, at IoU 0.75 and above. For counting, AP50 is the one that matters: a box needs to be on the right object, not to the pixel.

  • 40 photos
  • 10 photos
Precision against recall of the two fine-tuned detectors on the 30 test pallets, all boxes best score first, IoU 0.5. The detector trained on 40 photos keeps a precision near 1.0 up to a recall of about 0.6; AP is the area under each curve.
Text description of the chart

A chart with recall from 0 to 1 across and precision from 0 to 1 up. The blue curve, 40 photos, stays at a precision of 1.0 up to a recall of 0.5, is 0.99 at 0.6, 0.91 at 0.7, 0.79 at 0.8 and 0.59 at 0.9, and ends at a recall of 0.94 with a precision of 0.44. The orange dashed curve, 10 photos, stays near 1.0 up to a recall of 0.3, is 0.93 at 0.5, 0.84 at 0.6 and 0.69 at 0.7, and ends at a recall of 0.78 with a precision of 0.57. AP50 is 0.867 for 40 photos and 0.705 for 10 photos.

Check two of these numbers with your project:

  1. Go to your larkvision folder, with the virtual environment active and LARKVISION_DATA set (in a new terminal, see Meet the Larkfield images). Count with the detector trained on 10 photos, at its validation line:

    Terminal window
    vision count --method finetuned-10 --score 0.1
    finetuned-10 (score >= 0.1): count error 5.0 boxes a photo (bias +2.93), exact on 40% of photos
    loose 10 photos: error 2.4, bias +2.4
    medium 10 photos: error 0.7, bias +0.7
    tight 10 photos: error 11.9, bias +5.7
    AP at IoU 0.5: 0.705; kept boxes that cover two boxes: 12
  2. Compare with the detector trained on 40 photos, which you ran in Boxes and IoU: count error 2.5, tight 6.9, AP 0.867.

Four times the labels halve the count error. The tight pallets stay the hardest for both.

Common mistake

The loss becomes nan

Symptom: the course’s first run on 10 photos printed a normal loss for four epochs. At epoch 5, the loss was nan (“not a number”). Once one number is nan, every number computed from it is nan too, so the model is useless from that step on.

Cause: one training step was too large. With only 10 photos, one unlucky photo can give a very large error. The step that follows then changes the numbers so much that they overflow. After one nan, every later number is nan too.

Fix: limit the size of each step with gradient clipping. The course adds one line before each step: torch.nn.utils.clip_grad_norm_(params, 5.0). It shrinks a step whose size is above 5, and leaves smaller steps alone. With it, the 10-photo run trained normally (the orange line above). A smaller learning rate also helps, but makes training slower.

Consequence in AI work: small datasets make training unstable. Print the loss of every epoch, stop as soon as it is nan, and keep the protection in the code once you have added it.

No labels at all: an open-vocabulary detector

Section titled “No labels at all: an open-vocabulary detector”

Labels cost time. Is there a detector that needs none?

An open-vocabulary detector finds objects from a text that you write, such as “a cardboard box”. It works like CLIP, which you met in CLIP zero-shot: it puts small regions of the photo and your text into one space of numbers. Then it keeps the regions whose numbers are close to the text’s numbers.

The course uses OWLv2 (base size, Apache-2.0). It is a newer version of OWL-ViT. Another well-known one is Grounding DINO; the course uses one open-vocabulary detector only. OWLv2’s model is 620 MB and takes about 1.26 seconds a photo on a laptop CPU. That is about 15 times slower than the fine-tuned detector (87 milliseconds).

Its score line (0.075) and NMS line (0.3) were chosen on the 20 validation pallets, as for the fine-tuned detectors. Run it with your project:

Terminal window
vision count --method owlv2 --score 0.075 --nms 0.3
owlv2 (score >= 0.075, NMS 0.3): count error 14.0 boxes a photo (bias -12.13), exact on 0% of photos
loose 10 photos: error 1.9, bias +1.9
medium 10 photos: error 3.2, bias -1.4
tight 10 photos: error 36.9, bias -36.9
AP at IoU 0.5: 0.364; kept boxes that cover two boxes: 109

On loose pallets, OWLv2 is good with no labels: an error of 1.9 boxes a photo. It often adds one box around the whole pallet, so it counts a little too many. On tight pallets, it finds almost nothing: no box on 8 of the 10 photos, and one on the other 2. The error is 36.9 and the bias is −36.9.

Four pallet photos in two rows. Top row, a loose pallet with 9 boxes: OWLv2 finds 10 boxes, 9 right and one large orange box around the whole pallet; the fine-tuned detector finds 9 right boxes. Bottom row, a tight pallet with 22 boxes: OWLv2 finds 1 orange box around the whole pallet; the fine-tuned detector finds 22 boxes, 21 right and 1 wrong.
OWLv2 with the text 'a cardboard box' (score line 0.075, NMS 0.3) against the detector fine-tuned on 40 photos (score line 0.6), on the loose pallet S-061 and the tight pallet S-063. Blue: right box (IoU at least 0.5); orange: wrong box.
Text description of the diagram

Four synthetic pallet photos. Top left, OWLv2 on S-061 (9 boxes): 9 blue boxes, one on each cardboard box, and one orange box around the whole pallet: 10 boxes. Top right, the fine-tuned detector on S-061: 9 blue boxes. Bottom left, OWLv2 on S-063 (22 boxes, tight): only one orange box around the whole pallet. Bottom right, the fine-tuned detector on S-063: 22 boxes, 21 blue and 1 orange.

On a tight pallet, touching boxes form one big brown block. For OWLv2, that block is the best match for “a cardboard box”. The fine-tuned detector learned from 817 labelled boxes where one box ends and the next begins.

  • Fine-tuned, 40 photos
  • Fine-tuned, 10 photos
  • OWLv2, no labels
Count error by crowding. All three detectors are good on loose and medium pallets. On tight pallets, the error is 6.9 with 40 labelled photos, 11.9 with 10, and 36.9 with no labels.
Text description of the chart

A chart with three points across: loose, medium and tight. The fine-tuned detector with 40 photos has errors of 0.2, 0.4 and 6.9. With 10 photos: 2.4, 0.7 and 11.9. OWLv2 with no labels: 1.9, 3.2 and 36.9.

Fine-tuned on 40 photos Fine-tuned on 10 photos OWLv2, no labels
Count error (tight) 2.5 (6.9) 5.0 (11.9) 14.0 (36.9)
Labelled photos (boxes) 40 (817) 10 (197) 0
Training on a laptop CPU 107 s 60 s none
CPU time a photo 87 ms 99 ms 1,260 ms
Model size 74 MB 74 MB 620 MB
Licence BSD-3-Clause code; weights trained on COCO the same Apache-2.0

What do the labels cost? Assume a person draws one box in about 5 seconds (an assumption; measure it with your own tool). 817 boxes take about 68 minutes. At the course’s cost of a person’s time, 30 dollars an hour, that is about 34 dollars, once. For 10 photos (197 boxes), about 16 minutes and 8 dollars.

Fine-tune when you have a fixed kind of object and a few dozen photos to label. The labels cost little, and the detector is fast on a CPU.

Use an open-vocabulary detector to start before you have labels, for objects that stand apart, or to propose boxes that a person then corrects. That cuts the labelling time. Do not trust it on crowded scenes.

Use neither alone when a wrong count is expensive. On tight pallets, even the best detector is off by 6.9 boxes a photo: plan a check by a person there.

Guided practice: fine-tune in the notebook

Section titled “Guided practice: fine-tune in the notebook”
  1. Open the notebook from the box Do this lesson in a notebook at the top of this page.
  2. Run the setup cells. The first one installs PyTorch, which can take 2 minutes the first time.
  3. In section 1, run the cell. You see 40 training photos with 817 boxes; 30 test photos.
  4. In section 2, run the cell. It prints how many of the detector’s numbers will learn.
  5. In section 3, predict if the loss falls in every epoch. Then run the cell. It trains for about 2 minutes.
  6. In section 4, run the two cells. You see your AP50 and count error, and Check passed when your AP50 is close to 0.87.
  7. In section 6, compute OWLv2’s count error on tight pallets in owl_tight. The check cell prints Check passed.

Hamid’s team can label 10 pallets this week, or 40 pallets next month. Using the table above, predict for each choice: the count error on tight pallets, and the labelling time at 5 seconds a box. Then say which choice you would take, and why.

Show the answer
  • 10 pallets: about 16 minutes of labelling (197 boxes × 5 s). Count error 5.0 boxes a photo, 11.9 on tight pallets. AP50 0.705.
  • 40 pallets: about 68 minutes (817 boxes × 5 s). Count error 2.5, 6.9 on tight pallets. AP50 0.867.

The extra 52 minutes of labelling halve the count error. That is cheap compared with one wrong pallet count. A good plan: label 10 pallets this week, fine-tune, and use the detector with a check by a person on tight pallets. Then label the other 30 and fine-tune again. Keep the gradient clipping: the 10-photo run needs it.

Check your understanding

Why does the course freeze the backbone when it fine-tunes Faster R-CNN on 40 photos? Give two reasons.

Show the answer

The backbone already turns photos into useful features: edges, corners, textures. It learned them from millions of photos, and 40 pallet photos cannot teach it better ones. Freezing it keeps those features safe: the model cannot forget them while it fits 40 photos. It also means fewer numbers to learn (here about 16.0 of 18.9 million still learn), which saves some time and some labels. Training took 107 seconds on a laptop CPU. Only the region proposals and the new box head, with the classes background and box, learn.

Check your understanding

OWLv2 counts loose pallets with an error of 1.9 boxes, and tight pallets with an error of 36.9. Explain the difference, and say how you would use OWLv2 at Larkfield.

Show the answer

OWLv2 matches regions of the photo with the text “a cardboard box”. On a loose pallet, each box stands apart, so each one is a good match. On a tight pallet, touching boxes look like one big brown block, and the whole block is the best match: OWLv2 returns no box or one box. It never learned where one Larkfield box ends. Use it to propose boxes before you have labels, for example to speed up labelling, and never to count tight pallets.

Check your understanding

A colleague trains a detector on 10 photos and reports a count error of 3.1 on the test pallets, with a score line that they chose by trying lines on the test photos. What is wrong, and what should they do?

Show the answer

They chose the line on the photos that they then report on, so the number is too good: it was fitted to those 30 photos. The course chose each line on the 20 validation pallets, and only then measured once on the test pallets (count error 5.0 for 10 photos). They should choose the line on the validation photos, keep it, and report the test result once.

Recap

  • Fine-tuning starts from a trained detector and trains a part of it on your labels. With the backbone frozen and a new box head (background, box), 40 photos and 107 CPU seconds give AP50 0.867 and a count error of 2.5.
  • With 10 photos (60 s, 30 epochs), AP50 is 0.705 and the count error 5.0. Its first run broke (loss nan) until gradient clipping limited each step.
  • The score line is chosen on validation photos: 0.6 for 40 photos, 0.1 for 10.
  • An open-vocabulary detector (OWLv2, “a cardboard box”) needs no labels: error 1.9 on loose pallets, but 36.9 on tight ones, where touching boxes look like one. It is 620 MB and 1.26 s a photo.
  • Labels cost little here: about 68 minutes for 817 boxes, at an assumed 5 seconds a box.

You can now build a detector for Larkfield’s boxes and say what it costs. But Hamid does not want boxes: he wants a number, and the detector still struggles with tight pallets. Next, we compare counting from boxes with two ways that find no boxes at all.

Next: Counting, and tracking in brief