Fine-tune a detector on a few images
- Lesson 15
- 45 min
- Module 4: Detect and count
Where we are
The pretrained detectors know the objects of their training set, and a Larkfield box is not one of them. In this lesson, we fine-tune a small detector on a few labelled pallet photos on a CPU, and compare it with a detector that reads a text prompt.
Lesson 15 of 30 · Vision AI: See, Count and InspectFine-tune a detector on a few images
1. Seeing is a business problem
2. Baselines and pretrained classifiers
3. Embeddings, CLIP and few labels
4. Detect and count
6. Inspect for defects
7. Match, search and describe
8. Run vision in production
What you will be able to do
Teach a pretrained detector Larkfield’s boxes with 40 labelled photos and about two minutes on a laptop CPU. Measure it with AP and the count error, and compare it with an open-vocabulary detector that finds boxes from a text, with no labels at all.
NotebookDo this lesson in a notebook
The notebook fine-tunes Faster R-CNN on the 40 training pallets, measures it on the 30 test pallets, and replays the recorded boxes of the open-vocabulary detector OWLv2. Its setup cells install PyTorch and download the pallet photos and the detector’s weights (74 MB). About 45 minutes, with about 2 minutes of training on a laptop CPU.
Colab and Kaggle open your own copy. They need a free Google or Kaggle account. Stuck? See the solution notebook.
Before you startWhat to know and to have ready
What you need to know
- Pretrained detectors, AP and mAP50–95, from Pretrained detectors.
- Epochs and the training loss, from Batches, epochs and validation in the deep learning course (helpful, not required: this lesson explains what it uses).
- Image and text embeddings in one space, from The idea: one space for photos and words in CLIP zero-shot.
What you need on your computer
- For the notebook: nothing. It runs in Colab or Kaggle, on a CPU.
- For the project steps: your
larkvisionproject with thevision countcommand.
Tested with Python 3.12, PyTorch 2.15.0 and torchvision 0.29.2 on macOS (Apple M4 CPU). The Windows commands are the same; they were not run.
Teach the new worker
Section titled “Teach the new worker”Remember the new worker from the supermarket. They already know edges, corners, cardboard and shadows. They do not need to learn to see again. They only need to learn one new thing: what one Larkfield box looks like, and where it ends.
So Hamid shows them 40 pallets, and points at every box. After a few rounds, they find the boxes alone.
The problem: a detector without our class
Section titled “The problem: a detector without our class”In the last lesson, two COCO detectors gave AP50 0.000 and 0.013 on the pallets. They know 80 classes, and “box” is not one of them.
Training a new detector from nothing needs thousands of labelled photos. Larkfield has 40 labelled training pallets, with 817 boxes.
The idea: fine-tune the top, freeze the bottom
Section titled “The idea: fine-tune the top, freeze the bottom”Fine-tuning means: start from a trained model and train it a little more on your own labelled data. We do not change all of it.
Text description of the diagram
A diagram with a small “photo” square and three blocks connected by arrows. The first block, the backbone (MobileNetV3, learned on ImageNet and COCO), is grey with a dashed border and the words “frozen: not changed”. The second block, the region proposals (where an object may be), is blue and “trained”. The third block, the box head, now has 2 classes, background and box, and is blue and “trained”.
- The backbone turns the photo into features. It already knows edges, corners and textures, so we freeze it: its numbers do not change.
- The region proposals suggest places where an object may be. They learn where Larkfield’s boxes usually are.
- The box head names each region and fixes its box. We replace it with a new one that has two classes: background and box.
Freezing the backbone keeps what it learned from millions of photos: 40 pallet photos could not teach it better features, and could make it forget good ones. It also means fewer numbers to learn. In this model the frozen part is smaller than you may think: the notebook prints 15,982,677 of 18,930,229 numbers will learn. A part between the backbone and the proposals, the feature pyramid, still learns too.
A tiny example: what one epoch is
Section titled “A tiny example: what one epoch is”One epoch is one pass over all training photos. With 40 photos and 12 epochs, the detector sees 40 × 12 = 480 photos in all. For each photo, it computes a loss: one number that says how wrong its boxes and classes are. Each step changes the trained numbers a little, so that the loss goes down.
Half of the time, a photo is flipped left to right first. A flipped pallet is still a pallet, so the detector gets more variety for free. This is augmentation, as in Transfer learning.
Larkfield: fine-tune Faster R-CNN on a CPU
Section titled “Larkfield: fine-tune Faster R-CNN on a CPU”In the notebook, Priya fine-tunes torchvision’s Faster R-CNN MobileNetV3 with the course’s settings: backbone frozen, 12 epochs, flips. The cell that trains prints one line per epoch:
epoch 1: loss 1.490epoch 2: loss 1.002epoch 3: loss 0.931epoch 4: loss 0.930epoch 5: loss 0.824epoch 6: loss 0.771epoch 7: loss 0.775epoch 8: loss 0.769epoch 9: loss 0.700epoch 10: loss 0.695epoch 11: loss 0.681epoch 12: loss 0.701trained in 98 sThe course’s measurement run took 107 seconds, about 1.8 minutes, on the same laptop; this notebook run took 98. Your time will differ. The next cell measures the detector on the 30 test pallets:
AP50 0.867 | count error at score 0.6: 2.5 | by crowding: {'loose': 0.2, 'medium': 0.4, 'tight': 6.9}The same numbers as the course’s run: the code, the seed and the photos are the same.
The loss falls quickly in the first epochs, then slowly. It does not fall in every epoch: a small set of photos makes the loss jump a little.
- 40 photos, 12 epochs
- 10 photos, 30 epochs
Text description of the chart
A line chart with epochs from 1 to 30 across and the mean training loss from 0 to 2 up. The blue line, 40 photos, has 12 points: 1.49, 1.00, 0.93, 0.93, 0.82, 0.77, 0.77, 0.77, 0.70, 0.70, 0.68 and 0.70. The orange line, 10 photos, has 30 points: it starts at 1.94, falls to about 0.9 by epoch 6, then moves between 0.71 and 0.99 until epoch 30, where it is 0.83.
A low training loss is not the goal. The goal is good boxes on photos that the detector has never seen. So we measure on the 30 test pallets.
AP and the count, after fine-tuning
Section titled “AP and the count, after fine-tuning”The course’s measurements:
| Detector | Labelled photos | Training | AP50 | mAP50–95 | Count error (score line) |
|---|---|---|---|---|---|
| RT-DETR, COCO, no fine-tuning | 0 | none | 0.013 | 0.011 | 19.73 (0.5) |
| Faster R-CNN, fine-tuned | 10 (197 boxes) | 30 epochs, 60 s | 0.705 | 0.339 | 5.0 (0.1) |
| Faster R-CNN, fine-tuned | 40 (817 boxes) | 12 epochs, 107 s | 0.867 | 0.451 | 2.5 (0.6) |
The score line of each detector was chosen on the 20 validation pallets, where it gave the lowest count error. The 10-photo detector is less sure of itself, so its best line is much lower: 0.1.
Forty labelled photos and two minutes on a CPU take AP50 from 0.013 to 0.867. mAP50–95 is lower, 0.451, because it also asks for boxes that fit tightly, at IoU 0.75 and above. For counting, AP50 is the one that matters: a box needs to be on the right object, not to the pixel.
- 40 photos
- 10 photos
Text description of the chart
A chart with recall from 0 to 1 across and precision from 0 to 1 up. The blue curve, 40 photos, stays at a precision of 1.0 up to a recall of 0.5, is 0.99 at 0.6, 0.91 at 0.7, 0.79 at 0.8 and 0.59 at 0.9, and ends at a recall of 0.94 with a precision of 0.44. The orange dashed curve, 10 photos, stays near 1.0 up to a recall of 0.3, is 0.93 at 0.5, 0.84 at 0.6 and 0.69 at 0.7, and ends at a recall of 0.78 with a precision of 0.57. AP50 is 0.867 for 40 photos and 0.705 for 10 photos.
Check two of these numbers with your project:
-
Go to your
larkvisionfolder, with the virtual environment active andLARKVISION_DATAset (in a new terminal, see Meet the Larkfield images). Count with the detector trained on 10 photos, at its validation line:Terminal window vision count --method finetuned-10 --score 0.1finetuned-10 (score >= 0.1): count error 5.0 boxes a photo (bias +2.93), exact on 40% of photosloose 10 photos: error 2.4, bias +2.4medium 10 photos: error 0.7, bias +0.7tight 10 photos: error 11.9, bias +5.7AP at IoU 0.5: 0.705; kept boxes that cover two boxes: 12 -
Compare with the detector trained on 40 photos, which you ran in Boxes and IoU: count error 2.5, tight 6.9, AP 0.867.
Four times the labels halve the count error. The tight pallets stay the hardest for both.
Common mistake
The loss becomes nan
Symptom: the course’s first run on 10 photos printed a normal loss for four epochs. At epoch 5, the loss was nan (“not a number”). Once one number is nan, every number computed from it is nan too, so the model is useless from that step on.
Cause: one training step was too large. With only 10 photos, one unlucky photo can give a very large error. The step that follows then changes the numbers so much that they overflow. After one nan, every later number is nan too.
Fix: limit the size of each step with gradient clipping. The course adds one line before each step: torch.nn.utils.clip_grad_norm_(params, 5.0). It shrinks a step whose size is above 5, and leaves smaller steps alone. With it, the 10-photo run trained normally (the orange line above). A smaller learning rate also helps, but makes training slower.
Consequence in AI work: small datasets make training unstable. Print the loss of every epoch, stop as soon as it is nan, and keep the protection in the code once you have added it.
No labels at all: an open-vocabulary detector
Section titled “No labels at all: an open-vocabulary detector”Labels cost time. Is there a detector that needs none?
An open-vocabulary detector finds objects from a text that you write, such as “a cardboard box”. It works like CLIP, which you met in CLIP zero-shot: it puts small regions of the photo and your text into one space of numbers. Then it keeps the regions whose numbers are close to the text’s numbers.
The course uses OWLv2 (base size, Apache-2.0). It is a newer version of OWL-ViT. Another well-known one is Grounding DINO; the course uses one open-vocabulary detector only. OWLv2’s model is 620 MB and takes about 1.26 seconds a photo on a laptop CPU. That is about 15 times slower than the fine-tuned detector (87 milliseconds).
Its score line (0.075) and NMS line (0.3) were chosen on the 20 validation pallets, as for the fine-tuned detectors. Run it with your project:
vision count --method owlv2 --score 0.075 --nms 0.3owlv2 (score >= 0.075, NMS 0.3): count error 14.0 boxes a photo (bias -12.13), exact on 0% of photos loose 10 photos: error 1.9, bias +1.9 medium 10 photos: error 3.2, bias -1.4 tight 10 photos: error 36.9, bias -36.9 AP at IoU 0.5: 0.364; kept boxes that cover two boxes: 109On loose pallets, OWLv2 is good with no labels: an error of 1.9 boxes a photo. It often adds one box around the whole pallet, so it counts a little too many. On tight pallets, it finds almost nothing: no box on 8 of the 10 photos, and one on the other 2. The error is 36.9 and the bias is −36.9.

Text description of the diagram
Four synthetic pallet photos. Top left, OWLv2 on S-061 (9 boxes): 9 blue boxes, one on each cardboard box, and one orange box around the whole pallet: 10 boxes. Top right, the fine-tuned detector on S-061: 9 blue boxes. Bottom left, OWLv2 on S-063 (22 boxes, tight): only one orange box around the whole pallet. Bottom right, the fine-tuned detector on S-063: 22 boxes, 21 blue and 1 orange.
On a tight pallet, touching boxes form one big brown block. For OWLv2, that block is the best match for “a cardboard box”. The fine-tuned detector learned from 817 labelled boxes where one box ends and the next begins.
- Fine-tuned, 40 photos
- Fine-tuned, 10 photos
- OWLv2, no labels
Text description of the chart
A chart with three points across: loose, medium and tight. The fine-tuned detector with 40 photos has errors of 0.2, 0.4 and 6.9. With 10 photos: 2.4, 0.7 and 11.9. OWLv2 with no labels: 1.9, 3.2 and 36.9.
Which one, and what it costs
Section titled “Which one, and what it costs”| Fine-tuned on 40 photos | Fine-tuned on 10 photos | OWLv2, no labels | |
|---|---|---|---|
| Count error (tight) | 2.5 (6.9) | 5.0 (11.9) | 14.0 (36.9) |
| Labelled photos (boxes) | 40 (817) | 10 (197) | 0 |
| Training on a laptop CPU | 107 s | 60 s | none |
| CPU time a photo | 87 ms | 99 ms | 1,260 ms |
| Model size | 74 MB | 74 MB | 620 MB |
| Licence | BSD-3-Clause code; weights trained on COCO | the same | Apache-2.0 |
What do the labels cost? Assume a person draws one box in about 5 seconds (an assumption; measure it with your own tool). 817 boxes take about 68 minutes. At the course’s cost of a person’s time, 30 dollars an hour, that is about 34 dollars, once. For 10 photos (197 boxes), about 16 minutes and 8 dollars.
Fine-tune when you have a fixed kind of object and a few dozen photos to label. The labels cost little, and the detector is fast on a CPU.
Use an open-vocabulary detector to start before you have labels, for objects that stand apart, or to propose boxes that a person then corrects. That cuts the labelling time. Do not trust it on crowded scenes.
Use neither alone when a wrong count is expensive. On tight pallets, even the best detector is off by 6.9 boxes a photo: plan a check by a person there.
Guided practice: fine-tune in the notebook
Section titled “Guided practice: fine-tune in the notebook”- Open the notebook from the box Do this lesson in a notebook at the top of this page.
- Run the setup cells. The first one installs PyTorch, which can take 2 minutes the first time.
- In section 1, run the cell. You see
40 training photos with 817 boxes; 30 test photos. - In section 2, run the cell. It prints how many of the detector’s numbers will learn.
- In section 3, predict if the loss falls in every epoch. Then run the cell. It trains for about 2 minutes.
- In section 4, run the two cells. You see your AP50 and count error, and
Check passedwhen your AP50 is close to 0.87. - In section 6, compute OWLv2’s count error on tight pallets in
owl_tight. The check cell printsCheck passed.
Your turn: 10 photos or 40?
Section titled “Your turn: 10 photos or 40?”Hamid’s team can label 10 pallets this week, or 40 pallets next month. Using the table above, predict for each choice: the count error on tight pallets, and the labelling time at 5 seconds a box. Then say which choice you would take, and why.
Show the answer
- 10 pallets: about 16 minutes of labelling (197 boxes × 5 s). Count error 5.0 boxes a photo, 11.9 on tight pallets. AP50 0.705.
- 40 pallets: about 68 minutes (817 boxes × 5 s). Count error 2.5, 6.9 on tight pallets. AP50 0.867.
The extra 52 minutes of labelling halve the count error. That is cheap compared with one wrong pallet count. A good plan: label 10 pallets this week, fine-tune, and use the detector with a check by a person on tight pallets. Then label the other 30 and fine-tune again. Keep the gradient clipping: the 10-photo run needs it.
Knowledge check
Section titled “Knowledge check”Check your understanding
Why does the course freeze the backbone when it fine-tunes Faster R-CNN on 40 photos? Give two reasons.
Show the answer
The backbone already turns photos into useful features: edges, corners, textures. It learned them from millions of photos, and 40 pallet photos cannot teach it better ones. Freezing it keeps those features safe: the model cannot forget them while it fits 40 photos. It also means fewer numbers to learn (here about 16.0 of 18.9 million still learn), which saves some time and some labels. Training took 107 seconds on a laptop CPU. Only the region proposals and the new box head, with the classes background and box, learn.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
OWLv2 counts loose pallets with an error of 1.9 boxes, and tight pallets with an error of 36.9. Explain the difference, and say how you would use OWLv2 at Larkfield.
Show the answer
OWLv2 matches regions of the photo with the text “a cardboard box”. On a loose pallet, each box stands apart, so each one is a good match. On a tight pallet, touching boxes look like one big brown block, and the whole block is the best match: OWLv2 returns no box or one box. It never learned where one Larkfield box ends. Use it to propose boxes before you have labels, for example to speed up labelling, and never to count tight pallets.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
A colleague trains a detector on 10 photos and reports a count error of 3.1 on the test pallets, with a score line that they chose by trying lines on the test photos. What is wrong, and what should they do?
Show the answer
They chose the line on the photos that they then report on, so the number is too good: it was fitted to those 30 photos. The course chose each line on the 20 validation pallets, and only then measured once on the test pallets (count error 5.0 for 10 photos). They should choose the line on the validation photos, keep it, and report the test result once.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Recap
- Fine-tuning starts from a trained detector and trains a part of it on your labels. With the backbone frozen and a new box head (background, box), 40 photos and 107 CPU seconds give AP50 0.867 and a count error of 2.5.
- With 10 photos (60 s, 30 epochs), AP50 is 0.705 and the count error 5.0. Its first run broke (loss
nan) until gradient clipping limited each step. - The score line is chosen on validation photos: 0.6 for 40 photos, 0.1 for 10.
- An open-vocabulary detector (OWLv2, “a cardboard box”) needs no labels: error 1.9 on loose pallets, but 36.9 on tight ones, where touching boxes look like one. It is 620 MB and 1.26 s a photo.
- Labels cost little here: about 68 minutes for 817 boxes, at an assumed 5 seconds a box.
You can now build a detector for Larkfield’s boxes and say what it costs. But Hamid does not want boxes: he wants a number, and the detector still struggles with tight pallets. Next, we compare counting from boxes with two ways that find no boxes at all.