Skip to content
Français

Pretrained detectors

  • Lesson 14
  • 40 min
  • Module 4: Detect and count

Where we are

We can now judge one box with IoU. Next, we judge a whole detector. In this lesson, we run a torchvision detector and RT-DETR on pallet photos, read mAP by object size, and look at the licence of the popular YOLO models.

What you will be able to do

Run detectors that others trained on everyday photos, read what they find on Larkfield’s pallets, and judge a whole detector with one number: average precision (AP), by object size. Then decide whether a popular detector’s licence fits a closed product.

NotebookDo this lesson in a notebookColab · Kaggle · GitHub

The notebook runs torchvision’s Faster R-CNN on three pallet photos, replays the recorded boxes of Faster R-CNN and RT-DETR on all 30 test photos, and computes AP by object size. Its setup cells install PyTorch and download the pallet photos and the detector’s weights (74 MB), so it needs no local setup. About 40 minutes; the cells run in under a minute after the setup.

Colab and Kaggle open your own copy. They need a free Google or Kaggle account. Stuck? See the solution notebook.

Before you startWhat to know and to have readyAbout 40 minutes

What you need to know

What you need on your computer

  • For the notebook: nothing. It runs in Colab or Kaggle.
  • For the project steps: your larkvision project with the vision count command from the last lesson.

Tested with Python 3.12, PyTorch 2.15.0 and torchvision 0.29.2 on macOS. The Windows commands are the same; they were not run.

A new worker joins Hamid’s warehouse. Before, they worked in a big supermarket. They know thousands of products by name: kettles, keyboards, clocks, fridges. On their first day, they look at a pallet of Larkfield’s plain brown boxes. They have never sorted these boxes. So they name what they know: “that one looks like a fridge”.

A pretrained detector is that worker. It learned from many photos, but only the classes in those photos.

The problem: training a detector from nothing is expensive

Section titled “The problem: training a detector from nothing is expensive”

A detector learns from photos where a person drew every box. To learn what “an object” looks like at all, it needs many thousands of such photos. Larkfield has 40 labelled pallet photos.

So we start from a detector that someone else trained. A pretrained model is a model already trained on a large public dataset. We can use it as it is, or adapt it (the next lesson).

Most public detectors are trained on COCO (Common Objects in Context). It is a large public set of everyday photos with boxes for 80 classes: person, car, bottle, chair, clock, keyboard, stop sign and more. “Box” or “cardboard box” is not one of them.

We try two detectors, both trained on COCO:

Detector How it works Licence CPU time a photo
Faster R-CNN with a MobileNetV3 backbone (torchvision) Two stages: first propose regions that may hold an object, then name each region and fix its box BSD-3-Clause code; weights trained on COCO 138 ms
RT-DETR R18 A transformer that returns a fixed set of boxes in one pass; it is trained to give one box per object Apache-2.0 129 ms

The backbone is the part of the network that turns pixels into features: lists of numbers that describe small areas of the photo (as in Convolution and patches, in one lesson). The times are measured on a laptop CPU, one photo at a time.

In the notebook, Priya runs Faster R-CNN on three test pallets:

for sid in ("S-061", "S-063", "S-085"):
found = [f"{c} {s}" for c, s, _ in detect(sid) if s >= 0.5]
print(sid, images[sid]["crowding"], images[sid]["count"], "boxes ->", found or "nothing at score 0.5")
S-061 loose 9 boxes -> nothing at score 0.5
S-063 tight 22 boxes -> ['clock 0.87']
S-085 loose 7 boxes -> ['train 0.53']

The course team ran both detectors once on all 30 test photos and saved every box. These are the classes they found with a score of at least 0.5:

What RT-DETR calls Larkfield's boxes: 13 stop signs, 9 keyboards, 4 trains and 1 fridge. Faster R-CNN (not shown) finds 2 trains and 1 clock. Neither finds a single 'box', because COCO has no such class.
Text description of the chart

A column chart with four columns. RT-DETR found 13 stop signs, 9 keyboards, 4 trains and 1 refrigerator with a score of at least 0.5 on the 30 test pallets. Faster R-CNN, not in the chart, found 2 trains and 1 clock.

Three pallet photos. Left: one orange rectangle around the whole pallet, labelled clock 0.87. Middle: two top boxes labelled stop sign 0.68 and stop sign 0.58. Right: one rectangle around a crowded pallet, labelled keyboard 0.95.
Recorded boxes with a score of at least 0.5. Faster R-CNN calls the whole tight pallet S-063 a clock; RT-DETR calls two top boxes of S-067 stop signs, and the whole pallet S-075 a keyboard. The pallet photos are drawn by our code (synthetic).
Text description of the diagram

Three synthetic pallet photos side by side. On S-063, a tight pallet with 22 boxes, Faster R-CNN draws one rectangle around the whole pallet with the class clock and score 0.87. On S-067, a medium pallet with large boxes, RT-DETR draws two rectangles on the two top-right boxes with the class stop sign and scores 0.68 and 0.58. On S-075, a tight pallet with 46 boxes, RT-DETR draws one rectangle around the whole pallet with the class keyboard and score 0.95.

The guesses make sense. A tight pallet is a grid of small rectangles, like a keyboard. A box with a white label, seen from the front, has the shape of a road sign. The detectors are not broken. They answer the question that they were trained on.

Average precision: one number for a whole detector

Section titled “Average precision: one number for a whole detector”

To compare detectors, we need one number over all photos. Counting is one view. Average precision (AP) is the usual one.

Put every box of every photo in one list, best score first. Go down the list. At each box, ask: is it right (IoU at least 0.5 with a true box that is still free)? After each box, compute:

  • Precision: the share of the boxes so far that are right.
  • Recall: the share of all true boxes found so far.

A good detector keeps its precision high while its recall rises. AP measures the area under the curve of precision against recall. 1.0 is perfect: every true box is found before any wrong box.

A constructed example: 5 true boxes, and a detector that returns 6 boxes.

Box (best first) Right? Right so far Precision Recall
1 yes 1 1 ÷ 1 = 1.00 1 ÷ 5 = 0.20
2 yes 2 2 ÷ 2 = 1.00 0.40
3 no 2 2 ÷ 3 = 0.67 0.40
4 yes 3 3 ÷ 4 = 0.75 0.60
5 no 3 3 ÷ 5 = 0.60 0.60
6 yes 4 4 ÷ 6 = 0.67 0.80

One true box is never found, so recall stops at 0.80. For each step of recall, AP takes the best precision that is still possible from there on:

AP ≈ 0.2 × 1.00 + 0.2 × 1.00 + 0.2 × 0.75 + 0.2 × 0.67 = 0.68

  • Precision at each box
  • Best precision from here on
The constructed example: precision against recall after each of the 6 boxes (dots), and the step curve that AP measures (dashed). AP is the area under the dashed curve: about 0.68.
Text description of the chart

A chart with recall from 0 to 1 across and precision from 0 to 1 up. Six dots: (0.2, 1.00), (0.4, 1.00), (0.4, 0.67), (0.6, 0.75), (0.6, 0.60) and (0.8, 0.67). A dashed step line stays at 1.00 up to recall 0.4, then 0.75 up to 0.6, then 0.67 up to 0.8, then drops to 0. The area under it is about 0.68.

The course’s helper cp.average_precision does the same with 101 recall points, as COCO does. On this example it gives 0.686.

You will read these names in model cards:

  • AP50 (or AP at IoU 0.5): AP with the line at IoU 0.5.
  • mAP: the mean of AP over all classes. Larkfield has one class, “box”, so mAP equals AP.
  • mAP50–95: the mean of AP at ten IoU lines, 0.50, 0.55, … 0.95. It rewards boxes that fit tightly. COCO reports this one.

A model card’s mAP comes from COCO photos, not from yours. It is a vendor claim for your task until you measure it on your own photos.

COCO splits objects by area: small (under 32 × 32 pixels), medium (up to 96 × 96) and large. Detectors often miss small objects, so AP by size shows where a detector is weak.

The 619 test boxes of the pallets are 0 small, 566 medium and 53 large. So for the pallets, AP for “small” cannot be computed. On a shelf photo with many small products, AP for small objects is the number to watch.

Larkfield: AP and counts of the COCO detectors

Section titled “Larkfield: AP and counts of the COCO detectors”

The comparison table already has both detectors. Run them with your project. Every box of any class with a score of at least 0.5 counts as a box:

  1. In your larkvision folder, with the virtual environment active, count with Faster R-CNN:

    Terminal window
    vision count --method fasterrcnn-coco --score 0.5
    fasterrcnn-coco (score >= 0.5): count error 20.53 boxes a photo (bias -20.53), exact on 0% of photos
    loose 10 photos: error 8.0, bias -8.0
    medium 10 photos: error 16.6, bias -16.6
    tight 10 photos: error 37.0, bias -37.0
    AP at IoU 0.5: 0.000; kept boxes that cover two boxes: 11
  2. Count with RT-DETR:

    Terminal window
    vision count --method rtdetr-coco --score 0.5
    rtdetr-coco (score >= 0.5): count error 19.73 boxes a photo (bias -19.73), exact on 0% of photos
    loose 10 photos: error 6.8, bias -6.8
    medium 10 photos: error 16.1, bias -16.1
    tight 10 photos: error 36.3, bias -36.3
    AP at IoU 0.5: 0.013; kept boxes that cover two boxes: 203

AP50 is 0.000 for Faster R-CNN and 0.013 for RT-DETR. The bias equals the error with a minus sign: they count far too few on every photo. On the medium boxes, RT-DETR’s AP50 is 0.042, and on the large ones 0.067. Your notebook computes these.

The pallets have about 21 boxes a photo (619 ÷ 30 = 20.6). A count error of about 20 means that the detectors find almost nothing. They are no better than answering “0” every time.

Common mistake

Looking for a class that does not exist

Symptom: Tomás keeps only the boxes whose class is “box”, to remove the stop signs. Every pallet gets a count of 0.

Cause: COCO has no class “box”. The detector cannot return a class that it never learned. Filtering for it removes everything. The count error is then the mean true count: 20.6 boxes a photo.

Fix: read the model’s class list before you use it. torchvision lists it in weights.meta["categories"]; a model card on Hugging Face lists it in the model’s configuration. If your class is not there, you need a detector that reads a text (the next lesson), or labelled photos to fine-tune.

Consequence in AI work: a pretrained model’s outputs are limited to its training labels. “It is a strong model” says nothing about your labels. Check the overlap first.

You will meet YOLO (“You Only Look Once”) in almost every article about detection. YOLO models look at the whole photo once and return all boxes in one pass, which makes them fast. The best-known versions today come from the company Ultralytics, with an easy Python package.

This course does not use Ultralytics YOLO. The reason is its licence, not its quality.

Ultralytics publishes its YOLO code and weights under the GNU Affero General Public License version 3 (AGPL-3.0) (its licence file). It is a copyleft licence: you may use and change the code. But if you share a program built on it, you must share that program’s source code under the same licence. The “Affero” part adds a network rule. If users interact with your program over a network, for example through a web service, you must offer them its source code too (AGPL-3.0, section 13).

For a closed product, such as Larkfield’s internal image service, that is a problem. Using Ultralytics YOLO inside it could oblige Larkfield to publish the service’s code. Ultralytics sells an enterprise licence that removes this obligation.

Choice What you may do What it costs
torchvision detectors (BSD-3-Clause code) Use in a closed product; keep the licence notice Check the training data of the weights too
RT-DETR from Hugging Face (Apache-2.0) Use in a closed product; keep the notice The same check
Ultralytics YOLO under AGPL-3.0 Use it, but share your source code with your users, also over a network Your code becomes open
Ultralytics YOLO with an enterprise licence Use in a closed product A licence fee

Other YOLO-style models come from other authors, under other licences. Check each one.

This is not legal advice. Licence terms change: read the current licence page of each model and ask your legal team before you ship. A licence also covers only the code and the weights. The photos that the weights learned from have their own terms. When not to build it comes back to this.

Use it as it is when your objects are in its class list and look like its training photos. A COCO detector knows “chair”, “bowl”, “vase” and “potted plant”, so it may find some of Larkfield’s catalogue products with no training. Measure it on your own photos first.

Do not use it as it is when your objects are not in its class list, like the pallet boxes. Its scores then mean nothing for you: RT-DETR is 0.95 sure that a pallet is a keyboard.

What it costs: no labels and no training. About 130 milliseconds a photo on a laptop CPU, and a model of 74 to 81 MB. And a licence check of the code, the weights and their training data.

Guided practice: pretrained detectors in the notebook

Section titled “Guided practice: pretrained detectors in the notebook”
  1. Open the notebook from the box Do this lesson in a notebook at the top of this page.
  2. Run the setup cells. The first one installs PyTorch, which can take 2 minutes the first time.
  3. In section 1, run the cell. You see 30 photos, 619 true boxes.
  4. In section 2, predict what the detector says about a pallet. Then run the cell. It downloads the weights (74 MB) and prints what it finds on three photos.
  5. In section 3, run the cell. You see the classes of both detectors with a score of at least 0.5.
  6. In section 4, run the cell. You see AP50 0.000 and 0.013, and the true boxes by size.
  7. In section 5, compute RT-DETR’s AP on medium boxes in ap_medium. The check cell prints Check passed.

The counts above used a score line of 0.5. Predict: if you lower the score line to 0.1, will RT-DETR’s count error go up or down? Will its AP change? Then run vision count --method rtdetr-coco --score 0.1.

Show the answer

The AP does not change: it is computed from every recorded box at every score, so the score line plays no part in it. It stays at 0.013.

The count error changes, because more boxes pass the line:

rtdetr-coco (score >= 0.1): count error 16.27 boxes a photo (bias -0.8), exact on 0% of photos
loose 10 photos: error 7.5, bias +6.5
medium 10 photos: error 19.5, bias +11.9
tight 10 photos: error 21.8, bias -20.8
AP at IoU 0.5: 0.013; kept boxes that cover two boxes: 3621

The error falls from 19.73 to 16.27, and the bias is almost 0. That looks better, but it is not. Loose and medium pallets now get too many boxes, tight pallets still far too few, and the two cancel in the bias. The extra boxes are still stop signs, keyboards and other guesses. A lower line can make a count look better without making the detector any better. Always look at AP, or at the right and wrong boxes, next to the count.

Check your understanding

A detector returns 4 boxes, best first: right, wrong, right, right. There are 4 true boxes. Compute the precision and the recall after each box.

Show the answer

After box 1: precision 1 ÷ 1 = 1.00, recall 1 ÷ 4 = 0.25. After box 2: 1 ÷ 2 = 0.50, recall 0.25. After box 3: 2 ÷ 3 = 0.67, recall 0.50. After box 4: 3 ÷ 4 = 0.75, recall 0.75. One true box is never found, so recall stops at 0.75. The best precision from each recall on is 1.00, 0.75 and 0.75, so AP ≈ 0.25 × 1.00 + 0.25 × 0.75 + 0.25 × 0.75 = 0.625. The helper’s 101 recall points give 0.629.

Check your understanding

RT-DETR is 0.95 sure that the pallet S-075 is a keyboard. Tomás says: 'A score of 0.95 means it is almost certainly right.' What is wrong with this?

Show the answer

A score says how sure the detector is among the classes that it knows. It is not a probability that the answer is right for your task. RT-DETR knows 80 COCO classes and no “box”. For a grid of small rectangles, “keyboard” is its best match, so it is sure. On the 30 pallets, its AP50 for boxes is 0.013. Scores become useful only after you measure them on your own photos.

Check your understanding

Omar wants to put Ultralytics YOLO into Larkfield's closed image service, which staff use through a web page. What should he know about the AGPL-3.0 licence, and what are his options?

Show the answer

AGPL-3.0 is a copyleft licence with a network rule: if users interact with a program built on the code over a network, the program’s source code must be offered to them under the same licence. A web service used by staff is such a case. He has three options. He can publish the service’s code under AGPL-3.0. He can buy Ultralytics’ enterprise licence. Or he can use a detector with a permissive licence, such as torchvision’s Faster R-CNN (BSD-3-Clause) or RT-DETR (Apache-2.0). Then he must also check the terms of the photos that the weights learned from. He should ask the legal team and read the current licence page before deciding.

Recap

  • A pretrained detector knows only its training classes. COCO has 80, and no “box”.
  • On the pallets, Faster R-CNN finds a clock and trains (AP50 0.000), and RT-DETR finds 13 stop signs, 9 keyboards, 4 trains and 1 fridge (AP50 0.013). Both miss about 20 boxes a photo.
  • AP is the area under the precision–recall curve of all boxes, best score first. mAP50–95 averages it over IoU lines from 0.50 to 0.95. Split it by object size.
  • Ultralytics YOLO is AGPL-3.0: a closed product built on it must share its source code, also over a network, unless you buy a licence. This course uses BSD-3-Clause and Apache-2.0 detectors.

You can now judge a whole detector with AP and read its licence. But neither detector knows Larkfield’s boxes. Next, we teach one of them the class “box” with 40 labelled photos and two minutes on a CPU. Then we compare it with a detector that needs only a text.

Next: Fine-tune a detector on a few images