Pretrained detectors
- Lesson 14
- 40 min
- Module 4: Detect and count
Where we are
We can now judge one box with IoU. Next, we judge a whole detector. In this lesson, we run a torchvision detector and RT-DETR on pallet photos, read mAP by object size, and look at the licence of the popular YOLO models.
Lesson 14 of 30 · Vision AI: See, Count and InspectPretrained detectors
1. Seeing is a business problem
2. Baselines and pretrained classifiers
3. Embeddings, CLIP and few labels
4. Detect and count
6. Inspect for defects
7. Match, search and describe
8. Run vision in production
What you will be able to do
Run detectors that others trained on everyday photos, read what they find on Larkfield’s pallets, and judge a whole detector with one number: average precision (AP), by object size. Then decide whether a popular detector’s licence fits a closed product.
NotebookDo this lesson in a notebook
The notebook runs torchvision’s Faster R-CNN on three pallet photos, replays the recorded boxes of Faster R-CNN and RT-DETR on all 30 test photos, and computes AP by object size. Its setup cells install PyTorch and download the pallet photos and the detector’s weights (74 MB), so it needs no local setup. About 40 minutes; the cells run in under a minute after the setup.
Colab and Kaggle open your own copy. They need a free Google or Kaggle account. Stuck? See the solution notebook.
Before you startWhat to know and to have ready
What you need to know
- Boxes, scores, IoU and NMS, from Boxes and IoU.
- Precision and recall, from The precision–recall trade-off in the machine learning course.
What you need on your computer
- For the notebook: nothing. It runs in Colab or Kaggle.
- For the project steps: your
larkvisionproject with thevision countcommand from the last lesson.
Tested with Python 3.12, PyTorch 2.15.0 and torchvision 0.29.2 on macOS. The Windows commands are the same; they were not run.
A new worker from another shop
Section titled “A new worker from another shop”A new worker joins Hamid’s warehouse. Before, they worked in a big supermarket. They know thousands of products by name: kettles, keyboards, clocks, fridges. On their first day, they look at a pallet of Larkfield’s plain brown boxes. They have never sorted these boxes. So they name what they know: “that one looks like a fridge”.
A pretrained detector is that worker. It learned from many photos, but only the classes in those photos.
The problem: training a detector from nothing is expensive
Section titled “The problem: training a detector from nothing is expensive”A detector learns from photos where a person drew every box. To learn what “an object” looks like at all, it needs many thousands of such photos. Larkfield has 40 labelled pallet photos.
So we start from a detector that someone else trained. A pretrained model is a model already trained on a large public dataset. We can use it as it is, or adapt it (the next lesson).
The idea: detectors trained on COCO
Section titled “The idea: detectors trained on COCO”Most public detectors are trained on COCO (Common Objects in Context). It is a large public set of everyday photos with boxes for 80 classes: person, car, bottle, chair, clock, keyboard, stop sign and more. “Box” or “cardboard box” is not one of them.
We try two detectors, both trained on COCO:
| Detector | How it works | Licence | CPU time a photo |
|---|---|---|---|
| Faster R-CNN with a MobileNetV3 backbone (torchvision) | Two stages: first propose regions that may hold an object, then name each region and fix its box | BSD-3-Clause code; weights trained on COCO | 138 ms |
| RT-DETR R18 | A transformer that returns a fixed set of boxes in one pass; it is trained to give one box per object | Apache-2.0 | 129 ms |
The backbone is the part of the network that turns pixels into features: lists of numbers that describe small areas of the photo (as in Convolution and patches, in one lesson). The times are measured on a laptop CPU, one photo at a time.
What they see on a pallet
Section titled “What they see on a pallet”In the notebook, Priya runs Faster R-CNN on three test pallets:
for sid in ("S-061", "S-063", "S-085"): found = [f"{c} {s}" for c, s, _ in detect(sid) if s >= 0.5] print(sid, images[sid]["crowding"], images[sid]["count"], "boxes ->", found or "nothing at score 0.5")S-061 loose 9 boxes -> nothing at score 0.5S-063 tight 22 boxes -> ['clock 0.87']S-085 loose 7 boxes -> ['train 0.53']The course team ran both detectors once on all 30 test photos and saved every box. These are the classes they found with a score of at least 0.5:
Text description of the chart
A column chart with four columns. RT-DETR found 13 stop signs, 9 keyboards, 4 trains and 1 refrigerator with a score of at least 0.5 on the 30 test pallets. Faster R-CNN, not in the chart, found 2 trains and 1 clock.

Text description of the diagram
Three synthetic pallet photos side by side. On S-063, a tight pallet with 22 boxes, Faster R-CNN draws one rectangle around the whole pallet with the class clock and score 0.87. On S-067, a medium pallet with large boxes, RT-DETR draws two rectangles on the two top-right boxes with the class stop sign and scores 0.68 and 0.58. On S-075, a tight pallet with 46 boxes, RT-DETR draws one rectangle around the whole pallet with the class keyboard and score 0.95.
The guesses make sense. A tight pallet is a grid of small rectangles, like a keyboard. A box with a white label, seen from the front, has the shape of a road sign. The detectors are not broken. They answer the question that they were trained on.
Average precision: one number for a whole detector
Section titled “Average precision: one number for a whole detector”To compare detectors, we need one number over all photos. Counting is one view. Average precision (AP) is the usual one.
The idea
Section titled “The idea”Put every box of every photo in one list, best score first. Go down the list. At each box, ask: is it right (IoU at least 0.5 with a true box that is still free)? After each box, compute:
- Precision: the share of the boxes so far that are right.
- Recall: the share of all true boxes found so far.
A good detector keeps its precision high while its recall rises. AP measures the area under the curve of precision against recall. 1.0 is perfect: every true box is found before any wrong box.
A tiny example
Section titled “A tiny example”A constructed example: 5 true boxes, and a detector that returns 6 boxes.
| Box (best first) | Right? | Right so far | Precision | Recall |
|---|---|---|---|---|
| 1 | yes | 1 | 1 ÷ 1 = 1.00 | 1 ÷ 5 = 0.20 |
| 2 | yes | 2 | 2 ÷ 2 = 1.00 | 0.40 |
| 3 | no | 2 | 2 ÷ 3 = 0.67 | 0.40 |
| 4 | yes | 3 | 3 ÷ 4 = 0.75 | 0.60 |
| 5 | no | 3 | 3 ÷ 5 = 0.60 | 0.60 |
| 6 | yes | 4 | 4 ÷ 6 = 0.67 | 0.80 |
One true box is never found, so recall stops at 0.80. For each step of recall, AP takes the best precision that is still possible from there on:
AP ≈ 0.2 × 1.00 + 0.2 × 1.00 + 0.2 × 0.75 + 0.2 × 0.67 = 0.68
- Precision at each box
- Best precision from here on
Text description of the chart
A chart with recall from 0 to 1 across and precision from 0 to 1 up. Six dots: (0.2, 1.00), (0.4, 1.00), (0.4, 0.67), (0.6, 0.75), (0.6, 0.60) and (0.8, 0.67). A dashed step line stays at 1.00 up to recall 0.4, then 0.75 up to 0.6, then 0.67 up to 0.8, then drops to 0. The area under it is about 0.68.
The course’s helper cp.average_precision does the same with 101 recall points, as COCO does. On this example it gives 0.686.
mAP and its names
Section titled “mAP and its names”You will read these names in model cards:
- AP50 (or AP at IoU 0.5): AP with the line at IoU 0.5.
- mAP: the mean of AP over all classes. Larkfield has one class, “box”, so mAP equals AP.
- mAP50–95: the mean of AP at ten IoU lines, 0.50, 0.55, … 0.95. It rewards boxes that fit tightly. COCO reports this one.
A model card’s mAP comes from COCO photos, not from yours. It is a vendor claim for your task until you measure it on your own photos.
AP by object size
Section titled “AP by object size”COCO splits objects by area: small (under 32 × 32 pixels), medium (up to 96 × 96) and large. Detectors often miss small objects, so AP by size shows where a detector is weak.
The 619 test boxes of the pallets are 0 small, 566 medium and 53 large. So for the pallets, AP for “small” cannot be computed. On a shelf photo with many small products, AP for small objects is the number to watch.
Larkfield: AP and counts of the COCO detectors
Section titled “Larkfield: AP and counts of the COCO detectors”The comparison table already has both detectors. Run them with your project. Every box of any class with a score of at least 0.5 counts as a box:
-
In your
larkvisionfolder, with the virtual environment active, count with Faster R-CNN:Terminal window vision count --method fasterrcnn-coco --score 0.5fasterrcnn-coco (score >= 0.5): count error 20.53 boxes a photo (bias -20.53), exact on 0% of photosloose 10 photos: error 8.0, bias -8.0medium 10 photos: error 16.6, bias -16.6tight 10 photos: error 37.0, bias -37.0AP at IoU 0.5: 0.000; kept boxes that cover two boxes: 11 -
Count with RT-DETR:
Terminal window vision count --method rtdetr-coco --score 0.5rtdetr-coco (score >= 0.5): count error 19.73 boxes a photo (bias -19.73), exact on 0% of photosloose 10 photos: error 6.8, bias -6.8medium 10 photos: error 16.1, bias -16.1tight 10 photos: error 36.3, bias -36.3AP at IoU 0.5: 0.013; kept boxes that cover two boxes: 203
AP50 is 0.000 for Faster R-CNN and 0.013 for RT-DETR. The bias equals the error with a minus sign: they count far too few on every photo. On the medium boxes, RT-DETR’s AP50 is 0.042, and on the large ones 0.067. Your notebook computes these.
The pallets have about 21 boxes a photo (619 ÷ 30 = 20.6). A count error of about 20 means that the detectors find almost nothing. They are no better than answering “0” every time.
Common mistake
Looking for a class that does not exist
Symptom: Tomás keeps only the boxes whose class is “box”, to remove the stop signs. Every pallet gets a count of 0.
Cause: COCO has no class “box”. The detector cannot return a class that it never learned. Filtering for it removes everything. The count error is then the mean true count: 20.6 boxes a photo.
Fix: read the model’s class list before you use it. torchvision lists it in weights.meta["categories"]; a model card on Hugging Face lists it in the model’s configuration. If your class is not there, you need a detector that reads a text (the next lesson), or labelled photos to fine-tune.
Consequence in AI work: a pretrained model’s outputs are limited to its training labels. “It is a strong model” says nothing about your labels. Check the overlap first.
The YOLO family and its licence
Section titled “The YOLO family and its licence”You will meet YOLO (“You Only Look Once”) in almost every article about detection. YOLO models look at the whole photo once and return all boxes in one pass, which makes them fast. The best-known versions today come from the company Ultralytics, with an easy Python package.
This course does not use Ultralytics YOLO. The reason is its licence, not its quality.
What AGPL-3.0 asks
Section titled “What AGPL-3.0 asks”Ultralytics publishes its YOLO code and weights under the GNU Affero General Public License version 3 (AGPL-3.0) (its licence file). It is a copyleft licence: you may use and change the code. But if you share a program built on it, you must share that program’s source code under the same licence. The “Affero” part adds a network rule. If users interact with your program over a network, for example through a web service, you must offer them its source code too (AGPL-3.0, section 13).
For a closed product, such as Larkfield’s internal image service, that is a problem. Using Ultralytics YOLO inside it could oblige Larkfield to publish the service’s code. Ultralytics sells an enterprise licence that removes this obligation.
| Choice | What you may do | What it costs |
|---|---|---|
| torchvision detectors (BSD-3-Clause code) | Use in a closed product; keep the licence notice | Check the training data of the weights too |
| RT-DETR from Hugging Face (Apache-2.0) | Use in a closed product; keep the notice | The same check |
| Ultralytics YOLO under AGPL-3.0 | Use it, but share your source code with your users, also over a network | Your code becomes open |
| Ultralytics YOLO with an enterprise licence | Use in a closed product | A licence fee |
Other YOLO-style models come from other authors, under other licences. Check each one.
This is not legal advice. Licence terms change: read the current licence page of each model and ask your legal team before you ship. A licence also covers only the code and the weights. The photos that the weights learned from have their own terms. When not to build it comes back to this.
When to use a pretrained detector
Section titled “When to use a pretrained detector”Use it as it is when your objects are in its class list and look like its training photos. A COCO detector knows “chair”, “bowl”, “vase” and “potted plant”, so it may find some of Larkfield’s catalogue products with no training. Measure it on your own photos first.
Do not use it as it is when your objects are not in its class list, like the pallet boxes. Its scores then mean nothing for you: RT-DETR is 0.95 sure that a pallet is a keyboard.
What it costs: no labels and no training. About 130 milliseconds a photo on a laptop CPU, and a model of 74 to 81 MB. And a licence check of the code, the weights and their training data.
Guided practice: pretrained detectors in the notebook
Section titled “Guided practice: pretrained detectors in the notebook”- Open the notebook from the box Do this lesson in a notebook at the top of this page.
- Run the setup cells. The first one installs PyTorch, which can take 2 minutes the first time.
- In section 1, run the cell. You see
30 photos, 619 true boxes. - In section 2, predict what the detector says about a pallet. Then run the cell. It downloads the weights (74 MB) and prints what it finds on three photos.
- In section 3, run the cell. You see the classes of both detectors with a score of at least 0.5.
- In section 4, run the cell. You see AP50
0.000and0.013, and the true boxes by size. - In section 5, compute RT-DETR’s AP on medium boxes in
ap_medium. The check cell printsCheck passed.
Your turn: a lower score line
Section titled “Your turn: a lower score line”The counts above used a score line of 0.5. Predict: if you lower the score line to 0.1, will RT-DETR’s count error go up or down? Will its AP change? Then run vision count --method rtdetr-coco --score 0.1.
Show the answer
The AP does not change: it is computed from every recorded box at every score, so the score line plays no part in it. It stays at 0.013.
The count error changes, because more boxes pass the line:
rtdetr-coco (score >= 0.1): count error 16.27 boxes a photo (bias -0.8), exact on 0% of photos loose 10 photos: error 7.5, bias +6.5 medium 10 photos: error 19.5, bias +11.9 tight 10 photos: error 21.8, bias -20.8 AP at IoU 0.5: 0.013; kept boxes that cover two boxes: 3621The error falls from 19.73 to 16.27, and the bias is almost 0. That looks better, but it is not. Loose and medium pallets now get too many boxes, tight pallets still far too few, and the two cancel in the bias. The extra boxes are still stop signs, keyboards and other guesses. A lower line can make a count look better without making the detector any better. Always look at AP, or at the right and wrong boxes, next to the count.
Knowledge check
Section titled “Knowledge check”Check your understanding
A detector returns 4 boxes, best first: right, wrong, right, right. There are 4 true boxes. Compute the precision and the recall after each box.
Show the answer
After box 1: precision 1 ÷ 1 = 1.00, recall 1 ÷ 4 = 0.25. After box 2: 1 ÷ 2 = 0.50, recall 0.25. After box 3: 2 ÷ 3 = 0.67, recall 0.50. After box 4: 3 ÷ 4 = 0.75, recall 0.75. One true box is never found, so recall stops at 0.75. The best precision from each recall on is 1.00, 0.75 and 0.75, so AP ≈ 0.25 × 1.00 + 0.25 × 0.75 + 0.25 × 0.75 = 0.625. The helper’s 101 recall points give 0.629.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
RT-DETR is 0.95 sure that the pallet S-075 is a keyboard. Tomás says: 'A score of 0.95 means it is almost certainly right.' What is wrong with this?
Show the answer
A score says how sure the detector is among the classes that it knows. It is not a probability that the answer is right for your task. RT-DETR knows 80 COCO classes and no “box”. For a grid of small rectangles, “keyboard” is its best match, so it is sure. On the 30 pallets, its AP50 for boxes is 0.013. Scores become useful only after you measure them on your own photos.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
Omar wants to put Ultralytics YOLO into Larkfield's closed image service, which staff use through a web page. What should he know about the AGPL-3.0 licence, and what are his options?
Show the answer
AGPL-3.0 is a copyleft licence with a network rule: if users interact with a program built on the code over a network, the program’s source code must be offered to them under the same licence. A web service used by staff is such a case. He has three options. He can publish the service’s code under AGPL-3.0. He can buy Ultralytics’ enterprise licence. Or he can use a detector with a permissive licence, such as torchvision’s Faster R-CNN (BSD-3-Clause) or RT-DETR (Apache-2.0). Then he must also check the terms of the photos that the weights learned from. He should ask the legal team and read the current licence page before deciding.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Recap
- A pretrained detector knows only its training classes. COCO has 80, and no “box”.
- On the pallets, Faster R-CNN finds a clock and trains (AP50 0.000), and RT-DETR finds 13 stop signs, 9 keyboards, 4 trains and 1 fridge (AP50 0.013). Both miss about 20 boxes a photo.
- AP is the area under the precision–recall curve of all boxes, best score first. mAP50–95 averages it over IoU lines from 0.50 to 0.95. Split it by object size.
- Ultralytics YOLO is AGPL-3.0: a closed product built on it must share its source code, also over a network, unless you buy a licence. This course uses BSD-3-Clause and Apache-2.0 detectors.
You can now judge a whole detector with AP and read its licence. But neither detector knows Larkfield’s boxes. Next, we teach one of them the class “box” with 40 labelled photos and two minutes on a CPU. Then we compare it with a detector that needs only a text.