Skip to content
Français

Count and detect in crowded scenes

  • Case study 3
  • 150 min
  • Optional · 1 case study needed for the certificate

Where we are

In eight modules, we counted boxes on Larkfield's pallets. In this case study, we count objects in crowded open photos. We compare two detectors, a density method and a vision-language model on count error by crowding.

What you will be able to do

Count windows and cars in real open photos, from three objects to crowded scenes. You compare five counters by their count error by crowding and by what the errors cost a small business. On validation photos, you choose a counter and when a person counts instead. Then you test once, run the decision on new photos, and write it down for the owner.

Case studyPortfolio piece

Time
About 2 hours 30 minutes
Runs in
Colab, Kaggle or VS Code
Skills used
  • Count error by crowding
  • Pretrained against fine-tuned detectors
  • Density estimate
  • Vision-language model, recorded
  • A line to a person
  • Cost per 1,000 photos

Colab and Kaggle open your own copy. To use VS Code, download the notebook from GitHub. Stuck? See the solution notebook.

The notebook runs every step of this page, with the same code and the same output. It adds three tasks with checks. Its setup cells install the packages and download the photos (15 MB) and three models: RT-DETR (81 MB), Faster R-CNN (74 MB) and ResNet-18 (47 MB). After the downloads, it runs in about 4 minutes on a laptop CPU. The hosted model’s answers are recorded, so you need no key and no account.

Before you startWhat to know and to have readyAbout 150 minutes

What you need to know

This case study uses Module 4 most. These lessons matter:

What you need on your computer

Nothing for the notebook in Colab or Kaggle. To run it locally, use a virtual environment with Python 3.12 and Jupyter, as in Meet the Larkfield images. The notebook works in a new folder crowded-scenes. To start again, delete that folder. This case study does not use the larkvision project.

Tested on macOS with Python 3.12.13, PyTorch 2.15.0, torchvision 0.29.2, timm 1.0.31, Transformers 5.22.2 and scikit-learn 1.9.1. We ran it in Jupyter from a fresh start. Colab, Kaggle, Windows and Linux are not tested.

Clearline is a small cleaning company (a made-up company). It cleans the windows of office buildings and washes the cars in their car parks. A new customer sends a photo: the front of the building, or the car park. Marta, the owner, counts the windows or the cars and sends a quote.

Marta gets more photos every month, and counting a big building by eye is slow. She asks Diego, a data scientist: “Can the computer count the photos for me? I can still count some of them myself.” Leo, who writes the quotes with her, has a simpler idea: “Just ask one of the big AI models.”

A wrong count costs money in two ways. Too few, and Clearline cleans windows it is not paid for. Too many, and the quote is too high, and the customer goes to another company. Marta put a price on each mistake. These are assumptions, her estimates, not measured facts:

  • A window counted wrong costs 3 dollars: the price of cleaning one window.
  • A car counted wrong costs 12 dollars: the price of washing one car.
  • A photo counted by a person costs 0.50 dollars: about one minute at 30 dollars an hour. A person is always right.
  • The office can count at most 250 photos in every 1,000. The other photos must be quoted from the computer’s count.

This is Hamid’s pallet count from Module 4, with real photos and a price on every error.

You will build five counters. A counter takes a photo and returns a number. One extra rule turns a counter into a decision: a line to a person. A photo whose count is at or above the line goes to the office, while the budget lasts. Every other photo is quoted from its count.

Why does a line help? A counter makes its largest errors on crowded photos, and a crowded photo usually gets a high count. So the highest counts are the best photos to send to a person. It is the check zone of Set the line with the warehouse, with one line instead of two. You compare the counters by the cost per 1,000 photos, inside the budget of 250 photos to a person.

The photos come from Open Images V7, a large set of Flickr photos with boxes drawn and checked by people. The course’s catalogue photos come from it too. We use two of its classes: Window and Car.

Six photos in two rows, each with orange boxes around the counted objects. Top row, windows: a glass door with 3 window boxes; a wooden house on a hillside with 8; a large red-brick country house with 19. Bottom row, cars: a grey car with its doors open, 3 boxes; a classic silver sports car on grass with cars behind it, 6; a city crossroads seen from a tall building, with 30 small car boxes.
One validation photo of each class and crowding band, with Open Images' true boxes in orange. Photos (CC BY 2.0): charles roderick, pululante, Glen, Crosa, big-ashb and Bobbie Johnson on Flickr, from Open Images V7; boxes by Google (CC BY 4.0); boxes drawn by our code.
Text description of the diagram

Top row, windows. First, a glass door between brick walls: Open Images boxes 3 windows, the glass panels of the door and its frame. Second, a long wooden house on a grassy hillside: 8 windows. Third, a large red-brick country house with a formal garden: 19 windows over two floors and the roof. Bottom row, cars. First, a small grey car seen from the side with all its doors open: 3 boxes, the car and two cars far behind it. Second, a classic silver sports car on grass in front of a fence: 6 boxes, the sports car and five cars parked behind the fence. Third, a busy city crossroads seen from high above: 30 small boxes on the cars in the streets.

Each photo has one class, the thing that Marta counts in it. The true count is the number of boxes that Open Images’ annotators drew. The crowding band comes from the true count: 3 to 5, 6 to 9, or 10 or more objects.

Split Windows Cars What it is for
train 39 28 what the fine-tuned detector and the density estimate learn from
val (validation) 24 19 choose the score lines, the line to a person and the counter
test 36 31 measure once, at the end
new 12 9 the end-to-end run on new photos

Each split has the three bands in about equal shares. Crowded car photos are the exception: the sample has only 15 car photos with 10 cars or more (7 in the test split). We explain why below.

Before any model, Diego checks for photos that are there twice, counts that do not match their boxes, and the size of the boxes.

print("photos:", len(photos), "| different source photos:", photos["source_sha256"].nunique())
print("boxes:", len(boxes), "| counts match the boxes:",
(boxes.groupby("photo_id").size().reindex(photos["photo_id"]).values == photos["count"].values).all())
boxes["side"] = np.sqrt(boxes["w"] * boxes["h"]) # the side of a square with the same area, in pixels
print(boxes.groupby("class")["side"].describe(percentiles=[0.1, 0.5]).round(1)[["count", "10%", "50%", "max"]])
print(photos.groupby("class")["count"].describe()[["min", "50%", "max"]])
photos: 198 | different source photos: 198
boxes: 2153 | counts match the boxes: True
count 10% 50% max
class
car 606.0 11.8 43.2 550.8
window 1547.0 8.5 22.6 521.4
min 50% max
class
car 3.0 6.0 30.0
window 3.0 7.0 188.0

What this tells us. No photo is there twice, and every count matches its boxes. The photos are 640 pixels on the long side. A typical window box is only about 23 pixels across, and a typical car box about 43. One facade has 188 windows; the most crowded car photo has 30 cars.

A crowd has no count. When more than five objects touch and hide each other, Open Images’ annotators draw one group box around all of them. A group box says “many cars here”, not how many. Diego kept only photos with no group box of their class. That is why crowded car photos are rare. In the validation and test splits, 1,975 photos with no person have a group box of cars. Only 52 have 10 or more separate car boxes.

What is one window? Look at two test photos with their true boxes, next to the hosted model’s count:

Two building photos with orange boxes. Left: a brick and grey school building where each long band of glass panes has one box; 15 boxes, while the hosted model says 77. Right: a large modern building with a glass entrance hall; many small windows have boxes, but the glass hall has none; 55 boxes, while the hosted model says 224.
Two test photos with Open Images' true boxes (orange) and the hosted model's recorded count. Left: one box for a long band of panes. Right: the glass entrance hall has no box. Photos (CC BY 2.0): Denna Jones; U.S. Army Corps of Engineers Savannah District, on Flickr, from Open Images V7; boxes by Google (CC BY 4.0).
Text description of the diagram

Left: a three-storey building of brick and grey panels. Each floor has long bands of glass panes, and each band has one orange box. So a band of eight or ten panes counts as one window. There are 15 boxes. The hosted model says 77: it counts panes. Right: a large modern building with two wings and a glass entrance hall in the middle. Many small windows in the two wings have boxes. The tall glass wall of the hall has no box at all. There are 55 boxes. The hosted model says 224.

Both answers are “right” for some meaning of “window”. Marta has a third meaning: what her team cleans and charges for. Agree what one item is before you count. Here, the true count is the annotators’ count, and some of their counts are surely off by one or two.

  • Easier: Flickr photos are framed and sharp. A customer’s phone photo can be dark, tilted or partly hidden by a tree.
  • Harder: windows are tiny at 640 pixels, and the meaning of “window” changes from photo to photo.
  • Milder crowds: the most crowded real scenes have group boxes, so they are not in the sample. A full car park is harder than our “10 or more”.
  • Few photos: 36 window and 31 car test photos; 7 crowded car test photos. A window photo off by 10 windows moves the test cost by 1,000 ÷ 36 × 10 × 3 = 833 dollars per 1,000 photos. Small differences are noise.
  • One item is one photo.
  • The answer is a count of the photo’s class.
  • The count error is the mean absolute difference from the true counts (MAE), as in Module 4. The bias is the mean difference: positive means too many. The worst case is the crowding band with the largest error.
  • The cost is per 1,000 photos, with Marta’s prices. Photos at or above the line go to a person while the budget lasts (250 photos in 1,000). When more photos cross the line, the person takes those with the highest counts.
  • The lines and the counter are chosen on validation photos only, separately for windows and cars. They are different jobs with different prices.
  • The test photos are opened once, after the choice.

In the notebook, quote_cost gives the cost per 1,000 photos for a counter and a line, and choose_line keeps the cheapest line on the validation photos.

A flow diagram. At the top: the same customer photos of windows and cars. Five boxes below it, side by side: Mean count (the training mean, no look at the photo), RT-DETR (COCO detector, no training), Fine-tuned (Faster R-CNN, 10 epochs on a CPU), Density (ResNet-18 cells and ridge), Hosted model (asked how many, recorded). Below, three steps in a column: validation, choose the lines and a counter per class; test, once; new photos, quote or a person counts.
The case study's path: five counters count the same photos; the lines and a counter for each class are chosen on validation, the test is opened once, and the choice then meets new photos.
Text description of the diagram

At the top, a box: the same customer photos of windows and cars. Five lines lead down to five counters. Mean count gives every photo the training mean. RT-DETR is a COCO detector with no new training. Fine-tuned is Faster R-CNN, trained for 10 epochs on a CPU on 67 photos with boxes. Density uses ResNet-18 cells and a ridge regression, from the same 67 photos. The hosted model is asked “how many?”, and its answers are recorded. Below them, three boxes in a column. First, validation with 43 photos: choose the lines and a counter for each class, with at most 250 photos in 1,000 to a person. Then test with 67 photos, once. Last, 21 new photos: quote from the count, or a person counts.

The first counter never looks at the photo. It gives every photo the mean count of the training photos of its class: 12.0 windows or 6.6 cars. Any real counter must beat it.

RT-DETR R18 (Apache-2.0) learned from COCO, as in Pretrained detectors. COCO has a class “car”. It has no class “window”. Diego keeps RT-DETR’s boxes of the photo’s class and chooses the score line on validation: 0.35 for cars.

Predict. On Larkfield’s pallets, the COCO detectors had no “box” class and found stop signs and trains. What will RT-DETR count on a window photo?

It counts 0 windows on every photo, so its window error is the mean true count. On cars, it is the best counter of all.

Detector 2: Faster R-CNN, fine-tuned on 67 photos

Section titled “Detector 2: Faster R-CNN, fine-tuned on 67 photos”

This is the recipe of Fine-tune a detector on a few images. Faster R-CNN MobileNetV3, trained on COCO, gets a new last layer for two classes, window and car, with the backbone frozen. The detector learns from the 67 training photos and their 652 boxes, for 10 epochs, with flips and gradient clipping. On a laptop CPU, this took about 2.5 to 3 minutes (149 to 173 seconds in our runs).

frcnn = fasterrcnn_mobilenet_v3_large_fpn(weights=FasterRCNN_MobileNet_V3_Large_FPN_Weights.COCO_V1,
min_size=480, max_size=640, box_detections_per_img=300,
box_score_thresh=0.05)
frcnn.roi_heads.box_predictor = FastRCNNPredictor(frcnn.roi_heads.box_predictor.cls_score.in_features, 3)
for p in frcnn.backbone.body.parameters():
p.requires_grad = False # the backbone stays as it is (frozen), as in Module 4

Predict. RT-DETR already knows cars from COCO. Will a detector fine-tuned on only 28 car photos count cars better or worse?

This is the density estimate of Counting, and tracking in brief. ResNet-18 gives 256 numbers for each 16 × 16 pixel cell. A ridge regression learns how many box centres each cell holds. One ridge regression per class learns from the same 67 photos. The count is the sum over the cells; no box is found.

The hosted vision-language model, recorded

Section titled “The hosted vision-language model, recorded”

Leo’s idea: ask a large model. The course sent each validation, test and new photo once to the hosted model of Module 7. Each class has one question:

How many windows of buildings can you see in this photo? Count every window you can see, also small,
far or partly hidden ones. Answer with JSON only: {"count": <number>}.

The car question is the same with “cars”. Every answer was valid JSON. The notebook replays the 112 recorded answers. The recording budget (US$0.50 for this case study) ran out before the last 19 validation window photos. So the model has only 5 validation window answers (see the decision).

Now every counter has a count for every test photo. Diego compares them:

METHODS = {"baseline: mean count": baseline, "RT-DETR (COCO, no training)": rt_counts,
"Faster R-CNN fine-tuned": ft_counts, "density estimate": density, "hosted VLM (recorded)": vlm}
rows = [r for name, counts in METHODS.items() for r in count_rows(name, counts)]
method class count error bias exact 3-5 6-9 10+
baseline: mean count window 14.08 -5.58 0.000 8.42 4.25 29.58
baseline: mean count car 3.89 -1.24 0.065 3.24 0.98 10.00
RT-DETR (COCO, no training) window 17.58 -17.58 0.000 3.58 7.75 41.42
RT-DETR (COCO, no training) car 1.94 -0.84 0.258 0.67 1.75 4.43
Faster R-CNN fine-tuned window 10.92 3.97 0.000 4.58 7.58 20.58
Faster R-CNN fine-tuned car 3.39 -1.45 0.258 1.17 1.83 9.86
density estimate window 12.57 -1.52 0.000 7.98 8.08 21.65
density estimate car 4.18 -2.73 0.097 2.67 2.23 10.14
hosted VLM (recorded) window 21.22 20.89 0.111 4.08 17.50 42.08
hosted VLM (recorded) car 3.39 2.87 0.258 0.75 2.17 10.00

“exact” is the share of test photos counted exactly right. The last three columns are the count error in each crowding band.

  • Mean count
  • RT-DETR
  • Fine-tuned
  • Density
  • Hosted model
Count error by crowding on the 36 window test photos. Every counter is worst on the crowded band; the fine-tuned detector and the density estimate are the least bad there.
Text description of the chart

A chart with three points across: 3 to 5, 6 to 9, and 10 or more windows, and the count error from 0 to 45 up. Mean count: 8.42, 4.25, 29.58. RT-DETR: 3.58, 7.75, 41.42 (it counts no window, so its error is the true count). Fine-tuned: 4.58, 7.58, 20.58. Density: 7.98, 8.08, 21.65. Hosted model: 4.08, 17.50, 42.08.

  • Mean count
  • RT-DETR
  • Fine-tuned
  • Density
  • Hosted model
Count error by crowding on the 31 car test photos. RT-DETR, with no training, is the best in every band; all the others are off by about 10 cars on the crowded photos.
Text description of the chart

A chart with three points across: 3 to 5, 6 to 9, and 10 or more cars, and the count error from 0 to 11 up. Mean count: 3.24, 0.98, 10.00. RT-DETR: 0.67, 1.75, 4.43. Fine-tuned: 1.17, 1.83, 9.86. Density: 2.67, 2.23, 10.14. Hosted model: 0.75, 2.17, 10.00.

What this tells us.

  • Crowding is the worst case for every counter. On windows, the crowded band has errors of 20 to 42 windows a photo. On cars, 4 to 10.
  • COCO decides the detectors. For cars, RT-DETR with no training is the best counter: error 1.94, AP50 0.773. The detector fine-tuned on 28 car photos is worse: error 3.39, AP50 0.378. For windows, RT-DETR counts nothing.
  • The fine-tuned detector has the lowest window error (10.92), but bad boxes. Its AP50 on windows is only 0.069. Look at its boxes:
Four photos in two rows. Top: a motorway through wind turbines with many small cars; left, 30 true boxes in orange; right, RT-DETR's 17 blue boxes, which find the near cars and miss most far ones. Bottom: a four-storey stone building with many windows; left, 64 true boxes in orange around single windows; right, the fine-tuned detector's 80 blue boxes, many of them large boxes around groups of windows.
Left: the true boxes (orange). Right: the detector's boxes at its score line chosen on validation (blue). Top: RT-DETR on a crowded car photo (true 30, counted 17). Bottom: the fine-tuned detector on a facade (true 64, counted 80). Photos (CC BY 2.0): Kevin Dooley; w.marsh, on Flickr, from Open Images V7; boxes by Google (CC BY 4.0).
Text description of the diagram

Top row: a motorway that curves through a valley with wind turbines on the hills. The left photo has 30 orange boxes: large cars in front and many small cars further away. The right photo has RT-DETR’s 17 blue boxes: it finds the near cars and a few far ones, and misses most of the small far cars. Bottom row: a four-storey stone building with rows of tall windows. The left photo has 64 orange boxes, one around each window. The right photo has the fine-tuned detector’s 80 blue boxes: many are large boxes around groups of two to six windows, and they overlap. Its count is close, but its boxes are wrong.

A count of 80 for 64 windows looks close, but the boxes cover groups of windows. A count can be right for the wrong reason, as in Counting, and tracking in brief. RT-DETR’s car boxes, on the other hand, are right; it misses small far cars.

  • The hosted model counts something else. Its bias on windows is +20.89: on the photos above, it counted panes. On cars, its error (3.39) is close to the fine-tuned detector’s.
  • The density estimate is weak, as in Module 4. It has no clear win in any band. We kept it as simple as the course’s: one linear model per class.

The notebook adds these rows to the course’s comparison table, with the count task.

A small count error is not yet a decision. Marta needs to know which photos to count herself. For each counter and class, choose_line tries every line on the validation photos and keeps the cheapest. Then Diego chooses, for each class, the counter with the lowest validation cost. One more rule: a counter needs a count for every validation photo. The hosted model has only 5 of the 24 window answers, so it cannot be chosen for windows.

method class line validation cost test cost, no person test cost with the line photos to a person per 1,000
baseline: mean count window 12.0 26250.00 42250.00 31208.30 250.0
baseline: mean count car 7.0 21578.90 46838.70 39596.80 225.8
RT-DETR (COCO, no training) window 0.0 35250.00 52750.00 38208.30 250.0
RT-DETR (COCO, no training) car 9.0 8315.80 23225.80 13274.20 225.8
Faster R-CNN fine-tuned window 19.0 12625.00 32750.00 14458.30 250.0
Faster R-CNN fine-tuned car 7.0 36078.90 40645.20 14822.60 225.8
density estimate window 16.0 17500.00 37583.30 14041.70 250.0
density estimate car 8.0 24105.30 51096.80 35306.50 161.3
hosted VLM (recorded) window 12.0 4303.73 63670.43 12212.03 250.0
hosted VLM (recorded) car 11.0 7056.33 40648.93 9406.93 225.8
chosen on validation: {'window': 'Faster R-CNN fine-tuned', 'car': 'hosted VLM (recorded)'}

Costs are dollars per 1,000 photos. The hosted model’s costs include its bill: 3.73 dollars per 1,000 photos at the course’s price table (assumption). The local counters run on the office laptop for nearly nothing.

What this tells us. On validation, the fine-tuned detector is the cheapest for windows (12,625). The hosted model is the cheapest for cars (7,056), and RT-DETR is second (8,316). Diego chooses them now, before he looks at the test columns.

Test cost per 1,000 window photos with each counter's line to a person, chosen on validation, and Marta's prices (assumptions). Rand.: the mean count, and a person counts a random quarter of the photos. Mean: the mean count with its line. RT: RT-DETR. FT: fine-tuned. Dens.: density. Host: the hosted model.
Text description of the chart

A column chart with six columns. Random checks: 31,812 dollars per 1,000 photos. Mean count with its line: 31,208. RT-DETR: 38,208. Fine-tuned: 14,458. Density: 14,042. Hosted model: 12,212.

Test cost per 1,000 car photos with each counter's line to a person, chosen on validation, and Marta's prices (assumptions). Rand.: the mean count, and a person counts a random quarter of the photos. Mean: the mean count with its line. RT: RT-DETR. FT: fine-tuned. Dens.: density. Host: the hosted model.
Text description of the chart

A column chart with six columns. Random checks: 35,254 dollars per 1,000 photos. Mean count with its line: 39,597. RT-DETR: 13,274. Fine-tuned: 14,823. Density: 35,307. Hosted model: 9,407.

What this tells us.

  • Windows: the fine-tuned detector costs 14,458 dollars per 1,000 photos, against 31,812 with the mean count and random checks: 55% less. The density estimate (14,042) and the hosted model (12,212) are in the same range. A difference of 400 dollars is half a photo: noise.
  • Cars: the hosted model costs 9,407, against 35,254 with random checks: 73% less. RT-DETR, local and free, costs 13,274.
  • The line does most of the work. Without any person, the fine-tuned detector costs 32,750 on windows. The line sends the 9 photos with the highest counts to the office and cuts the cost by more than half.
  • The cost is still high. About 14 dollars a window photo is a large error on a quote. Marta should not quote from these counts without a margin.

Common mistake

Choosing on 5 photos

Symptom: Leo reads the validation column and says: “The hosted model is the cheapest for windows by far: 4,304 against 12,625. Use it.” On test it is indeed the cheapest (12,212).

Cause: its validation cost comes from 5 photos, not 24, because the recording budget ran out. Five photos can easily be five easy photos. On the test photos, its count error on windows is 21.22, the worst of all. Its line works only because it overcounts the crowded facades even more.

Fix: compare counters on the same validation photos. Record the missing answers (about 0.07 dollars more) before the choice, or choose among the counters with full answers, as Diego did.

Consequence in AI work: a number from fewer examples is not the same number. Check how many photos stand behind each row before you compare rows.

Diego writes to Marta in her words:

  • Windows: use the fine-tuned detector. Count the photo yourself when the detector says 19 windows or more: about a quarter of the photos.
  • Cars: use the hosted model with a line at 11 cars. A free local option is almost as good: RT-DETR with a line at 9 cars (13,274 against 9,407 dollars per 1,000 photos on test).
  • The expected cost is about 14,458 dollars per 1,000 window photos and 9,407 per 1,000 car photos, at Marta’s prices, against about 31,800 and 35,300 with the mean count and random checks.
  • Still counted by a person: every photo above the line, and a small random sample of the others each month, to measure the real error on Marta’s own photos.
  • Before you quote from a count, agree what one window is. Then label 50 of your own customer photos with that rule, and set the lines again.

And what not to use it for:

  • Not for a final price. A window quote is off by about 5 to 8 windows on photos under the line. Quote a range, or confirm on site.
  • Not for full car parks. The sample had no car park of more than 30 cars. A full car park goes to a person.
  • Not for photos with people in them sent to the hosted model. Blur faces first, as in People in images, and check the provider’s contract.

The decision now meets 21 new photos that no step has seen: 12 window photos and 9 car photos. Clearline quotes each one from the chosen counter’s count, or sends it to a person:

use = {"window": ft_counts, "car": vlm}
for cls in CLASSES:
line = choose_line(use[cls], cls)
new = split_photos(cls, "new")
c = np.array([use[cls][i] for i in new["photo_id"]])
print(f"{cls}: line {line}, count of each photo:", np.round(c).astype(int).tolist())
print(f"{' ' * len(cls)} true counts: ", new["count"].tolist())
print(f"{' ' * len(cls)} cost per 1,000 photos and photos to a person per 1,000:", quote_cost(use[cls], cls, "new", line))
window: line 19.0, count of each photo: [0, 43, 30, 50, 3, 5, 0, 2, 12, 10, 14, 3]
true counts: [14, 37, 13, 19, 4, 4, 3, 5, 6, 6, 6, 7]
cost per 1,000 photos and photos to a person per 1,000: (11125.0, 250.0)
car: line 11.0, count of each photo: [11, 4, 6, 7, 3, 5, 13, 9, 8]
true counts: [10, 4, 4, 4, 3, 6, 7, 7, 8]
cost per 1,000 photos and photos to a person per 1,000: (10777.8, 222.2)

What this tells us.

  • Windows: three photos went to the office (counts 43, 30 and 50). One of them has only 13 windows: the person caught the overcount. One photo with 14 windows got a count of 0 and was quoted for 0 windows. Marta got that photo because it shows windows, so a count of 0 is a sign of failure. A simple rule catches it: send every count under 3 to a person too.
  • Cars: two photos went to the office. The other seven were quoted with errors of 0 to 3 cars.
  • 21 photos are a first look at the decision at work, not a new measurement.

Marta asks: “What if I hire a part-time helper, so the office can count 500 photos in 1,000?” The notebook chooses the lines again on validation for each budget and measures on test: an experiment for planning, not a new choice.

Photos to a person, per 1,000 Windows: fine-tuned Cars: hosted model
0 32,750 40,645
100 24,375 27,113
250 (today) 14,458 9,403
500 4,833 4,887

The cars column leaves out the model’s bill of 3.73 dollars per 1,000 photos. With 500 checks, windows cost a third: 4,833. The extra 250 checks cost 125 dollars per 1,000 photos and save almost 10,000. Here the next dollar is better spent on counting time than on a bigger model.

  • Compared with Larkfield’s pallets. On the drawn pallets, the fine-tuned detector had an AP50 of 0.867 and the hosted model counted all 30 test pallets exactly. On real windows, the same recipe gives an AP50 of 0.069, and the hosted model counts 11% of the window test photos exactly. Drawn scenes are clean; real scenes are not. Measure on your own photos.
  • A class that COCO knows is a different problem from one it does not. For cars, start from a COCO detector. For windows, fine-tuning is the only detector route, and 39 photos are not enough for good boxes.
  • The definition matters as much as the model. Write the counting rule down, with pictures, before you label or measure.
  • Small objects. Half the windows are under 23 pixels across. Larger photos would help the detectors and cost more time.
  • Not tested here: more window photos, larger photos, OWLv2 with the text “a window”, the rule “send counts under 3 to a person”, and the missing validation answers. One try with smaller anchor boxes gave a window error of 9.31 and a worse car error; it is not in the comparison.
  • Speed: on a laptop CPU, RT-DETR took about 120 milliseconds a photo, the fine-tuned detector about 120 and the density estimate about 45. The hosted model took about 9 seconds an answer.
  • The licences. Each photo is CC BY 2.0: if you share your work, credit its author (in manifest.csv) and say what you changed. The boxes are CC BY 4.0 (Google).
  • The model weights. RT-DETR is Apache-2.0 and torchvision’s code is BSD-3-Clause, but both learned from COCO, whose photos have their own licences. ResNet-18 learned from ImageNet. As When not to build it explains, ask your legal team before you ship them.
  • People and number plates. We removed every photo with a person, by labels, two detectors and a look at every photo. Customers’ photos will show people and number plates. Blur faces before a photo leaves the office, and never read or keep number plates.
  • A hosted model sees the customer’s photo. Check where the provider keeps it, and for how long. RT-DETR sends nothing.
  • The costs are assumptions. Show Marta’s prices next to every cost, and change them when the real numbers come in. The lines move with them.

It shows that you can turn a counting model into a decision with a price, and that you check what the labels mean. Make it yours with one change: another Open Images class (for example Flower, Bottle or Chair), another budget, or the rule “counts under 3 go to a person”.

What to put in the repository:

  • README.md (template below).
  • The notebook, run from top to bottom, with its outputs.
  • The comparison table as a CSV: counter, class, count error, bias, error by band, AP50, line, validation cost, test cost.
  • MODEL_CARD.md: the intended use (a first count for a quote, with a person above the line) and what not to use it for (final prices, full car parks, photos with people sent to a hosted model). Add the data, the lines, the cost assumptions and the limits.
  • The data’s source and licence with the attribution: the authors of the photos you show, and Google for the boxes. A link and the checksum are enough; you do not need to share the photos.

A README template:

# Counting windows and cars for cleaning quotes
## Problem
A cleaning company quotes from a customer's photo. A window counted wrong costs
3 dollars, a car 12; a person counts a photo for 0.50 and can count 250 photos
in 1,000 (assumptions).
## Data and licence
Open Images V7: 198 photos (CC BY 2.0, each by its Flickr author), boxes CC BY 4.0
(Google). No group boxes, no people. 67 train, 43 validation, 67 test, 21 new.
## Approach
Five counters on the same photos: training mean, RT-DETR (COCO, no training),
Faster R-CNN fine-tuned on 67 photos, a density estimate, a hosted
vision-language model (recorded). A line to a person, chosen on validation.
## Results (test set, used once)
Windows: fine-tuned detector, 14,458 dollars per 1,000 photos (random checks
31,812). Cars: hosted model 9,407; RT-DETR (local) 13,274.
## Limits
"Window" means different things to annotators and models; window boxes are
small; few crowded car photos; costs are assumptions.

How to describe it in an interview: “I compared two detectors, a density estimate and a vision-language model on counting windows and cars in real photos, and turned the counts into a decision with a line to a person, chosen on validation inside a budget. That cut the expected cost by more than half, and the biggest error was a different definition of ‘one window’.”

Go further:

  • Add the rule “counts under 3 go to a person”. Predict first: how much does it save on the new photos?
  • Fine-tune with 800-pixel photos and measure the window AP and the time per photo.
  • Run OWLv2 with the text “a window of a building”, as in Fine-tune a detector on a few images, and add its row.
  • Write your own counting rule for windows, relabel 20 test photos, and measure every counter again.

Check your understanding

RT-DETR, with no training, had a car count error of 1.94. The same kind of detector, fine-tuned on 28 car photos, had 3.39. Diego's colleague says fine-tuning always helps. What do you answer?

Show the answer

Fine-tuning helps when the pretrained model lacks something: a class (windows, Larkfield’s boxes) or a kind of photo. COCO has many thousands of photos of cars, so a COCO detector already knows cars well (AP50 0.773 here). The fine-tuned detector learned two classes at once from only 28 car photos and 39 window photos. Its car AP50 fell to 0.378. With so few photos, fine-tuning can make a strong model worse. Start from the pretrained model, measure it on your own validation photos, and fine-tune only when it falls short.

Check your understanding

The hosted model counted 77 windows on a school building with 15 window boxes, and 224 on a building with 55. Is the model wrong?

Show the answer

It is not wrong in a simple way: it counts a different thing. The annotators drew one box around each long band of glass panes, and skipped the glass wall of the entrance hall. The model counted panes. The count error measures agreement with the annotators’ meaning of “window”, not the truth. For Marta, the right meaning is what her team cleans and charges for. Write that rule down with example pictures, label her own photos with it, and measure again. The question can also move the answer: “count each window frame, not each pane”.

Check your understanding

Marta can either buy a better counter or pay a helper so the office can count 500 photos in 1,000 instead of 250. Which do you advise first, with the numbers of this case study?

Show the answer

The helper first. With 500 photos to a person, the window test cost falls from 14,458 to 4,833 dollars per 1,000 photos, and the car cost from 9,403 to 4,887. The extra 250 checks cost 125 dollars per 1,000 photos. No counter came close to that gain: the best window counters were all between about 12,000 and 14,500 at today’s budget. A better counter becomes worth it when the number of photos grows, or when a clear counting rule and her own labelled photos make a detector much more accurate.

Recap

  • You counted windows and cars in 198 real open photos (Open Images V7, CC BY 2.0 photos, CC BY 4.0 boxes) with five counters. You measured the count error by crowding. Every counter was worst on the crowded band.
  • For cars, RT-DETR with no training was the best counter (error 1.94, AP50 0.773); fine-tuning on 28 photos made it worse. For windows, which COCO lacks, the fine-tuned detector had the lowest error (10.92) but bad boxes (AP50 0.069).
  • The hosted model counted panes: a window bias of +20.89. Agree what one item is before you count.
  • A line to a person, chosen on validation inside a budget of 250 photos in 1,000, did most of the work. On test: 14,458 dollars per 1,000 window photos and 9,407 per 1,000 car photos, against about 31,800 and 35,300 with random checks.
  • Compare counters on the same validation photos: a row from 5 photos is not a row from 24.
  • More counting time beat a better model: 500 checks per 1,000 photos cut the window cost to 4,833.

Next, the final assignment: a second warehouse with a different camera and a new product category, where you bring the whole course together.

Next: Final assignment