Skip to content
Français

Classify products and pets from photos

  • Case study 2
  • 150 min
  • Optional · 1 case study needed for the certificate

Where we are

In eight modules, we classified Larkfield's catalogue with rules, classifiers and embeddings. In this case study, we use real open photos with many similar classes. We compare four methods against the number of labelled images per class.

What you will be able to do

Classify 37 breeds of cats and dogs from real open photos, and choose a method by what it needs and what it costs. You will compare four families of methods against the number of labelled photos per breed. You will find the breeds they confuse, and set a line for automatic answers from the cost of a mistake. Then you run the whole method on new photos and write it up as a portfolio piece.

Case studyPortfolio piece

Time
About 2.5 hours
Runs in
Colab, Kaggle or VS Code
Skills used
  • Fine-grained classification
  • Colour histogram baseline
  • CLIP prompt sets
  • Linear probe
  • Probe with prompts
  • Fine-tuning
  • Learning curve
  • Worst classes
  • Cost-based line
  • End-to-end run

Colab and Kaggle open your own copy. To use VS Code, download the notebook from GitHub. Stuck? See the solution notebook.

The notebook runs every step of this page, with the same code and the same output, plus three tasks with checks. Its setup cells install PyTorch, timm and open_clip. They download three large files once: the photo archive from the dataset’s own site (about 790 MB), CLIP (605 MB) and ResNet-18 (47 MB). After the downloads, the notebook runs in about 2 minutes on a laptop.

Before you startWhat to know and to have readyAbout 150 minutes

What you need to know

This case study uses Modules 2 and 3 most:

What you need on your computer (only for the local route; in Colab or Kaggle you need nothing)

  • Python 3.12 and a new virtual environment in an empty folder. The notebook’s first cell installs the pinned packages.
  • About 3 GB of free disk: the archive, the unpacked photos and the two models. To start again, delete the folder pets-case-study. To free the space at the end, delete it too, and the models in your Hugging Face cache.

Tested with Python 3.12.13, PyTorch 2.15.0, timm 1.0.31, open_clip 4.2.1 and scikit-learn 1.9.1 on macOS (Apple M4, CPU), in Jupyter from a fresh start. Windows, Linux, Colab and Kaggle were not tested.

Riverside Animal Rescue (a fictional charity) runs three shelters. About 60 cats and dogs arrive on a normal day (an assumption for this case study). Each animal gets an adoption listing on the website, with a photo and a breed. Today a member of staff looks at each photo and types the breed. It is slow, and two people often type different breeds for the same dog.

Hana leads the listings team. She asks a simple question: “Can a model suggest the breed from the photo, and fill it in when it is sure?”

Each answer can go wrong in two ways, and Hana puts a rough price on each (assumptions):

  • A check by a person: someone looks at the photo and chooses the breed. That takes about a minute, so it costs 0.50 dollars, as in the rest of this course.
  • A wrong breed on a listing: an adopter comes for the wrong kind of dog, and staff must correct the listing. The shelter also loses some trust. Hana counts it as 5 dollars.

She adds one rule before any model exists. Some laws name pit bull types. For example, Ontario’s Dog Owners’ Liability Act, section 1 (1) says that “pit bull” includes “a Staffordshire bull terrier” and “an American pit bull terrier”. So a person must always check three breeds that look alike: the American Pit Bull Terrier, the Staffordshire Bull Terrier and the American Bulldog. The American Bulldog is Hana’s own addition, because people mix it up with the other two. You will see that the data agrees with her.

The same question comes up in a shop. Larkfield’s catalogue team sorts photos into 8 broad categories. A finer list, such as 37 kinds of vase or chair, is the same problem as 37 breeds: many classes that look almost the same. No open set of 37 look-alike products exists with a licence that we can use, so pets stand in for products.

You will build a breed suggester. A model gives the three most likely breeds for a photo, and a rule decides when to fill in the first one automatically.

A visitor walks through the shelter. She can tell every cat from every dog at a glance. Then she stops at two grey cats. Are they both British Shorthairs? The vet nurse looks for two seconds: “No. The left one has copper eyes and a thick round face. The right one has green eyes and a thin face. That one is a Russian Blue.”

Both women see the same photo. Only the nurse knows which small details matter.

The problem: the baseline that worked on Larkfield fails here

Section titled “The problem: the baseline that worked on Larkfield fails here”

In Rules and colour histograms, a colour histogram with 5 nearest neighbours reached 0.263 on Larkfield’s 8 categories. That is about twice the 0.125 of pure chance. On the 37 breeds, with 20 labelled photos per breed, it reaches 0.044. Pure chance is 1 ÷ 37 = 0.027.

A black cat and a black dog have almost the same colour histogram. So do the two grey cats above. Colour describes the whole photo, and the answer hides in a few pixels: the eyes, the ears, the shape of the face.

Fine-grained classification sorts photos into many classes of one kind of thing: breeds of dog, models of car, or styles of vase. The classes share their shape and their colours. They differ in small details. A method for it needs features that learned those details, and it needs a few labels to learn where one class ends.

Here are three cat breeds and three details. The values are typical for each breed.

Breed Coat colour Eye colour Face
British Shorthair grey copper round, thick cheeks
Russian Blue grey green thin, wedge-shaped
Bombay black copper round

Coat colour separates the Bombay from the two grey cats. It cannot separate the British Shorthair from the Russian Blue: both are grey. Eye colour does that. But the eyes are perhaps 1% of the photo, so they barely move a colour histogram. A model that learned faces from millions of photos sees them.

Eight photos in four pairs. A pit bull terrier that the model called a Staffordshire Bull Terrier, and a Staffordshire Bull Terrier called right. A Birman called a Ragdoll, and a Ragdoll called right. A British Shorthair called a Russian Blue, and a Russian Blue called right. An American Bulldog called a Boxer, and a Boxer called right. Each mistake has a red frame.
Four pairs of breeds that look alike. Above each photo: the true breed and the chosen method's answer. A red frame marks a mistake. Photos: the Oxford-IIIT Pet Dataset (Parkhi et al.), CC BY-SA 4.0; the copyright of each photo stays with its owner. This figure (cut square and resized) is shared under CC BY-SA 4.0.
Text description of the diagram

Eight photos in four pairs, each with its true breed and the answer of the method we choose later (CLIP probe with prompts). Pair 1: an American Pit Bull Terrier puppy in snow, answered Staffordshire Bull Terrier (a mistake); a Staffordshire Bull Terrier, answered right. Pair 2: a Birman, answered Ragdoll (a mistake); a Ragdoll, answered right. Pair 3: a British Shorthair, answered Russian Blue (a mistake); a Russian Blue, answered right. Pair 4: an American Bulldog lying on a floor, answered Boxer (a mistake); a running Boxer, answered right.

The data is the Oxford-IIIT Pet Dataset: 7,349 photos of 37 breeds, 12 of cats and 25 of dogs, about 200 photos per breed. The authors split it into two official lists: 3,680 photos for training and validation, and 3,669 for testing. Each file name holds the breed: Abyssinian_26.jpg is photo 26 of an Abyssinian cat.

A grid of 37 photos, one for each breed, with its name: 12 cat breeds and 25 dog breeds.
One test photo of each of the 37 breeds. Photos: the Oxford-IIIT Pet Dataset (Parkhi et al.), CC BY-SA 4.0; the copyright of each photo stays with its owner. This figure (cut square and resized) is shared under CC BY-SA 4.0.
Text description of the diagram

A grid of 37 square photos in alphabetical order, from Abyssinian to Yorkshire Terrier. The cats are Abyssinian, Bengal, Birman, Bombay, British Shorthair, Egyptian Mau, Maine Coon, Persian, Ragdoll, Russian Blue, Siamese and Sphynx. The other 25 are dogs, for example Basset Hound, Beagle, Boxer, Chihuahua, Pug, Samoyed and Shiba Inu.

We do not need all 7,349 photos. The course team chose a fixed subset with a committed script and a fixed random seed. For each breed:

  • a pool of 20 labelled photos to train on, and 10 validation photos to choose settings (from the official training list);
  • 25 test photos to compare the methods, and 4 new photos for the end-to-end run (from the official test list).

Many pet photos show their owners. Before choosing, a person detector looked at 2,443 candidate photos and found a person in 260 of them (11%). Those photos are not used. A detector can miss a hand or a leg, so every photo on this page and in the notebook was also looked at by eye.

The notebook’s first data cell counts the parts:

print(photos.groupby("part", sort=False).size().to_dict())
{'pool': 740, 'val': 370, 'test': 925, 'new': 148}
37 breeds: 12 cats and 25 dogs

What this tells us. Every breed has the same number of photos in each part. So the accuracy treats all breeds the same, and the worst breed has 25 test photos. One test photo is about 0.1 points of accuracy. With 925 photos, a difference under about 2 points can be chance: treat it as a tie.

Three rules keep the comparison honest. Every setting is chosen on the validation photos. The test photos only compare the methods. The new photos are used once, at the end.

Five boxes. Colour histogram plus 5-NN: no model, 128 colour bins, the 5 nearest labelled photos, 20 labels a breed. CLIP zero-shot: a frozen model and one text for each breed, no labels. Linear probe: a frozen model and a logistic regression on its embeddings, 1 to 20 labels a breed. Probe plus prompts: the probe's and the prompts' probabilities mixed, 1 to 20 labels a breed. Fine-tuning: the last block of ResNet-18 and a new last layer learn, 5 to 20 labels a breed.
The methods of this case study, from the simplest to the one that changes the network. Grey: no labels.
Text description of the diagram

Five boxes in a row (one under the other on a phone). The colour histogram with 5 nearest neighbours uses no model: it compares 128 colour bins with 20 labelled photos per breed, and nothing learns. CLIP zero-shot uses a frozen model and one text per breed, with no labels; the texts name the breeds. The linear probe trains a logistic regression on the frozen embeddings, with 1 to 20 labels per breed. The probe with prompts mixes the probabilities of the probe and of the prompts. Fine-tuning trains the last block of ResNet-18 and a new last layer, with 5 to 20 labels per breed.

The histogram is the course’s own (8 × 4 × 4 bins of hue, saturation and value). Each test photo takes the most common breed among its 5 nearest labelled photos:

colour histogram + 5-NN, 20 photos per breed: accuracy 0.044 | chance: 0.027

It finds 41 of the 925 test photos. It even confuses cats with dogs: 381 of its 884 mistakes name the wrong species. The baseline tells us one useful thing: colour is not the answer here.

CLIP compares the photo with one text per breed, as in CLIP zero-shot. It needs no labelled photo. The text around the breed’s name is the prompt set, and the course team tried four on the validation photos:

Prompt set Validation Test
{name} (the name only) 0.822 0.851
a photo of a {name}. 0.838 0.861
a photo of a {name}, a type of pet. 0.870 0.893
a photo of a {name}, a type of {species}. 0.884 0.896

The last one says “cat” or “dog”. It wins on validation, so it is the chosen prompt set. Its test accuracy is 0.896 with no label at all. The species helps most with cats, whose names say nothing: “Ragdoll”, “Birman” and “Bombay” could be anything. With the species, cat accuracy rises from 0.703 (name only) to 0.830.

A linear probe is a logistic regression on CLIP’s embeddings, as in Linear probes and few-shot. That lesson used the setting C = 2 for Larkfield’s 8 classes. With 37 classes, the course team chose C again on the validation photos: C = 10 gave 0.897, against 0.889 for C = 2. A higher C means a weaker pull towards small weights, so the probe can learn sharper lines between 37 classes.

With all 20 labelled photos per breed, the probe reaches 0.872 on the test. That is lower than zero-shot (0.896). With 5 photos per breed it reaches 0.761, and with one photo, 0.504.

Why do 740 labels lose against none? The prompts carry something that 20 photos cannot teach: the names. CLIP learned from many millions of captioned photos on the internet, and pet owners love to write the breed in a caption. The probe throws that knowledge away and learns each breed from 20 photos.

We can keep both. The probe gives 37 probabilities, and so do the prompts. A simple mix takes a weighted average of their logarithms. Then it scales the result back into probabilities that add up to 1. The weight of the prompts is called alpha: 0 means the probe alone, 1 means the prompts alone.

A tiny example with two breeds, A and B:

A B
The probe 0.70 0.30
The prompts 0.20 0.80
Mix with alpha = 0.25 0.57 0.43
Mix with alpha = 0.5 0.43 0.57

With alpha = 0.25: for A, 0.75 × ln 0.70 + 0.25 × ln 0.20 = −0.670; for B, 0.75 × ln 0.30 + 0.25 × ln 0.80 = −0.959. Then e−0.670 = 0.512 and e−0.959 = 0.383. Divided by their sum (0.895), they give 0.57 and 0.43. The answer changes with alpha, so alpha is chosen on the validation photos: 0.25 for 5 to 20 photos per breed, 0.5 for 1 and 2.

The mix wins at every size. With 20 photos per breed it reaches 0.912, and with one photo per breed it already reaches 0.900. The labels now only correct the prompts where they are wrong.

Fine-tuning changes the network itself. As in Transfer learning, ResNet-18 keeps its first layers frozen, and its last block and a new last layer learn from the labelled photos. The course’s recipe stays the same: AdamW, a learning rate of 0.001, batches of 32, no augmentation. The number of epochs was chosen on validation: 5 of 15.

First, ResNet-18 frozen, with a linear probe on its embeddings: 0.875 with 20 photos per breed. Then fine-tuned: 0.846. Fine-tuning lost. With 20 photos per breed, the last block learns the training photos too well, and it forgets some of what ImageNet taught it. With 10 photos per breed it reaches 0.787, and with 5, 0.714 (the mean of 3 runs each). Training 15 epochs, with a check on the validation photos after each one, took about 3 minutes on the laptop’s CPU.

All the methods ran on the same 925 test photos, with the course’s helper compare.py. The task column of these rows is pets.

  • CLIP probe + prompts
  • CLIP probe
  • ResNet-18 probe
  • ResNet-18 fine-tuned
  • CLIP zero-shot (no labels)
Accuracy against labelled photos per breed, on the 925 test photos. Each point below 20 is the mean of 10 random draws (3 for fine-tuning). The dashed line is CLIP zero-shot with the chosen prompt set. The colour histogram (0.035 to 0.044) is far below the axis, which starts at 0.4.
Text description of the chart

A line chart. The x axis has five steps: 1, 2, 5, 10 and 20 labelled photos per breed. The y axis is the test accuracy, from 0.4 to 1. CLIP probe with prompts: 0.900, 0.903, 0.901, 0.907 and 0.912. CLIP probe: 0.504, 0.630, 0.761, 0.821 and 0.872. ResNet-18 probe: 0.649, 0.739, 0.827, 0.855 and 0.875. ResNet-18 fine-tuned: 0.714 with 5, 0.787 with 10 and 0.846 with 20. A dashed horizontal line at 0.896 is CLIP zero-shot with no labels.

The table holds the rows with 20 labelled photos per breed. The worst breed is the one with the lowest recall: the share of its 25 test photos that the method gets right.

Method Accuracy Worst breed (recall) Labelled photos Model Time per photo
Colour histogram + 5-NN 0.044 British Shorthair (0.00) 740 none 2.2 ms
CLIP zero-shot, species prompt 0.896 American Pit Bull Terrier (0.56) 0 605 MB 9.3 ms
CLIP + linear probe 0.872 American Pit Bull Terrier (0.36) 740 605 MB 9.3 ms
CLIP + linear probe + prompts 0.912 American Pit Bull Terrier (0.40) 740 605 MB 9.3 ms
ResNet-18 frozen + linear probe 0.875 Russian Blue (0.56) 740 47 MB 14.9 ms
ResNet-18 fine-tuned, 5 epochs 0.846 American Pit Bull Terrier (0.36) 740 47 MB 14.9 ms

The time is the time to read a photo and compute its embedding, in batches of 32 with 4 threads on an Apple M4 CPU. The probe and the mix add well under a millisecond. CLIP’s weights are MIT. ResNet-18’s weights are Apache-2.0, but they were trained on ImageNet, whose own terms are for non-commercial research (see When not to build it).

How to read it:

  • The prompts are worth hundreds of labels. Zero-shot with no label beats every method with 740 labels, except the mix.
  • The mix is the best at every size, and its curve is almost flat: one photo per breed gives 0.900, twenty give 0.912. A gain of about 1 point, near the noise level of this test.
  • Fine-tuning lost to a frozen probe on the same network. On a CPU, with 20 photos per breed, it is not worth its minutes here.

ResNet-18 was trained on ImageNet’s 1,000 classes. 22 of our 25 dog breeds are ImageNet classes, but only 3 of the 12 cat breeds (Persian, Siamese and “Egyptian cat”).

  • ResNet-18 probe
  • CLIP zero-shot
  • CLIP probe + prompts
Test accuracy on the 300 cat photos and the 625 dog photos: ResNet-18 with a probe, CLIP zero-shot with the species prompt, and the CLIP probe with prompts (20 labelled photos per breed). The axis starts at 0.6.
Text description of the chart

A column chart with three pairs of columns, on an axis from 0.6 to 1. ResNet-18 with a probe: cats 0.78, dogs 0.92. CLIP zero-shot with the species prompt: cats 0.83, dogs 0.93. CLIP probe with prompts: cats 0.90, dogs 0.92.

ResNet-18 gets 0.918 of the dogs right, but only 0.783 of the cats: the cats it never learned as classes. The mix closes the gap (0.897 against 0.920). This is the honest reason why the pretrained models do well here: they have seen these breeds before.

The chosen method is the CLIP probe with prompts. It makes 81 mistakes on 925 test photos, and none of them names the wrong species. Its five worst breeds:

The five worst breeds (recall on 25 test photos each):
american_pit_bull_terrier 0.40
staffordshire_bull_terrier 0.72
birman 0.72
maine_coon 0.80
egyptian_mau 0.80

The mistakes come in pairs. The American Pit Bull Terrier and the Staffordshire Bull Terrier are confused 14 times, more than one in six of all mistakes. Then Birman and Ragdoll, 9 times: two long-haired cats with dark points and blue eyes. Then English Cocker Spaniel and English Setter (5), and American Bulldog and Boxer (4).

The pit bull pair is hard for people too. Breed clubs describe them differently, but many dogs fit both. ImageNet even puts “American Staffordshire terrier” and “American pit bull terrier” into one class. The labels in this dataset come from the photo owners, so some of them may be wrong. When the labels themselves disagree, no model can be right every time.

Common mistake

The person detector missed a hand

Symptom: the course team looked at the first test photo of each breed for the overview figure. One photo of a Scottish Terrier puppy shows a person’s hand under the puppy. The person detector had passed it.

Cause: a person detector is trained to find whole people, or large parts of them. A hand at the edge of a photo, a leg or a reflection is easy to miss. The detector removed 260 photos, but it gives no promise about the rest.

Fix: never show or publish a photo only because a detector passed it. Look at every photo that you show, by eye, on a contact sheet. The course team replaced that photo in the figure. Staff should be able to remove any photo with a person.

Consequence in AI work: an automatic privacy filter is a first pass, not a guarantee. A report that says “no people” must say how it was checked, and what the check can miss.

Hana does not need a breed for every photo. She needs a rule: when may the model fill in the breed by itself? The rule has one line: if the top probability reaches the line, the breed is filled in automatically. Below the line, a person checks.

A flow for one photo. The probe with prompts gives 37 probabilities. If the top breed is pit bull, Staffordshire or American bulldog, a person always chooses from the top 3. Otherwise, if the top probability is at least 0.47, the breed is filled in automatically; if not, a person chooses from the top 3.
The decision for one photo, with the line chosen on the validation photos.
Text description of the diagram

A flow from left to right (top to bottom on a phone). A new photo goes through the probe with prompts, which gives 37 probabilities. First question: is the top breed American Pit Bull Terrier, Staffordshire Bull Terrier or American Bulldog? If yes, a person always checks and chooses from the top 3. If no, second question: is the top probability at least 0.47? If yes, the breed is filled in on the listing. If no, a person checks and chooses from the top 3.

A low line fills in more breeds, with more wrong ones at 5 dollars each. A high line sends more photos to a person, at 0.50 dollars each. The course team tried every line from 0 to 1 on the validation photos:

  • Validation (370 photos): the line is chosen here
  • Test (925 photos)
The cost per 1,000 photos for every line, with checks at 0.50 dollars and wrong breeds at 5 dollars (assumptions). The three breeds of Hana's rule always cost a check. A line of 1 sends every photo to a person: 500 dollars.
Text description of the chart

A line chart. The x axis is the line, from 0 to 1. The y axis is the cost in dollars per 1,000 photos, from 0 to 550. Both curves start high at a line of 0 (validation 274, test 349: many wrong breeds filled in), fall to a minimum near 0.5, and rise to 500 at a line of 1 (every photo checked). The validation minimum is at 0.47, at 175.7 dollars. The test curve at 0.47 is 204.9 dollars; its own minimum is at 0.55, at 154.6 dollars.

The cheapest line on validation is 0.47, at 175.7 dollars per 1,000 photos. On the test photos, the same line gives:

Test (925 photos) Per 1,000 photos
Filled in automatically 756 (82%) —
Wrong among those 21 (97.2% right) —
Sent to a person 169 —
Cost — 204.9 dollars
No line (fill in all but the three breeds) — 348.6 dollars
A person checks every photo (today) — 500 dollars

The test cost is higher than the validation cost: 204.9 against 175.7. Validation has only 10 photos per breed, so the line is a little lucky there. On the test photos, a line of 0.55 would have cost 154.6 dollars, but we may not choose on the test. This is the same lesson as the line that did not hold: a line chosen on few photos must be checked on new ones.

Riverside has about 60 animals a day. So 204.9 dollars per 1,000 photos is about 12 dollars a day, against 30 dollars if a person checks every photo. About 11 photos a day go to a person, and about 1.4 wrong breeds a day reach a listing. The money is small. The real gain is time: the team checks 11 photos instead of 60.

What the analysis tells Hana, in plain words:

  1. Start today with zero-shot. CLIP with the species prompt needs no labels and gets 0.896 of the test photos right. Use it to suggest three breeds, and let staff choose. Nothing is filled in automatically yet.
  2. Label 20 photos per breed from your own shelters while you use it: 740 photos, about 12 hours of one person’s time at a minute each. Then switch to the probe with prompts (0.912 on our test, 0.900 already with one photo per breed).
  3. Fill in the breed automatically when the top probability is at least 0.47. Below that, staff choose from the top three. The true breed was among them for 146 of the 148 new photos.
  4. Always send the pit bull, Staffordshire and American Bulldog to a person, whatever the probability. The model is worst exactly there.
  5. Add “mixed or not sure” for staff. Most shelter dogs are mixed breeds, and the model can only answer one of 37.
  6. Measure again on your own photos before you trust the line, with at least 10 photos per breed.

The function suggest is the whole method for one photo. It reads the photo, computes its CLIP embedding, mixes the probe and the prompts, and decides:

def suggest(file):
"""One new photo -> the three most likely breeds and the zone: the whole method, end to end."""
with torch.no_grad():
e = unit(clip.encode_image(preprocess(Image.open(file).convert("RGB"))[None]).numpy())
log_probe = np.log(probe20.predict_proba(e) + 1e-9)
log_prompts = np.log(softmax(100 * e @ T.T) + 1e-9)
p = softmax((1 - ALPHA[20]) * log_probe + ALPHA[20] * log_prompts)[0]
top3 = [(BREEDS[i], round(float(p[i]), 2)) for i in np.argsort(-p)[:3]]
zone = "always check" if top3[0][0] in ALWAYS_CHECK else "check" if top3[0][1] < LINE else "auto-fill"
return top3, zone

The 148 new photos were never used to train or to choose anything. Six of them:

beagle_44 auto-fill beagle 0.83, basset_hound 0.14, saint_bernard 0.01
Bengal_43 auto-fill bengal 0.78, egyptian_mau 0.16, abyssinian 0.03
Abyssinian_92 auto-fill abyssinian 0.93, bombay 0.02, siamese 0.02
British_Shorthair_84 auto-fill russian_blue 0.55, british_shorthair 0.30, bombay 0.06
american_pit_bull_terrier_197 always check staffordshire_bull_terrier 0.71, american_pit_bull_terrier 0.24, american_bulldog 0.02
chihuahua_60 check chihuahua 0.26, miniature_pinscher 0.14, scottish_terrier 0.09
Six new photos with the zone, the model's top breed and probability, and the true breed: a beagle, a Bengal and an Abyssinian filled in right; a British Shorthair filled in as Russian Blue, with a red frame; a pit bull answered Staffordshire, always checked; a chihuahua at 0.26, checked.
Six of the 148 new photos, with the zone, the model's top breed and its probability, and the true breed. A red frame marks a wrong breed filled in automatically. Photos: the Oxford-IIIT Pet Dataset (Parkhi et al.), CC BY-SA 4.0; the copyright of each photo stays with its owner. This figure (cut square and resized) is shared under CC BY-SA 4.0.
Text description of the diagram

Six photos in two rows. A beagle, filled in as beagle at 0.83. A Bengal cat, filled in as Bengal at 0.78. An Abyssinian, filled in at 0.93. A grey British Shorthair, filled in as Russian Blue at 0.55: a mistake, with a red frame. A black and white pit bull terrier, answered Staffordshire Bull Terrier at 0.71 and always checked. A small chihuahua next to a “security dog” sign, answered chihuahua at only 0.26 and checked.

What this tells us. Three photos are filled in right. The British Shorthair is filled in as a Russian Blue at 0.55: above the line, and wrong. The line lets such mistakes through, at 5 dollars each. The pit bull is answered “Staffordshire” at 0.71. Without Hana’s rule, that wrong breed would reach the listing. The chihuahua is right, but the model is unsure (0.26), so a person checks.

On all 148 new photos:

{'auto-fill': 120, 'check': 16, 'always check': 12}
auto-filled and wrong: 4 of 120
the true breed is among the three suggestions: 146 of 148

What this tells us. 81% of the new photos are filled in, and 96.7% of those are right, close to the test (97.2%).

Pets are easier than a shop’s own categories in one way. The breeds are famous. ImageNet has 25 of the 37 as classes, and CLIP read their names in captions. Larkfield’s catalogue has its own line between a plant pot and a vase. There, the CLIP probe with 20 photos per class (0.929) did a little better than zero-shot (0.925), as Linear probes and few-shot showed. For a product list that the internet has never named, expect the labels to matter more.

Pets are harder in another way. There are 37 classes, not 8, and the worst pair is very close.

The photos are kinder than real life. They are typical, pure-bred animals in good photos from the internet. Shelter photos show mixed breeds, puppies, old animals, cages, bad light and flash. Expect a lower accuracy, and measure it.

Small numbers. The test has 25 photos per breed, so one photo moves a breed’s recall by 0.04.

What we did not try. Fine-tuning with augmentation, more layers or more photos per breed could beat the probe. A larger CLIP model may also do better, at a higher cost per photo.

  • A breed from a photo is a guess about looks. It says nothing certain about ancestry, temperament, health or risk. Never use it to decide whether an animal is safe, whether a family may adopt, or what an insurance costs.
  • Some breeds carry legal weight. In some places, housing rules or laws name certain breeds. For example, the UK’s Dangerous Dogs Act 1991, section 1 (1) (a) applies to “any dog of the type known as the pit bull terrier”. The model is worst exactly on those breeds. Keep Hana’s rule: a person decides them, and the listing says that a person checked.
  • Let the model say “I do not know”. A list of 37 breeds forces a wrong answer for a mixed animal. The listing should say “looks like” when it is a guess.
  • People in the photos. Shelter photos often show staff, volunteers or adopters. Run the model on your own computer, blur faces by default, as in People in images, and never send a photo with a person to a hosted model.
  • The licence travels. The photos are CC BY-SA 4.0. Credit the dataset in your work. Anything you share that is made from the photos, such as a figure or a table of photo names, must use the same licence.

This case study shows that you can choose a vision method from evidence: a fair test, a learning curve, the worst classes and a cost. Make it yours with one change: other costs, another “always check” list, or DINOv2 instead of CLIP. Then make it public, and make it easy to read in five minutes.

What to put in the repository:

  • README.md, the page that people read first (template below).
  • The notebook, run from top to bottom, with its outputs.
  • results.csv: one row per method, from the comparison table (accuracy, worst breed, labelled photos, model size, time per photo).
  • MODEL_CARD.md: the intended use (suggest a breed; a person checks) and what it is not for (safety, adoption or insurance decisions). Add the data, the evaluation, the worst pairs, the line and its cost, and the limits.
  • The dataset credit and licence from the box above. Do not upload the photos: link the dataset and give the archive’s SHA-256.

A README template:

# Breed suggestions for adoption listings
## Problem
A rescue lists about 60 animals a day. Staff type each breed by eye. A check costs
about 0.50 dollars, a wrong breed on a listing about 5 dollars (assumptions).
## Data and licence
Oxford-IIIT Pet Dataset (Parkhi et al., CVPR 2012), 37 breeds, CC BY-SA 4.0,
https://www.robots.ox.ac.uk/~vgg/data/pets/. Fixed subset per breed: 20 train,
10 validation, 25 test, 4 new. Photos with a person (detector + eye) removed.
## Approach
Colour histogram + 5-NN; CLIP ViT-B/32 zero-shot (4 prompt sets); CLIP and ResNet-18
linear probes; probe + prompts; ResNet-18 fine-tuned. All settings chosen on
validation; 1 to 20 labelled photos per breed.
## Results (925 test photos)
Histogram 0.044; zero-shot 0.896; probe 0.872; probe + prompts 0.912; fine-tuned 0.846.
Worst pair: pit bull vs Staffordshire (14 of 81 mistakes).
Line 0.47: 82% filled in, 97.2% of those right, 204.9 dollars per 1,000 photos
against 500 for checking every photo.
## Limits
Pure-bred internet photos; pretrained models saw these breeds; 25 test photos per
breed; mixed breeds not covered; a breed is a guess about looks.

How to describe it in an interview, in two sentences: “I compared four families of image classifiers on 37 look-alike pet breeds: CLIP’s prompts beat 740 labels, and a probe mixed with the prompts was best even with one labelled photo per breed. I chose the line for automatic answers from the cost of each error, which cut the cost by 59% against checking every photo, and the riskiest breeds always go to a person.”

Go further:

  • Replace CLIP with DINOv2 small (the course’s other embedding). Predict first: without text, can it reach the mix?
  • Add a “mixed breed” class with photos of mixed dogs from an allowed source, and measure how often the model admits it.
  • Try a learning curve with 50 photos per breed. Does the probe alone pass zero-shot?
  • Measure the cost again with a check at 0.25 dollars. Predict first: does the line go up or down?

Check your understanding

The CLIP probe trained on 740 labelled photos (0.872) scores lower than CLIP zero-shot with no label (0.896). Explain how that can happen, and what the mix of the two shows.

Show the answer

The probe learns each breed only from its 20 labelled photos, and it ignores what CLIP knows about words. Zero-shot uses the breed names: CLIP learned from many captioned internet photos, and pet owners often write the breed in a caption. So the names carry knowledge that 20 photos cannot teach. The mix keeps both: the prompts bring the names, and the labels correct the prompts where they are wrong. It reaches 0.900 with one photo per breed and 0.912 with 20, better than either alone. For a product list that the internet has never named, the labels would matter more.

Check your understanding

Hana's rule sends the pit bull, the Staffordshire and the American Bulldog to a person whatever the probability. Give two reasons from the data and one from outside it.

Show the answer

From the data: the American Pit Bull Terrier is the worst breed (recall 0.40). The pit bull–Staffordshire pair holds 14 of the 81 mistakes, so the model is weakest exactly there. Even a high probability can be wrong: a new pit bull photo was answered “Staffordshire” at 0.71, above the line. The labels themselves are uncertain: ImageNet even merges the two names into one class. From outside the data: in some places, housing rules or laws name these breeds, so a wrong breed on a listing can harm an adopter or an animal. A person must decide and own those answers.

Check your understanding

The line 0.47 cost 175.7 dollars per 1,000 photos on validation and 204.9 on test. Why did it rise, and what should Hana do before she trusts it?

Show the answer

The line was chosen on only 370 validation photos, 10 per breed. The cheapest line on a small set fits that set a little by luck, so a new set costs more. The same happened in the course’s inspection lesson, where a line chosen on 12 damaged photos did not hold. Hana should measure the line again on her own shelters’ photos, at least 10 per breed, and keep the “check” route. If the cost on her photos is much higher, she should raise the line or label more photos before she lets the model fill in breeds.

Recap

  • Fine-grained classification sorts many classes of one kind. A colour histogram fails (0.044 on 37 breeds, chance 0.027): the answer hides in small details.
  • The prompts were worth more than 740 labels: CLIP zero-shot with the species in the prompt reached 0.896; the probe alone 0.872.
  • The probe with the prompts won at every size: 0.900 with one photo per breed, 0.912 with 20. Fine-tuning ResNet-18 lost to its own frozen probe (0.846 against 0.875).
  • The worst breeds come in pairs: pit bull and Staffordshire (14 mistakes), Birman and Ragdoll (9). Pretrained models do well here partly because they have seen these breeds before.
  • The decision: fill in automatically at a top probability of 0.47 or more, always check three breeds, and choose from three suggestions otherwise. Test cost 204.9 dollars per 1,000 photos against 500; on new photos, 4 of 120 filled-in breeds were wrong.

Next, the last case study: count objects in crowded open photos, where boxes overlap and a detector can count two objects as one.

Next: Count and detect in crowded scenes