Classify products and pets from photos
- Case study 2
- 150 min
- Optional · 1 case study needed for the certificate
Where we are
In eight modules, we classified Larkfield's catalogue with rules, classifiers and embeddings. In this case study, we use real open photos with many similar classes. We compare four methods against the number of labelled images per class.
Case study 2 of 3 · Vision AI: See, Count and InspectClassify products and pets from photos
1. Seeing is a business problem
2. Baselines and pretrained classifiers
3. Embeddings, CLIP and few labels
4. Detect and count
6. Inspect for defects
7. Match, search and describe
8. Run vision in production
What you will be able to do
Classify 37 breeds of cats and dogs from real open photos, and choose a method by what it needs and what it costs. You will compare four families of methods against the number of labelled photos per breed. You will find the breeds they confuse, and set a line for automatic answers from the cost of a mistake. Then you run the whole method on new photos and write it up as a portfolio piece.
The notebook runs every step of this page, with the same code and the same output, plus three tasks with checks. Its setup cells install PyTorch, timm and open_clip. They download three large files once: the photo archive from the dataset’s own site (about 790 MB), CLIP (605 MB) and ResNet-18 (47 MB). After the downloads, the notebook runs in about 2 minutes on a laptop.
Before you startWhat to know and to have ready
What you need to know
This case study uses Modules 2 and 3 most:
- The colour histogram and nearest neighbours, from Rules and colour histograms.
- A frozen network with a new last layer, and fine-tuning, from Transfer learning.
- Prompts and prompt sets, from CLIP zero-shot.
- The linear probe and the learning curve, from Linear probes and few-shot.
- Two lines and three zones, from Set the line with the warehouse. This case study uses one line.
What you need on your computer (only for the local route; in Colab or Kaggle you need nothing)
- Python 3.12 and a new virtual environment in an empty folder. The notebook’s first cell installs the pinned packages.
- About 3 GB of free disk: the archive, the unpacked photos and the two models. To start again, delete the folder
pets-case-study. To free the space at the end, delete it too, and the models in your Hugging Face cache.
Tested with Python 3.12.13, PyTorch 2.15.0, timm 1.0.31, open_clip 4.2.1 and scikit-learn 1.9.1 on macOS (Apple M4, CPU), in Jupyter from a fresh start. Windows, Linux, Colab and Kaggle were not tested.
The situation
Section titled “The situation”Riverside Animal Rescue (a fictional charity) runs three shelters. About 60 cats and dogs arrive on a normal day (an assumption for this case study). Each animal gets an adoption listing on the website, with a photo and a breed. Today a member of staff looks at each photo and types the breed. It is slow, and two people often type different breeds for the same dog.
Hana leads the listings team. She asks a simple question: “Can a model suggest the breed from the photo, and fill it in when it is sure?”
Each answer can go wrong in two ways, and Hana puts a rough price on each (assumptions):
- A check by a person: someone looks at the photo and chooses the breed. That takes about a minute, so it costs 0.50 dollars, as in the rest of this course.
- A wrong breed on a listing: an adopter comes for the wrong kind of dog, and staff must correct the listing. The shelter also loses some trust. Hana counts it as 5 dollars.
She adds one rule before any model exists. Some laws name pit bull types. For example, Ontario’s Dog Owners’ Liability Act, section 1 (1) says that “pit bull” includes “a Staffordshire bull terrier” and “an American pit bull terrier”. So a person must always check three breeds that look alike: the American Pit Bull Terrier, the Staffordshire Bull Terrier and the American Bulldog. The American Bulldog is Hana’s own addition, because people mix it up with the other two. You will see that the data agrees with her.
The same question comes up in a shop. Larkfield’s catalogue team sorts photos into 8 broad categories. A finer list, such as 37 kinds of vase or chair, is the same problem as 37 breeds: many classes that look almost the same. No open set of 37 look-alike products exists with a licence that we can use, so pets stand in for products.
What you will build
Section titled “What you will build”You will build a breed suggester. A model gives the three most likely breeds for a photo, and a rule decides when to fill in the first one automatically.
Fine-grained classification
Section titled “Fine-grained classification”A visitor and a vet nurse
Section titled “A visitor and a vet nurse”A visitor walks through the shelter. She can tell every cat from every dog at a glance. Then she stops at two grey cats. Are they both British Shorthairs? The vet nurse looks for two seconds: “No. The left one has copper eyes and a thick round face. The right one has green eyes and a thin face. That one is a Russian Blue.”
Both women see the same photo. Only the nurse knows which small details matter.
The problem: the baseline that worked on Larkfield fails here
Section titled “The problem: the baseline that worked on Larkfield fails here”In Rules and colour histograms, a colour histogram with 5 nearest neighbours reached 0.263 on Larkfield’s 8 categories. That is about twice the 0.125 of pure chance. On the 37 breeds, with 20 labelled photos per breed, it reaches 0.044. Pure chance is 1 ÷ 37 = 0.027.
A black cat and a black dog have almost the same colour histogram. So do the two grey cats above. Colour describes the whole photo, and the answer hides in a few pixels: the eyes, the ears, the shape of the face.
The idea: classes inside one kind
Section titled “The idea: classes inside one kind”Fine-grained classification sorts photos into many classes of one kind of thing: breeds of dog, models of car, or styles of vase. The classes share their shape and their colours. They differ in small details. A method for it needs features that learned those details, and it needs a few labels to learn where one class ends.
A tiny worked example
Section titled “A tiny worked example”Here are three cat breeds and three details. The values are typical for each breed.
| Breed | Coat colour | Eye colour | Face |
|---|---|---|---|
| British Shorthair | grey | copper | round, thick cheeks |
| Russian Blue | grey | green | thin, wedge-shaped |
| Bombay | black | copper | round |
Coat colour separates the Bombay from the two grey cats. It cannot separate the British Shorthair from the Russian Blue: both are grey. Eye colour does that. But the eyes are perhaps 1% of the photo, so they barely move a colour histogram. A model that learned faces from millions of photos sees them.

Text description of the diagram
Eight photos in four pairs, each with its true breed and the answer of the method we choose later (CLIP probe with prompts). Pair 1: an American Pit Bull Terrier puppy in snow, answered Staffordshire Bull Terrier (a mistake); a Staffordshire Bull Terrier, answered right. Pair 2: a Birman, answered Ragdoll (a mistake); a Ragdoll, answered right. Pair 3: a British Shorthair, answered Russian Blue (a mistake); a Russian Blue, answered right. Pair 4: an American Bulldog lying on a floor, answered Boxer (a mistake); a running Boxer, answered right.
The data
Section titled “The data”The data is the Oxford-IIIT Pet Dataset: 7,349 photos of 37 breeds, 12 of cats and 25 of dogs, about 200 photos per breed. The authors split it into two official lists: 3,680 photos for training and validation, and 3,669 for testing. Each file name holds the breed: Abyssinian_26.jpg is photo 26 of an Abyssinian cat.

Text description of the diagram
A grid of 37 square photos in alphabetical order, from Abyssinian to Yorkshire Terrier. The cats are Abyssinian, Bengal, Birman, Bombay, British Shorthair, Egyptian Mau, Maine Coon, Persian, Ragdoll, Russian Blue, Siamese and Sphynx. The other 25 are dogs, for example Basset Hound, Beagle, Boxer, Chihuahua, Pug, Samoyed and Shiba Inu.
The subset and the people check
Section titled “The subset and the people check”We do not need all 7,349 photos. The course team chose a fixed subset with a committed script and a fixed random seed. For each breed:
- a pool of 20 labelled photos to train on, and 10 validation photos to choose settings (from the official training list);
- 25 test photos to compare the methods, and 4 new photos for the end-to-end run (from the official test list).
Many pet photos show their owners. Before choosing, a person detector looked at 2,443 candidate photos and found a person in 260 of them (11%). Those photos are not used. A detector can miss a hand or a leg, so every photo on this page and in the notebook was also looked at by eye.
The notebook’s first data cell counts the parts:
print(photos.groupby("part", sort=False).size().to_dict()){'pool': 740, 'val': 370, 'test': 925, 'new': 148}37 breeds: 12 cats and 25 dogsWhat this tells us. Every breed has the same number of photos in each part. So the accuracy treats all breeds the same, and the worst breed has 25 test photos. One test photo is about 0.1 points of accuracy. With 925 photos, a difference under about 2 points can be chance: treat it as a tie.
Three rules keep the comparison honest. Every setting is chosen on the validation photos. The test photos only compare the methods. The new photos are used once, at the end.
The four families of methods
Section titled “The four families of methods”Text description of the diagram
Five boxes in a row (one under the other on a phone). The colour histogram with 5 nearest neighbours uses no model: it compares 128 colour bins with 20 labelled photos per breed, and nothing learns. CLIP zero-shot uses a frozen model and one text per breed, with no labels; the texts name the breeds. The linear probe trains a logistic regression on the frozen embeddings, with 1 to 20 labels per breed. The probe with prompts mixes the probabilities of the probe and of the prompts. Fine-tuning trains the last block of ResNet-18 and a new last layer, with 5 to 20 labels per breed.
The baseline: a colour histogram
Section titled “The baseline: a colour histogram”The histogram is the course’s own (8 × 4 × 4 bins of hue, saturation and value). Each test photo takes the most common breed among its 5 nearest labelled photos:
colour histogram + 5-NN, 20 photos per breed: accuracy 0.044 | chance: 0.027It finds 41 of the 925 test photos. It even confuses cats with dogs: 381 of its 884 mistakes name the wrong species. The baseline tells us one useful thing: colour is not the answer here.
CLIP zero-shot: the words of the prompt
Section titled “CLIP zero-shot: the words of the prompt”CLIP compares the photo with one text per breed, as in CLIP zero-shot. It needs no labelled photo. The text around the breed’s name is the prompt set, and the course team tried four on the validation photos:
| Prompt set | Validation | Test |
|---|---|---|
{name} (the name only) |
0.822 | 0.851 |
a photo of a {name}. |
0.838 | 0.861 |
a photo of a {name}, a type of pet. |
0.870 | 0.893 |
a photo of a {name}, a type of {species}. |
0.884 | 0.896 |
The last one says “cat” or “dog”. It wins on validation, so it is the chosen prompt set. Its test accuracy is 0.896 with no label at all. The species helps most with cats, whose names say nothing: “Ragdoll”, “Birman” and “Bombay” could be anything. With the species, cat accuracy rises from 0.703 (name only) to 0.830.
A linear probe
Section titled “A linear probe”A linear probe is a logistic regression on CLIP’s embeddings, as in Linear probes and few-shot. That lesson used the setting C = 2 for Larkfield’s 8 classes. With 37 classes, the course team chose C again on the validation photos: C = 10 gave 0.897, against 0.889 for C = 2. A higher C means a weaker pull towards small weights, so the probe can learn sharper lines between 37 classes.
With all 20 labelled photos per breed, the probe reaches 0.872 on the test. That is lower than zero-shot (0.896). With 5 photos per breed it reaches 0.761, and with one photo, 0.504.
Why do 740 labels lose against none? The prompts carry something that 20 photos cannot teach: the names. CLIP learned from many millions of captioned photos on the internet, and pet owners love to write the breed in a caption. The probe throws that knowledge away and learns each breed from 20 photos.
The probe and the prompts together
Section titled “The probe and the prompts together”We can keep both. The probe gives 37 probabilities, and so do the prompts. A simple mix takes a weighted average of their logarithms. Then it scales the result back into probabilities that add up to 1. The weight of the prompts is called alpha: 0 means the probe alone, 1 means the prompts alone.
A tiny example with two breeds, A and B:
| A | B | |
|---|---|---|
| The probe | 0.70 | 0.30 |
| The prompts | 0.20 | 0.80 |
| Mix with alpha = 0.25 | 0.57 | 0.43 |
| Mix with alpha = 0.5 | 0.43 | 0.57 |
With alpha = 0.25: for A, 0.75 × ln 0.70 + 0.25 × ln 0.20 = −0.670; for B, 0.75 × ln 0.30 + 0.25 × ln 0.80 = −0.959. Then e−0.670 = 0.512 and e−0.959 = 0.383. Divided by their sum (0.895), they give 0.57 and 0.43. The answer changes with alpha, so alpha is chosen on the validation photos: 0.25 for 5 to 20 photos per breed, 0.5 for 1 and 2.
The mix wins at every size. With 20 photos per breed it reaches 0.912, and with one photo per breed it already reaches 0.900. The labels now only correct the prompts where they are wrong.
Fine-tuning ResNet-18
Section titled “Fine-tuning ResNet-18”Fine-tuning changes the network itself. As in Transfer learning, ResNet-18 keeps its first layers frozen, and its last block and a new last layer learn from the labelled photos. The course’s recipe stays the same: AdamW, a learning rate of 0.001, batches of 32, no augmentation. The number of epochs was chosen on validation: 5 of 15.
First, ResNet-18 frozen, with a linear probe on its embeddings: 0.875 with 20 photos per breed. Then fine-tuned: 0.846. Fine-tuning lost. With 20 photos per breed, the last block learns the training photos too well, and it forgets some of what ImageNet taught it. With 10 photos per breed it reaches 0.787, and with 5, 0.714 (the mean of 3 runs each). Training 15 epochs, with a check on the validation photos after each one, took about 3 minutes on the laptop’s CPU.
A fair comparison
Section titled “A fair comparison”All the methods ran on the same 925 test photos, with the course’s helper compare.py. The task column of these rows is pets.
- CLIP probe + prompts
- CLIP probe
- ResNet-18 probe
- ResNet-18 fine-tuned
- CLIP zero-shot (no labels)
Text description of the chart
A line chart. The x axis has five steps: 1, 2, 5, 10 and 20 labelled photos per breed. The y axis is the test accuracy, from 0.4 to 1. CLIP probe with prompts: 0.900, 0.903, 0.901, 0.907 and 0.912. CLIP probe: 0.504, 0.630, 0.761, 0.821 and 0.872. ResNet-18 probe: 0.649, 0.739, 0.827, 0.855 and 0.875. ResNet-18 fine-tuned: 0.714 with 5, 0.787 with 10 and 0.846 with 20. A dashed horizontal line at 0.896 is CLIP zero-shot with no labels.
The table holds the rows with 20 labelled photos per breed. The worst breed is the one with the lowest recall: the share of its 25 test photos that the method gets right.
| Method | Accuracy | Worst breed (recall) | Labelled photos | Model | Time per photo |
|---|---|---|---|---|---|
| Colour histogram + 5-NN | 0.044 | British Shorthair (0.00) | 740 | none | 2.2 ms |
| CLIP zero-shot, species prompt | 0.896 | American Pit Bull Terrier (0.56) | 0 | 605 MB | 9.3 ms |
| CLIP + linear probe | 0.872 | American Pit Bull Terrier (0.36) | 740 | 605 MB | 9.3 ms |
| CLIP + linear probe + prompts | 0.912 | American Pit Bull Terrier (0.40) | 740 | 605 MB | 9.3 ms |
| ResNet-18 frozen + linear probe | 0.875 | Russian Blue (0.56) | 740 | 47 MB | 14.9 ms |
| ResNet-18 fine-tuned, 5 epochs | 0.846 | American Pit Bull Terrier (0.36) | 740 | 47 MB | 14.9 ms |
The time is the time to read a photo and compute its embedding, in batches of 32 with 4 threads on an Apple M4 CPU. The probe and the mix add well under a millisecond. CLIP’s weights are MIT. ResNet-18’s weights are Apache-2.0, but they were trained on ImageNet, whose own terms are for non-commercial research (see When not to build it).
How to read it:
- The prompts are worth hundreds of labels. Zero-shot with no label beats every method with 740 labels, except the mix.
- The mix is the best at every size, and its curve is almost flat: one photo per breed gives 0.900, twenty give 0.912. A gain of about 1 point, near the noise level of this test.
- Fine-tuning lost to a frozen probe on the same network. On a CPU, with 20 photos per breed, it is not worth its minutes here.
Why ResNet-18 knows dogs better than cats
Section titled “Why ResNet-18 knows dogs better than cats”ResNet-18 was trained on ImageNet’s 1,000 classes. 22 of our 25 dog breeds are ImageNet classes, but only 3 of the 12 cat breeds (Persian, Siamese and “Egyptian cat”).
- ResNet-18 probe
- CLIP zero-shot
- CLIP probe + prompts
Text description of the chart
A column chart with three pairs of columns, on an axis from 0.6 to 1. ResNet-18 with a probe: cats 0.78, dogs 0.92. CLIP zero-shot with the species prompt: cats 0.83, dogs 0.93. CLIP probe with prompts: cats 0.90, dogs 0.92.
ResNet-18 gets 0.918 of the dogs right, but only 0.783 of the cats: the cats it never learned as classes. The mix closes the gap (0.897 against 0.920). This is the honest reason why the pretrained models do well here: they have seen these breeds before.
The worst breeds
Section titled “The worst breeds”The chosen method is the CLIP probe with prompts. It makes 81 mistakes on 925 test photos, and none of them names the wrong species. Its five worst breeds:
The five worst breeds (recall on 25 test photos each):american_pit_bull_terrier 0.40staffordshire_bull_terrier 0.72birman 0.72maine_coon 0.80egyptian_mau 0.80The mistakes come in pairs. The American Pit Bull Terrier and the Staffordshire Bull Terrier are confused 14 times, more than one in six of all mistakes. Then Birman and Ragdoll, 9 times: two long-haired cats with dark points and blue eyes. Then English Cocker Spaniel and English Setter (5), and American Bulldog and Boxer (4).
The pit bull pair is hard for people too. Breed clubs describe them differently, but many dogs fit both. ImageNet even puts “American Staffordshire terrier” and “American pit bull terrier” into one class. The labels in this dataset come from the photo owners, so some of them may be wrong. When the labels themselves disagree, no model can be right every time.
Common mistake
The person detector missed a hand
Symptom: the course team looked at the first test photo of each breed for the overview figure. One photo of a Scottish Terrier puppy shows a person’s hand under the puppy. The person detector had passed it.
Cause: a person detector is trained to find whole people, or large parts of them. A hand at the edge of a photo, a leg or a reflection is easy to miss. The detector removed 260 photos, but it gives no promise about the rest.
Fix: never show or publish a photo only because a detector passed it. Look at every photo that you show, by eye, on a contact sheet. The course team replaced that photo in the figure. Staff should be able to remove any photo with a person.
Consequence in AI work: an automatic privacy filter is a first pass, not a guarantee. A report that says “no people” must say how it was checked, and what the check can miss.
From probabilities to a decision
Section titled “From probabilities to a decision”Hana does not need a breed for every photo. She needs a rule: when may the model fill in the breed by itself? The rule has one line: if the top probability reaches the line, the breed is filled in automatically. Below the line, a person checks.
Text description of the diagram
A flow from left to right (top to bottom on a phone). A new photo goes through the probe with prompts, which gives 37 probabilities. First question: is the top breed American Pit Bull Terrier, Staffordshire Bull Terrier or American Bulldog? If yes, a person always checks and chooses from the top 3. If no, second question: is the top probability at least 0.47? If yes, the breed is filled in on the listing. If no, a person checks and chooses from the top 3.
Choosing the line on validation
Section titled “Choosing the line on validation”A low line fills in more breeds, with more wrong ones at 5 dollars each. A high line sends more photos to a person, at 0.50 dollars each. The course team tried every line from 0 to 1 on the validation photos:
- Validation (370 photos): the line is chosen here
- Test (925 photos)
Text description of the chart
A line chart. The x axis is the line, from 0 to 1. The y axis is the cost in dollars per 1,000 photos, from 0 to 550. Both curves start high at a line of 0 (validation 274, test 349: many wrong breeds filled in), fall to a minimum near 0.5, and rise to 500 at a line of 1 (every photo checked). The validation minimum is at 0.47, at 175.7 dollars. The test curve at 0.47 is 204.9 dollars; its own minimum is at 0.55, at 154.6 dollars.
The cheapest line on validation is 0.47, at 175.7 dollars per 1,000 photos. On the test photos, the same line gives:
| Test (925 photos) | Per 1,000 photos | |
|---|---|---|
| Filled in automatically | 756 (82%) | — |
| Wrong among those | 21 (97.2% right) | — |
| Sent to a person | 169 | — |
| Cost | — | 204.9 dollars |
| No line (fill in all but the three breeds) | — | 348.6 dollars |
| A person checks every photo (today) | — | 500 dollars |
The test cost is higher than the validation cost: 204.9 against 175.7. Validation has only 10 photos per breed, so the line is a little lucky there. On the test photos, a line of 0.55 would have cost 154.6 dollars, but we may not choose on the test. This is the same lesson as the line that did not hold: a line chosen on few photos must be checked on new ones.
Riverside has about 60 animals a day. So 204.9 dollars per 1,000 photos is about 12 dollars a day, against 30 dollars if a person checks every photo. About 11 photos a day go to a person, and about 1.4 wrong breeds a day reach a listing. The money is small. The real gain is time: the team checks 11 photos instead of 60.
The recommendation
Section titled “The recommendation”What the analysis tells Hana, in plain words:
- Start today with zero-shot. CLIP with the species prompt needs no labels and gets 0.896 of the test photos right. Use it to suggest three breeds, and let staff choose. Nothing is filled in automatically yet.
- Label 20 photos per breed from your own shelters while you use it: 740 photos, about 12 hours of one person’s time at a minute each. Then switch to the probe with prompts (0.912 on our test, 0.900 already with one photo per breed).
- Fill in the breed automatically when the top probability is at least 0.47. Below that, staff choose from the top three. The true breed was among them for 146 of the 148 new photos.
- Always send the pit bull, Staffordshire and American Bulldog to a person, whatever the probability. The model is worst exactly there.
- Add “mixed or not sure” for staff. Most shelter dogs are mixed breeds, and the model can only answer one of 37.
- Measure again on your own photos before you trust the line, with at least 10 photos per breed.
End to end on new photos
Section titled “End to end on new photos”The function suggest is the whole method for one photo. It reads the photo, computes its CLIP embedding, mixes the probe and the prompts, and decides:
def suggest(file): """One new photo -> the three most likely breeds and the zone: the whole method, end to end.""" with torch.no_grad(): e = unit(clip.encode_image(preprocess(Image.open(file).convert("RGB"))[None]).numpy()) log_probe = np.log(probe20.predict_proba(e) + 1e-9) log_prompts = np.log(softmax(100 * e @ T.T) + 1e-9) p = softmax((1 - ALPHA[20]) * log_probe + ALPHA[20] * log_prompts)[0] top3 = [(BREEDS[i], round(float(p[i]), 2)) for i in np.argsort(-p)[:3]] zone = "always check" if top3[0][0] in ALWAYS_CHECK else "check" if top3[0][1] < LINE else "auto-fill" return top3, zoneThe 148 new photos were never used to train or to choose anything. Six of them:
beagle_44 auto-fill beagle 0.83, basset_hound 0.14, saint_bernard 0.01Bengal_43 auto-fill bengal 0.78, egyptian_mau 0.16, abyssinian 0.03Abyssinian_92 auto-fill abyssinian 0.93, bombay 0.02, siamese 0.02British_Shorthair_84 auto-fill russian_blue 0.55, british_shorthair 0.30, bombay 0.06american_pit_bull_terrier_197 always check staffordshire_bull_terrier 0.71, american_pit_bull_terrier 0.24, american_bulldog 0.02chihuahua_60 check chihuahua 0.26, miniature_pinscher 0.14, scottish_terrier 0.09
Text description of the diagram
Six photos in two rows. A beagle, filled in as beagle at 0.83. A Bengal cat, filled in as Bengal at 0.78. An Abyssinian, filled in at 0.93. A grey British Shorthair, filled in as Russian Blue at 0.55: a mistake, with a red frame. A black and white pit bull terrier, answered Staffordshire Bull Terrier at 0.71 and always checked. A small chihuahua next to a “security dog” sign, answered chihuahua at only 0.26 and checked.
What this tells us. Three photos are filled in right. The British Shorthair is filled in as a Russian Blue at 0.55: above the line, and wrong. The line lets such mistakes through, at 5 dollars each. The pit bull is answered “Staffordshire” at 0.71. Without Hana’s rule, that wrong breed would reach the listing. The chihuahua is right, but the model is unsure (0.26), so a person checks.
On all 148 new photos:
{'auto-fill': 120, 'check': 16, 'always check': 12}auto-filled and wrong: 4 of 120the true breed is among the three suggestions: 146 of 148What this tells us. 81% of the new photos are filled in, and 96.7% of those are right, close to the test (97.2%).
Insights and limits
Section titled “Insights and limits”Pets are easier than a shop’s own categories in one way. The breeds are famous. ImageNet has 25 of the 37 as classes, and CLIP read their names in captions. Larkfield’s catalogue has its own line between a plant pot and a vase. There, the CLIP probe with 20 photos per class (0.929) did a little better than zero-shot (0.925), as Linear probes and few-shot showed. For a product list that the internet has never named, expect the labels to matter more.
Pets are harder in another way. There are 37 classes, not 8, and the worst pair is very close.
The photos are kinder than real life. They are typical, pure-bred animals in good photos from the internet. Shelter photos show mixed breeds, puppies, old animals, cages, bad light and flash. Expect a lower accuracy, and measure it.
Small numbers. The test has 25 photos per breed, so one photo moves a breed’s recall by 0.04.
What we did not try. Fine-tuning with augmentation, more layers or more photos per breed could beat the probe. A larger CLIP model may also do better, at a higher cost per photo.
Responsible use
Section titled “Responsible use”- A breed from a photo is a guess about looks. It says nothing certain about ancestry, temperament, health or risk. Never use it to decide whether an animal is safe, whether a family may adopt, or what an insurance costs.
- Some breeds carry legal weight. In some places, housing rules or laws name certain breeds. For example, the UK’s Dangerous Dogs Act 1991, section 1 (1) (a) applies to “any dog of the type known as the pit bull terrier”. The model is worst exactly on those breeds. Keep Hana’s rule: a person decides them, and the listing says that a person checked.
- Let the model say “I do not know”. A list of 37 breeds forces a wrong answer for a mixed animal. The listing should say “looks like” when it is a guess.
- People in the photos. Shelter photos often show staff, volunteers or adopters. Run the model on your own computer, blur faces by default, as in People in images, and never send a photo with a person to a hosted model.
- The licence travels. The photos are CC BY-SA 4.0. Credit the dataset in your work. Anything you share that is made from the photos, such as a figure or a table of photo names, must use the same licence.
Make it a portfolio piece
Section titled “Make it a portfolio piece”This case study shows that you can choose a vision method from evidence: a fair test, a learning curve, the worst classes and a cost. Make it yours with one change: other costs, another “always check” list, or DINOv2 instead of CLIP. Then make it public, and make it easy to read in five minutes.
What to put in the repository:
README.md, the page that people read first (template below).- The notebook, run from top to bottom, with its outputs.
results.csv: one row per method, from the comparison table (accuracy, worst breed, labelled photos, model size, time per photo).MODEL_CARD.md: the intended use (suggest a breed; a person checks) and what it is not for (safety, adoption or insurance decisions). Add the data, the evaluation, the worst pairs, the line and its cost, and the limits.- The dataset credit and licence from the box above. Do not upload the photos: link the dataset and give the archive’s SHA-256.
A README template:
# Breed suggestions for adoption listings
## ProblemA rescue lists about 60 animals a day. Staff type each breed by eye. A check costsabout 0.50 dollars, a wrong breed on a listing about 5 dollars (assumptions).
## Data and licenceOxford-IIIT Pet Dataset (Parkhi et al., CVPR 2012), 37 breeds, CC BY-SA 4.0,https://www.robots.ox.ac.uk/~vgg/data/pets/. Fixed subset per breed: 20 train,10 validation, 25 test, 4 new. Photos with a person (detector + eye) removed.
## ApproachColour histogram + 5-NN; CLIP ViT-B/32 zero-shot (4 prompt sets); CLIP and ResNet-18linear probes; probe + prompts; ResNet-18 fine-tuned. All settings chosen onvalidation; 1 to 20 labelled photos per breed.
## Results (925 test photos)Histogram 0.044; zero-shot 0.896; probe 0.872; probe + prompts 0.912; fine-tuned 0.846.Worst pair: pit bull vs Staffordshire (14 of 81 mistakes).Line 0.47: 82% filled in, 97.2% of those right, 204.9 dollars per 1,000 photosagainst 500 for checking every photo.
## LimitsPure-bred internet photos; pretrained models saw these breeds; 25 test photos perbreed; mixed breeds not covered; a breed is a guess about looks.How to describe it in an interview, in two sentences: “I compared four families of image classifiers on 37 look-alike pet breeds: CLIP’s prompts beat 740 labels, and a probe mixed with the prompts was best even with one labelled photo per breed. I chose the line for automatic answers from the cost of each error, which cut the cost by 59% against checking every photo, and the riskiest breeds always go to a person.”
Go further:
- Replace CLIP with DINOv2 small (the course’s other embedding). Predict first: without text, can it reach the mix?
- Add a “mixed breed” class with photos of mixed dogs from an allowed source, and measure how often the model admits it.
- Try a learning curve with 50 photos per breed. Does the probe alone pass zero-shot?
- Measure the cost again with a check at 0.25 dollars. Predict first: does the line go up or down?
Knowledge check
Section titled “Knowledge check”Check your understanding
The CLIP probe trained on 740 labelled photos (0.872) scores lower than CLIP zero-shot with no label (0.896). Explain how that can happen, and what the mix of the two shows.
Show the answer
The probe learns each breed only from its 20 labelled photos, and it ignores what CLIP knows about words. Zero-shot uses the breed names: CLIP learned from many captioned internet photos, and pet owners often write the breed in a caption. So the names carry knowledge that 20 photos cannot teach. The mix keeps both: the prompts bring the names, and the labels correct the prompts where they are wrong. It reaches 0.900 with one photo per breed and 0.912 with 20, better than either alone. For a product list that the internet has never named, the labels would matter more.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
Hana's rule sends the pit bull, the Staffordshire and the American Bulldog to a person whatever the probability. Give two reasons from the data and one from outside it.
Show the answer
From the data: the American Pit Bull Terrier is the worst breed (recall 0.40). The pit bull–Staffordshire pair holds 14 of the 81 mistakes, so the model is weakest exactly there. Even a high probability can be wrong: a new pit bull photo was answered “Staffordshire” at 0.71, above the line. The labels themselves are uncertain: ImageNet even merges the two names into one class. From outside the data: in some places, housing rules or laws name these breeds, so a wrong breed on a listing can harm an adopter or an animal. A person must decide and own those answers.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
The line 0.47 cost 175.7 dollars per 1,000 photos on validation and 204.9 on test. Why did it rise, and what should Hana do before she trusts it?
Show the answer
The line was chosen on only 370 validation photos, 10 per breed. The cheapest line on a small set fits that set a little by luck, so a new set costs more. The same happened in the course’s inspection lesson, where a line chosen on 12 damaged photos did not hold. Hana should measure the line again on her own shelters’ photos, at least 10 per breed, and keep the “check” route. If the cost on her photos is much higher, she should raise the line or label more photos before she lets the model fill in breeds.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Recap
- Fine-grained classification sorts many classes of one kind. A colour histogram fails (0.044 on 37 breeds, chance 0.027): the answer hides in small details.
- The prompts were worth more than 740 labels: CLIP zero-shot with the species in the prompt reached 0.896; the probe alone 0.872.
- The probe with the prompts won at every size: 0.900 with one photo per breed, 0.912 with 20. Fine-tuning ResNet-18 lost to its own frozen probe (0.846 against 0.875).
- The worst breeds come in pairs: pit bull and Staffordshire (14 mistakes), Birman and Ragdoll (9). Pretrained models do well here partly because they have seen these breeds before.
- The decision: fill in automatically at a top probability of 0.47 or more, always check three breeds, and choose from three suggestions otherwise. Test cost 204.9 dollars per 1,000 photos against 500; on new photos, 4 of 120 filled-in breeds were wrong.
Next, the last case study: count objects in crowded open photos, where boxes overlap and a detector can count two objects as one.