Two ways to find damage
- Lesson 20
- 35 min
- Module 6: Inspect for defects
Where we are
In Module 5, we cut objects out with masks and measured the size of a damage. Hamid’s main question is still open: is this returned candle damaged at all? In this lesson, we compare two ways to answer it, the way a new warehouse worker learns: from a list of known faults, or from many good candles.
Lesson 20 of 30 · Vision AI: See, Count and InspectTwo ways to find damage
1. Seeing is a business problem
2. Baselines and pretrained classifiers
3. Embeddings, CLIP and few labels
4. Detect and count
6. Inspect for defects
7. Match, search and describe
8. Run vision in production
What you will be able to do
Choose how an inspector learns what damage looks like. Damaged returns are rare, and every damage is a little different. So the usual way, with many labelled examples of each kind, may not work.
NotebookDo this lesson in a notebook
The notebook has every code cell, task and check of this lesson. It turns the returns photos into embeddings and builds both inspectors. Then it compares them by AUROC and by Hamid’s cost per day. Its setup cells install the packages (with PyTorch) and download the returns photos, so it needs no local setup. About 35 minutes.
Colab and Kaggle open your own copy. They need a free Google or Kaggle account. Stuck? See the solution notebook.
Before you startWhat to know and to have ready
What you need to know
- Hamid’s returns, the 600 returns a day, and what each mistake costs, from The return that cost twice.
- The returns photos and their splits, from Meet the Larkfield images.
- Embeddings: a pretrained model turns a photo into a list of numbers, and similar photos get close lists, from Image embeddings.
- Train, validation and test sets, from Train, validation and test sets in Machine Learning: From Problem to Reliable Model.
What you need on your computer
Nothing for this lesson: the notebook runs in Colab, Kaggle or Jupyter. The project command for inspection comes in Set the line with the warehouse.
The notebook was tested with Python 3.12, PyTorch 2.15.0, timm 1.0.31 and scikit-learn 1.9.1, in Jupyter on macOS (laptop CPU).
How a new worker learns to spot a bad candle
Section titled “How a new worker learns to spot a bad candle”Think of a new worker in Hamid’s returns warehouse. Her job is to look at returned tea lights and say “good” or “damaged”. There are two ways to train her.
The first way is a binder of faults. Hamid shows her photos of every kind of damage he has seen. There is a chunk of wax missing, a dark spot, a bent wick, a crushed packet. She learns each fault from its pictures. On her first day, she checks every candle against the binder.
The second way is many good candles. Hamid gives her a box of good tea lights and says: “Look at these all week. Learn what normal looks like.” After a week, she knows the smooth wax, the straight wick and the clean edge. Then anything that looks “off” catches her eye, even a fault that no binder showed her.
Both ways work, and each one fails in its own way. The binder worker misses a new kind of fault. The second worker also stops at things that are only unusual: a candle under a different lamp, or a new brand of packet. In this lesson, we build both workers as models.
The problem: damage is rare and every damage is different
Section titled “The problem: damage is rare and every damage is different”The returns set has photos of four tea lights each, from the VisA inspection set. Here are one good photo and three damaged ones. The circle shows where the true damage is.

Text description of the diagram
Four photos in a row (two by two on a phone). Each photo shows four round tea lights, seen from above, on a dark brown cloth. The first photo is a good item. In the second, an orange circle surrounds the edge of one candle, where a chunk of wax is missing. In the third, the circle surrounds a dark spot of a few pixels on the wax. In the fourth, it surrounds a wick that leans to one side. Without the circles, the damage in the third and fourth photos is hard to see.
Two facts about these photos decide how we can build an inspector.
Damage is rare. Of the 405 training photos, only 25 are damaged: about 6%. The validation set has 12 damaged photos among 112, and the test set 18 among 218.
Damage is varied. The 25 damaged training photos show seven kinds of damage, and some kinds have only two or three photos. “Weird candle wick” has 3, “extra wax in candle” has 2. The test set even has one photo whose damage is labelled “other”: no training photo shows that kind. In real use, Hamid will also meet damage that nobody has photographed yet.
So the binder of faults is thin. It has a few pictures of each fault, and no picture of the next new one.
Way 1: supervised, learn from labelled damage
Section titled “Way 1: supervised, learn from labelled damage”The first way is the one you know from the catalogue. Label every training photo “good” or “damaged”, and train a classifier on the labels. This is supervised inspection: the model learns from examples whose right answers are known.
Priya reuses the embeddings of Image embeddings. ResNet-18 turns each returns photo into 512 numbers. A logistic regression learns from the 405 training embeddings which ones are damaged. The model’s output for a new photo is a probability of “damaged”, from 0 to 1.
One detail matters here. With 380 good photos and 25 damaged ones, a model can be right 94% of the time by calling everything good. So Priya gives the damaged class the same total weight as the good class (class_weight="balanced" in scikit-learn). Each damaged photo then counts about 15 times as much as a good one.
Way 2: anomaly-based, learn only what good looks like
Section titled “Way 2: anomaly-based, learn only what good looks like”The second way is the worker with the box of good candles. Train only on good photos. For a new photo, measure how different it is from everything good that the model has seen. A photo that is far from all good photos is probably damaged.
The word for “different from normal” is anomaly, so this is anomaly-based inspection. Its output is an anomaly score: a number that grows as the photo looks less like a good candle. It is not a probability, and its scale depends on the method. A score of 0.3 can be low for one method and high for another.
The simplest anomaly score uses the same embeddings. For a new photo, find the most similar good training photo, and take how far away it is: 1 minus their cosine similarity. A good photo has a near twin among the 380 good photos, so its score is small. This is the nearest-neighbour distance of Far from the neighbours: kNN distance in Anomaly Detection: Find What Is Unusual, now with photos.
Text description of the diagram
The diagram has two columns. The left column, “Supervised”, has three boxes joined by arrows. First: it learns from good photos and labelled damaged photos, 380 good and 25 damaged. Second: a classifier learns the difference between the two. Third: it says “damaged” with a probability, and it finds damage like the examples. Under the column: it misses a new kind of damage, and it needs many damaged examples. The right column, “Anomaly-based”, also has three boxes. First: it learns from good photos only, 380 good. Second: a memory of what good candles look like. Third: it says how different the photo is from good, as an anomaly score. Under the column: it finds any kind of damage, and it also flags new light or a new camera.
A tiny example: the bent wick that the binder missed
Section titled “A tiny example: the bent wick that the binder missed”Here is each way with two numbers per photo instead of 512, so you can follow every step. Imagine the numbers are “how even the wax looks” and “how straight the wick looks”, each from 0 to 1.
| Photo | Wax evenness | Wick straightness | Label |
|---|---|---|---|
| G1 | 0.90 | 0.90 | good |
| G2 | 0.85 | 0.95 | good |
| G3 | 0.95 | 0.85 | good |
| D1 | 0.30 | 0.90 | damaged (chunk of wax missing) |
The supervised model sees one damaged example. It learns a rule that separates D1 from the good photos: “wax evenness below 0.6 means damaged”. The anomaly model ignores D1 and keeps only G1, G2 and G3.
Now two new photos arrive: N1 is a good candle at (0.88, 0.90), and N2 has a bent wick at (0.90, 0.40).
| New photo | Supervised rule | Distance to the nearest good photo | Anomaly score |
|---|---|---|---|
| N1 (good) | evenness 0.88, so “good” | to G1: √(0.02² + 0.00²) = 0.02 | 0.02: low |
| N2 (bent wick) | evenness 0.90, so “good”: missed | to G3: √(0.05² + 0.45²) = 0.45 | 0.45: high |
The supervised rule looks only at the wax, because its one damaged example was about wax. It calls the bent wick good. The anomaly score has no idea what a wick fault is. It only sees that N2 is far from every good photo, so N2 stands out.
The tiny example also shows the price of the anomaly way. A good candle photographed under a yellow lamp would also move away from G1, G2 and G3, and get a high score. The anomaly model cannot tell “damaged” from “new”. Module 8 meets exactly this problem with a new camera.
How to compare two inspectors
Section titled “How to compare two inspectors”The two models give different kinds of numbers: a probability and a distance. To compare them fairly, we need a measure that does not care about the scale.
AUROC: does damage score higher than good?
Section titled “AUROC: does damage score higher than good?”Take one damaged photo and one good photo at random. A useful inspector gives the damaged one the higher score. AUROC is how often that happens, over all pairs of one damaged and one good photo. The name means “area under the ROC curve”, a curve of all possible lines. The precision–recall trade-off in the machine learning course draws one and calls the area ROC AUC. You only need the pairs idea:
- AUROC 1.0: every damaged photo scores above every good photo.
- AUROC 0.5: a coin flip. The score knows nothing.
A tiny example: 2 damaged photos score 0.8 and 0.4, and 3 good photos score 0.1, 0.3 and 0.5. There are 2 × 3 = 6 pairs. The 0.8 is above all three good scores: 3 pairs right. The 0.4 is above 0.1 and 0.3 but below 0.5: 2 pairs right. So 5 of 6 pairs are right, and the AUROC is 5 ÷ 6 = 0.83.
AUROC does not depend on where we put a line, so it is a good first comparison. It does not say what an inspector costs Hamid.
Cost per day: what Hamid pays
Section titled “Cost per day: what Hamid pays”The course’s helper compare.py turns scores into Hamid’s money. It sets two lines on the validation photos. Below one line an item passes, above the other it fails, and between them a person checks it. Then it measures the test photos with those lines. A damaged item sold as new costs 45 dollars, a good item sent to the outlet costs 28, and a check costs 0.50. These are the course’s assumptions. The team can do 100 checks a day. Set the line with the warehouse explains the lines. In this lesson, we use them as a ready-made judge.
Larkfield: the two ways on the real returns
Section titled “Larkfield: the two ways on the real returns”Priya runs both ways in the notebook, on the same 218 test photos. The supervised model:
from sklearn.linear_model import LogisticRegressionclf = LogisticRegression(C=1.0, max_iter=5000, class_weight="balanced").fit(Xtr, dtr)sup_val, sup_test = clf.predict_proba(Xval)[:, 1], clf.predict_proba(Xte)[:, 1]print("AUROC on test:", cp.roc_auc(sup_test, dte))AUROC on test: 0.914The anomaly score, from the good training photos only:
good = Xtr[~dtr]anom_val, anom_test = 1 - (Xval @ good.T).max(1), 1 - (Xte @ good.T).max(1)print("AUROC on test:", cp.roc_auc(anom_test, dte))AUROC on test: 0.73Then Hamid’s cost per day, with the lines chosen on validation:
supervised (ResNet-18 + logistic regression) cost 1106.4 a day | AUROC 0.914 | worst different colour spot 0.4anomaly (nearest good photo) cost 1578.4 a day | AUROC 0.73 | worst damaged corner of packaging 0.0“Worst” is the kind of damage that the inspector catches least often. The supervised model stops only 40% of the “different colour spot” photos. The anomaly score stops none of the “damaged corner of packaging” photos.
Text description of the chart
A column chart with three columns. No inspection: 2,229.4 dollars a day. The whole-photo anomaly score: 1,578.4 dollars a day. The supervised ResNet-18 with logistic regression: 1,106.4 dollars a day.
Both inspectors save money compared with no inspection, where every damaged return goes back on sale: 2,229.4 dollars a day. Here, the supervised model wins clearly. Does that mean that the anomaly way is wrong?
Why the whole-photo score is weak
Section titled “Why the whole-photo score is weak”No. Our anomaly score compares whole photos. The embedding squeezes four candles and the cloth into 512 numbers. A damage is small: the median damage covers about 64 pixels of a 224 × 224 photo, which has 50,176 pixels. That is about 0.13% of the photo. A dark spot that small hardly changes the whole-photo embedding, so the damaged photo still has a near twin among the good photos.
The idea is right, but the unit is too big. The worker with the box of good candles does not compare whole trays. She looks at each part of each candle. The next lesson does the same with patches: small squares of the photo, each compared with good squares. On the same test photos, that method, PatchCore, costs 368.8 dollars a day.
When to use each way
Section titled “When to use each way”Use supervised inspection when you have many labelled examples of each kind of damage, and the kinds do not change much. It answers the question “is this one of the faults we know?”, and its probability is easy to explain. It needs the most labels: each new kind of damage needs new labelled photos. And it is blind to damage it never saw.
Use anomaly-based inspection when damage is rare, varied or new, and good items are easy to collect. Hamid’s good returns are 94% of the photos, so they cost almost nothing to gather. It needs a careful choice of the unit: patches, not whole photos. And its score flags any change, such as new light or a new camera. Every new “normal” must go into its memory.
Many teams use both. They start with anomaly-based inspection, because it needs no damaged labels. People check the photos that it flags, so labelled damage piles up. Later, a supervised model can learn the common faults, while the anomaly score keeps watching for new ones.
Tomás asks: “Why not ask a vision-language model if the candle is damaged?” Vision-language models answers him in Module 7.
Guided practice: both ways in the notebook
Section titled “Guided practice: both ways in the notebook”- Open the notebook from the box Do this lesson in a notebook at the top of this page.
- Run the setup cells. The data cell ends with a
Ready:line. - In section 1, run the cell. You see 380 good and 25 damaged photos in train, and four damaged photos.
- In section 2, run the embedding cell. You see
(405, 512) (112, 512) (218, 512). - In section 3, run the supervised model. You see
AUROC on test: 0.914. - In section 4, write your prediction, then run the anomaly score. You see
AUROC on test: 0.73. - In section 5, run the cost cell. You see
1106.4and1578.4dollars a day, then the course’s table. - In section 6, write the AUROC of the anomaly score in
anomaly_auc. - Run the check cell. It prints
Check passed.
Show the answer to step 8
anomaly_auc = cp.roc_auc(anom_test, dte)print(anomaly_auc)0.73Your turn: write to Hamid
Section titled “Your turn: write to Hamid”Section 7 of the notebook asks for two sentences to Hamid: which way would you start with, and what must his team label first? Use the numbers above. Then compare with our answer.
Show our answer
Start with the anomaly-based way, because it needs only good photos, and they are 94% of the returns. Score small parts of each photo, not the whole photo, because the whole-photo score misses small damage (AUROC 0.73). Hamid’s team should first label the photos that the inspector sends to a person, with the kind of damage. These labels show where the inspector is wrong, and later they can train a supervised model for the common faults.
Common mistake
A high accuracy that catches nothing
Symptom: Tomás trains a classifier on the returns and reports “91.7% accuracy on the test photos”. He is pleased.
Cause: the test set has 200 good photos and 18 damaged ones. A model that calls every candle good is right 200 times out of 218: 91.7%. It catches no damage at all. With a rare class, accuracy mostly counts the good items.
Fix: compare inspectors by AUROC, which looks only at pairs of a damaged and a good photo. Then compare the cost per day, which counts each mistake at its price. For the “everything is good” model, the cost per day is the same as no inspection: 2,229.4 dollars.
Consequence in AI work: a rare-class model chosen by accuracy can look excellent and do nothing. The anomaly detection course shows the same trap with payments, in Why accuracy lies here.
Knowledge check
Section titled “Knowledge check”Check your understanding
Hamid's team has photographed 25 damaged candles and 380 good ones. Next month, a new supplier's packets arrive, and some are torn in a way nobody has seen. Which way of inspection is more likely to flag the torn packets, and why?
Show the answer
The anomaly-based way. It learned only what good candles and packets look like, so anything far from them gets a high score, including a new kind of tear. The supervised model learned the difference between good photos and the 25 damaged ones. A torn packet looks like none of its damaged examples, so it may call it good. The anomaly way may also flag the new supplier’s good packets, because they look new. Hamid’s team should check those flags, and good photos of the new packets should go into the memory.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
Two damaged photos score 0.9 and 0.2. Three good photos score 0.1, 0.4 and 0.6. What is the AUROC? Is this inspector useful?
Show the answer
There are 2 × 3 = 6 pairs of one damaged and one good photo. The 0.9 is above all three good scores: 3 right pairs. The 0.2 is above only 0.1: 1 right pair. So 4 of 6 pairs are right, and the AUROC is 4 ÷ 6 = 0.67. It is better than a coin (0.5), but weak: one damaged photo scores lower than most good photos. Whatever the line, that damaged candle either passes or takes two good candles with it.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
The whole-photo anomaly score has an AUROC of 0.73 on the test photos. Explain in two sentences why it is weak, and what would make it stronger.
Show the answer
The embedding describes the whole photo, four candles and the cloth. A typical damage covers only about 0.13% of its pixels, so a damaged photo still looks almost like a good one. Comparing small parts of the photo (patches) with good parts would make it stronger. A damaged patch is very different from every good patch, even when the whole photo is not.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Recap
- Damage is rare: 25 of 405 training photos. It is varied: seven kinds, some with two or three photos, and an “other” kind in the test set.
- Supervised inspection learns good and damaged photos with labels, and gives a probability. It finds damage like its examples and misses new kinds.
- Anomaly-based inspection learns only good photos, and gives an anomaly score: how far a photo is from good. It flags any kind of damage, and also anything new.
- AUROC is how often a damaged photo scores above a good one: 1.0 is perfect, 0.5 is a coin. The cost per day turns the mistakes into Hamid’s money.
- On the test photos: supervised ResNet-18 AUROC 0.914 and 1,106.4 dollars a day; the whole-photo anomaly score 0.73 and 1,578.4; no inspection 2,229.4.
You can now build both kinds of inspector and judge them. But the anomaly score compares whole photos, and a small spot gets lost. Next, you compare small patches with a memory of good patches, and see where the model thinks the damage is.
Next: Anomaly-based inspection