Anomaly-based inspection
- Lesson 21
- 45 min
- Module 6: Inspect for defects
Where we are
In the last lesson, we learned only from good candles and scored each photo by how far it is from them. The idea was right, but a whole-photo score lost the small damage. In this lesson, we compare small patches with a memory of good patches, with PatchCore and PaDiM in anomalib, and we read the heat maps that show where each model sees damage.
Lesson 21 of 30 · Vision AI: See, Count and InspectAnomaly-based inspection
1. Seeing is a business problem
2. Baselines and pretrained classifiers
3. Embeddings, CLIP and few labels
4. Detect and count
6. Inspect for defects
7. Match, search and describe
8. Run vision in production
What you will be able to do
Find small damage on a returned item: compare each small part of its photo with good parts, and see where the damage is. A score for the whole photo misses damage that covers a tiny part of it.
NotebookDo this lesson in a notebook
The notebook has every code cell, task and check of this lesson. It builds a small PatchCore by hand, then runs anomalib’s PatchCore and PaDiM on the 380 good training photos. It draws heat maps and compares the inspectors by AUROC and by Hamid’s cost per day. Its setup cells install the packages (with PyTorch and anomalib) and download the returns photos. About 45 minutes, of which about 3 minutes of computing on a laptop, and more on a free Colab or Kaggle CPU.
Colab and Kaggle open your own copy. They need a free Google or Kaggle account. Stuck? See the solution notebook.
Before you startWhat to know and to have ready
What you need to know
- Supervised and anomaly-based inspection, the anomaly score, AUROC and the cost per day, from Two ways to find damage.
- How a convolutional network keeps a grid of small regions in its middle layers, from Convolution and patches in one lesson.
What you need on your computer
Nothing for this lesson: the notebook runs in Colab, Kaggle or Jupyter. It downloads ResNet-18 (47 MB) and the wide ResNet-50 that anomalib uses (about 270 MB).
The notebook was tested with Python 3.12, PyTorch 2.15.0, timm 1.0.31 and anomalib 2.7.0, in Jupyter on macOS (laptop CPU).
A worker with a magnifying glass
Section titled “A worker with a magnifying glass”Think again of the new worker who learned from a box of good tea lights. When she checks a returned packet, she does not look at the whole tray and ask “does this tray look normal?”. She looks at each part: this piece of wax, this wick, this edge. A tiny dark spot hardly changes the look of the tray. But that one piece of wax looks like no piece of wax she has ever seen.
Her memory is not a memory of whole trays. It is a memory of parts: hundreds of pieces of good wax, good wicks and good edges. She compares each part of a new candle with the parts she knows. If one part has no match, the candle is suspect, and she knows where to point.
The problem: the whole photo hides the spot
Section titled “The problem: the whole photo hides the spot”In the last lesson, the anomaly score compared whole photos, and it was weak: an AUROC of 0.73 on the test photos. The median damage covers about 64 pixels of a 224 × 224 photo, about 0.13% of it. The photo’s embedding hardly moves, so the damaged photo still looks like its nearest good photo.
The fix is the worker’s habit: compare parts, not trays.
The idea: a memory of good patches
Section titled “The idea: a memory of good patches”Cut each photo into a grid of small squares. Each square is a patch. For each patch, find the most similar patch in a memory of patches from good photos. A patch of normal wax has a close match. A patch of damage does not. The photo’s score is the distance of its worst patch: the patch that is furthest from every good patch.
This method is PatchCore (Roth and others, 2022). “Patch” because it compares patches. “Core” because its memory is a coreset: a small set of patches, chosen so that every kind of good patch has a close match in it. A memory of every patch of every good photo would be large and slow to search.
Text description of the diagram
The diagram has three steps. Step one shows a photo cut into a grid of 4 by 4 patches; one patch is orange, the others blue. An arrow leads to step two, a grey box: a memory of patches of 380 good photos. An arrow leads to step three, two boxes. Most patches have a distance of 0.2, because a good patch is near them. The odd patch has a distance of 0.9, because it is far from all good patches. A line of text under the steps says: photo score = the largest patch distance. Heat map = the distance of every patch, shown as a picture.
A tiny example: four patches
Section titled “A tiny example: four patches”Imagine that each patch is described by one number, its brightness, and that a photo has only four patches. The memory holds five good patches: 0.47, 0.49, 0.50, 0.53 and 0.55.
| Patch | Value | Nearest good patch | Distance |
|---|---|---|---|
| A (wax) | 0.50 | 0.50 | 0.00 |
| B (wax) | 0.52 | 0.53 | 0.01 |
| C (wick) | 0.48 | 0.47 or 0.49 | 0.01 |
| D (dark spot) | 0.20 | 0.47 | 0.27 |
The photo’s score is the largest distance: 0.27, from patch D. The average distance is only (0.00 + 0.01 + 0.01 + 0.27) ÷ 4 = 0.07. One odd patch of four already shrinks to a quarter in the average. A real photo has 784 patches, so the average would hide a small spot even more. That is why PatchCore takes the largest distance.
Where the patches come from
Section titled “Where the patches come from”A patch is not a square of raw pixels. Two pixel squares of the same wax can differ a lot when the light falls a little differently. PatchCore takes its patches from the middle layers of a pretrained network, such as ResNet. Convolution and patches in one lesson showed that each position in a middle layer describes one region of the photo. Its many numbers describe edges, textures and small shapes. The last layer would mix all positions together, and we would be back to one embedding for the whole photo.
In the notebook, the small PatchCore uses ResNet-18. A photo of 224 × 224 pixels gives a grid of 28 × 28 = 784 positions, and each position gets 384 numbers. Each patch also averages its neighbours, so a patch “sees” a little more than its own square.
p = patches("data/" + good_train["file"].iloc[0])print(p.shape)(784, 384)The 380 good training photos give 297,920 patches. The small PatchCore keeps 20,000 of them at random as its memory. anomalib’s PatchCore chooses them with the coreset method instead.
297,920 good patches, 20,000 kept in the memoryHeat maps: where the model sees damage
Section titled “Heat maps: where the model sees damage”Draw the distance of every patch on the 28 × 28 grid, and you get a heat map. It is bright where a patch is far from every good patch. It turns a score into a place that a person can look at.
Here are three damaged test candles. The top row shows each photo with its true damage circled, from the damage mask. The bottom row shows the heat map of anomalib’s PatchCore over the same photo.

Text description of the diagram
Top row, three photos of four tea lights each, with an orange circle around the true damage. In the first photo, the damage is a lump of extra wax on the top right candle. In the second, extra wax lies near the wick of the bottom right candle. In the third, wax has melted out at the left edge of the bottom left candle. Bottom row: the same three photos under a heat map. It is dark purple where the patches look good, and bright yellow or orange where they do not. In each photo, the bright area sits on the circled damage. In the second photo, the bright area is weaker than in the first.
In these three photos, the bright area sits on the damage. They are the three best of the 18 damaged test photos. How good are heat maps in general?
- With the small PatchCore of the notebook, the worst patch lies inside the true damage in 13 of the 18 damaged test photos. In the other 5, the photo may still score high, but the bright spot is elsewhere.
- The course team compared anomalib’s heat maps with the true masks. Bright parts (the top 2% of all heat-map values) against the true damage give a mean Dice of only 0.175. Dice is the overlap measure of Measure from masks: 1.0 is the same shape, 0 no overlap.
So a heat map points to the area, not to the outline. Use it to show a person where to look first. Do not use it to measure the size of a damage: the masks of Module 5 do that.
PaDiM: one bell curve per position
Section titled “PaDiM: one bell curve per position”anomalib has a second classic method for patches, PaDiM (Defard and others, 2021). PaDiM also takes patches from the middle layers. But it keeps no memory of patches. For each position of the grid, it stores the average good patch at that position and how much good patches vary there. A new patch is odd if it is far from the average of its own position, measured in units of that position’s normal variation.
Think of it as one bell curve per position, the normal-distribution shape of Essential Mathematics and Statistics for AI. PaDiM learns in seconds, because it only computes averages: 10 seconds on the 380 good photos, against 94 seconds for PatchCore with a wide ResNet-50. Its weakness is the positions. It works only if the same thing is always at the same place. Hamid’s photos are taken in a fixed tray, so the positions are roughly stable. But a packet placed a little to the left breaks the averages.
Larkfield: four inspectors on the same photos
Section titled “Larkfield: four inspectors on the same photos”Priya runs the small PatchCore by hand, then anomalib’s PatchCore and PaDiM. anomalib reads the photos from folders, learns from the folder of good photos, and gives every new photo a score from 0 to 1 and a heat map:
model = Patchcore(backbone="wide_resnet50_2", coreset_sampling_ratio=0.01, visualizer=False)folder = Folder(name="candles", root=work, normal_dir="train_good", abnormal_dir="train_damaged", num_workers=0, train_batch_size=16, eval_batch_size=16, seed=0)engine = Engine(accelerator="cpu", devices=1, logger=False, default_root_dir=work / "results", enable_progress_bar=False)engine.fit(model=model, datamodule=folder)coreset_sampling_ratio=0.01 keeps 1% of the good patches in the memory. The damaged training photos do not teach the model. anomalib uses them only to scale its scores between 0 and 1.
The course’s recorded run, on the 218 test photos, with the lines chosen on validation:
| Inspector | AUROC | Cost a day | Learns in | Per photo | Size |
|---|---|---|---|---|---|
| PaDiM, ResNet-18 | 0.824 | 993.6 | 10 s | 26.5 ms | 47 MB |
| Small PatchCore (ResNet-18, random memory) | 0.847 | 1,290.8 | not timed | 17.3 ms | 78 MB |
| PatchCore, ResNet-18 (anomalib) | 0.919 | 706.0 | 50 s | 20.4 ms | 47 MB |
| PatchCore, wide ResNet-50 (anomalib) | 0.917 | 368.8 | 94 s | 47.8 ms | 267 MB |
One row needs care. anomalib’s PatchCore with ResNet-18 (706.0 dollars a day) needs about 113 checks a day on the test photos (112.8), over Hamid’s budget of 100. So its cost is not reachable as it stands.
Text description of the chart
A column chart with four columns. PaDiM with ResNet-18: 993.6 dollars a day. The small PatchCore: 1,290.8. anomalib’s PatchCore with ResNet-18: 706.0. anomalib’s PatchCore with a wide ResNet-50: 368.8.
Patches work. Every patch method beats the whole-photo score of the last lesson (0.73, 1,578.4 dollars a day). anomalib’s PatchCore with a wide ResNet-50 is the cheapest: 368.8 dollars a day, against 2,229.4 with no inspection.
Close AUROC, far-apart costs
Section titled “Close AUROC, far-apart costs”Look at the two anomalib PatchCores. Their AUROC is almost the same (0.919 and 0.917), but one costs almost twice as much a day. Why?
AUROC counts all 3,600 pairs of one damaged and one good photo (18 × 200). The cost depends only on the few photos near the lines. With 18 damaged test photos, one damaged photo that passes adds 600 ÷ 218 × 45 = 124 dollars a day. A difference of two or three photos makes the gap. The same is true of the test set itself: it is small, so every cost here moves a lot with a few photos.
Your run will differ
Section titled “Your run will differ”anomalib’s PatchCore and PaDiM do not give exactly the same scores at every run. The table above is the course’s recorded run, made once. Then we ran the notebook five more times on one laptop. PatchCore with a wide ResNet-50 gave an AUROC from 0.926 to 0.939, and a cost from 370.2 to 1,194.5 dollars a day. PaDiM gave from 0.797 to 0.841, and from 1,149.1 to 1,875.7 dollars a day. Same code, same photos, same splits. With so few damaged photos, a small change in the scores moves a photo across a line. Then the cost moves by hundreds of dollars. Treat the costs as a range, not as exact numbers, and use the recorded run when you compare with the course’s table.
The small PatchCore and Amira’s service
Section titled “The small PatchCore and Amira’s service”anomalib’s PatchCore is the cheapest. Why does the project use the small one?
Amira, Larkfield’s developer, must ship the inspector in the image service of Module 8. anomalib needs PyTorch, Lightning and their many packages, and the wide ResNet-50 weighs 267 MB. The small PatchCore needs only the first layers of a ResNet-18. Amira can export them to ONNX, a format that a light library, ONNX Runtime, runs without PyTorch. Made smaller with 8-bit numbers, it is 2.9 MB, plus a memory of 15 MB. It needs only ONNX Runtime and NumPy, and it scores a photo in 15 ms. The price is quality: a test AUROC of 0.836 against 0.917.
That is a real trade-off, not a free choice. The better model exists, and the lesson Fast enough on a CPU measures what each choice costs in time and size. For every choice, write down the price in both directions.
When to use anomaly-based inspection, and what it costs
Section titled “When to use anomaly-based inspection, and what it costs”Use it when the items look the same from photo to photo: the same product, the same tray, the same light. It fits when good items are easy to collect and damage is rare and varied. Hamid’s tea lights fit well.
Be careful when good items vary a lot. Many colours or shapes of the same product, or photos from many angles, make “normal” wide. A damage can then hide among normal differences. Do not use it alone when you must name the kind of damage: it says “odd here”, not “chunk of wax missing”.
What it costs: you store and update a memory of good patches, or their statistics. You run a backbone on every photo. And you need new good photos when “normal” changes. A new camera or new light makes every patch look odd: Module 8 shows it. And the score is not a decision. A line must turn it into pass or fail. The alert budget in Anomaly Detection: Find What Is Unusual explains how to set a line when only a few examples are labelled. The next lesson sets Hamid’s lines.
Guided practice: patches and anomalib in the notebook
Section titled “Guided practice: patches and anomalib in the notebook”- Open the notebook from the box Do this lesson in a notebook at the top of this page.
- Run the setup cells. The install cell lists anomalib 2.7.0, and the data cell ends with
Ready:. - In section 1, run the cell. You see
(784, 384). - In section 2, run the memory cell. You see
297,920 good patches, 20,000 kept in the memory. - In section 3, write your prediction, then run the heat-map cell. The brightest patches sit on the extra wax, top right.
- In section 4, run the scoring cell. You see
small PatchCore, AUROC on test: 0.847. - Run the next cell. Scores go from about 5 to 19, and a line at 0.5 fails all 200 good photos.
- In section 5, run the anomalib cell. It learns for about 2 minutes, then prints an AUROC between about 0.91 and 0.94.
- Run the check cell. It prints
Check passed. - In section 6, run PaDiM. It learns in seconds.
- In section 7, run the cost cell. Compare your three costs with the table above.
Your turn: the mean instead of the largest distance
Section titled “Your turn: the mean instead of the largest distance”Change one thing: score each photo by the mean of its 784 patch distances instead of the largest one. The tiny example says that the mean hides a small spot. Before you run it, predict: will the AUROC on the test photos be higher or lower than 0.847? In the notebook, section 8, put the AUROC in mean_auc.
Show the answer
mean_auc = cp.roc_auc([d.mean() for d in small_test_dist], dte)print(mean_auc)0.875Higher, which surprised us too. On these 18 damaged test photos, the mean does better than the largest distance (0.875 against 0.847). Five of them have wax melted out of the candle, a large damage that raises many patches at once. The tiny example is still right about one small spot. So a rule that sounds right must still be measured. And with 18 damaged photos, a difference of 0.03 could change with other photos. Compare such choices on the validation photos, never on the test photos.
Common mistake
A line at 0.5 on a distance
Symptom: Tomás puts the small PatchCore in front of Hamid’s team with the rule “fail when the score is above 0.5”. On the test photos, every candle fails. In the notebook:
print(f"scores from {small_test.min():.2f} to {small_test.max():.2f}")print(cp.inspection_costs(small_test, dte, 0.5, 0.5))scores from 5.61 to 19.27{'items': 218, 'damaged': 18, 'missed': 0, 'good_failed': 200, 'checks': 0, 'checks_per_day': 0.0, 'damaged_sold_as_new_per_day': 0.0, 'good_marked_damaged_per_day': 550.5, 'cost_per_day': 15412.8, 'over_budget': False}Cause: an anomaly score is a distance, not a probability. The small PatchCore’s scores go from about 5 to 19, so 0.5 is below every photo. anomalib scales its scores to 0–1, but that scale also depends on its data, so 0.5 means nothing special there either.
Fix: never reuse a line from another model or another scale. Choose the lines on the validation photos of this model, from the costs and the budget, as the next lesson does.
Consequence in AI work: all 200 good candles go to the outlet: 15,412.8 dollars a day, almost seven times the cost of no inspection. A line copied from a probability model or from a blog post can make an inspector worse than none.
Knowledge check
Section titled “Knowledge check”Check your understanding
A photo has six patches with distances 0.05, 0.04, 0.06, 0.05, 0.03 and 0.61 to their nearest good patch. What is its PatchCore score? What would the mean be, and why does PatchCore use the largest distance?
Show the answer
The score is the largest distance, 0.61. The mean is (0.05 + 0.04 + 0.06 + 0.05 + 0.03 + 0.61) ÷ 6 = 0.84 ÷ 6 = 0.14. PatchCore uses the largest distance because one odd patch is enough to make an item suspect. A small damage covers only a few of the 784 patches of a real photo. The mean shrinks one odd patch by the number of patches. Still, measure the choice: on our 18 damaged test photos, the mean did better, because several damages were large.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
The heat map of a damaged candle is bright on the damage. Can Hamid use the bright area to measure how big the damage is? Explain with the course's numbers.
Show the answer
No. A heat map points to the area, not to the outline. Against the true masks, the bright parts of anomalib’s heat maps give a mean Dice of only 0.175. In the small PatchCore, the worst patch was inside the true damage in 13 of 18 photos. Use the heat map to show a person where to look first. To measure a damage, use a mask, as in Measure from masks in Module 5.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
Amira's service uses the small PatchCore (test AUROC 0.836), not anomalib's PatchCore with a wide ResNet-50 (0.917, 368.8 dollars a day). Give one reason for her choice and one cost of it.
Show the answer
Reason: the small PatchCore runs with ONNX Runtime and NumPy only. It is 2.9 MB plus a 15 MB memory and scores a photo in 15 ms on a CPU. anomalib needs PyTorch, Lightning and 267 MB of weights. Cost: it finds damage less well, so more damaged candles pass or more good candles fail. On the test photos, its float version costs 1,290.8 dollars a day against 368.8. The choice must be written down with both numbers, so it can be changed when the service can carry a bigger model.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Recap
- PatchCore compares each patch of a photo with a memory of good patches. The photo’s score is its largest patch distance, so one small odd patch is enough.
- Patches come from the middle layers of a pretrained network: 784 positions of 384 numbers with ResNet-18 at 224 pixels.
- A heat map shows every patch distance. It points to the area (the worst patch was inside the damage in 13 of 18 photos) but not the outline (mean Dice 0.175).
- PaDiM keeps one average and spread per position. It learns in 10 seconds, but needs items at the same place.
- On the test photos, anomalib’s PatchCore with a wide ResNet-50 has an AUROC of 0.917 and costs 368.8 dollars a day. The service’s small PatchCore has 0.836. Costs move a lot with a few photos, and between runs.
You can now score every return and show where the damage is. But a score is not a decision. Next, you turn scores into pass, check and fail with Hamid, from the cost of each mistake and his 100 checks a day.