Skip to content
Français

Anomaly-based inspection

  • Lesson 21
  • 45 min
  • Module 6: Inspect for defects

Where we are

In the last lesson, we learned only from good candles and scored each photo by how far it is from them. The idea was right, but a whole-photo score lost the small damage. In this lesson, we compare small patches with a memory of good patches, with PatchCore and PaDiM in anomalib, and we read the heat maps that show where each model sees damage.

Plays from YouTube. Nothing loads from YouTube until you press play. Privacy

What you will be able to do

Find small damage on a returned item: compare each small part of its photo with good parts, and see where the damage is. A score for the whole photo misses damage that covers a tiny part of it.

NotebookDo this lesson in a notebookColab · Kaggle · GitHub

The notebook has every code cell, task and check of this lesson. It builds a small PatchCore by hand, then runs anomalib’s PatchCore and PaDiM on the 380 good training photos. It draws heat maps and compares the inspectors by AUROC and by Hamid’s cost per day. Its setup cells install the packages (with PyTorch and anomalib) and download the returns photos. About 45 minutes, of which about 3 minutes of computing on a laptop, and more on a free Colab or Kaggle CPU.

Colab and Kaggle open your own copy. They need a free Google or Kaggle account. Stuck? See the solution notebook.

Before you startWhat to know and to have readyAbout 45 minutes

What you need to know

What you need on your computer

Nothing for this lesson: the notebook runs in Colab, Kaggle or Jupyter. It downloads ResNet-18 (47 MB) and the wide ResNet-50 that anomalib uses (about 270 MB).

The notebook was tested with Python 3.12, PyTorch 2.15.0, timm 1.0.31 and anomalib 2.7.0, in Jupyter on macOS (laptop CPU).

Think again of the new worker who learned from a box of good tea lights. When she checks a returned packet, she does not look at the whole tray and ask “does this tray look normal?”. She looks at each part: this piece of wax, this wick, this edge. A tiny dark spot hardly changes the look of the tray. But that one piece of wax looks like no piece of wax she has ever seen.

Her memory is not a memory of whole trays. It is a memory of parts: hundreds of pieces of good wax, good wicks and good edges. She compares each part of a new candle with the parts she knows. If one part has no match, the candle is suspect, and she knows where to point.

The problem: the whole photo hides the spot

Section titled “The problem: the whole photo hides the spot”

In the last lesson, the anomaly score compared whole photos, and it was weak: an AUROC of 0.73 on the test photos. The median damage covers about 64 pixels of a 224 × 224 photo, about 0.13% of it. The photo’s embedding hardly moves, so the damaged photo still looks like its nearest good photo.

The fix is the worker’s habit: compare parts, not trays.

Cut each photo into a grid of small squares. Each square is a patch. For each patch, find the most similar patch in a memory of patches from good photos. A patch of normal wax has a close match. A patch of damage does not. The photo’s score is the distance of its worst patch: the patch that is furthest from every good patch.

This method is PatchCore (Roth and others, 2022). “Patch” because it compares patches. “Core” because its memory is a coreset: a small set of patches, chosen so that every kind of good patch has a close match in it. A memory of every patch of every good photo would be large and slow to search.

Three steps from left to right. One: a photo cut into a 4 by 4 grid of patches, one patch outlined in orange. Two: each patch is compared with a memory of patches of 380 good photos. Three: most patches have a distance of 0.2, near a good one; the odd patch has a distance of 0.9, far from all. Below: photo score is the largest patch distance; heat map is the distance of every patch, as a picture.
PatchCore in three steps: cut the photo into patches, find each patch's nearest good patch, and take the distance as how odd the patch is. The numbers are made up to show the idea.
Text description of the diagram

The diagram has three steps. Step one shows a photo cut into a grid of 4 by 4 patches; one patch is orange, the others blue. An arrow leads to step two, a grey box: a memory of patches of 380 good photos. An arrow leads to step three, two boxes. Most patches have a distance of 0.2, because a good patch is near them. The odd patch has a distance of 0.9, because it is far from all good patches. A line of text under the steps says: photo score = the largest patch distance. Heat map = the distance of every patch, shown as a picture.

Imagine that each patch is described by one number, its brightness, and that a photo has only four patches. The memory holds five good patches: 0.47, 0.49, 0.50, 0.53 and 0.55.

Patch Value Nearest good patch Distance
A (wax) 0.50 0.50 0.00
B (wax) 0.52 0.53 0.01
C (wick) 0.48 0.47 or 0.49 0.01
D (dark spot) 0.20 0.47 0.27

The photo’s score is the largest distance: 0.27, from patch D. The average distance is only (0.00 + 0.01 + 0.01 + 0.27) ÷ 4 = 0.07. One odd patch of four already shrinks to a quarter in the average. A real photo has 784 patches, so the average would hide a small spot even more. That is why PatchCore takes the largest distance.

A patch is not a square of raw pixels. Two pixel squares of the same wax can differ a lot when the light falls a little differently. PatchCore takes its patches from the middle layers of a pretrained network, such as ResNet. Convolution and patches in one lesson showed that each position in a middle layer describes one region of the photo. Its many numbers describe edges, textures and small shapes. The last layer would mix all positions together, and we would be back to one embedding for the whole photo.

In the notebook, the small PatchCore uses ResNet-18. A photo of 224 × 224 pixels gives a grid of 28 × 28 = 784 positions, and each position gets 384 numbers. Each patch also averages its neighbours, so a patch “sees” a little more than its own square.

p = patches("data/" + good_train["file"].iloc[0])
print(p.shape)
(784, 384)

The 380 good training photos give 297,920 patches. The small PatchCore keeps 20,000 of them at random as its memory. anomalib’s PatchCore chooses them with the coreset method instead.

297,920 good patches, 20,000 kept in the memory

Draw the distance of every patch on the 28 × 28 grid, and you get a heat map. It is bright where a patch is far from every good patch. It turns a score into a place that a person can look at.

Here are three damaged test candles. The top row shows each photo with its true damage circled, from the damage mask. The bottom row shows the heat map of anomalib’s PatchCore over the same photo.

Six pictures. Top row: three photos of four tea lights with an orange circle around the damage: extra wax at the top right candle, extra wax near the wick of the bottom right candle, and wax melted out at the left edge of the bottom left candle. Bottom row: the same photos covered by a purple heat map, with one bright yellow and orange area at the place of each circled damage.
PatchCore heat maps on three damaged test candles: extra wax in candle (R-33640732), extra wax in candle (R-a796ccf6), wax melted out of the candle (R-d17d264e). Photos: VisA dataset, Zou, Jeong, Pemula, Zhang and Dabeer (Amazon), CC BY 4.0, resized; circles and heat maps drawn by our code.
Text description of the diagram

Top row, three photos of four tea lights each, with an orange circle around the true damage. In the first photo, the damage is a lump of extra wax on the top right candle. In the second, extra wax lies near the wick of the bottom right candle. In the third, wax has melted out at the left edge of the bottom left candle. Bottom row: the same three photos under a heat map. It is dark purple where the patches look good, and bright yellow or orange where they do not. In each photo, the bright area sits on the circled damage. In the second photo, the bright area is weaker than in the first.

In these three photos, the bright area sits on the damage. They are the three best of the 18 damaged test photos. How good are heat maps in general?

  • With the small PatchCore of the notebook, the worst patch lies inside the true damage in 13 of the 18 damaged test photos. In the other 5, the photo may still score high, but the bright spot is elsewhere.
  • The course team compared anomalib’s heat maps with the true masks. Bright parts (the top 2% of all heat-map values) against the true damage give a mean Dice of only 0.175. Dice is the overlap measure of Measure from masks: 1.0 is the same shape, 0 no overlap.

So a heat map points to the area, not to the outline. Use it to show a person where to look first. Do not use it to measure the size of a damage: the masks of Module 5 do that.

anomalib has a second classic method for patches, PaDiM (Defard and others, 2021). PaDiM also takes patches from the middle layers. But it keeps no memory of patches. For each position of the grid, it stores the average good patch at that position and how much good patches vary there. A new patch is odd if it is far from the average of its own position, measured in units of that position’s normal variation.

Think of it as one bell curve per position, the normal-distribution shape of Essential Mathematics and Statistics for AI. PaDiM learns in seconds, because it only computes averages: 10 seconds on the 380 good photos, against 94 seconds for PatchCore with a wide ResNet-50. Its weakness is the positions. It works only if the same thing is always at the same place. Hamid’s photos are taken in a fixed tray, so the positions are roughly stable. But a packet placed a little to the left breaks the averages.

Larkfield: four inspectors on the same photos

Section titled “Larkfield: four inspectors on the same photos”

Priya runs the small PatchCore by hand, then anomalib’s PatchCore and PaDiM. anomalib reads the photos from folders, learns from the folder of good photos, and gives every new photo a score from 0 to 1 and a heat map:

model = Patchcore(backbone="wide_resnet50_2", coreset_sampling_ratio=0.01, visualizer=False)
folder = Folder(name="candles", root=work, normal_dir="train_good", abnormal_dir="train_damaged",
num_workers=0, train_batch_size=16, eval_batch_size=16, seed=0)
engine = Engine(accelerator="cpu", devices=1, logger=False, default_root_dir=work / "results",
enable_progress_bar=False)
engine.fit(model=model, datamodule=folder)

coreset_sampling_ratio=0.01 keeps 1% of the good patches in the memory. The damaged training photos do not teach the model. anomalib uses them only to scale its scores between 0 and 1.

The course’s recorded run, on the 218 test photos, with the lines chosen on validation:

Inspector AUROC Cost a day Learns in Per photo Size
PaDiM, ResNet-18 0.824 993.6 10 s 26.5 ms 47 MB
Small PatchCore (ResNet-18, random memory) 0.847 1,290.8 not timed 17.3 ms 78 MB
PatchCore, ResNet-18 (anomalib) 0.919 706.0 50 s 20.4 ms 47 MB
PatchCore, wide ResNet-50 (anomalib) 0.917 368.8 94 s 47.8 ms 267 MB

One row needs care. anomalib’s PatchCore with ResNet-18 (706.0 dollars a day) needs about 113 checks a day on the test photos (112.8), over Hamid’s budget of 100. So its cost is not reachable as it stands.

Cost per day on the 218 test photos (lines chosen on validation, the course's cost assumptions). Small PC is the small PatchCore; PC R-18 and PC WR-50 are anomalib's PatchCore with a ResNet-18 and a wide ResNet-50. No inspection costs 2,229.4.
Text description of the chart

A column chart with four columns. PaDiM with ResNet-18: 993.6 dollars a day. The small PatchCore: 1,290.8. anomalib’s PatchCore with ResNet-18: 706.0. anomalib’s PatchCore with a wide ResNet-50: 368.8.

Patches work. Every patch method beats the whole-photo score of the last lesson (0.73, 1,578.4 dollars a day). anomalib’s PatchCore with a wide ResNet-50 is the cheapest: 368.8 dollars a day, against 2,229.4 with no inspection.

Look at the two anomalib PatchCores. Their AUROC is almost the same (0.919 and 0.917), but one costs almost twice as much a day. Why?

AUROC counts all 3,600 pairs of one damaged and one good photo (18 × 200). The cost depends only on the few photos near the lines. With 18 damaged test photos, one damaged photo that passes adds 600 ÷ 218 × 45 = 124 dollars a day. A difference of two or three photos makes the gap. The same is true of the test set itself: it is small, so every cost here moves a lot with a few photos.

anomalib’s PatchCore and PaDiM do not give exactly the same scores at every run. The table above is the course’s recorded run, made once. Then we ran the notebook five more times on one laptop. PatchCore with a wide ResNet-50 gave an AUROC from 0.926 to 0.939, and a cost from 370.2 to 1,194.5 dollars a day. PaDiM gave from 0.797 to 0.841, and from 1,149.1 to 1,875.7 dollars a day. Same code, same photos, same splits. With so few damaged photos, a small change in the scores moves a photo across a line. Then the cost moves by hundreds of dollars. Treat the costs as a range, not as exact numbers, and use the recorded run when you compare with the course’s table.

anomalib’s PatchCore is the cheapest. Why does the project use the small one?

Amira, Larkfield’s developer, must ship the inspector in the image service of Module 8. anomalib needs PyTorch, Lightning and their many packages, and the wide ResNet-50 weighs 267 MB. The small PatchCore needs only the first layers of a ResNet-18. Amira can export them to ONNX, a format that a light library, ONNX Runtime, runs without PyTorch. Made smaller with 8-bit numbers, it is 2.9 MB, plus a memory of 15 MB. It needs only ONNX Runtime and NumPy, and it scores a photo in 15 ms. The price is quality: a test AUROC of 0.836 against 0.917.

That is a real trade-off, not a free choice. The better model exists, and the lesson Fast enough on a CPU measures what each choice costs in time and size. For every choice, write down the price in both directions.

When to use anomaly-based inspection, and what it costs

Section titled “When to use anomaly-based inspection, and what it costs”

Use it when the items look the same from photo to photo: the same product, the same tray, the same light. It fits when good items are easy to collect and damage is rare and varied. Hamid’s tea lights fit well.

Be careful when good items vary a lot. Many colours or shapes of the same product, or photos from many angles, make “normal” wide. A damage can then hide among normal differences. Do not use it alone when you must name the kind of damage: it says “odd here”, not “chunk of wax missing”.

What it costs: you store and update a memory of good patches, or their statistics. You run a backbone on every photo. And you need new good photos when “normal” changes. A new camera or new light makes every patch look odd: Module 8 shows it. And the score is not a decision. A line must turn it into pass or fail. The alert budget in Anomaly Detection: Find What Is Unusual explains how to set a line when only a few examples are labelled. The next lesson sets Hamid’s lines.

Guided practice: patches and anomalib in the notebook

Section titled “Guided practice: patches and anomalib in the notebook”
  1. Open the notebook from the box Do this lesson in a notebook at the top of this page.
  2. Run the setup cells. The install cell lists anomalib 2.7.0, and the data cell ends with Ready:.
  3. In section 1, run the cell. You see (784, 384).
  4. In section 2, run the memory cell. You see 297,920 good patches, 20,000 kept in the memory.
  5. In section 3, write your prediction, then run the heat-map cell. The brightest patches sit on the extra wax, top right.
  6. In section 4, run the scoring cell. You see small PatchCore, AUROC on test: 0.847.
  7. Run the next cell. Scores go from about 5 to 19, and a line at 0.5 fails all 200 good photos.
  8. In section 5, run the anomalib cell. It learns for about 2 minutes, then prints an AUROC between about 0.91 and 0.94.
  9. Run the check cell. It prints Check passed.
  10. In section 6, run PaDiM. It learns in seconds.
  11. In section 7, run the cost cell. Compare your three costs with the table above.

Your turn: the mean instead of the largest distance

Section titled “Your turn: the mean instead of the largest distance”

Change one thing: score each photo by the mean of its 784 patch distances instead of the largest one. The tiny example says that the mean hides a small spot. Before you run it, predict: will the AUROC on the test photos be higher or lower than 0.847? In the notebook, section 8, put the AUROC in mean_auc.

Show the answer
mean_auc = cp.roc_auc([d.mean() for d in small_test_dist], dte)
print(mean_auc)
0.875

Higher, which surprised us too. On these 18 damaged test photos, the mean does better than the largest distance (0.875 against 0.847). Five of them have wax melted out of the candle, a large damage that raises many patches at once. The tiny example is still right about one small spot. So a rule that sounds right must still be measured. And with 18 damaged photos, a difference of 0.03 could change with other photos. Compare such choices on the validation photos, never on the test photos.

Common mistake

A line at 0.5 on a distance

Symptom: Tomás puts the small PatchCore in front of Hamid’s team with the rule “fail when the score is above 0.5”. On the test photos, every candle fails. In the notebook:

print(f"scores from {small_test.min():.2f} to {small_test.max():.2f}")
print(cp.inspection_costs(small_test, dte, 0.5, 0.5))
scores from 5.61 to 19.27
{'items': 218, 'damaged': 18, 'missed': 0, 'good_failed': 200, 'checks': 0, 'checks_per_day': 0.0, 'damaged_sold_as_new_per_day': 0.0, 'good_marked_damaged_per_day': 550.5, 'cost_per_day': 15412.8, 'over_budget': False}

Cause: an anomaly score is a distance, not a probability. The small PatchCore’s scores go from about 5 to 19, so 0.5 is below every photo. anomalib scales its scores to 0–1, but that scale also depends on its data, so 0.5 means nothing special there either.

Fix: never reuse a line from another model or another scale. Choose the lines on the validation photos of this model, from the costs and the budget, as the next lesson does.

Consequence in AI work: all 200 good candles go to the outlet: 15,412.8 dollars a day, almost seven times the cost of no inspection. A line copied from a probability model or from a blog post can make an inspector worse than none.

Check your understanding

A photo has six patches with distances 0.05, 0.04, 0.06, 0.05, 0.03 and 0.61 to their nearest good patch. What is its PatchCore score? What would the mean be, and why does PatchCore use the largest distance?

Show the answer

The score is the largest distance, 0.61. The mean is (0.05 + 0.04 + 0.06 + 0.05 + 0.03 + 0.61) ÷ 6 = 0.84 ÷ 6 = 0.14. PatchCore uses the largest distance because one odd patch is enough to make an item suspect. A small damage covers only a few of the 784 patches of a real photo. The mean shrinks one odd patch by the number of patches. Still, measure the choice: on our 18 damaged test photos, the mean did better, because several damages were large.

Check your understanding

The heat map of a damaged candle is bright on the damage. Can Hamid use the bright area to measure how big the damage is? Explain with the course's numbers.

Show the answer

No. A heat map points to the area, not to the outline. Against the true masks, the bright parts of anomalib’s heat maps give a mean Dice of only 0.175. In the small PatchCore, the worst patch was inside the true damage in 13 of 18 photos. Use the heat map to show a person where to look first. To measure a damage, use a mask, as in Measure from masks in Module 5.

Check your understanding

Amira's service uses the small PatchCore (test AUROC 0.836), not anomalib's PatchCore with a wide ResNet-50 (0.917, 368.8 dollars a day). Give one reason for her choice and one cost of it.

Show the answer

Reason: the small PatchCore runs with ONNX Runtime and NumPy only. It is 2.9 MB plus a 15 MB memory and scores a photo in 15 ms on a CPU. anomalib needs PyTorch, Lightning and 267 MB of weights. Cost: it finds damage less well, so more damaged candles pass or more good candles fail. On the test photos, its float version costs 1,290.8 dollars a day against 368.8. The choice must be written down with both numbers, so it can be changed when the service can carry a bigger model.

Recap

  • PatchCore compares each patch of a photo with a memory of good patches. The photo’s score is its largest patch distance, so one small odd patch is enough.
  • Patches come from the middle layers of a pretrained network: 784 positions of 384 numbers with ResNet-18 at 224 pixels.
  • A heat map shows every patch distance. It points to the area (the worst patch was inside the damage in 13 of 18 photos) but not the outline (mean Dice 0.175).
  • PaDiM keeps one average and spread per position. It learns in 10 seconds, but needs items at the same place.
  • On the test photos, anomalib’s PatchCore with a wide ResNet-50 has an AUROC of 0.917 and costs 368.8 dollars a day. The service’s small PatchCore has 0.836. Costs move a lot with a few photos, and between runs.

You can now score every return and show where the damage is. But a score is not a decision. Next, you turn scores into pass, check and fail with Hamid, from the cost of each mistake and his 100 checks a day.

Next: Set the line with the warehouse