Skip to content
Français

Visual inspection with few defects

  • Case study 1
  • 120 min
  • Optional · 1 case study needed for the certificate

Where we are

In eight modules, we inspected Larkfield's returns with labels and without them. In this case study, we use real open inspection images. We compare a supervised classifier, PatchCore and a CLIP prompt, and choose thresholds from a budget.

What you will be able to do

Build a visual inspector for two kinds of circuit board from real open photos, with only 10 damaged photos of each. You compare four inspectors on the same photos and under one measure. You choose pass, check and fail lines from an inspection budget, test once, and write the decision for the quality lead. The result is a portfolio piece.

Case studyPortfolio piece

Licence
CC BY 4.0
Time
About 2 hours
Runs in
Colab, Kaggle or VS Code
Skills used
  • Inspection budget
  • Supervised against anomaly-based
  • CLIP zero-shot
  • Pass–check–fail lines
  • Heat maps
  • Cost per 1,000 items

Colab and Kaggle open your own copy. To use VS Code, download the notebook from GitHub. Stuck? See the solution notebook.

The notebook runs every step of this page, with the same code and the same output, plus three tasks with checks. Its setup cells install the packages, download the photos (37 MB), ResNet-18 (47 MB) and CLIP (605 MB). After the downloads, it runs in about 2 minutes on a laptop CPU.

Before you startWhat to know and to have readyAbout 120 minutes

What you need to know

This case study uses Module 6 most. These lessons matter:

What you need on your computer

Nothing for the notebook in Colab or Kaggle. To run it locally, use a virtual environment with Python 3.12 and Jupyter, as in Meet the Larkfield images. The notebook works in a new folder visa-boards. To start again, delete that folder. This case study does not use the larkvision project.

Tested on macOS with Python 3.12.13, PyTorch 2.15.0, timm 1.0.31, open_clip 4.2.1 and scikit-learn 1.9.1. We ran it in Jupyter from a fresh start. Colab, Kaggle, Windows and Linux are not tested.

Fernhill Circuits is a small electronics maker (a made-up company). It builds two kinds of circuit board: an ultrasonic distance sensor and a battery charger. A camera above the line takes one photo of every finished board.

Today, Ayumi, the quality lead, has one checker: a person who looks at boards by hand. He can check 100 boards in every 1,000. Everything else is shipped without a look. Some faulty boards reach customers, and they come back.

Ayumi put a price on each mistake. These are assumptions, her estimates, not measured facts:

  • A faulty board that is shipped costs 60 dollars: the customer’s return, a new board and the shipping.
  • A good board that is scrapped costs 15 dollars: its parts and the assembly time.
  • One check by the checker costs 0.50 dollars: one minute at 30 dollars an hour.

A faulty board shipped costs four times as much as a good board scrapped. Ayumi asks Mateus, a data scientist: “Can the camera tell my checker which boards to look at? We have very few photos of faulty boards.”

That is the problem of Module 6 in a new factory. In the course, Hamid’s returns desk had 25 damaged candle photos. Here, each board type has only 10.

You will build four inspectors. On this page, an inspector is a method, not a person: it gives each photo a score: high means “probably damaged”. Then two lines on the score sort each board into three zones, as in Set the line with the warehouse:

  • pass: the board is shipped;
  • check: the checker looks at it, and is always right;
  • fail: the board is scrapped.

You compare the inspectors by one measure: the cost per 1,000 boards, inside the budget of 100 checks per 1,000. You choose the method and the lines on validation photos, test once, and run the choice on new boards. At the end, you write the decision for Ayumi: which method, which lines, what the checker still checks, and what it costs.

The photos come from VisA, the Visual Anomaly dataset of Amazon. It has 12 kinds of object, each with about 500 to 1,000 good photos and 100 damaged photos. Each damage has a mask: a black-and-white picture of where the damage is. The course’s tea lights are VisA’s candle class. Here we use two other classes, the circuit boards that VisA calls pcb1 and pcb4. On this page they are board 1 and board 4.

Eight photos of circuit boards in two rows. Top row: a blue ultrasonic sensor board with two round sensors on grey cloth; one good, then three damaged ones with an orange circle: a bent pin, a melted spot near the pins, and a board whose sensor meshes are missing. Bottom row: a small blue charger board on black cloth; one good, then three damaged ones with an orange circle: an extra part, a missing part and a burnt spot.
Board 1 (top) and board 4 (bottom): one good photo, then three damaged ones with the damage circled. Board 1: bent, melt, missing. Board 4: extra, missing, burnt. Photos: VisA dataset, Zou, Jeong, Pemula, Zhang and Dabeer (Amazon), CC BY 4.0, resized; circles drawn by our code from the masks.
Text description of the diagram

Two rows of four photos. The top row shows board 1, an ultrasonic sensor board: blue, with two round silver sensors and four pins, on grey cloth. The first photo is a good board. It is turned half a turn compared with the others. The second has a bent pin, circled at the bottom left of the pins. The third has a small melted spot near the pins. The fourth has both sensor meshes missing: two flat white discs are left, and the circle covers the whole board. The bottom row shows board 4, a small charger board on black cloth, with a USB socket and a black chip. The first photo is good. The second has an extra part on the socket, the third a missing small part, and the fourth a burnt spot at the bottom.

Each board type has 800 photos, in five splits:

Split Per board type What it is for
train 200 good + 10 damaged what the inspectors learn from
val (validation) 100 good + 20 damaged choose the method and the lines
test 300 good + 40 damaged measure once, at the end
new 100 good + 10 damaged the end-to-end run on new boards
extra 20 damaged one experiment: what more labels would buy

The notebook’s setup cell downloads the sample and checks its checksum. Mateus loads the manifest. It has one row per photo:

boards = pd.read_csv("data/visa-pcb/manifest.csv")
PARTS = ["pcb1", "pcb4"]
SPLITS = ["train", "val", "test", "new", "extra"]
print(boards.groupby(["part", "split", "status"]).size().unstack(fill_value=0))

The output:

status damaged good
part split
pcb1 extra 20 0
new 10 100
test 40 300
train 10 200
val 20 100
pcb4 extra 20 0
new 10 100
test 40 300
train 10 200
val 20 100

Two checks before any model: is any photo in the sample twice, and do the photos of one board type have one size?

print("photos:", len(boards), "| different source files:", boards["source_sha256"].nunique())
print(boards.groupby("part")[["source_width", "source_height"]].agg(["min", "max"]))
sizes = {Image.open("data/visa-pcb/" + f).size for f in boards["file"]}
print("sizes in the sample:", sorted(sizes))
photos: 1600 | different source files: 1600
source_width source_height
min max min max
part
pcb1 1404 1404 1070 1070
pcb4 1358 1358 1104 1104
sizes in the sample: [(320, 244), (320, 260)]

What this tells us. Every photo comes from a different source file, so no photo is in two splits. VisA’s photos are large (about 1,400 × 1,100 pixels); the sample is 320 pixels wide, to keep the download small. The damaged photos have from one to seven damage types per board type. Several types can appear on one board, such as “melt; scratch”.

Be honest about what this data can show:

  • Easier: a fixed camera, fixed light, one board per photo, near the centre. A real line has new batches, dust and, one day, a new camera (as in Domain shift).
  • Harder: one test board in 8.5 is damaged. A real line has far fewer faulty boards. So the costs per 1,000 boards are higher here than they would be on a real line.
  • Few examples: 20 damaged validation photos per board type. Move one damaged photo to the wrong side of a line, and the validation cost changes by 1,000 ÷ 120 × 60 = 500 dollars per 1,000 boards. On the test photos, one photo moves it by 176 dollars. Small differences are noise.
  • One item is one photo of one board.
  • The answer is a zone: pass, check or fail.
  • The measure is the cost per 1,000 boards of one type, with Ayumi’s prices. The worst case is the damage type that passes most often.
  • The budget is 100 checks per 1,000 boards. Lines that need more checks are not allowed.
  • The lines are chosen on the validation photos only, separately for each board type, because the two boards look different.
  • The method is one for both board types: the one with the lowest mean validation cost. One method is simpler to run and to look after.
  • The test photos are opened once, after the choice.

The notebook turns Ayumi’s prices into the cost table of the course’s compare.py. Its key names come from the returns desk, so “damaged sold as new” here means “faulty board shipped”:

COSTS = {
"damaged_sold_as_new": 60.0, # a faulty board shipped: the customer's return, a new board, shipping
"good_marked_damaged": 15.0, # a good board scrapped: its parts and the assembly time
"check_by_a_person": 0.5, # one minute of the checker at 30 dollars an hour
"checks_per_day": 100, # the checker's budget: 100 checks per 1,000 boards
"returns_per_day": 1000, # costs are given per 1,000 boards
}

The function board_row scores one method on one board type, as compare.inspection_row does in Module 6. cp.choose_lines finds the cheapest pair of lines on validation inside the budget. cp.inspection_costs measures them on test. The notebook has its full code.

Three reference rows frame every result. Each board type has the same numbers, because each has 40 damaged boards in 340 test photos:

Reference Cost per 1,000 boards
No inspection: every faulty board is shipped 7,059
The checker looks at 100 random boards 6,403
A perfect inspector: it checks only the damaged boards 59

Every inspector gets the same photos and the same splits. They differ in what they learn from.

A flow diagram. At the top: the same photos of each board type. Four boxes below it, side by side: Baseline (difference from the average good board, learns from good photos), Supervised (ResNet-18 and logistic regression, learns from 200 good and 10 damaged), PatchCore (a memory of good patches, learns from good photos only), CLIP prompt (flawless against damaged, learns from nothing). Below, three steps in a column: validation, choose the method and the lines; test, once, the cost per 1,000 boards; new boards, pass, check or fail.
The case study's path: four inspectors score the same photos; the lines and the method are chosen on validation, the test is opened once, and the choice then meets new boards.
Text description of the diagram

At the top, a box: the same photos of each board type. Four lines lead down to four inspectors. The baseline compares each photo with the average good board and learns from good photos. The supervised inspector uses ResNet-18 and logistic regression and learns from 200 good and 10 damaged photos. PatchCore keeps a memory of good patches and learns from good photos only. The CLIP prompt compares “flawless” with “damaged” and learns from nothing. Below them, three boxes in a column. First, validation (100 good and 20 damaged photos): choose the method and the pass, check and fail lines, with the budget of 100 checks per 1,000 boards. Second, test (300 good and 40 damaged), opened once: the cost per 1,000 boards and the worst damage type. Third, new boards (100 good and 10 damaged), where the decision works: pass, check or fail.

A baseline: difference from the average good board

Section titled “A baseline: difference from the average good board”

Think of a worker who holds a perfect board next to each new one and looks for any difference. The baseline does that with numbers. It makes each photo small and grey (64 × 48 pixels). It takes the average of the 200 good training photos. Each photo’s score is its mean difference from that average. It needs no model.

Predict. Board 1 lies in about the same place in each photo (sometimes turned half a turn). Board 4 moves a little more. On which board will the average-board baseline do better?

pcb1 AUROC on test: 0.916
pcb4 AUROC on test: 0.531

AUROC is the chance that a damaged photo scores higher than a good one (Two ways to find damage); 0.5 is a coin. On board 1, the simple rule is strong: 0.916. On board 4, it is a coin. A small shift of the board changes many pixels, so the average of the good boards is a blur.

ResNet-18, the network of Modules 2 and 6, turns each photo into an embedding: 512 numbers that describe the whole photo. A logistic regression then learns good against damaged from the 200 good and 10 damaged training photos of each board type. class_weight="balanced" makes the 10 damaged photos count as much as the 200 good ones.

def supervised_scores(part, train_splits=("train",)):
"""Supervised: logistic regression on the ResNet-18 embeddings of the labelled training photos."""
rows = boards[boards["part"] == part]
tr = rows[rows["split"].isin(train_splits)]
X = np.stack([EMBEDDING[i] for i in tr["item_id"]])
model = LogisticRegression(C=1.0, max_iter=5000, class_weight="balanced").fit(X, tr["status"] == "damaged")
p = model.predict_proba(np.stack([EMBEDDING[i] for i in rows["item_id"]]))[:, 1]
return dict(zip(rows["item_id"], p.astype(float)))

The risk is the one of Module 6: 10 examples cannot show every kind of damage. A new kind may look “good” to it.

PatchCore learns from good photos only. One pass of ResNet-18 also gives 784 patches per photo: a 28 × 28 grid of small squares, each described by 384 numbers. The course’s small PatchCore keeps a memory of 20,000 patches of the 200 good training photos of each board type. A photo’s score is the distance from its oddest patch to the nearest good patch. The full idea is in Anomaly-based inspection.

It needs no damaged photo, so it can catch a kind of damage that nobody has seen. Its risk: anything new looks odd, also a good board in a new position.

CLIP (CLIP zero-shot) compares a photo with texts. Mateus writes two sets of texts. One has four “good” words (flawless, perfect, intact, good), the other four “bad” words (damaged, defective, broken, scratched). Each word goes into four templates, such as “a photo of a damaged circuit board.” The score is CLIP’s share of “damaged” between the two sets. No board photo is used to learn.

Predict. On the course’s candles, this kind of prompt gave an AUROC of 0.808. Will it do better or worse on circuit boards?

Every inspector now has a score for every photo. Mateus compares them on the validation photos only. For each method and board type, the cheapest lines inside the budget, and their cost:

METHODS = {"baseline": base, "supervised": sup, "patchcore": pc, "clip": clip}
rows = [board_row(name, part, scores[part]) | {"part": part} for name, scores in METHODS.items() for part in PARTS]
validation = pd.DataFrame([{"method": r["method"], "part": r["part"], "validation cost": r["validation_cost"]} for r in rows])
print(validation.pivot(index="method", columns="part", values="validation cost").assign(mean=lambda t: t.mean(axis=1)).round(0).astype(int))
part pcb1 pcb4 mean
method
baseline 1175 8425 4800
clip 9175 4292 6733
patchcore 2421 1300 1860
supervised 1788 671 1229

What this tells us. The supervised classifier has the lowest mean validation cost: 1,229 dollars per 1,000 boards. Mateus chooses it now, before any test photo is opened.

Look at the rows by board type too. On board 1 alone, the baseline is the cheapest on validation (1,175). A choice per board type would take the baseline for board 1. Keep that in mind for the test.

Common mistake

A line at 0.5 for CLIP

Symptom: CLIP gives a share of “damaged” from 0 to 1. A first idea: fail every board above 0.5. On board 1 that costs 7,059 dollars per 1,000 boards: exactly the cost of no inspection.

Cause: every CLIP score on board 1 lies between 0.008 and 0.056. A line at 0.5 passes every board. The score looks like a probability, but nothing made it one.

Fix: choose the lines from the scores of the validation photos, as cp.choose_lines does. Never take a line from the look of the numbers.

Now Mateus opens the test photos for every method, with the lines from validation:

method part AUROC test cost checks per 1,000 faulty shipped good scrapped worst type
baseline pcb1 0.916 2058.8 147.1 4 29 melt 0.864
baseline pcb4 0.531 9094.1 100.0 2 197 extra 0.833
supervised pcb1 0.956 1025.0 108.8 4 6 scratch 0.75
supervised pcb4 0.913 2239.7 67.6 6 26 scratch 0.714
patchcore pcb1 0.876 3666.2 97.1 7 54 melt 0.682
patchcore pcb4 0.975 1354.4 61.8 0 30 burnt 1.0
clip pcb1 0.416 6452.9 111.8 34 9 scratch 0.0
clip pcb4 0.678 5670.6 47.1 23 36 missing 0.231
mean test cost: {'baseline': 5576, 'clip': 6062, 'patchcore': 2510, 'supervised': 1632}

“Faulty shipped” and “good scrapped” are counts of the 340 test photos. The worst type is the damage type found least often, with the share found (not “pass”).

Test cost per 1,000 boards, mean of the two board types, with lines chosen on validation and Ayumi's prices (assumptions). None: no inspection. Random: the checker looks at 100 random boards in 1,000. Base: the average-board baseline. PC: PatchCore. Sup.: supervised. A perfect inspector would cost 59.
Text description of the chart

A column chart with six columns. No inspection: 7,059 dollars per 1,000 boards. 100 random checks: 6,403. CLIP prompt: 6,062. Baseline: 5,576. PatchCore: 2,510. Supervised: 1,632.

What this tells us.

  • The supervised classifier wins on test too: 1,632 dollars per 1,000 boards, against 7,059 with no inspection. That is 77% less.
  • PatchCore is second (2,510), and it is the best on board 4 (1,354): it found all 40 damaged test boards of board 4. On board 1, it scraps 54 good boards. We did not find out why; the heat maps of those good boards are the place to start.
  • The choice per board type would have been worse. The baseline was the cheapest on board 1’s validation photos. On board 1’s test photos it costs 2,059: twice the supervised classifier.
  • CLIP failed. Its AUROC on board 1 is 0.416: worse than a coin. It found 0 of 8 scratches. CLIP learned from photos with captions from the web; “a damaged circuit board” in a caption rarely means a melted spot of one millimetre.

The supervised classifier needs 108.8 checks per 1,000 board 1 photos on test: a little over the budget. The lines kept it inside on validation. Expect the number of checks to move, and plan some slack.

Mean cost per 1,000 boards of the two board types: on the validation photos, where the lines were chosen, and on the test photos. Sup.: supervised. PC: PatchCore.
Text description of the chart

A column chart with four columns. Supervised: 1,229 on validation, 1,632 on test. PatchCore: 1,860 on validation, 2,510 on test.

Both inspectors cost about a third more on test than on validation. That is normal: the lines were chosen to fit the validation photos, so validation looks too good. The test number is the one to report. With 20 damaged validation photos, a gap like this is expected; in Module 6, Hamid’s lines on 12 damaged photos held much worse.

How many damaged test boards of each type did the two best inspectors find (zone check or fail)?

Board Damage type Supervised PatchCore
1 bent 7 of 7 7 of 7
1 melt 19 of 22 15 of 22
1 missing 8 of 8 8 of 8
1 scratch 6 of 8 8 of 8
4 burnt 4 of 4 4 of 4
4 damage 6 of 7 7 of 7
4 dirt 15 of 16 16 of 16
4 extra 11 of 12 12 of 12
4 missing 10 of 13 13 of 13
4 scratch 5 of 7 7 of 7
4 wrong place 5 of 6 6 of 6

One board can have several damage types, so the rows add up to more than 40. The supervised classifier misses some boards of every type on board 4 except burnt: it has seen only 10 damaged board 4 photos. PatchCore misses melts on board 1: a melted spot is small and close in colour to the board. The two inspectors fail on different boards. That is a hint for the next step.

A heat map shows how odd each patch is. It tells the checker where to look on a board in the check zone.

Six pictures in two rows. Top row: three damaged boards with the damage circled in orange: board 1 with both sensor meshes missing, board 4 with an extra part near the top left, board 1 with a melted spot near the pins. Bottom row: the same photos under a heat map: very bright on the two missing meshes; a bright spot at the top left of board 4; a weak glow around the sensors of the third board, not on the circled melt.
Small PatchCore heat maps on three damaged test boards: board 1, missing (B-2c82e6fe); board 4, extra (B-33c1f46c); board 1, melt (B-7a8900cc), which PatchCore passed. The colours of each board type use the same scale: bright yellow is above PatchCore's check line. Photos: VisA dataset, Zou, Jeong, Pemula, Zhang and Dabeer (Amazon), CC BY 4.0, resized; circles and heat maps drawn by our code.
Text description of the diagram

Top row: three photos of damaged boards with an orange circle around the true damage. First, board 1 with both sensor meshes missing; the circle covers the whole board. Second, board 4 with an extra part near the top left corner, small circle. Third, board 1 with a melted spot near the pins. Bottom row: the same photos under a heat map, dark purple where patches look good and orange to yellow where they look odd. First: two very bright yellow discs exactly on the missing meshes. Second: an orange spot at the top left, on the extra part, and a weak glow elsewhere. Third: a weak orange glow around the two sensors and the edges, and nothing bright on the melted spot. This board scored under the check line and passed.

On the first two boards, the bright patches sit on the damage. On the third, PatchCore looks at the sensors, not at the melt, and the board passes.

How good are the heat maps in general? We compared the bright parts (the top 2% of the heat-map values) with the true masks of the 80 damaged test boards. The mean Dice is 0.182: 0.101 on board 1 and 0.263 on board 4. Dice is the overlap of two shapes: 1 is the same shape, 0 no overlap (Measure from masks). The course’s candles gave 0.175. A heat map points to an area. It does not draw the outline of the damage.

Mateus writes to Ayumi in her words:

  • Use the supervised inspector, one per board type, with these lines. Board 1: check from 0.352, fail from 0.592. Board 4: check from 0.112, fail from 0.202.
  • Boards in the check zone go to the checker, about 70 to 110 per 1,000 boards. Show him the PatchCore heat map with each board, to tell him where to look.
  • The expected cost is about 1,632 dollars per 1,000 boards, against 7,059 today with no inspection. This is at the test set’s share of faulty boards (one in 8.5). With fewer faulty boards, both numbers fall.
  • Still checked by a person: every board in the check zone, plus a small random sample of passed boards each week. The sample finds the faulty boards that the supervised inspector passes, and each one becomes a new label.
  • Label every board the checker looks at. Set the lines again when each board type has 30 damaged photos (see below).

And what not to use it for:

  • Not for a new board type or a new camera without new photos and new lines. The scores move, as in Domain shift.
  • Not as proof that a shipped board is good. It misses about 1 faulty board in 8 on the test photos.

The decision now meets 110 new boards of each type: 100 good and 10 damaged, photos that no step has seen. Each board goes to pass, check or fail with the lines above:

for part in PARTS:
r = next(x for x in rows if x["method"] == "supervised" and x["part"] == part)
new = boards[(boards["part"] == part) & (boards["split"] == "new")]
zone = cp.zones([sup[part][i] for i in new["item_id"]], *r["lines"])
print(part, f"lines {r['lines'][0]:.3f} and {r['lines'][1]:.3f}:",
pd.crosstab(new["status"].values, zone).to_dict("index"))
pcb1 lines 0.352 and 0.592: {'damaged': {'check': 4, 'fail': 3, 'pass': 3}, 'good': {'check': 10, 'fail': 2, 'pass': 88}}
pcb4 lines 0.112 and 0.202: {'damaged': {'check': 1, 'fail': 7, 'pass': 2}, 'good': {'check': 6, 'fail': 7, 'pass': 87}}

What this tells us.

  • The inspector stopped 15 of the 20 damaged new boards (check or fail). 5 passed: on board 1 a scratch, a missing part and a melt; on board 4 two missing parts.
  • Board 1 sent 14 boards in 110 to the checker: 127 per 1,000, over the budget of 100. On test it was 108.8. The checks a day are not fixed; they move with each batch.
  • Board 4 scrapped 7 good boards in 110. Each costs 15 dollars, and Ayumi may want the checker to look at some failed boards too.
  • 110 boards are a small batch. These are a first look at the decision at work, not a new measurement.

Ayumi asks two questions. First: what if the checker is ill, and only 50 checks per 1,000 are possible? The lines move to stay inside the budget. The supervised inspector’s test cost rises from 1,632 to 2,033 dollars per 1,000 boards.

Second: what if her team labels 20 more damaged photos of each board type? The notebook trains the supervised classifier on 30 damaged photos instead of 10, chooses the lines on validation again, and measures on test. This is an experiment to plan the next step, not a new choice.

Test cost per 1,000 boards of the supervised inspector, mean of the two board types, trained on 10 or on 30 damaged photos per board type. Lines chosen on validation each time.
Text description of the chart

A column chart with two columns. 10 damaged training photos: 1,632 dollars per 1,000 boards. 30 damaged training photos: 784.

Twenty more labels per board type halve the cost: from 1,632 to 784 dollars per 1,000 boards (board 1: 808.8, board 4: 758.8). That is the strongest result of this case study. With few damaged photos, the next labels can be worth more than a new model. They cost little if the checker records what he finds.

  • Compared with the course’s candles. On Hamid’s tea lights, anomalib’s PatchCore with a wide ResNet-50 was the cheapest. The supervised ResNet-18 had an AUROC of 0.914 there. Here the supervised classifier wins with 0.956 and 0.913, and the small PatchCore is strong only on board 4 (0.975). The best family depends on the product. Measure it on your own photos.
  • CLIP prompts are not inspectors for small industrial damage: 0.416 and 0.678 here, against 0.808 on the candles. Use CLIP to sort products, not to find a melt.
  • A simple baseline can be strong where the product sits in the same place (0.916 on board 1). It costs nothing to try, and it tells you what a model must beat.
  • The lines need slack. Checks on test and on new boards went above the budget on board 1.
  • Not tested here: anomalib’s PatchCore with a wide ResNet-50 (better on the candles; 267 MB of weights). A larger photo size: 320 pixels wide is small for a melt of a few millimetres. A mix of the supervised and PatchCore scores. Each one is a good next experiment.
  • Speed: on a laptop CPU, ResNet-18 read the 1,600 photos in about 27 seconds, including loading the model: about 17 ms a photo. CLIP took about 28 ms a photo in the same way. Both are fast enough for a line that makes a board every few seconds.
  • The licence. VisA is CC BY 4.0. If you share your work, credit the authors and say what you changed, as the dataset card does. Do not use MVTec AD, MPDD or KolektorSDD2 for commercial work: their licences forbid it.
  • The model weights. ResNet-18’s weights are Apache-2.0, but they learned from ImageNet. As When not to build it explains, “Apache-2.0 weights” does not settle product use: ask your legal team before you ship them.
  • The costs are assumptions. Show Ayumi’s prices next to every cost, and change them when the real numbers come in. The lines move with them.
  • People stay in the loop. The checker decides the boards in the check zone, and someone samples the passed boards. In a medical or safety product, a missed faulty board costs far more than 60 dollars. There, set the lines with far more damaged photos and much more care.
  • No people in the photos. This set shows only boards. A real line camera may see workers’ hands or faces; blur them by default, as in People in images.

This case study shows an employer that you can take few labelled defects to a decision with a price. Make it yours with one change: another VisA class (for example pcb2 or capsules), another budget, or the mix of the supervised and PatchCore scores. Then make it public, and easy to read in five minutes.

What to put in the repository:

  • README.md (template below).
  • The notebook, run from top to bottom, with its outputs.
  • The comparison table as a CSV: method, board type, AUROC, validation cost, test cost, checks per 1,000, worst damage type.
  • MODEL_CARD.md: the intended use (sort boards into pass, check and fail) and what not to use it for (new board types or cameras without new lines). Add the data, the lines, the cost assumptions and the limits.
  • The data’s source and licence with the attribution. CC BY 4.0 allows you to share the sample, but a link and the checksum are enough.

A README template:

# Visual inspection of circuit boards with 10 damaged photos
## Problem
An electronics maker can check 100 boards in 1,000 by hand. A faulty board shipped
costs 60 dollars, a good board scrapped 15, a check 0.50 (assumptions).
## Data and licence
VisA (Zou et al., Amazon), classes pcb1 and pcb4, CC BY 4.0. Fixed sample:
per board type 200+10 train, 100+20 validation, 300+40 test, 100+10 new.
## Approach
Four inspectors on the same photos: difference from the average good board,
ResNet-18 + logistic regression (10 damaged photos), PatchCore (good photos only),
CLIP prompt (no labels). Pass-check-fail lines chosen on validation inside the
budget; test used once.
## Results (test set, used once)
Supervised 1,632 dollars per 1,000 boards (no inspection 7,059); PatchCore 2,510;
CLIP 6,062. With 30 damaged photos: 784.
## Limits
One fixed camera; one test board in 8.5 is faulty, far more than on a
real line; 20 damaged validation photos per board type; costs are assumptions.

How to describe it in an interview: “I compared a supervised classifier, an anomaly detector and a zero-shot CLIP prompt for circuit-board inspection with only 10 damaged photos per board. I chose pass, check and fail lines from an inspection budget. On a held-out test, the chosen inspector cut the expected cost by 77% against no inspection. Twenty more labelled defects halved the cost again, so the next step is labelling, not a bigger model.”

Go further:

  • Average the supervised and PatchCore scores (after scaling each to its validation range). Predict first: will it beat both?
  • Repeat the comparison at 448 pixels instead of 224, and measure the time per photo.
  • Run anomalib’s PatchCore with a wide ResNet-50, as in Anomaly-based inspection, and add its row.
  • Change the share of faulty boards to 1% by weighting the test photos, and see how the lines and the costs move.

Check your understanding

On board 1's validation photos, the average-board baseline was the cheapest method (1,175 against 1,788 for the supervised classifier). Mateus still chose the supervised classifier for both boards. On test, the baseline cost 2,059 and the supervised classifier 1,025. Explain why the validation ranking was not reliable here.

Show the answer

Board 1’s validation set has only 20 damaged photos. Move one damaged photo to the wrong side of a line, and the validation cost changes by 1,000 ÷ 120 × 60 = 500 dollars per 1,000 boards. So a gap of about 600 dollars is about one photo: noise. The lines are also fitted to these same photos, so every validation cost looks too good. One method for both boards rests on 40 damaged photos instead of 20. Over both, the supervised classifier was clearly cheaper (1,229 against 4,800). The test confirmed it. The lesson: with few damaged photos, choose with as many photos as you can, and do not trust small differences.

Check your understanding

CLIP's prompt reached an AUROC of 0.808 on the course's candles but 0.416 on board 1. Tomás says CLIP is broken. What do you answer, and what would you use CLIP for in this factory?

Show the answer

CLIP is not broken: it does what it learned. It learned from web photos and their captions. Captions describe whole objects (“a circuit board”), and rarely a small melt or scratch. Large candle damage, such as wax melted out, may be closer to what captions say than a bent pin is. So a prompt finds large, visible damage at best. On board 1 it found 0 of 8 scratches and 1 of 22 melts. In this factory, use CLIP to sort which product is in a photo, or to search the photo archive, not to find small damage. For damage, use labelled photos or a model of the good boards, and measure it.

Check your understanding

Ayumi can either pay for a second checker (100 more checks per 1,000 boards) or ask the first checker to record what he finds, so that each board type has 30 damaged photos in a few weeks. Which do you advise first, with the numbers of this case study?

Show the answer

Labelling first. With 30 damaged photos per board type, the test cost of the supervised inspector fell from 1,632 to 784 dollars per 1,000 boards, about half. The checker already looks at the boards in the check zone, so recording what he finds costs little. More checks also help. A budget of 200 checks per 1,000 cut the cost on board 1 from 1,025 to 556. But checks cost a salary every day. The labels are bought once. After the labels, set the lines again on validation, test once, and then decide whether a second checker still pays.

Recap

  • You built four inspectors for two circuit boards from real open photos (VisA, CC BY 4.0), with only 10 damaged photos per board type. You compared them under one measure: the cost per 1,000 boards inside a budget of 100 checks.
  • On validation, the supervised classifier was the cheapest (1,229). On test, once, it cost 1,632 dollars per 1,000 boards, against 7,059 with no inspection. PatchCore was second (2,510) and the best on board 4.
  • The average-board baseline was strong only where the board sits still (AUROC 0.916 on board 1, 0.531 on board 4). The CLIP prompt failed on small damage (0.416 and 0.678).
  • Heat maps point to the area, not the outline (Dice 0.182). Use them to tell the checker where to look.
  • On new boards, 15 of 20 faulty boards were stopped, and the checks went over the budget on board 1. Plan slack.
  • 20 more damaged photos per board type halved the cost (784): with few defects, labels are worth more than a bigger model.

Next, a different kind of answer: which product, or which breed of pet, is in a photo, with many classes that look alike.

Next: Classify products and pets from photos