Skip to content
Français

Set the line with the warehouse

  • Lesson 22
  • 40 min
  • Module 6: Inspect for defects

Where we are

In the last lesson, PatchCore gave every returned candle an anomaly score and a heat map. A score is not a decision: Hamid must know what to do with each item. In this lesson, we turn scores into pass, check and fail with Hamid, from the cost of each mistake and his 100 checks a day, the way his team already sorts returns into three bins.

What you will be able to do

Turn an inspector’s scores into decisions that Hamid’s team can carry out: back on sale, checked by a person, or sent to the outlet. The best lines depend on the cost of each mistake and on the team’s checks a day. So they are set with the warehouse, not by the data scientist alone.

NotebookDo this lesson in a notebookColab · Kaggle · GitHub

The notebook has every code cell, task and check of this lesson. It works on the recorded scores of anomalib’s PatchCore, so it downloads no model and runs in seconds. It sorts ten returns by hand and chooses the lines for 100 and for 50 checks a day. About 30 minutes.

Colab and Kaggle open your own copy. They need a free Google or Kaggle account. Stuck? See the solution notebook.

Before you startWhat to know and to have readyAbout 40 minutes

What you need to know

What you need on your computer (for the project steps; the notebook needs none of it)

  • The labs repository with the course’s data, from Meet the Larkfield images.
  • Your larkvision project at the end of Module 5 (snapshot end-of-m05), beside nextia-learning-labs, with its .venv environment. If vision masks does not run, copy the snapshot end-of-m05 again, as in Meet the Larkfield images.

Tested on macOS with Python 3.12.13, NumPy 2.5.4 and pytest 9.2.0. The Windows commands are written but were not run.

Hamid’s team already sorts returns in a simple way. A worker looks at a returned candle. If it is clearly fine, it goes back to the shelf. If it is clearly broken, it goes to the outlet bin. If she is not sure, she puts it in a third bin. A senior worker looks at that bin at the end of the day.

The third bin is the clever part: easy items go fast, and hard ones get more care. But the senior worker can look at only so many items a day.

Our inspector gives a score, not a bin. In this lesson, we draw the borders of the three bins on the score.

With one line, every item either passes or fails. Nobody checks anything. Then every mistake of the inspector reaches the customer or the outlet.

With PatchCore’s scores and no checks at all, the best single line on the validation photos is at 0.4106. On the test photos, it sells 5.5 damaged items a day as new and sends 74.3 good items a day to the outlet:

patchcore_wide_resnet50: lines chosen on the validation photos for 0 checks a day: check from 0.4106, fail from 0.4106
Test (218 photos, 18 damaged; AUROC 0.917), per day at 600 returns: 0 checks, 5.5 damaged sold as new, 74.3 good items lost; cost 2328 dollars a day

That is 2,328 dollars a day: more than no inspection at all, which costs 2,229.4 dollars a day on the same photos. Many good candles score higher than the lowest damaged ones. The scores overlap, and the overlap needs a person.

Use two lines on the score. Below the check line, the item passes and goes back on sale as new. At or above the fail line, the item fails and goes to the outlet. Between the lines, a person checks it. These three ranges are the pass–check–fail zones.

A score axis from low score on the left to high score on the right, cut by a dashed check line and a solid fail line into three zones. Pass, in green: back on sale as new; mistake: a damaged item sold as new costs 45 dollars. Check, in grey: a person looks; each check costs 0.50 dollars, at most 100 a day. Fail, in orange: to the outlet; mistake: a good item lost costs 28 dollars.
Two lines cut the score into three zones. Each zone has its own action and its own possible mistake, with the course's cost assumptions.
Text description of the diagram

A horizontal arrow goes from “low score” on the left to “high score” on the right. Under it are three boxes in a row. The first box, green, is “PASS”: the item goes back on sale as new. Its mistake is a damaged item sold as new, which costs 45 dollars. A dashed line, the check line, leads to the second box, grey, “CHECK”: a person looks. Each check costs 0.50 dollars, and the team can do at most 100 a day. A solid line, the fail line, leads to the third box, orange, “FAIL”: the item goes to the outlet. Its mistake is a good item lost, which costs 28 dollars. On a phone, the three zones are stacked from top to bottom.

A person who checks is always right, in this course’s model. So the check zone has no mistakes, only a cost: 0.50 dollars a check, one minute of a worker at 30 dollars an hour. A damaged item in the pass zone is sold as new: 45 dollars. A good item in the fail zone is lost: 28 dollars.

Here are ten returns with their scores. The lines are: check from 0.4, fail from 0.6.

Score 0.12 0.25 0.31 0.38 0.45 0.52 0.58 0.63 0.71 0.90
Damaged? no no yes no no yes no no yes yes
Zone pass pass pass pass check check check fail fail fail

The damaged item at 0.31 passes: it is sold as new. The good item at 0.63 fails: it is lost. Three items are checked. Each of the ten items stands for 60 of Hamid’s 600 returns a day, so a day costs:

Part Per day Price Dollars a day
Damaged sold as new 1 × 60 = 60 45 2,700
Good items lost 1 × 60 = 60 28 1,680
Checks 3 × 60 = 180 0.50 90
Total 4,470

And 180 checks a day is over Hamid’s budget of 100. The notebook computes the same with the course’s helper:

print(cp.inspection_costs(tiny["score"], tiny["damaged"], 0.4, 0.6))
{'items': 10, 'damaged': 4, 'missed': 1, 'good_failed': 1, 'checks': 3, 'checks_per_day': 180.0, 'damaged_sold_as_new_per_day': 60.0, 'good_marked_damaged_per_day': 60.0, 'cost_per_day': 4470.0, 'over_budget': True}

A lower check line catches more damaged items but sends more good items to a person. A lower fail line saves checks but loses more good items. choose_lines in the course’s helper takes 201 points along the validation scores and tries every pair of them as the two lines. It keeps the cheapest pair whose checks stay inside the budget.

We choose on the validation photos and measure on the test photos. Lines chosen on a set fit that set’s accidents, so their cost on that set is too good.

Larkfield: Hamid’s lines for 100 checks a day

Section titled “Larkfield: Hamid’s lines for 100 checks a day”

Priya and Hamid choose the lines for PatchCore with a wide ResNet-50 in the notebook:

lines = cp.choose_lines(sv, dv)
on_test = cp.inspection_costs(st, dt, lines["check_line"], lines["fail_line"])
check from 0.4106, fail from 0.6441
validation: {'checks_per_day': 75.0, 'damaged_sold_as_new_per_day': 0.0, 'good_marked_damaged_per_day': 0.0, 'cost_per_day': 37.5}
test: {'checks_per_day': 88.1, 'damaged_sold_as_new_per_day': 5.5, 'good_marked_damaged_per_day': 2.8, 'cost_per_day': 368.8}

On the validation photos, the lines are perfect: no damaged item passes, no good item fails, and 75 checks a day cost 37.5 dollars. On the test photos, two damaged candles score below the check line: 0.244 and 0.339, both with a damaged corner of the packet. And one good candle scores above the fail line. The cost is 368.8 dollars a day: ten times the validation cost, but still far below no inspection.

Now try the lines yourself in the explorer.

Explore

Pass, check and fail zones for Hamid's returns

passcheckfail0200.171 to 0.192: 2 good photos0.193 to 0.214: 6 good photos0.214 to 0.235: 25 good photos0.235 to 0.256: 28 good photos0.256 to 0.277: 31 good photos0.277 to 0.299: 21 good photos0.299 to 0.320: 13 good photos0.320 to 0.341: 19 good photos0.341 to 0.362: 16 good photos0.362 to 0.384: 6 good photos0.384 to 0.405: 4 good photos0.405 to 0.426: 5 good photos0.426 to 0.448: 5 good photos0.448 to 0.469: 3 good photos0.469 to 0.490: 5 good photos0.490 to 0.511: 1 good photos0.511 to 0.532: 3 good photos0.532 to 0.554: 1 good photos0.554 to 0.575: 1 good photos0.575 to 0.596: 1 good photos0.617 to 0.639: 2 good photos0.639 to 0.660: 2 good photosGood returns (200)R-a9048b86: damaged corner of packaging; different colour spot, score 0.244, passR-74669eb6: damaged corner of packaging, score 0.339, passR-c3b7e506: damaged corner of packaging, score 0.413, checkR-344c8b55: different colour spot, score 0.424, checkR-de7e819e: different colour spot, score 0.434, checkR-a796ccf6: extra wax in candle, score 0.557, checkR-0d4c03e5: wax melded out of the candle, score 0.560, checkR-15b594e6: different colour spot, score 0.629, checkR-d17d264e: wax melded out of the candle, score 0.671, failR-5918b873: chunk of wax missing, score 0.799, failR-33640732: extra wax in candle, score 0.845, failR-800400d7: wax melded out of the candle, score 0.851, failR-5cf1ce29: foreign particals on candle; wax melded out of the candle, score 0.968, failR-8cb58274: damaged corner of packaging; different colour spot; other, score 1.000, failR-1cc8cc10: weird candle wick, score 1.000, failR-31ebdb39: wax melded out of the candle, score 1.000, failR-aa4a48cb: foreign particals on candle, score 1.000, failR-ef0b6f29: extra wax in candle, score 1.000, failDamaged returns (18): one dot each0.20.40.60.81Anomaly score (higher = less like a good candle)check from 0.411fail from 0.644
passcheckfail025500.150 to 0.184: 1 good photos0.184 to 0.218: 10 good photos0.218 to 0.252: 45 good photos0.252 to 0.286: 48 good photos0.286 to 0.320: 22 good photos0.320 to 0.354: 29 good photos0.354 to 0.388: 14 good photos0.388 to 0.422: 6 good photos0.422 to 0.456: 7 good photos0.456 to 0.490: 7 good photos0.490 to 0.524: 3 good photos0.524 to 0.558: 3 good photos0.558 to 0.592: 1 good photos0.592 to 0.626: 1 good photos0.626 to 0.660: 3 good photosGood returns (200)R-a9048b86: damaged corner of packaging; different colour spot, score 0.244, passR-74669eb6: damaged corner of packaging, score 0.339, passR-c3b7e506: damaged corner of packaging, score 0.413, checkR-344c8b55: different colour spot, score 0.424, checkR-de7e819e: different colour spot, score 0.434, checkR-a796ccf6: extra wax in candle, score 0.557, checkR-0d4c03e5: wax melded out of the candle, score 0.560, checkR-15b594e6: different colour spot, score 0.629, checkR-d17d264e: wax melded out of the candle, score 0.671, failR-5918b873: chunk of wax missing, score 0.799, failR-33640732: extra wax in candle, score 0.845, failR-800400d7: wax melded out of the candle, score 0.851, failR-5cf1ce29: foreign particals on candle; wax melded out of the candle, score 0.968, failR-8cb58274: damaged corner of packaging; different colour spot; other, score 1.000, failR-1cc8cc10: weird candle wick, score 1.000, failR-31ebdb39: wax melded out of the candle, score 1.000, failR-aa4a48cb: foreign particals on candle, score 1.000, failR-ef0b6f29: extra wax in candle, score 1.000, failDamaged returns (18): one dot each0.250.50.751Anomaly score (higher = less like a good candle)check from 0.411fail from 0.644
validation
112 photos
test
218 photos
Checks a day (budget 100)7588.1
Damaged items sold as new a day05.5
Good items lost a day02.8
Damaged photos that pass0 of 122 of 18
Cost a day (assumptions)37.5368.8

On the test photos, these lines send 88.1 returns a day to a person (budget 100), sell 5.5 damaged items a day as new and send 2.8 good items a day to the outlet: 368.8 dollars a day. With no inspection, the same photos cost 2,229.4 dollars a day. These are the cheapest lines on the validation photos for 100 checks a day.

The lines are chosen on the validation photos and measured on the test photos. Costs are the course’s assumptions: a damaged item sold as new 45 dollars, a good item lost 28 dollars, a check 0.50 dollars; a person who checks is always right. Per day at 600 returns.

The recorded scores of anomalib's PatchCore (wide ResNet-50) and of the supervised ResNet-18 on the 112 validation and 218 test returns. Pick a budget to choose the cheapest lines on validation, or move each line yourself.
Text description of the explorer

The explorer shows two panels on one score axis from 0.15 to 1. The top panel is a histogram of the good returns; the bottom panel shows each damaged return as one dot. Behind them, three coloured zones: pass (green) left of the check line, check (grey) between the lines, and fail (orange) right of the fail line. A damaged dot in the pass zone has a dark ring: it is sold as new. Controls: the inspector (PatchCore or the supervised ResNet-18), the budget (25 to 150 checks a day), a slider for each line, and the photos in the picture. A table has one column for the validation photos and one for the test photos. Its rows are the checks a day, the damaged items sold as new, the good items lost, the damaged photos that pass, and the cost a day. At the start, PatchCore with 100 checks: check from 0.411, fail from 0.644; validation 37.5 dollars a day, test 368.8. With 50 checks: fail from 0.486; test 1,272.9 dollars a day, with 35.8 good items lost a day.

The cost goes up a lot: from 368.8 to 1,272.9 dollars a day on the test photos. The project command gives the same lines:

patchcore_wide_resnet50: lines chosen on the validation photos for 50 checks a day: check from 0.4106, fail from 0.4859
Test (218 photos, 18 damaged; AUROC 0.917), per day at 600 returns: 47 checks, 5.5 damaged sold as new, 35.8 good items lost; cost 1273 dollars a day

The check line stays at 0.4106, because moving it up would sell more damaged candles at 45 dollars each. So the only way to halve the checks is to bring the fail line down, from 0.6441 to 0.4859, into the good candles. Good items lost go from 2.8 to 35.8 a day. The checks were cheap (0.50 dollars) and each one protected a 28-dollar candle.

Hamid now has a number for his manager: each check that the team gives up costs about 22 dollars a day: (1,272.9 − 368.8) ÷ (88.1 − 46.8) = 904.1 ÷ 41.3.

The honest surprise: the line did not hold

Section titled “The honest surprise: the line did not hold”

Why did the test cost ten times the validation cost? Look at the damaged photos that set the line.

damaged validation scores: [0.412 0.439 0.445 0.478 0.481 0.615 0.62 0.682 0.713 0.763 0.888 0.892]
damaged test scores: [0.244 0.339 0.413 0.424 0.434 0.557 0.56 0.629 0.671 0.799 0.845 0.851
0.968 1. 1. 1. 1. 1. ]
check line 0.4106

The validation set has only 12 damaged photos. The check line sits just below the lowest one, 0.412. In the notebook, Priya removes that one photo and chooses again:

without R-46ef3aca: check from 0.4219, fail from 0.6458
test cost: 488.5

One photo less, and the test cost moves from 368.8 to 488.5 dollars a day. A line set on 12 examples is closer to a guess than to a measurement.

The course team kept a hidden test set: 135 returns, 15 of them damaged, not in labs and never used for any choice. It plays the role of next month’s returns. They measured the chosen lines on it once:

Cost per day of anomalib's PatchCore (wide ResNet-50) with the lines chosen on validation for 100 checks a day: on validation, on test, and once on the hidden test. No inspection on test, for comparison.
Text description of the chart

A column chart with four columns. Validation: 37.5 dollars a day. Test: 368.8. Hidden test: 1,468.9. No inspection (test photos): 2,229.4.

On the hidden test, the same lines cost 1,468.9 dollars a day. The AUROC was still good, 0.891, but 6 of the 15 damaged candles passed. Their scores were between 0.279 and 0.363, below the check line. Four of the six were “different colour spot”: the small spots that a heat map can miss. The inspector still saves money: with no inspection, these photos would cost 3,000 dollars a day. But the day costs four times what the test set promised.

The sets were small: 18 and 12 damaged photos cannot show every kind of damage. Evaluate without a full answer key in the anomaly detection course meets the same limit with payments.

Hamid does not throw the inspector away. He changes how he uses it:

  • Keep a person on a sample of passed items. On the test photos, the lines used about 88 of his 100 checks. He gives the other 12 or so to random items from the pass zone.
  • Label every checked item. Each check produces a labelled photo for free. When the validation set has many more damaged photos, Priya sets the lines again.
  • Watch the zone shares. If the share of fail or check moves far from its validation level, something changed, such as the light or the camera. Module 8 turns this into an alarm.
  • Write the uncertainty down. The report says “368.8 dollars a day on test and 1,468.9 on the hidden test”, not “369 dollars a day”.

Your larkvision project gets the zones as a command. It reads the recorded scores in data/predictions/, then chooses and measures the lines as the notebook does.

The heart of the command:

scores = json.loads((Path(d) / f"predictions/returns_{args.method}.json").read_text())
costs = D.costs(d)
if args.budget is not None:
costs = dict(costs, checks_per_day=args.budget)
val, test = D.returns(d, "val"), D.returns(d, "test")
sv = [scores[r["item_id"]] for r in val]
dv = [r["status"] == "damaged" for r in val]
lines = choose_lines(sv, dv, costs)
st = [scores[r["item_id"]] for r in test]
dt = [r["status"] == "damaged" for r in test]
c = inspection_costs(st, dt, lines["check_line"], lines["fail_line"], costs)

This lesson adds one command, vision zones, and its three tests from the snapshot end-of-m06.

  1. Open a terminal and go to your larkvision folder. Activate the virtual environment: source .venv/bin/activate on macOS and Linux, .venv\Scripts\Activate.ps1 on Windows.

  2. If this is a new terminal, set LARKVISION_DATA again. It lives only in the terminal where you set it:

    Terminal window
    export LARKVISION_DATA="$(cd ../nextia-learning-labs/C42/data && pwd)"
  3. Copy the new command, its tests and the README from the snapshot:

    Terminal window
    S=../nextia-learning-labs/C42/snapshots/end-of-m06
    cp "$S/larkvision/commands/zones.py" larkvision/commands/
    cp "$S/tests/test_zones.py" tests/
    cp "$S/README.md" .
  4. Run the tests:

    Terminal window
    python -m pytest -q

    The last line is 22 passed. If it says 19 passed, the file tests/test_zones.py is missing.

  5. List the recorded scores:

    Terminal window
    vision zones --list

    You see six methods, from padim_resnet18 to supervised_resnet18.

  6. Choose the lines for PatchCore and 100 checks a day:

    Terminal window
    vision zones --method patchcore_wide_resnet50 --budget 100
    patchcore_wide_resnet50: lines chosen on the validation photos for 100 checks a day: check from 0.4106, fail from 0.6441
    Test (218 photos, 18 damaged; AUROC 0.917), per day at 600 returns: 88 checks, 5.5 damaged sold as new, 2.8 good items lost; cost 369 dollars a day
  7. Run the same command with --budget 50. The fail line moves to 0.4859 and the cost to 1273 dollars a day.

  8. Show your two runs in the comparison table:

    Terminal window
    vision compare --task inspect --mine

    You see returns test with 368.8, and returns test, 50 checks a day with 1272.9.

If vision zones says no data folder found, the terminal does not have LARKVISION_DATA: set it as in step 2. If your project does not run at all, copy the snapshot end-of-m06 again, as in Meet the Larkfield images.

Use two lines when a person can check some items and the two mistakes have different prices. The check zone is where the inspector’s scores overlap, and a minute of a person there is much cheaper than a wrong decision.

Use one line only when nobody can check, and know what the overlap costs. For Hamid, no checks cost more than no inspection.

What it costs: the lines are only as good as the damaged photos they were chosen on. And the prices are assumptions that Hamid and his manager must agree on. When a price changes, choose the lines again. Thresholds with few labels and review budgets are the core of The alert budget and From score to alert in the anomaly detection course.

Guided practice: the lines in the notebook

Section titled “Guided practice: the lines in the notebook”
  1. Open the notebook from the box Do this lesson in a notebook at the top of this page.
  2. Run the setup cells. The data cell ends with a Ready: line.
  3. In section 1, predict the zones of the ten returns, then run the cell. You see cost_per_day: 4470.0 and over_budget: True.
  4. In section 2, run the cell. You see AUROC on test: 0.917.
  5. In section 3, run the cell. You see check from 0.4106, fail from 0.6441, then 37.5 on validation and 368.8 on test.
  6. In section 4, write your prediction, then run the cell. You see 1272.9.
  7. In section 5, run both cells. Without the lowest damaged photo, the check line moves to 0.4219 and the test cost to 488.5.

Hamid’s team may lose another worker. Change one thing: the budget, to 25 checks a day. Before you run it, predict: will the cost on the test photos be closer to 1,272.9 or to no inspection (2,229.4)? In the notebook, section 6, put the test cost per day in cost_25, then run the check cell.

Show the answer
quarter = dict(cp.COSTS, checks_per_day=25)
lines_25 = cp.choose_lines(sv, dv, quarter)
cost_25 = cp.inspection_costs(st, dt, lines_25["check_line"], lines_25["fail_line"], quarter)["cost_per_day"]
print(cost_25)
1727.1

The check line stays at 0.4106 again, and the fail line comes down to 0.4532. On the test photos, the team checks 30.3 items a day, and 52.3 good items a day go to the outlet. That is over 25 checks: the lines fit the budget on validation, not exactly on test. At 1,727.1 dollars a day, the inspector saves less than a quarter of the cost of no inspection.

Common mistake

Choosing the lines on the test photos

Symptom: Tomás sees 368.8 dollars a day on the test photos and wants a better number. He runs choose_lines on the test scores instead of the validation scores and gets lines with a test cost of 293.1 dollars a day. He reports it.

Cause: the lines now fit the accidents of the 18 damaged test photos. Its cost is too good, like the validation cost of 37.5. The course team measured Tomás’s lines once on the hidden test: 1,468.9 dollars a day, exactly as bad as the honest lines.

Fix: choose the lines on the validation photos. Measure on the test photos once, and report that number as it is. When the validation set is too small, label more damaged photos; do not borrow the test set.

Consequence in AI work: a test number that took part in a choice promises too much. Hamid plans around 293 dollars a day and gets five times that.

Check your understanding

Lines: check from 0.5, fail from 0.8. Five returns score 0.20 (good), 0.45 (damaged), 0.60 (good), 0.70 (damaged) and 0.85 (good). Each return stands for 120 returns a day. What does the day cost with the course's prices, and is it within 100 checks?

Show the answer

Zones: 0.20 pass (good, no mistake), 0.45 pass (damaged: sold as new), 0.60 check, 0.70 check, 0.85 fail (good: lost). Per day: 120 damaged items sold as new × 45 = 5,400 dollars. 120 good items lost × 28 = 3,360. 240 checks × 0.50 = 120. Total 8,880 dollars a day. 240 checks is over the budget of 100, so these lines cannot be used: Hamid’s team cannot do the checks.

Check your understanding

Hamid's budget falls from 100 to 50 checks a day. Explain why the cost on the test photos goes up from 368.8 to 1,272.9 dollars a day, and not only by the price of the lost checks.

Show the answer

To halve the checks, the lines must move. Moving the check line up would sell more damaged candles at 45 dollars each. So the cheapest move is to bring the fail line down, from 0.6441 to 0.4859. That sends many good candles straight to the outlet instead of to a person. Good items lost go from 2.8 to 35.8 a day, at 28 dollars each. A check cost only 0.50 dollars and protected a 28-dollar candle, so every check given up costs about 22 dollars a day.

Check your understanding

The lines cost 37.5 dollars a day on validation, 368.8 on test and 1,468.9 on the hidden test. Nobody chose anything on the test or hidden photos. What went wrong, and what should Hamid do now?

Show the answer

Nothing in the method went wrong; the lines rest on too few damaged photos. The validation set has 12, and the check line sits just below the lowest one. So one photo decides it: without it, the test cost is 488.5. The hidden set had more low-scoring damage: 6 of 15 damaged candles passed, four of them small colour spots. Hamid should keep the inspector, but put part of his checks on a random sample of passed items. His team should label every checked photo. Priya sets the lines again when there are many more damaged photos. They watch the share of each zone and report a range instead of one number.

Recap

  • Two lines, three zones: an item passes below the check line and fails from the fail line up. Between them, a person checks it. One line with no checks cost 2,328 dollars a day, more than no inspection.
  • Choose the lines on validation with the prices and the budget, then measure once on test. For 100 checks a day: check from 0.4106, fail from 0.6441; 37.5 dollars a day on validation, 368.8 on test.
  • The budget matters a lot. At 50 checks, the fail line comes down into the good candles: 1,272.9 dollars a day. At 25: 1,727.1.
  • The line did not hold. It rested on 12 damaged photos. On the hidden test, measured once, the same lines cost 1,468.9 dollars a day: 6 of 15 damaged candles passed.
  • Keep a person on a sample of passed items, label every check, and choose the lines again when there are more damaged photos.
  • vision zones chooses and measures the lines in the project.

You can now turn the inspector’s scores into decisions that Hamid’s team can carry out, and you know how far to trust them. But Larkfield’s photos raise other questions: a customer sends a phone photo of a product, and the catalogue must find it. Next, you match photos with embeddings and nearest neighbours.

Next: Find the same product