Set the line with the warehouse
- Lesson 22
- 40 min
- Module 6: Inspect for defects
Where we are
In the last lesson, PatchCore gave every returned candle an anomaly score and a heat map. A score is not a decision: Hamid must know what to do with each item. In this lesson, we turn scores into pass, check and fail with Hamid, from the cost of each mistake and his 100 checks a day, the way his team already sorts returns into three bins.
Lesson 22 of 30 · Vision AI: See, Count and InspectSet the line with the warehouse
1. Seeing is a business problem
2. Baselines and pretrained classifiers
3. Embeddings, CLIP and few labels
4. Detect and count
6. Inspect for defects
7. Match, search and describe
8. Run vision in production
What you will be able to do
Turn an inspector’s scores into decisions that Hamid’s team can carry out: back on sale, checked by a person, or sent to the outlet. The best lines depend on the cost of each mistake and on the team’s checks a day. So they are set with the warehouse, not by the data scientist alone.
NotebookDo this lesson in a notebook
The notebook has every code cell, task and check of this lesson. It works on the recorded scores of anomalib’s PatchCore, so it downloads no model and runs in seconds. It sorts ten returns by hand and chooses the lines for 100 and for 50 checks a day. About 30 minutes.
Colab and Kaggle open your own copy. They need a free Google or Kaggle account. Stuck? See the solution notebook.
Before you startWhat to know and to have ready
What you need to know
- Anomaly scores, AUROC and PatchCore, from Anomaly-based inspection.
- The cost of each mistake and Hamid’s 100 checks a day, from The return that cost twice.
- Why we choose on validation and measure once on test, from Train, validation and test sets in the machine learning course.
What you need on your computer (for the project steps; the notebook needs none of it)
- The labs repository with the course’s data, from Meet the Larkfield images.
- Your
larkvisionproject at the end of Module 5 (snapshotend-of-m05), besidenextia-learning-labs, with its.venvenvironment. Ifvision masksdoes not run, copy the snapshotend-of-m05again, as in Meet the Larkfield images.
Tested on macOS with Python 3.12.13, NumPy 2.5.4 and pytest 9.2.0. The Windows commands are written but were not run.
Three bins at the returns desk
Section titled “Three bins at the returns desk”Hamid’s team already sorts returns in a simple way. A worker looks at a returned candle. If it is clearly fine, it goes back to the shelf. If it is clearly broken, it goes to the outlet bin. If she is not sure, she puts it in a third bin. A senior worker looks at that bin at the end of the day.
The third bin is the clever part: easy items go fast, and hard ones get more care. But the senior worker can look at only so many items a day.
Our inspector gives a score, not a bin. In this lesson, we draw the borders of the three bins on the score.
The problem: one line forces a bad choice
Section titled “The problem: one line forces a bad choice”With one line, every item either passes or fails. Nobody checks anything. Then every mistake of the inspector reaches the customer or the outlet.
With PatchCore’s scores and no checks at all, the best single line on the validation photos is at 0.4106. On the test photos, it sells 5.5 damaged items a day as new and sends 74.3 good items a day to the outlet:
patchcore_wide_resnet50: lines chosen on the validation photos for 0 checks a day: check from 0.4106, fail from 0.4106Test (218 photos, 18 damaged; AUROC 0.917), per day at 600 returns: 0 checks, 5.5 damaged sold as new, 74.3 good items lost; cost 2328 dollars a dayThat is 2,328 dollars a day: more than no inspection at all, which costs 2,229.4 dollars a day on the same photos. Many good candles score higher than the lowest damaged ones. The scores overlap, and the overlap needs a person.
The idea: two lines and three zones
Section titled “The idea: two lines and three zones”Use two lines on the score. Below the check line, the item passes and goes back on sale as new. At or above the fail line, the item fails and goes to the outlet. Between the lines, a person checks it. These three ranges are the pass–check–fail zones.
Text description of the diagram
A horizontal arrow goes from “low score” on the left to “high score” on the right. Under it are three boxes in a row. The first box, green, is “PASS”: the item goes back on sale as new. Its mistake is a damaged item sold as new, which costs 45 dollars. A dashed line, the check line, leads to the second box, grey, “CHECK”: a person looks. Each check costs 0.50 dollars, and the team can do at most 100 a day. A solid line, the fail line, leads to the third box, orange, “FAIL”: the item goes to the outlet. Its mistake is a good item lost, which costs 28 dollars. On a phone, the three zones are stacked from top to bottom.
A person who checks is always right, in this course’s model. So the check zone has no mistakes, only a cost: 0.50 dollars a check, one minute of a worker at 30 dollars an hour. A damaged item in the pass zone is sold as new: 45 dollars. A good item in the fail zone is lost: 28 dollars.
A tiny example: ten returns
Section titled “A tiny example: ten returns”Here are ten returns with their scores. The lines are: check from 0.4, fail from 0.6.
| Score | 0.12 | 0.25 | 0.31 | 0.38 | 0.45 | 0.52 | 0.58 | 0.63 | 0.71 | 0.90 |
|---|---|---|---|---|---|---|---|---|---|---|
| Damaged? | no | no | yes | no | no | yes | no | no | yes | yes |
| Zone | pass | pass | pass | pass | check | check | check | fail | fail | fail |
The damaged item at 0.31 passes: it is sold as new. The good item at 0.63 fails: it is lost. Three items are checked. Each of the ten items stands for 60 of Hamid’s 600 returns a day, so a day costs:
| Part | Per day | Price | Dollars a day |
|---|---|---|---|
| Damaged sold as new | 1 × 60 = 60 | 45 | 2,700 |
| Good items lost | 1 × 60 = 60 | 28 | 1,680 |
| Checks | 3 × 60 = 180 | 0.50 | 90 |
| Total | 4,470 |
And 180 checks a day is over Hamid’s budget of 100. The notebook computes the same with the course’s helper:
print(cp.inspection_costs(tiny["score"], tiny["damaged"], 0.4, 0.6)){'items': 10, 'damaged': 4, 'missed': 1, 'good_failed': 1, 'checks': 3, 'checks_per_day': 180.0, 'damaged_sold_as_new_per_day': 60.0, 'good_marked_damaged_per_day': 60.0, 'cost_per_day': 4470.0, 'over_budget': True}How the lines are chosen
Section titled “How the lines are chosen”A lower check line catches more damaged items but sends more good items to a person. A lower fail line saves checks but loses more good items. choose_lines in the course’s helper takes 201 points along the validation scores and tries every pair of them as the two lines. It keeps the cheapest pair whose checks stay inside the budget.
We choose on the validation photos and measure on the test photos. Lines chosen on a set fit that set’s accidents, so their cost on that set is too good.
Larkfield: Hamid’s lines for 100 checks a day
Section titled “Larkfield: Hamid’s lines for 100 checks a day”Priya and Hamid choose the lines for PatchCore with a wide ResNet-50 in the notebook:
lines = cp.choose_lines(sv, dv)on_test = cp.inspection_costs(st, dt, lines["check_line"], lines["fail_line"])check from 0.4106, fail from 0.6441validation: {'checks_per_day': 75.0, 'damaged_sold_as_new_per_day': 0.0, 'good_marked_damaged_per_day': 0.0, 'cost_per_day': 37.5}test: {'checks_per_day': 88.1, 'damaged_sold_as_new_per_day': 5.5, 'good_marked_damaged_per_day': 2.8, 'cost_per_day': 368.8}On the validation photos, the lines are perfect: no damaged item passes, no good item fails, and 75 checks a day cost 37.5 dollars. On the test photos, two damaged candles score below the check line: 0.244 and 0.339, both with a damaged corner of the packet. And one good candle scores above the fail line. The cost is 368.8 dollars a day: ten times the validation cost, but still far below no inspection.
Now try the lines yourself in the explorer.
Explore
Pass, check and fail zones for Hamid's returns
Predict first: If Hamid's budget falls from 100 to 50 checks a day, does the cost on the test photos go up a little or a lot?
| validation 112 photos | test 218 photos | |
|---|---|---|
| Checks a day (budget 100) | 75 | 88.1 |
| Damaged items sold as new a day | 0 | 5.5 |
| Good items lost a day | 0 | 2.8 |
| Damaged photos that pass | 0 of 12 | 2 of 18 |
| Cost a day (assumptions) | 37.5 | 368.8 |
On the test photos, these lines send 88.1 returns a day to a person (budget 100), sell 5.5 damaged items a day as new and send 2.8 good items a day to the outlet: 368.8 dollars a day. With no inspection, the same photos cost 2,229.4 dollars a day. These are the cheapest lines on the validation photos for 100 checks a day.
The lines are chosen on the validation photos and measured on the test photos. Costs are the course’s assumptions: a damaged item sold as new 45 dollars, a good item lost 28 dollars, a check 0.50 dollars; a person who checks is always right. Per day at 600 returns.
Text description of the explorer
The explorer shows two panels on one score axis from 0.15 to 1. The top panel is a histogram of the good returns; the bottom panel shows each damaged return as one dot. Behind them, three coloured zones: pass (green) left of the check line, check (grey) between the lines, and fail (orange) right of the fail line. A damaged dot in the pass zone has a dark ring: it is sold as new. Controls: the inspector (PatchCore or the supervised ResNet-18), the budget (25 to 150 checks a day), a slider for each line, and the photos in the picture. A table has one column for the validation photos and one for the test photos. Its rows are the checks a day, the damaged items sold as new, the good items lost, the damaged photos that pass, and the cost a day. At the start, PatchCore with 100 checks: check from 0.411, fail from 0.644; validation 37.5 dollars a day, test 368.8. With 50 checks: fail from 0.486; test 1,272.9 dollars a day, with 35.8 good items lost a day.
What the budget of 50 does
Section titled “What the budget of 50 does”The cost goes up a lot: from 368.8 to 1,272.9 dollars a day on the test photos. The project command gives the same lines:
patchcore_wide_resnet50: lines chosen on the validation photos for 50 checks a day: check from 0.4106, fail from 0.4859Test (218 photos, 18 damaged; AUROC 0.917), per day at 600 returns: 47 checks, 5.5 damaged sold as new, 35.8 good items lost; cost 1273 dollars a dayThe check line stays at 0.4106, because moving it up would sell more damaged candles at 45 dollars each. So the only way to halve the checks is to bring the fail line down, from 0.6441 to 0.4859, into the good candles. Good items lost go from 2.8 to 35.8 a day. The checks were cheap (0.50 dollars) and each one protected a 28-dollar candle.
Hamid now has a number for his manager: each check that the team gives up costs about 22 dollars a day: (1,272.9 − 368.8) ÷ (88.1 − 46.8) = 904.1 ÷ 41.3.
The honest surprise: the line did not hold
Section titled “The honest surprise: the line did not hold”Why did the test cost ten times the validation cost? Look at the damaged photos that set the line.
damaged validation scores: [0.412 0.439 0.445 0.478 0.481 0.615 0.62 0.682 0.713 0.763 0.888 0.892]damaged test scores: [0.244 0.339 0.413 0.424 0.434 0.557 0.56 0.629 0.671 0.799 0.845 0.851 0.968 1. 1. 1. 1. 1. ]check line 0.4106The validation set has only 12 damaged photos. The check line sits just below the lowest one, 0.412. In the notebook, Priya removes that one photo and chooses again:
without R-46ef3aca: check from 0.4219, fail from 0.6458test cost: 488.5One photo less, and the test cost moves from 368.8 to 488.5 dollars a day. A line set on 12 examples is closer to a guess than to a measurement.
One look at the hidden test
Section titled “One look at the hidden test”The course team kept a hidden test set: 135 returns, 15 of them damaged, not in labs and never used for any choice. It plays the role of next month’s returns. They measured the chosen lines on it once:
Text description of the chart
A column chart with four columns. Validation: 37.5 dollars a day. Test: 368.8. Hidden test: 1,468.9. No inspection (test photos): 2,229.4.
On the hidden test, the same lines cost 1,468.9 dollars a day. The AUROC was still good, 0.891, but 6 of the 15 damaged candles passed. Their scores were between 0.279 and 0.363, below the check line. Four of the six were “different colour spot”: the small spots that a heat map can miss. The inspector still saves money: with no inspection, these photos would cost 3,000 dollars a day. But the day costs four times what the test set promised.
The sets were small: 18 and 12 damaged photos cannot show every kind of damage. Evaluate without a full answer key in the anomaly detection course meets the same limit with payments.
What Hamid does with this
Section titled “What Hamid does with this”Hamid does not throw the inspector away. He changes how he uses it:
- Keep a person on a sample of passed items. On the test photos, the lines used about 88 of his 100 checks. He gives the other 12 or so to random items from the pass zone.
- Label every checked item. Each check produces a labelled photo for free. When the validation set has many more damaged photos, Priya sets the lines again.
- Watch the zone shares. If the share of fail or check moves far from its validation level, something changed, such as the light or the camera. Module 8 turns this into an alarm.
- Write the uncertainty down. The report says “368.8 dollars a day on test and 1,468.9 on the hidden test”, not “369 dollars a day”.
The project: the vision zones command
Section titled “The project: the vision zones command”Your larkvision project gets the zones as a command. It reads the recorded scores in data/predictions/, then chooses and measures the lines as the notebook does.
The heart of the command:
scores = json.loads((Path(d) / f"predictions/returns_{args.method}.json").read_text())costs = D.costs(d)if args.budget is not None: costs = dict(costs, checks_per_day=args.budget)val, test = D.returns(d, "val"), D.returns(d, "test")sv = [scores[r["item_id"]] for r in val]dv = [r["status"] == "damaged" for r in val]lines = choose_lines(sv, dv, costs)st = [scores[r["item_id"]] for r in test]dt = [r["status"] == "damaged" for r in test]c = inspection_costs(st, dt, lines["check_line"], lines["fail_line"], costs)Set up the project for this lesson
Section titled “Set up the project for this lesson”This lesson adds one command, vision zones, and its three tests from the snapshot end-of-m06.
-
Open a terminal and go to your
larkvisionfolder. Activate the virtual environment:source .venv/bin/activateon macOS and Linux,.venv\Scripts\Activate.ps1on Windows. -
If this is a new terminal, set
LARKVISION_DATAagain. It lives only in the terminal where you set it:Terminal window export LARKVISION_DATA="$(cd ../nextia-learning-labs/C42/data && pwd)"Terminal window $env:LARKVISION_DATA = (Resolve-Path ..\nextia-learning-labs\C42\data).Path -
Copy the new command, its tests and the README from the snapshot:
Terminal window S=../nextia-learning-labs/C42/snapshots/end-of-m06cp "$S/larkvision/commands/zones.py" larkvision/commands/cp "$S/tests/test_zones.py" tests/cp "$S/README.md" .Terminal window $S = "..\nextia-learning-labs\C42\snapshots\end-of-m06"Copy-Item "$S\larkvision\commands\zones.py" larkvision\commands\Copy-Item "$S\tests\test_zones.py" tests\Copy-Item "$S\README.md" . -
Run the tests:
Terminal window python -m pytest -qThe last line is
22 passed. If it says19 passed, the filetests/test_zones.pyis missing. -
List the recorded scores:
Terminal window vision zones --listYou see six methods, from
padim_resnet18tosupervised_resnet18. -
Choose the lines for PatchCore and 100 checks a day:
Terminal window vision zones --method patchcore_wide_resnet50 --budget 100patchcore_wide_resnet50: lines chosen on the validation photos for 100 checks a day: check from 0.4106, fail from 0.6441Test (218 photos, 18 damaged; AUROC 0.917), per day at 600 returns: 88 checks, 5.5 damaged sold as new, 2.8 good items lost; cost 369 dollars a day -
Run the same command with
--budget 50. The fail line moves to0.4859and the cost to1273dollars a day. -
Show your two runs in the comparison table:
Terminal window vision compare --task inspect --mineYou see
returns testwith368.8, andreturns test, 50 checks a daywith1272.9.
If vision zones says no data folder found, the terminal does not have LARKVISION_DATA: set it as in step 2. If your project does not run at all, copy the snapshot end-of-m06 again, as in Meet the Larkfield images.
When to use two lines, and what they cost
Section titled “When to use two lines, and what they cost”Use two lines when a person can check some items and the two mistakes have different prices. The check zone is where the inspector’s scores overlap, and a minute of a person there is much cheaper than a wrong decision.
Use one line only when nobody can check, and know what the overlap costs. For Hamid, no checks cost more than no inspection.
What it costs: the lines are only as good as the damaged photos they were chosen on. And the prices are assumptions that Hamid and his manager must agree on. When a price changes, choose the lines again. Thresholds with few labels and review budgets are the core of The alert budget and From score to alert in the anomaly detection course.
Guided practice: the lines in the notebook
Section titled “Guided practice: the lines in the notebook”- Open the notebook from the box Do this lesson in a notebook at the top of this page.
- Run the setup cells. The data cell ends with a
Ready:line. - In section 1, predict the zones of the ten returns, then run the cell. You see
cost_per_day: 4470.0andover_budget: True. - In section 2, run the cell. You see
AUROC on test: 0.917. - In section 3, run the cell. You see
check from 0.4106, fail from 0.6441, then37.5on validation and368.8on test. - In section 4, write your prediction, then run the cell. You see
1272.9. - In section 5, run both cells. Without the lowest damaged photo, the check line moves to
0.4219and the test cost to488.5.
Your turn: a budget of 25 checks
Section titled “Your turn: a budget of 25 checks”Hamid’s team may lose another worker. Change one thing: the budget, to 25 checks a day. Before you run it, predict: will the cost on the test photos be closer to 1,272.9 or to no inspection (2,229.4)? In the notebook, section 6, put the test cost per day in cost_25, then run the check cell.
Show the answer
quarter = dict(cp.COSTS, checks_per_day=25)lines_25 = cp.choose_lines(sv, dv, quarter)cost_25 = cp.inspection_costs(st, dt, lines_25["check_line"], lines_25["fail_line"], quarter)["cost_per_day"]print(cost_25)1727.1The check line stays at 0.4106 again, and the fail line comes down to 0.4532. On the test photos, the team checks 30.3 items a day, and 52.3 good items a day go to the outlet. That is over 25 checks: the lines fit the budget on validation, not exactly on test. At 1,727.1 dollars a day, the inspector saves less than a quarter of the cost of no inspection.
Common mistake
Choosing the lines on the test photos
Symptom: Tomás sees 368.8 dollars a day on the test photos and wants a better number. He runs choose_lines on the test scores instead of the validation scores and gets lines with a test cost of 293.1 dollars a day. He reports it.
Cause: the lines now fit the accidents of the 18 damaged test photos. Its cost is too good, like the validation cost of 37.5. The course team measured Tomás’s lines once on the hidden test: 1,468.9 dollars a day, exactly as bad as the honest lines.
Fix: choose the lines on the validation photos. Measure on the test photos once, and report that number as it is. When the validation set is too small, label more damaged photos; do not borrow the test set.
Consequence in AI work: a test number that took part in a choice promises too much. Hamid plans around 293 dollars a day and gets five times that.
Knowledge check
Section titled “Knowledge check”Check your understanding
Lines: check from 0.5, fail from 0.8. Five returns score 0.20 (good), 0.45 (damaged), 0.60 (good), 0.70 (damaged) and 0.85 (good). Each return stands for 120 returns a day. What does the day cost with the course's prices, and is it within 100 checks?
Show the answer
Zones: 0.20 pass (good, no mistake), 0.45 pass (damaged: sold as new), 0.60 check, 0.70 check, 0.85 fail (good: lost). Per day: 120 damaged items sold as new × 45 = 5,400 dollars. 120 good items lost × 28 = 3,360. 240 checks × 0.50 = 120. Total 8,880 dollars a day. 240 checks is over the budget of 100, so these lines cannot be used: Hamid’s team cannot do the checks.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
Hamid's budget falls from 100 to 50 checks a day. Explain why the cost on the test photos goes up from 368.8 to 1,272.9 dollars a day, and not only by the price of the lost checks.
Show the answer
To halve the checks, the lines must move. Moving the check line up would sell more damaged candles at 45 dollars each. So the cheapest move is to bring the fail line down, from 0.6441 to 0.4859. That sends many good candles straight to the outlet instead of to a person. Good items lost go from 2.8 to 35.8 a day, at 28 dollars each. A check cost only 0.50 dollars and protected a 28-dollar candle, so every check given up costs about 22 dollars a day.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
The lines cost 37.5 dollars a day on validation, 368.8 on test and 1,468.9 on the hidden test. Nobody chose anything on the test or hidden photos. What went wrong, and what should Hamid do now?
Show the answer
Nothing in the method went wrong; the lines rest on too few damaged photos. The validation set has 12, and the check line sits just below the lowest one. So one photo decides it: without it, the test cost is 488.5. The hidden set had more low-scoring damage: 6 of 15 damaged candles passed, four of them small colour spots. Hamid should keep the inspector, but put part of his checks on a random sample of passed items. His team should label every checked photo. Priya sets the lines again when there are many more damaged photos. They watch the share of each zone and report a range instead of one number.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Recap
- Two lines, three zones: an item passes below the check line and fails from the fail line up. Between them, a person checks it. One line with no checks cost 2,328 dollars a day, more than no inspection.
- Choose the lines on validation with the prices and the budget, then measure once on test. For 100 checks a day: check from 0.4106, fail from 0.6441; 37.5 dollars a day on validation, 368.8 on test.
- The budget matters a lot. At 50 checks, the fail line comes down into the good candles: 1,272.9 dollars a day. At 25: 1,727.1.
- The line did not hold. It rested on 12 damaged photos. On the hidden test, measured once, the same lines cost 1,468.9 dollars a day: 6 of 15 damaged candles passed.
- Keep a person on a sample of passed items, label every check, and choose the lines again when there are more damaged photos.
vision zoneschooses and measures the lines in the project.
You can now turn the inspector’s scores into decisions that Hamid’s team can carry out, and you know how far to trust them. But Larkfield’s photos raise other questions: a customer sends a phone photo of a product, and the catalogue must find it. Next, you match photos with embeddings and nearest neighbours.
Next: Find the same product