Skip to content
Français

Final assignment

  • After the case studies
  • about 3 hr

Where we are

Each module added one piece and tried it alone: the framing, the baselines, the embeddings, the detectors, the inspection zones and the service. In this assignment, you use them all at once. Larkfield opens a second warehouse with a new camera and a new product category.

What you will be able to do

Use every module of the course on one new situation, and deliver a vision pack that Hamid and Omar can act on. The photos look like the course’s photos, but a new camera and a new product category change them. So you must test each choice again, not copy it.

Before you startWhat to know and to have readyAbout 180 minutes
  • You completed Modules 1 to 8. Each part below names the lessons that it uses.
  • You do not need the case studies of Module 9 for this assignment. (The certificate asks for at least one of them.) To see a similar inspection on real data first, open Visual inspection with few defects. It compares four inspectors on the same photos and sets the lines from a budget.
  • Your larkvision project works at the end of Module 8, with its .venv environment and 32 passed, from Domain shift: Set up the project for this module. The folder nextia-learning-labs is beside it, as in Meet the Larkfield images.
  • For Parts 2 and 3 you need PyTorch, timm, open_clip and Transformers, as in the notebooks of Modules 2 and 3. You can do these parts in a notebook (local, Colab or Kaggle). Parts 4 and 5 need the project and a terminal.
  • About 3 hours of work, plus run time. In the author’s solution, the longest run took under 2 minutes on a laptop CPU.
  • Tested on macOS with Python 3.12.13, the project’s pinned packages, torch 2.15.0, timm 1.0.31, open-clip-torch 4.2.1 and transformers 5.22.2. The Windows commands are written, but they were not run.

The situation: a second warehouse and a new category

Section titled “The situation: a second warehouse and a new category”

Larkfield is growing in two ways at once.

  • A second returns warehouse opened on 1 April. Its camera is not the camera of the first warehouse. Hamid, the returns warehouse lead, runs both warehouses. The second one also gets about 600 returns a day, and his team there can do 100 checks a day. Both numbers are the course’s assumptions.
  • A new product category. Omar, the product manager, adds kettles to the catalogue. The photos come from the Open Images class “Kettle”, which also holds teapots and coffee pots. So Omar’s category is really “kettles and teapots”. Customers send phone photos of these products, and the shop must find the right catalogue item.

Hamid asks for a vision pack before he trusts any machine in the second warehouse. Omar asks the question that decides the catalogue part: “What does a mistake cost our customers?” Tomás, the AI engineer, has an answer ready: “Send every photo to the hosted vision-language model. It can see kettles and damage.” Amira owns the image service. She asks one thing: “Does the service still work with the new camera?”

Priya asks you to answer all of them with evidence. These are the kinds of photos that you work with:

Five photos in a row. First, four white tea lights on a dark cloth: a return from the first warehouse. Second, four tea lights that look yellow, smaller, with a flat grey border around the cloth: a return from the second warehouse. Third, a chrome electric kettle on a kitchen counter. Fourth, a white teapot with a green pattern and a bamboo handle. Fifth, a silver teapot photographed at an angle on a brown table, in poor light: a customer photo.
Left: returns R-fb5f129e from the first warehouse and W2-003 from the second warehouse (VisA dataset, Amazon, CC BY 4.0; the second camera's yellow light, wider view and noise are synthetic, made by our code). Middle: kettle catalogue photos K-001 (photo: Perry Goh) and K-005 (photo: Nikita), CC BY 2.0, via Open Images. Right: customer photo KC-007, simulated by our code from one of the kettle photos (authors in credits.csv). Which kettle it shows is part of the task.
Text description of the diagram

Five photos in a row (two rows on a phone). The first warehouse’s photo shows four white tea lights from above on a dark cloth. The second warehouse’s photo shows the same kind of tea lights, but yellow, smaller and with a flat grey border: the camera sees a wider area. Then two catalogue photos of the new category: a chrome electric kettle, and a white teapot with a bamboo handle. The last photo is a customer photo of a silver teapot. It is turned, smaller in the frame, on a textured brown table, in dim light.

A person still sees good tea lights in the second photo. A model that learned the first camera may not. And the new category is not as clean as its name: the kettle photos also hold teapots, jugs and coffee makers.

All the returns and customer photos are changed or made by our code, and Larkfield is fictional. No photo shows a real person.

The data is in the labs repository, in C42/final/data/. The course’s own data, C42/data/, stays as it was: the first warehouse and the eight earlier categories.

File or folder Rows What one row is Columns
returns/W2-*.jpg, returns_manifest.csv 95 One returned tea light photo from the second warehouse, 320 × 291 pixels, from 1 to 30 April. No labels. item_id, file, day, licence, author, source_url, synthetic, note
catalogue/kettle/K-*.jpg, catalogue_manifest.csv 102 One catalogue photo of the new category, 256 × 256: 64 train, 38 test image_id, file, split, label (always kettle), source_url, licence, author, synthetic, note
customer_photos/KC-*.jpg, customer_photos_manifest.csv 102 One customer photo of a kettle, 320 × 240. Which kettle it shows is not given. photo_id, file, licence, synthetic, note
credits.csv 102 The author and licence of each kettle source photo, in an order that does not reveal the matches source_url, author, licence

The folder also has dataset-card.md and SHA256SUMS, and is about 5 MB. The IDs are new: an ID does not show which photo is damaged or which kettle a customer photo shows. The course team keeps those answers, and a hidden copy of the second warehouse’s statuses. Nobody chooses anything with them.

Check the files. Run this in your larkvision folder:

Terminal window
git -C ../nextia-learning-labs pull
(cd ../nextia-learning-labs/C42/final/data && shasum -a 256 -c SHA256SUMS | grep -vc ': OK$')

The number that it prints is the number of wrong files. It must be 0.

Make a folder vision-pack/ in your larkvision folder. Put every file of the assignment in it. What to submit lists the files. In each new terminal, activate .venv and set LARKVISION_DATA and LARKVISION_MODELS again, as in Module 8.

The pack follows the course, one module after another.

The data box on top: 95 returns photos from the second warehouse with no labels, 102 kettle photos, 102 customer photos. Below, six boxes in order: 1 Frame, 2 Catalogue, 3 Search (green, hand in kettle_matches.csv), 4 Inspection (green, hand in w2_zones.csv), 5 Service, 6 Hamid's page. An orange box at the bottom: the course team scores the two green files once, on the answers that it keeps; nothing is chosen with that score; you test the catalogue once yourself.
The six parts of the pack. Two files go to the course team, because only it has their answers. You test the catalogue yourself, once, because its test labels are public.
Text description of the diagram

At the top, a grey box: the data, 95 returns photos from the second warehouse with no labels, 102 kettle photos and 102 customer photos. Below it, six boxes joined by arrows, in two rows of three. 1 Frame: kinds of answer, two cameras, costs, the review budget. 2 Catalogue: nine classes, a baseline and three families, by class. 3 Search, in green: customer photo to kettle, recall at 1 and at 5; hand in kettle_matches.csv. 4 Inspection, in green: find the shift, fix it, set lines from the budget; hand in w2_zones.csv. 5 Service: the month, with review, faces, latency and alarms. 6 Hamid’s page: the decision, its cost, what a person still checks. At the bottom, an orange box: the course team scores the two green files once, on the answers that it keeps. Nothing is chosen with that score. You test the catalogue once yourself.

Uses The return that cost twice, Five kinds of answer and When not to build it.

Write vision-pack/FRAMING.md. Use the course’s costs. They are assumptions: write them as assumptions.

  1. Name the three requests: inspect the second warehouse’s returns, sort the catalogue with the new category, and find a customer’s kettle. For each one, write the kind of answer (label, box, mask, match or text) and who uses it.
  2. Write the camera set-up of each warehouse, as you see it in the photos: the light, how much of the table the camera sees, the sharpness. Say what you would ask the warehouse to fix before any model.
  3. Write the cost of each mistake: a damaged item sold as new costs 45 dollars, a good item marked damaged costs 28, a person’s check costs 0.50.
  4. Write the daily review budget: 100 checks for about 600 returns a day, so about 1 return in 6. Say what happens to a return that the team cannot check.
  5. Write Omar’s costs for the catalogue and the search in words: a kettle filed as a vase, and a customer shown the wrong kettle. There is no course number for these. Say so, and say how you would find one.
  6. Answer Tomás in two or three sentences, before you measure anything: which request could a vision-language model help with, and what would you check first?

Uses Rules and colour histograms, Hand-made features, Image embeddings, CLIP zero-shot, Linear probes and few-shot and Labels: how many, and how good.

The catalogue now has nine classes: the eight of the course and kettle. The course’s embedding files, data/embeddings/*.npz, have no kettle photos. So embed the 102 kettle photos (and, for Part 3, the 102 customer photos) with the same pinned models as the notebooks.

  1. Check your embeddings first. Embed three old catalogue photos with your code. Compare them with the course’s file: the cosine similarity must be 1.000 (to three decimals). If it is not, your preprocessing differs, and every later number is wrong.
  2. Look at the kettle photos. Make a contact sheet of the 64 training photos. Write a short labelling guide for Omar’s category: what counts as a kettle or teapot, and what does not (a jug, a coffee maker, a mug). Do not delete photos to make the numbers better. Write down which photos the guide would change.
  3. Make a validation set for kettles. The kettles have only train and test. Keep some training kettles apart with a fixed seed (for example 16 of 64), or use cross-validation on the training photos. Never choose with the 38 test kettles.
  4. Run a baseline and at least three method families on the same photos with the same evaluation. For example: a colour histogram or HOG (the baseline), and CLIP zero-shot with a prompt for the new class. Then linear probes on CLIP, DINOv2 or ResNet-18 embeddings, with and without balanced classes.
  5. Write your rule for the choice in REPORT.md, before you run the test: for example “the best validation accuracy, unless the kettle recall is lower”. Then choose on validation.
  6. Test once. Train the chosen method on all training photos, and score the 198 test photos: 160 old and 38 kettles. Write the accuracy, the recall and precision of each class, the worst class and the confusion matrix. Write vision-pack/catalogue_predictions.csv with image_id, predicted.
  7. Per camera. The catalogue has a second “camera” too: the new studio of Module 8 (160 photos, warmer light). Score your chosen method there, on the eight old classes.

Uses Find the same product and Check what a vision-language model says.

A customer sends a phone photo of a kettle. The search must show the right catalogue photo in its first five results. You do not have the answers. The course team scores your file once.

  1. Use the whole catalogue as the gallery: the 733 public photos of the course and the 102 kettle photos (835 photos). A search that finds only kettles because it sees only kettles is not a test.
  2. Run a baseline (a colour histogram or a perceptual hash) and at least two embedding models.
  3. Check without answers. For each method, compute the share of customer photos whose first match is a kettle photo. It is an upper limit on the recall at 1. A first match that is not a kettle is always wrong, but a kettle can be the wrong kettle.
  4. Look at 10 customer photos and their top 5 for your best method. Write what the wrong matches have in common, if anything.
  5. Choose one method with a reason that you write before the course team’s score. Write vision-pack/kettle_matches.csv: photo_id, match_1, match_2, match_3, match_4, match_5, one row for each of the 102 customer photos, best first.

The record matching course, not yet published, treats pairs and clusters of records in depth. You do not need it here.

Uses Two ways to find damage, Anomaly-based inspection, Set the line with the warehouse and Domain shift.

This is the hard part. The inspectors of the course learned the first camera. The second warehouse has no labelled photo at all. You must find out what the new camera does to each inspector, fix what you can, and set lines without labels from the new camera.

First, the first warehouse, where you have labels

Section titled “First, the first warehouse, where you have labels”
  1. Take a baseline and at least three families of inspectors from Module 6. For example: no inspection and “a person checks 100 random returns” (the baselines), the small PatchCore of the service (anomaly-based), a supervised classifier on embeddings, and the CLIP prompt.
  2. Choose the lines of each one on the first warehouse’s validation photos with choose_lines, for a budget of 100 checks. Score them on its test photos: AUROC, cost per day, checks per day. Your rows must match the course’s rows. If they do not, fix your code first.
  1. Find the shift. Score the 95 photos with each inspector. Compare the scores with the first warehouse’s validation scores. Use the PSI and the shares of fail and check with the old lines. Add a domain classifier on embeddings, with a control.
  2. Write your rule for the choice in REPORT.md, before you try any fix. It must use no label from the new camera. For example: “the fix that brings the zone shares back near their validation level, with checks inside the budget”.
  3. Choose a fix, and justify it. Read the fixes in Domain shift: The fixes, from cheap to expensive. Try at least two of them. Then choose one with your rule from step 2, and explain in REPORT.md why it fits the second warehouse and what it costs. Never put a photo with a face into a model or its memory.
  4. Set the lines from the review budget. You have no labelled damage from the new camera, so choose_lines cannot run. Decide how to set the two lines without those labels, and justify it with Hamid’s budget of 100 checks a day. Say which days you used to set them, and keep them for the rest of the month.
  5. Write which photos you would label first, and why. The check zone gives labels for free. A random sample of the pass zone finds the damage that passes.
  6. Write vision-pack/w2_zones.csv: item_id, day, score, zone, one row for each of the 95 photos. The course team scores it once with the course’s costs. Do not change your method after you see that score.

Uses Fast enough on a CPU, The image service and its review queue and People in images.

The command vision replay reads the course’s month. For the second warehouse, call the same function from a short script. Save it as vision-pack/replay_w2.py:

import csv
from pathlib import Path
from larkvision import data as D
from larkvision.commands.replay import replay
from larkvision.inspector import Inspector
from larkvision.paths import data_dir, models_dir
from larkvision.privacy import default_finder
final = Path("../nextia-learning-labs/C42/final/data")
with open(final / "returns_manifest.csv", encoding="utf-8") as f:
month = list(csv.DictReader(f)) # each row has item_id, file and day
inspector = Inspector(models_dir() / "inspector")
results, summary = replay(final, month, inspector, default_finder(models_dir()),
"vision-pack/review-w2", D.costs(data_dir()))
for week in summary:
print(week["week"], week["photos"], week["pass"], week["check"], week["fail"],
week["mean_score"], round(week["psi"], 2), week["alarms"], week["cost_alarms"])

Run it from your larkvision folder with python vision-pack/replay_w2.py. Every result has the zone, the score, the faces, the review flag and the latency.

  1. Replay the month with the service as it is. Write down the zones of each week, the alarms and the time per photo. This is what would happen if Amira changed nothing.
  2. Make a new inspector folder for the second warehouse, for example vision-pack/models-w2/inspector/, with your fix from Part 4. The Inspector class reads three files: model.onnx, bank.npy (the memory) and config.json. The config has version, check_line, fail_line, validation_good_scores (the reference for the PSI) and validation_shares. Give it a new version name.
  3. Replay the month again with the new folder. Check that the review folder has every unsure photo and every photo with a face, and that each face is blurred.
  4. Write the latency on your laptop: the median and the 95th percentile per photo, with the face finder. Compare it with a stated target, for example “under 100 ms a photo on one CPU core”.
  5. Read every alarm. For each one, say if it is a real change or noise, with the number of photos behind it. Say what Hamid’s team should do when it fires.
  6. Save the output of both runs as vision-pack/replay.txt.

Uses Set the line with the warehouse: What Hamid does with this and When not to build it.

Write vision-pack/HAMID.md: one page for Hamid, who does not read code. End it with a short part for Omar.

  1. Start with the decision: what the system does in the second warehouse from tomorrow, and what a person does.
  2. Give the expected cost a day, with the course’s costs as assumptions, and with an honest range. Your lines were set on a few dozen photos. Say so.
  3. Name what the system cannot see: at least one kind of damage or one condition, with the evidence. For example: a spot that is smaller than a pixel at 224, a new kind of damage, a camera that moves again.
  4. Say what a person still checks: the check zone, a random sample of the pass zone, every alarm week, and the first weeks of a new camera.
  5. Answer Omar: the kettle recall in the catalogue, the search result, and what a wrong match costs a customer, in words. Answer Tomás with one number from your table.
  6. Check every number on the page against your files. If an AI assistant drafted any part, say so, and check its numbers and its meaning yourself.

The folder vision-pack/ in your larkvision folder:

vision-pack/
FRAMING.md Part 1
comparison.json Parts 2 to 4: the course's columns, one row for each method and setting
catalogue_predictions.csv Part 2: image_id, predicted (198 test photos)
kettle_matches.csv Part 3: photo_id, match_1 ... match_5 (102 customer photos)
w2_zones.csv Part 4: item_id, day, score, zone (95 photos)
replay_w2.py Part 5, with any other script or notebook that you used
models-w2/inspector/ Part 5: model.onnx, bank.npy, config.json
replay.txt Part 5: both replays
REPORT.md Parts 1 to 5, with the commands that make every number
HAMID.md Part 6
requirements.txt exact versions

Build comparison.json with the course’s Table and its row helpers (classification_row, inspection_row, match_row in larkvision/compare.py). Put the camera or the set in setting, for example catalogue test (9 classes), new studio, returns test (first warehouse) and second warehouse (scored by the course team). A row for the second warehouse has no cost until the course team scores it. Leave the value empty, and fill it in once.

REPORT.md has these sections:

# Vision pack for the second warehouse and the kettles
## Framing (short; the full text is in FRAMING.md)
## Data checks (files, checksums, what you saw in the photos, the labelling guide)
## Catalogue: rule, table, test (Part 2: the rule written first, validation, the one test, per class, new studio)
## Kettle search (Part 3: the methods, the check without answers, the 10 photos you looked at)
## Inspection: first camera (Part 4: your rows against the course's rows)
## Inspection: the new camera (Part 4: the shift, the rule written first, the fixes, the lines from the budget)
## The service over the month (Part 5: both replays, latency, review folder, faces, each alarm)
## Scores from the course team (filled in once; no change after it)
## Limits (what the pack cannot see, and what you would label next)
## How to reproduce (commands, versions, run times, a fresh environment)

A reviewer marks each item demonstrated, partially demonstrated or not yet demonstrated, with a concrete reason. Without a reviewer, use the table on your own pack, or ask a colleague. Each item has points: all of them when demonstrated, half when partially demonstrated, none when not yet demonstrated. The points show where your pack is weakest. But the course is complete only when all six items are demonstrated: 100 points. You can improve an item and send it again.

Required item Points Demonstrated when Partially demonstrated, for example Not yet demonstrated, for example
1. Framing note (Module 1) 10 Each request has its kind of answer and its user. Both cameras are described. The costs and the budget are there, labelled as assumptions. Tomás gets a first answer. The camera set-up is missing, or the costs have no label. No kind of answer, or no budget.
2. Comparison table (Modules 2, 3, 6 and 7) 25 A baseline and at least three families for the catalogue and for the inspection, and a baseline and two models for the search. Same photos, same evaluation, the course’s columns. Results by class and by camera (old and new studio, first and second warehouse). The catalogue rule is written before the one test. Results by class are missing for one task. Or the first-camera rows do not match the course’s. No baseline. Or accuracy only. Or the kettle test photos are used to choose.
3. Inspection method (Modules 6 and 8) 20 The shift is shown with at least two signals. At least two fixes are tried, with a label-free rule written first. The lines come from the review budget, set on the first days only. The pack says what to label first. Only one fix, or lines set on the whole month. The old lines kept with no check. Or lines tuned on photos that you labelled by eye.
4. The service over the month (Module 8) 20 Two replays: the service as it is, and the new inspector folder. Unsure and face photos go to review, blurred. Latency is measured and compared with a target. Every alarm is read, with the number of photos behind it. No latency target. Or the alarms are listed but not read. No replay. Or a face kept unblurred.
5. Recommendation for Hamid (Modules 6, 7 and 8) 15 One page, no code. The decision and its cost with a range come first. It names what the system cannot see, with evidence, and what a person still checks. Omar and Tomás get answers with numbers. The limits are general, with no evidence. No limits. Or a claim that the system finds every damaged item.
6. A run that someone else can repeat 10 REPORT.md has the commands, versions and run times. The scripts or notebooks run in a fresh environment and give the same numbers. The embedding check gives 1.000. One step needs a file that is not in the pack. The numbers cannot be made again.

Not accepted, whatever the rest shows (the item is “not yet demonstrated”):

  • A model judged by overall accuracy alone. The kettles are 38 of 198 test photos, and damage is rare. Give the worst class and the cost. (Items 2 and 3.)
  • Lines tuned on the hidden test. That includes lines changed after the course team’s score, and lines tuned on candles that you marked by eye. (Item 3.)
  • A method chosen without a baseline in the comparison. (Item 2.)
  • A photo with a person stored or sent to a hosted model without blurring. That includes a face photo in the review folder, in the inspector’s memory, or in a call to Tomás’s model. (Items 3 and 4.)

A good pack is honest before it is impressive. It has three things.

  • Honest measures under the course’s costs. The cost a day, the checks a day, the worst class and the recall, each with its label: measured, assumed or scored once by the course team. The assumptions are next to them.
  • A simple baseline in the comparison. For the inspection, “no inspection” and “a person checks 100 random returns”. For the catalogue and the search, a colour histogram. If a model does not beat its baseline, the pack says so.
  • A decision with its cost. “From tomorrow, the service does this. It costs about this much a day, within this range. A person still checks this.”

These ranges come from the author’s own solution. The author built it from the public data only, with the course’s code. Then the course team scored it once against the answers that it keeps. Your methods can differ. Then compare the method and the size of the result, not the exact number.

  • The catalogue keeps its accuracy. With nine classes, the CLIP and DINOv2 probes were at about 0.89 to 0.94 on the test. A probe found about nine kettles in ten or more. CLIP zero-shot with a kettle prompt found only about three in four: the others became vases and mugs. On the old classes, the same method gave the course’s number.
  • A small validation set cannot separate close methods. In the author’s run, the rule picked zero-shot by two validation photos. On the test, the balanced probe was better by about five points. Write a rule that also looks at the new class’s recall.
  • The search is nearly solved by the right model. DINOv2 found the right kettle in its first five for almost every customer photo, and first for more than 95% of them. CLIP found it first for only about 6 in 10, and the colour histogram and the perceptual hash for almost none.
  • The new camera breaks the service completely. With the old memory and the old lines, every photo fails, and both alarms fire in every week. That costs about three times as much as no inspection at all.
  • A camera rule can make things worse. The author’s colour and border rules brought the scores down. But then the inspector ranked damaged and good candles about as well as a coin. Measure every fix.
  • New photos in the memory, and lines from the budget, help a lot. The author’s best label-free fixes cost about half to two thirds of “no inspection”. That is also less than “a person checks 100 random returns”. After the same fix with the old lines, most damage passed: about as costly as no inspection.
  • No family is safe on the new camera. Without a fix, every family ranked damage worse than on the first camera. The CLIP prompt fell close to a coin, and the supervised classifiers lost more than the PatchCore.
  • The lines from a few dozen photos do not hold exactly. The author’s check share went over the budget in two weeks. The PSI against a reference of about 25 photos alarmed in every week. One damaged photo more or less moves the cost by about 280 dollars a day (600 ÷ 95 × 45). Say so in the range.
  • The service stays fast. About 20 to 30 ms a photo on the course’s laptop, with the face finder, before and after the new memory.

Common mistake

Moving the line instead of fixing the photos

Symptom: on 1 April, every photo of the second warehouse fails. Someone raises the fail line until the fail share looks normal.

Cause: every patch of the new camera is far from the old memory. The score went up for good and damaged candles alike.

Fix: fix what the inspector compares with (new-camera photos in its memory, checked by a person), then set the lines again from the budget. Measure on photos that did not set the lines.

Consequence in AI work: a moved line hides the shift. Damaged candles go back on sale as new, and the next alarm never fires.

Common mistake

A photo scored against itself

Symptom: you add the first week’s photos to the memory and set the lines on their scores. In the rest of the month, far more photos than planned go to check or fail.

Cause: each photo of the first week finds some of its own patches in the memory. Its largest distance gets smaller, so its score is too low, and so are the lines. In the author’s run, such lines sent about 43% of the later photos to check or fail, instead of about 20%.

Fix: score each photo against a memory without its own patches (leave one out), or set the lines on other photos.

Consequence in AI work: lines from scores that are too low break the check budget in the first busy week. Then Hamid’s team stops trusting the zones.

  • The kettle test photos used to choose. There is no kettle validation set. Make one from the training photos.
  • A search gallery of kettles only. The customer cannot say “it is a kettle” first. Search the whole catalogue.
  • The category taken at its name. “Kettle” holds teapots, jugs and coffee makers. Write the guide, and tell Omar what his category really holds.
  • A face in the memory or in the review folder. The face finder must run before anything is stored. A painted face also gives a very high score and moves the lines.
  • Daily alarms. About 3 photos a day give a daily share that jumps by 33 points for one photo. Read weekly numbers, with the count of photos.
  • A cost with no range. With 95 photos, one damaged photo moves the cost a lot. Give a range, or say how many photos it rests on.
  • Accuracy for the search. The search has no classes to be accurate about. Use recall at 1 and at 5.
Show the public check values

Open this only after you have your own numbers. These values come from the public data, so you can check them. They are not the answers to the assignment.

  • Data: 95 returns photos from 2026-04-01 to 2026-04-30; the replay weeks have 26, 24, 24 and 21 photos. 102 kettle photos (64 train, 38 test) and 102 customer photos. Every file in SHA256SUMS is OK.
  • Embedding check: your CLIP, DINOv2 and ResNet-18 embeddings of the course’s photos have a cosine similarity of 1.000 with data/embeddings/*.npz.
  • Catalogue (198 test photos): colour histogram + 5 neighbours about 0.25, HOG + logistic regression about 0.35. The balanced CLIP probe, trained with the 64 kettles: 0.939 on all nine classes, and 0.944 on the 160 old photos, the same as the course’s row.
  • Search, check without answers (gallery of 835 photos): the share of first matches that are kettle photos is about 0.99 for DINOv2, 0.86 for CLIP, 0.74 for ResNet-18 and under 0.05 for the colour histogram and the perceptual hash.
  • First warehouse, as the course: small PatchCore 988.1 dollars a day (AUROC 0.836), supervised ResNet-18 1,106.4 (0.914), CLIP prompt 1,012.8 (0.808).
  • The service as it is, over the second warehouse: all 95 photos fail. The mean score is about 21 in every week, against about 7 for the good validation photos. The PSI is 8.28 and both alarms fire in all four weeks.

Recap

You can now take a vision problem from a request to a decision. You frame each request by its kind of answer and its costs. You compare a baseline and several method families on the same photos, by class and by camera. You notice when a new camera breaks a good inspector, fix it without labels, and set the lines from the review budget. You run the service with its review folder, its face blur and its alarms, and explain to Hamid what it costs and what a person still checks. The case studies of Module 9 use the same steps on real data: Visual inspection with few defects, Classify products and pets from photos and Count and detect in crowded scenes.