The image service and its review queue
- Lesson 29
- 45 min
- Module 8: Run vision in production
Where we are
Our inspector is now fast and light enough for the desk's computer. It still lives in a notebook and a command. In this lesson, Amira serves it as a small image service. It answers with a confidence, falls back to a person when it cannot judge, and keeps a review folder. Then we turn reviewed photos into new labels without bias.
Lesson 29 of 30 · Vision AI: See, Count and InspectThe image service and its review queue
1. Seeing is a business problem
2. Baselines and pretrained classifiers
3. Embeddings, CLIP and few labels
4. Detect and count
6. Inspect for defects
7. Match, search and describe
8. Run vision in production
What you will be able to do
Put an inspector where the returns desk can use it, without letting it guess. A service that always answers is dangerous: it judges a black photo as confidently as a good one. Amira’s service says how sure it is and hands unclear cases to a person with the reason. It keeps what the person decides, so the next model learns from it.
NotebookDo this lesson in a notebook
The notebook builds the service’s decision step by hand with the service’s own inspector: the score, the zone, the confidence, the fallback and the review folder. Then it scores the validation and test returns and shows which photos a person sees. Its setup cells install ONNX Runtime and download the returns photos and the inspector (18 MB). About 35 minutes.
Colab and Kaggle open your own copy. They need a free Google or Kaggle account. Stuck? See the solution notebook.
Before you startWhat to know and to have ready
What you need to know
- Pass, check and fail lines, from Set the line with the warehouse.
- The service’s inspector (the small PatchCore in int8), from Fast enough on a CPU.
What you need on your computer (for the project steps; the notebook needs none of it)
- Your
larkvisionproject at the end of Module 8 (snapshotC42/snapshots/end-of-m08), set up in Domain shift, with its.venvenvironment. curl, a small program that sends web requests. macOS, Linux and Windows 10 or later include it (on Windows, typecurl.exe).
Tested on macOS with Python 3.12.13, FastAPI 0.162.2, Uvicorn 0.54.0 and ONNX Runtime 1.31.1. The Windows commands are written but were not run.
The “not sure” tray
Section titled “The “not sure” tray”At a good returns counter, the worker has three trays: back on sale, outlet, and “not sure”. She also has a rule for bad days. If the scanner breaks, she does not guess. She puts the item in the “not sure” tray with a note. At the end of the day, a senior worker empties that tray. Each decision she makes there teaches the team something.
Amira’s image service copies this counter. The inspector gives the first answer. The service adds the “not sure” tray, the note, and the rule for bad days.
The problem: a model always answers
Section titled “The problem: a model always answers”The inspector is a function: photo in, score out. It never says “I cannot judge this”. Priya sent it three photos that no worker would judge. One is black (the lens cap was on), one is white (a flash on a white wall), and one is a thumbnail of 64 × 58 pixels.
| Photo | Score | Zone with the lines 8.761 / 10.826 |
|---|---|---|
| Black, 320 × 291 | 15.853 | fail |
| White, 320 × 291 | 17.595 | fail |
| A good candle, made 64 × 58 pixels | 12.359 | fail |
All three “fail”. At the desk, each one would send an item to the outlet: 28 dollars lost if the item was good. The candle in the thumbnail is good: its full-size photo passes with a score of 5.832. The model has no way to say “this is not a photo I know”. The service must say it.
The idea: a model plus three rules
Section titled “The idea: a model plus three rules”A service is a program that waits for requests and answers them, here over HTTP, the protocol of the web. The desk’s software sends a photo; the service sends back a small JSON answer. The rules around the model are what make it safe to use.
Text description of the diagram
A row of boxes (a column on a phone). A photo of a returned item enters box 1, “Can we judge it?”: is it a photo, not too dark, not too small? If not, an orange arrow says “fallback = check”. Box 2 is the inspector: score, zone and confidence. An orange arrow from it says “between the lines or model error”. Box 3 is the face finder, which blurs every face before any copy is kept. An orange arrow says “a face”. The last box, green, is the answer: pass, check or fail, with reasons. Under the row, the orange arrows end in one box: “review/<day>/: a blurred copy + the reasons (never the original photo)”. A note says that a person looks at the review folder and decides.
- A confidence. With every answer, the service says how far the score is from the nearest line. Far from a line, the zone is clear. Near a line, a small change in the photo could change the zone.
- A fallback. When the service cannot judge (the photo is too dark, too small, not a photo, or the model fails), it does not guess. It answers “check”: a person decides.
- A review folder. Every photo that needs a person is copied to
review/<day>/, with a small file of reasons. The face finder of the next lesson blurs faces before the copy is written.
A tiny example: the confidence
Section titled “A tiny example: the confidence”The service’s lines are 8.761 (check) and 10.826 (fail). The confidence is the distance to the nearest line, as a share of that line, at most 1. Between the lines it is 0: the zone itself says “a person decides”.
| Photo | Score | Zone | Confidence |
|---|---|---|---|
| A good candle | 5.832 | pass | (8.761 − 5.832) ÷ 8.761 = 0.334 |
| A candle with extra wax | 9.924 | check | between the lines: 0 |
| A candle with a bent wick | 19.445 | fail | (19.445 − 10.826) ÷ 10.826 = 0.796 |
This confidence is not a probability. It does not say “79.6% sure the wick is bent”. It only says how far the score is from a line, so the desk can sort answers from “clear” to “close”.
Larkfield: Amira’s service
Section titled “Larkfield: Amira’s service”The service is in larkvision/service.py. It uses FastAPI, a Python library for web services. The heart of it:
@app.post("/inspect")async def inspect(file: UploadFile = File(...), item_id: str = Query("upload")): if not ITEM_ID.match(item_id): raise HTTPException(422, "item_id: letters, digits, '-' and '_' only, at most 40") raw = await file.read() if len(raw) > 10_000_000: raise HTTPException(413, "the photo is larger than 10 MB") try: img = Image.open(io.BytesIO(raw)) img.load() except (UnidentifiedImageError, OSError): raise HTTPException(422, "the file is not a photo that we can read") from None return judge(img.convert("RGB"), item_id, inspector, find_faces, review_dir)The checks come before the model. The item ID may contain only letters, digits, - and _, because it becomes a file name in the review folder: ../x would write outside it. A file that is not a photo gets the HTTP status 422 (the request is wrong), not a score. judge in larkvision/review.py does the rest: the fallback, the score, the zone, the confidence, the faces and the review copy.
Run the service
Section titled “Run the service”-
Open a terminal in your
larkvisionfolder and activate the environment (source .venv/bin/activate, or.venv\Scripts\Activate.ps1on Windows). -
In a new terminal, set
LARKVISION_DATAandLARKVISION_MODELSagain, with the commands of step 6 in Domain shift. -
Start the service:
Terminal window vision serveINFO: Started server process [17455]INFO: Waiting for application startup.INFO: Application startup complete.INFO: Uvicorn running on http://127.0.0.1:8042 (Press CTRL+C to quit)The number in brackets is the process number: yours will differ. Leave this terminal open.
-
Open a second terminal in the same folder. Ask the service if it is ready:
Terminal window curl http://127.0.0.1:8042/health{"status":"ok","model_version":"inspector-1","check_line":8.761079921722413,"fail_line":10.826346116065979} -
Send a good candle. On macOS and Linux:
Terminal window curl -F "file=@../nextia-learning-labs/C42/data/returns/test/R-6a26ad64.jpg" "http://127.0.0.1:8042/inspect?item_id=R-6a26ad64"On Windows:
curl.exe -F "file=@..\nextia-learning-labs\C42\data\returns\test\R-6a26ad64.jpg" "http://127.0.0.1:8042/inspect?item_id=R-6a26ad64".{"item_id":"R-6a26ad64","zone":"pass","score":5.8323,"confidence":0.334,"faces":0,"review":false,"reasons":[],"model_version":"inspector-1","latency_ms":86.7}The first request is slower (86.7 ms here): the model warms up.
-
Send the candle with extra wax (
R-33640732.jpg, same command with that name twice). It lands between the lines:{"item_id":"R-33640732","zone":"check","score":9.9238,"confidence":0.0,"faces":0,"review":true,"reasons":["unsure: the score is between the two lines"],"model_version":"inspector-1","latency_ms":22.0} -
Send a file that is not a photo, for example
README.md, withitem_id=R-x. Add-ito curl to see the status line:HTTP/1.1 422 Unprocessable Entity{"detail":"the file is not a photo that we can read"} -
Make a black photo and send it with
item_id=R-dark:Terminal window python -c "from PIL import Image; Image.new('RGB', (320, 291), (8, 8, 8)).save('dark.jpg')"curl -F "file=@dark.jpg" "http://127.0.0.1:8042/inspect?item_id=R-dark"{"item_id":"R-dark","zone":"check","score":null,"confidence":0.0,"faces":0,"review":true,"reasons":["fallback: photo too dark (mean brightness 8 of 255)"],"model_version":"inspector-1","latency_ms":2.3}The fallback answered: no score, and a person decides. Delete
dark.jpgafterwards. -
Look in the review folder. It has
review/today/R-33640732.jpgand.json, and the same forR-dark. The good candle is not there: nobody needs to look at it. -
Stop the service: go back to its terminal and press Ctrl+C. Check that the last line is
Finished server process.
If curl says Connection refused, the service is not running, or it runs on another port. If vision serve says address already in use, another program uses port 8042: run vision serve --port 8043 and use 8043 in the commands.
How many photos reach a person
Section titled “How many photos reach a person”The service’s lines were chosen on the validation photos for Hamid’s 100 checks a day, as in Module 6. Try other budgets with the service’s own scores: the 112 validation and 218 test returns, scored by the int8 inspector.
Explore
The service's pass, check and fail zones
Predict first: Hamid's budget falls from 100 to 50 checks a day. For PatchCore with a wide ResNet-50, the fail line came down. For the service's inspector, which line moves?
| validation 112 photos | test 218 photos | |
|---|---|---|
| Checks a day (budget 100) | 96.4 | 88.1 |
| Damaged items sold as new a day | 21.4 | 19.3 |
| Good items lost a day | 0 | 2.8 |
| Damaged photos that pass | 4 of 12 | 7 of 18 |
| Cost a day (assumptions) | 1,012.5 | 988.1 |
On the test photos, these lines send 88.1 returns a day to a person (budget 100), sell 19.3 damaged items a day as new and send 2.8 good items a day to the outlet: 988.1 dollars a day. With no inspection, the same photos cost 2,229.4 dollars a day. These are the cheapest lines on the validation photos for 100 checks a day.
The lines are chosen on the validation photos and measured on the test photos. Costs are the course’s assumptions: a damaged item sold as new 45 dollars, a good item lost 28 dollars, a check 0.50 dollars; a person who checks is always right. Per day at 600 returns.
Text description of the explorer
The explorer shows two panels on one score axis from 5 to 20. The top panel is a histogram of the good returns; the bottom panel shows each damaged return as one dot. Behind them, three coloured zones: pass (green) left of the check line, check (grey) between the lines, and fail (orange) right of the fail line. A damaged dot in the pass zone has a dark ring: it is sold as new. Controls: the budget (25 to 150 checks a day), a slider for each line, and the photos in the picture. A table has one column for the validation photos and one for the test photos, with the checks a day, the damaged items sold as new, the good items lost, the damaged photos that pass, and the cost a day. At the start, with 100 checks: check from 8.761, fail from 10.826; validation 1,012.5 dollars a day, test 988.1. With 50 checks: check from 9.743, fail unchanged; test 1,092.7 dollars a day.
What the budget of 50 does here
Section titled “What the budget of 50 does here”This time the check line moves up, from 8.761 to 9.743, and the fail line stays. The cost on the test photos goes from 988.1 to 1,092.7 dollars a day: one more damaged candle in 218 passes. For this inspector, the cheapest way to halve the checks is to pass the photos just above the old check line, not to fail more photos. Each inspector has its own best move, so choose the lines again for each one.
With 25 checks, the cost is 1,282.6 dollars a day, and the test photos need 27.5 checks a day: over the budget. Lines chosen on 112 photos do not fit the next photos exactly.
Reviewed photos become labels, without bias
Section titled “Reviewed photos become labels, without bias”Every photo in the review folder gets a decision from a person: good or damaged. That is a new label for free. It is tempting to collect these labels and set the lines again from them. Look at which photos a person sees first.
| Zone (218 test photos) | Photos | Damaged | Does a person see it? |
|---|---|---|---|
| pass | 177 | 7 | no |
| check | 32 | 3 | yes |
| fail | 9 | 8 | no |
A person sees only the check zone: photos with scores between 8.761 and 10.826. The 7 damaged candles that pass (small colour spots, damaged packet corners, scores from 6.6 to 8.5) are never seen. Labels from the review folder alone say nothing about them. Lines set again from those labels will miss them again. This is selection bias: the labels come from a part of the photos that a rule selected.
Text description of the diagram
On the left, three stacked boxes: pass (177 test photos, 7 damaged, 4%), check (32 photos, 3 damaged, 9%) and fail (9 photos, 8 damaged, 89%). A solid arrow goes from check, and two dashed blue arrows labelled “a random sample” go from pass and from fail, to a box “A person labels it (without seeing the score)”. An arrow leads on to “New labels, each with WHY it was picked (zone or random)”. An orange note says: with only the check zone, every label lies between the lines, and the 7 damaged candles that pass are never seen.
Three rules keep the labels honest:
- Add a random sample of the other zones. Hamid gives about 12 of his 100 checks a day to random items from the pass zone. A few more go to the fail zone.
- Record why each photo was picked: “check zone” or “random sample”. Later, Priya can weigh each group by its true share of the returns.
- Hide the score from the person. A person who sees “score 10.7, almost fail” looks for damage that may not be there.
How much does the random sample find? On the test photos, the pass zone holds 177 of 218 photos. At 600 returns a day, that is about 487 passed items a day. About 19.3 of them are damaged: the “damaged sold as new” of the cost. Twelve random checks from 487 items find about 12 × 19.3 ÷ 487 = 0.48 damaged candles a day: about 14 in a month. Few, but they are exactly the ones the inspector cannot see. This is an estimate from the test photos, not a measurement at the desk.
The review folder of the month
Section titled “The review folder of the month”vision replay (from Domain shift) also fills a review folder: 12 photos for the month, each in a folder for its day. Six are in days 1–7, four in days 8–15, and one in each of the last two weeks. After the camera change, the folder is almost empty, because the inspector fails everything with confidence. An empty review folder is not good news. Watch its size with the cost alarm.
When to use a service like this, and what it costs
Section titled “When to use a service like this, and what it costs”Use a service with a confidence, a fallback and a review folder whenever a model’s answer becomes an action with a price. Here the price is an item sold or lost.
Do not hide the fallback behind a default answer such as “pass”. And do not let the review folder grow without a person who empties it every day: then it is a log, not a review.
What it costs: a person’s time for every check and every random sample, and a place to store the blurred review copies and their labels. Someone must also look at the alarms. Running a service in production needs more than this lesson: health checks, versions, safe releases, logs and a way back. Run the live service and Controlled rollouts in MLOps and LLMOps: Operating AI in Production cover them.
Guided practice: the decision step in the notebook
Section titled “Guided practice: the decision step in the notebook”- Open the notebook from the box Do this lesson in a notebook at the top of this page.
- Run the setup cells. The data cell ends with a
Ready:line. - In section 1, load the inspector. You see
model inspector-1 check line 8.761 fail line 10.826. - In section 2, predict the three zones, then run the cell. You see
pass,checkandfail, with the confidences 0.334, 0.0 and 0.796. - In section 3, run the fallback cell. You see
photo too darkandphoto too small, thenNonefor a real photo. - In section 4, run
judge. The review folder gets the unsure candle and the dark photo. - In section 5, score every photo. You see
AUROC on test: 0.836and'cost_per_day': 988.1. - In section 6, count the damaged photos in each zone.
Your turn: what the review folder never shows
Section titled “Your turn: what the review folder never shows”In the notebook, section 7, put the number of damaged test photos in the pass zone in missed_in_pass, then run the check cell. Predict first: more or fewer than in the check zone?
Show the answer
missed_in_pass = int(((test["zone"] == "pass") & test["damaged"]).sum())print(missed_in_pass)7More: 7 in the pass zone, against 3 in the check zone. Their scores are from 6.60 to 8.54. Two have a different colour spot, two a damaged packet corner, and one both. One has extra wax, and one has wax melted out. These go back on sale as new, and no person sees them unless a random sample picks them.
Common mistake
New lines from the review folder only
Symptom: after a month, Priya has 900 labelled photos from the review folder. Tomás proposes to set the lines again on them alone. The new lines look excellent on those photos. The next month, damaged candles with small colour spots keep coming back from customers.
Cause: every photo in the review folder had a score between the two lines. The new lines learned only that narrow range. The damaged candles with low scores were never in the folder, so nothing could teach the lines about them.
Fix: keep a random sample of every zone in the labels, and record why each photo was picked. Choose the lines on a validation set that includes the random sample.
Consequence in AI work: a model that learns only from the cases it was unsure about becomes more and more sure about its blind spots.
Knowledge check
Section titled “Knowledge check”Check your understanding
With lines 8.761 (check) and 10.826 (fail), a photo scores 12.0. What zone and confidence does the service give, and what does that confidence mean?
Show the answer
12.0 is above the fail line, so the zone is fail. The confidence is (12.0 − 10.826) ÷ 10.826 = 0.108. It means the score is about 11% above the fail line: a fairly close call. It is not a probability that the item is damaged.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
The desk's camera sends a black photo because the lens cap is on. What does the bare inspector do with it, and what does Amira's service do?
Show the answer
The bare inspector gives it a score like any other photo: 15.853, above the fail line. So the item would go to the outlet, and a good item would cost 28 dollars. The service checks the photo before the model. Its mean brightness is 8 of 255, below 25, so the fallback answers “check” with no score. A blurred copy and the reason “photo too dark” go to the review folder, and a person decides.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
Priya wants to set the lines again with labels from the review folder. Name the bias, and two rules that fix it.
Show the answer
Selection bias. The review folder holds only photos with scores between the lines. So damaged candles with low scores (7 of the 18 damaged test photos) are never labelled. Fixes: send a small random sample of the pass and fail zones to a person too. Record why each photo was picked (zone or random), so each group can be weighed. Hide the score from the person who labels.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Recap
- A model always answers: the bare inspector “failed” a black photo, a white photo and a thumbnail of a good candle.
- Amira’s service adds three rules: a confidence (the distance to the nearest line, not a probability), a fallback (“check” when it cannot judge) and a review folder with blurred copies and reasons.
vision serveruns it on port 8042:/health, and/inspectwith a photo; a file that is not a photo gets HTTP 422. Stop it with Ctrl+C.- On the test photos, the service’s inspector has AUROC 0.836 and costs 988.1 dollars a day at 100 checks. At 50 checks its check line moves up: 1,092.7.
- A person sees only the check zone. 7 of the 18 damaged test candles pass and never reach the review folder. Add a random sample of every zone, record why each photo was picked, and hide the score.
The service now sends unsure photos to a person. Some photos need care for another reason: a person leans over the table, and their face is in the photo. Next, the service blurs faces by default and keeps as little as it can.
Next: People in images