Skip to content
Français

The image service and its review queue

  • Lesson 29
  • 45 min
  • Module 8: Run vision in production

Where we are

Our inspector is now fast and light enough for the desk's computer. It still lives in a notebook and a command. In this lesson, Amira serves it as a small image service. It answers with a confidence, falls back to a person when it cannot judge, and keeps a review folder. Then we turn reviewed photos into new labels without bias.

What you will be able to do

Put an inspector where the returns desk can use it, without letting it guess. A service that always answers is dangerous: it judges a black photo as confidently as a good one. Amira’s service says how sure it is and hands unclear cases to a person with the reason. It keeps what the person decides, so the next model learns from it.

NotebookDo this lesson in a notebookColab · Kaggle · GitHub

The notebook builds the service’s decision step by hand with the service’s own inspector: the score, the zone, the confidence, the fallback and the review folder. Then it scores the validation and test returns and shows which photos a person sees. Its setup cells install ONNX Runtime and download the returns photos and the inspector (18 MB). About 35 minutes.

Colab and Kaggle open your own copy. They need a free Google or Kaggle account. Stuck? See the solution notebook.

Before you startWhat to know and to have readyAbout 45 minutes

What you need to know

What you need on your computer (for the project steps; the notebook needs none of it)

  • Your larkvision project at the end of Module 8 (snapshot C42/snapshots/end-of-m08), set up in Domain shift, with its .venv environment.
  • curl, a small program that sends web requests. macOS, Linux and Windows 10 or later include it (on Windows, type curl.exe).

Tested on macOS with Python 3.12.13, FastAPI 0.162.2, Uvicorn 0.54.0 and ONNX Runtime 1.31.1. The Windows commands are written but were not run.

At a good returns counter, the worker has three trays: back on sale, outlet, and “not sure”. She also has a rule for bad days. If the scanner breaks, she does not guess. She puts the item in the “not sure” tray with a note. At the end of the day, a senior worker empties that tray. Each decision she makes there teaches the team something.

Amira’s image service copies this counter. The inspector gives the first answer. The service adds the “not sure” tray, the note, and the rule for bad days.

The inspector is a function: photo in, score out. It never says “I cannot judge this”. Priya sent it three photos that no worker would judge. One is black (the lens cap was on), one is white (a flash on a white wall), and one is a thumbnail of 64 × 58 pixels.

Photo Score Zone with the lines 8.761 / 10.826
Black, 320 × 291 15.853 fail
White, 320 × 291 17.595 fail
A good candle, made 64 × 58 pixels 12.359 fail

All three “fail”. At the desk, each one would send an item to the outlet: 28 dollars lost if the item was good. The candle in the thumbnail is good: its full-size photo passes with a score of 5.832. The model has no way to say “this is not a photo I know”. The service must say it.

A service is a program that waits for requests and answers them, here over HTTP, the protocol of the web. The desk’s software sends a photo; the service sends back a small JSON answer. The rules around the model are what make it safe to use.

A flow from left to right. A photo of a returned item goes to 1. Can we judge it (a photo, not too dark, not too small)? If not: fallback, check. Then 2. the inspector: score, zone and confidence. If the score is between the lines or the model fails: to review. Then 3. the face finder blurs every face before any copy. If there is a face: to review. The flow ends in the answer: pass, check or fail with reasons. All review arrows lead to the review folder: a blurred copy and the reasons, never the original photo. A person looks at the review folder and decides.
One request through Amira's service. Every orange arrow sends a blurred copy and its reasons to the review folder for a person.
Text description of the diagram

A row of boxes (a column on a phone). A photo of a returned item enters box 1, “Can we judge it?”: is it a photo, not too dark, not too small? If not, an orange arrow says “fallback = check”. Box 2 is the inspector: score, zone and confidence. An orange arrow from it says “between the lines or model error”. Box 3 is the face finder, which blurs every face before any copy is kept. An orange arrow says “a face”. The last box, green, is the answer: pass, check or fail, with reasons. Under the row, the orange arrows end in one box: “review/<day>/: a blurred copy + the reasons (never the original photo)”. A note says that a person looks at the review folder and decides.

  1. A confidence. With every answer, the service says how far the score is from the nearest line. Far from a line, the zone is clear. Near a line, a small change in the photo could change the zone.
  2. A fallback. When the service cannot judge (the photo is too dark, too small, not a photo, or the model fails), it does not guess. It answers “check”: a person decides.
  3. A review folder. Every photo that needs a person is copied to review/<day>/, with a small file of reasons. The face finder of the next lesson blurs faces before the copy is written.

The service’s lines are 8.761 (check) and 10.826 (fail). The confidence is the distance to the nearest line, as a share of that line, at most 1. Between the lines it is 0: the zone itself says “a person decides”.

Photo Score Zone Confidence
A good candle 5.832 pass (8.761 − 5.832) ÷ 8.761 = 0.334
A candle with extra wax 9.924 check between the lines: 0
A candle with a bent wick 19.445 fail (19.445 − 10.826) ÷ 10.826 = 0.796

This confidence is not a probability. It does not say “79.6% sure the wick is bent”. It only says how far the score is from a line, so the desk can sort answers from “clear” to “close”.

The service is in larkvision/service.py. It uses FastAPI, a Python library for web services. The heart of it:

@app.post("/inspect")
async def inspect(file: UploadFile = File(...), item_id: str = Query("upload")):
if not ITEM_ID.match(item_id):
raise HTTPException(422, "item_id: letters, digits, '-' and '_' only, at most 40")
raw = await file.read()
if len(raw) > 10_000_000:
raise HTTPException(413, "the photo is larger than 10 MB")
try:
img = Image.open(io.BytesIO(raw))
img.load()
except (UnidentifiedImageError, OSError):
raise HTTPException(422, "the file is not a photo that we can read") from None
return judge(img.convert("RGB"), item_id, inspector, find_faces, review_dir)

The checks come before the model. The item ID may contain only letters, digits, - and _, because it becomes a file name in the review folder: ../x would write outside it. A file that is not a photo gets the HTTP status 422 (the request is wrong), not a score. judge in larkvision/review.py does the rest: the fallback, the score, the zone, the confidence, the faces and the review copy.

  1. Open a terminal in your larkvision folder and activate the environment (source .venv/bin/activate, or .venv\Scripts\Activate.ps1 on Windows).

  2. In a new terminal, set LARKVISION_DATA and LARKVISION_MODELS again, with the commands of step 6 in Domain shift.

  3. Start the service:

    Terminal window
    vision serve
    INFO: Started server process [17455]
    INFO: Waiting for application startup.
    INFO: Application startup complete.
    INFO: Uvicorn running on http://127.0.0.1:8042 (Press CTRL+C to quit)

    The number in brackets is the process number: yours will differ. Leave this terminal open.

  4. Open a second terminal in the same folder. Ask the service if it is ready:

    Terminal window
    curl http://127.0.0.1:8042/health
    {"status":"ok","model_version":"inspector-1","check_line":8.761079921722413,"fail_line":10.826346116065979}
  5. Send a good candle. On macOS and Linux:

    Terminal window
    curl -F "file=@../nextia-learning-labs/C42/data/returns/test/R-6a26ad64.jpg" "http://127.0.0.1:8042/inspect?item_id=R-6a26ad64"

    On Windows: curl.exe -F "file=@..\nextia-learning-labs\C42\data\returns\test\R-6a26ad64.jpg" "http://127.0.0.1:8042/inspect?item_id=R-6a26ad64".

    {"item_id":"R-6a26ad64","zone":"pass","score":5.8323,"confidence":0.334,"faces":0,"review":false,"reasons":[],"model_version":"inspector-1","latency_ms":86.7}

    The first request is slower (86.7 ms here): the model warms up.

  6. Send the candle with extra wax (R-33640732.jpg, same command with that name twice). It lands between the lines:

    {"item_id":"R-33640732","zone":"check","score":9.9238,"confidence":0.0,"faces":0,"review":true,"reasons":["unsure: the score is between the two lines"],"model_version":"inspector-1","latency_ms":22.0}
  7. Send a file that is not a photo, for example README.md, with item_id=R-x. Add -i to curl to see the status line:

    HTTP/1.1 422 Unprocessable Entity
    {"detail":"the file is not a photo that we can read"}
  8. Make a black photo and send it with item_id=R-dark:

    Terminal window
    python -c "from PIL import Image; Image.new('RGB', (320, 291), (8, 8, 8)).save('dark.jpg')"
    curl -F "file=@dark.jpg" "http://127.0.0.1:8042/inspect?item_id=R-dark"
    {"item_id":"R-dark","zone":"check","score":null,"confidence":0.0,"faces":0,"review":true,"reasons":["fallback: photo too dark (mean brightness 8 of 255)"],"model_version":"inspector-1","latency_ms":2.3}

    The fallback answered: no score, and a person decides. Delete dark.jpg afterwards.

  9. Look in the review folder. It has review/today/R-33640732.jpg and .json, and the same for R-dark. The good candle is not there: nobody needs to look at it.

  10. Stop the service: go back to its terminal and press Ctrl+C. Check that the last line is Finished server process.

If curl says Connection refused, the service is not running, or it runs on another port. If vision serve says address already in use, another program uses port 8042: run vision serve --port 8043 and use 8043 in the commands.

The service’s lines were chosen on the validation photos for Hamid’s 100 checks a day, as in Module 6. Try other budgets with the service’s own scores: the 112 validation and 218 test returns, scored by the int8 inspector.

Explore

The service's pass, check and fail zones

passcheckfail025505.375 to 5.750: 1 good photos5.750 to 6.125: 14 good photos6.125 to 6.500: 50 good photos6.500 to 6.875: 34 good photos6.875 to 7.250: 26 good photos7.250 to 7.625: 14 good photos7.625 to 8.000: 14 good photos8.000 to 8.375: 7 good photos8.375 to 8.750: 10 good photos8.750 to 9.125: 1 good photos9.125 to 9.500: 8 good photos9.500 to 9.875: 7 good photos9.875 to 10.250: 4 good photos10.250 to 10.625: 7 good photos10.625 to 11.000: 2 good photos11.000 to 11.375: 1 good photosGood returns (200)R-344c8b55: different colour spot, score 6.604, passR-a9048b86: damaged corner of packaging; different colour spot, score 6.756, passR-c3b7e506: damaged corner of packaging, score 7.108, passR-74669eb6: damaged corner of packaging, score 7.209, passR-a796ccf6: extra wax in candle, score 7.225, passR-de7e819e: different colour spot, score 8.384, passR-0d4c03e5: wax melded out of the candle, score 8.544, passR-d17d264e: wax melded out of the candle, score 9.330, checkR-33640732: extra wax in candle, score 9.924, checkR-15b594e6: different colour spot, score 10.100, checkR-aa4a48cb: foreign particals on candle, score 11.639, failR-800400d7: wax melded out of the candle, score 11.699, failR-31ebdb39: wax melded out of the candle, score 11.857, failR-5918b873: chunk of wax missing, score 12.889, failR-8cb58274: damaged corner of packaging; different colour spot; other, score 12.985, failR-ef0b6f29: extra wax in candle, score 13.530, failR-5cf1ce29: foreign particals on candle; wax melded out of the candle, score 14.409, failR-1cc8cc10: weird candle wick, score 19.445, failDamaged returns (18): one dot each68101214161820Anomaly score (higher = less like a good candle)check from 8.761fail from 10.826
passcheckfail0505.600 to 6.200: 20 good photos6.200 to 6.800: 76 good photos6.800 to 7.400: 34 good photos7.400 to 8.000: 23 good photos8.000 to 8.600: 15 good photos8.600 to 9.200: 4 good photos9.200 to 9.800: 13 good photos9.800 to 10.400: 9 good photos10.400 to 11.000: 5 good photos11.000 to 11.600: 1 good photosGood returns (200)R-344c8b55: different colour spot, score 6.604, passR-a9048b86: damaged corner of packaging; different colour spot, score 6.756, passR-c3b7e506: damaged corner of packaging, score 7.108, passR-74669eb6: damaged corner of packaging, score 7.209, passR-a796ccf6: extra wax in candle, score 7.225, passR-de7e819e: different colour spot, score 8.384, passR-0d4c03e5: wax melded out of the candle, score 8.544, passR-d17d264e: wax melded out of the candle, score 9.330, checkR-33640732: extra wax in candle, score 9.924, checkR-15b594e6: different colour spot, score 10.100, checkR-aa4a48cb: foreign particals on candle, score 11.639, failR-800400d7: wax melded out of the candle, score 11.699, failR-31ebdb39: wax melded out of the candle, score 11.857, failR-5918b873: chunk of wax missing, score 12.889, failR-8cb58274: damaged corner of packaging; different colour spot; other, score 12.985, failR-ef0b6f29: extra wax in candle, score 13.530, failR-5cf1ce29: foreign particals on candle; wax melded out of the candle, score 14.409, failR-1cc8cc10: weird candle wick, score 19.445, failDamaged returns (18): one dot each5101520Anomaly score (higher = less like a good candle)check from 8.761fail from 10.826
validation
112 photos
test
218 photos
Checks a day (budget 100)96.488.1
Damaged items sold as new a day21.419.3
Good items lost a day02.8
Damaged photos that pass4 of 127 of 18
Cost a day (assumptions)1,012.5988.1

On the test photos, these lines send 88.1 returns a day to a person (budget 100), sell 19.3 damaged items a day as new and send 2.8 good items a day to the outlet: 988.1 dollars a day. With no inspection, the same photos cost 2,229.4 dollars a day. These are the cheapest lines on the validation photos for 100 checks a day.

The lines are chosen on the validation photos and measured on the test photos. Costs are the course’s assumptions: a damaged item sold as new 45 dollars, a good item lost 28 dollars, a check 0.50 dollars; a person who checks is always right. Per day at 600 returns.

The scores of the service's inspector (the small PatchCore in ONNX int8) on the 112 validation and 218 test returns. Pick a budget to choose the cheapest lines on validation, or move each line yourself.
Text description of the explorer

The explorer shows two panels on one score axis from 5 to 20. The top panel is a histogram of the good returns; the bottom panel shows each damaged return as one dot. Behind them, three coloured zones: pass (green) left of the check line, check (grey) between the lines, and fail (orange) right of the fail line. A damaged dot in the pass zone has a dark ring: it is sold as new. Controls: the budget (25 to 150 checks a day), a slider for each line, and the photos in the picture. A table has one column for the validation photos and one for the test photos, with the checks a day, the damaged items sold as new, the good items lost, the damaged photos that pass, and the cost a day. At the start, with 100 checks: check from 8.761, fail from 10.826; validation 1,012.5 dollars a day, test 988.1. With 50 checks: check from 9.743, fail unchanged; test 1,092.7 dollars a day.

This time the check line moves up, from 8.761 to 9.743, and the fail line stays. The cost on the test photos goes from 988.1 to 1,092.7 dollars a day: one more damaged candle in 218 passes. For this inspector, the cheapest way to halve the checks is to pass the photos just above the old check line, not to fail more photos. Each inspector has its own best move, so choose the lines again for each one.

With 25 checks, the cost is 1,282.6 dollars a day, and the test photos need 27.5 checks a day: over the budget. Lines chosen on 112 photos do not fit the next photos exactly.

Reviewed photos become labels, without bias

Section titled “Reviewed photos become labels, without bias”

Every photo in the review folder gets a decision from a person: good or damaged. That is a new label for free. It is tempting to collect these labels and set the lines again from them. Look at which photos a person sees first.

Zone (218 test photos) Photos Damaged Does a person see it?
pass 177 7 no
check 32 3 yes
fail 9 8 no

A person sees only the check zone: photos with scores between 8.761 and 10.826. The 7 damaged candles that pass (small colour spots, damaged packet corners, scores from 6.6 to 8.5) are never seen. Labels from the review folder alone say nothing about them. Lines set again from those labels will miss them again. This is selection bias: the labels come from a part of the photos that a rule selected.

Three boxes on the left: pass, 177 test photos, 7 damaged (4%); check, 32 test photos, 3 damaged (9%); fail, 9 test photos, 8 damaged (89%). A solid arrow goes from check, and dashed arrows labelled a random sample go from pass and fail, to a box: a person labels it, without seeing the score. Then to a box: new labels, each with why it was picked, zone or random. A note in orange: only the check zone means labels between the lines only; the 7 damaged that pass are never seen.
Feeding reviewed photos back into labels: all checks, plus a small random sample of the pass and fail zones, each label with the reason it was picked. Counts: the service's inspector on the 218 test photos.
Text description of the diagram

On the left, three stacked boxes: pass (177 test photos, 7 damaged, 4%), check (32 photos, 3 damaged, 9%) and fail (9 photos, 8 damaged, 89%). A solid arrow goes from check, and two dashed blue arrows labelled “a random sample” go from pass and from fail, to a box “A person labels it (without seeing the score)”. An arrow leads on to “New labels, each with WHY it was picked (zone or random)”. An orange note says: with only the check zone, every label lies between the lines, and the 7 damaged candles that pass are never seen.

Three rules keep the labels honest:

  1. Add a random sample of the other zones. Hamid gives about 12 of his 100 checks a day to random items from the pass zone. A few more go to the fail zone.
  2. Record why each photo was picked: “check zone” or “random sample”. Later, Priya can weigh each group by its true share of the returns.
  3. Hide the score from the person. A person who sees “score 10.7, almost fail” looks for damage that may not be there.

How much does the random sample find? On the test photos, the pass zone holds 177 of 218 photos. At 600 returns a day, that is about 487 passed items a day. About 19.3 of them are damaged: the “damaged sold as new” of the cost. Twelve random checks from 487 items find about 12 × 19.3 ÷ 487 = 0.48 damaged candles a day: about 14 in a month. Few, but they are exactly the ones the inspector cannot see. This is an estimate from the test photos, not a measurement at the desk.

vision replay (from Domain shift) also fills a review folder: 12 photos for the month, each in a folder for its day. Six are in days 1–7, four in days 8–15, and one in each of the last two weeks. After the camera change, the folder is almost empty, because the inspector fails everything with confidence. An empty review folder is not good news. Watch its size with the cost alarm.

When to use a service like this, and what it costs

Section titled “When to use a service like this, and what it costs”

Use a service with a confidence, a fallback and a review folder whenever a model’s answer becomes an action with a price. Here the price is an item sold or lost.

Do not hide the fallback behind a default answer such as “pass”. And do not let the review folder grow without a person who empties it every day: then it is a log, not a review.

What it costs: a person’s time for every check and every random sample, and a place to store the blurred review copies and their labels. Someone must also look at the alarms. Running a service in production needs more than this lesson: health checks, versions, safe releases, logs and a way back. Run the live service and Controlled rollouts in MLOps and LLMOps: Operating AI in Production cover them.

Guided practice: the decision step in the notebook

Section titled “Guided practice: the decision step in the notebook”
  1. Open the notebook from the box Do this lesson in a notebook at the top of this page.
  2. Run the setup cells. The data cell ends with a Ready: line.
  3. In section 1, load the inspector. You see model inspector-1 check line 8.761 fail line 10.826.
  4. In section 2, predict the three zones, then run the cell. You see pass, check and fail, with the confidences 0.334, 0.0 and 0.796.
  5. In section 3, run the fallback cell. You see photo too dark and photo too small, then None for a real photo.
  6. In section 4, run judge. The review folder gets the unsure candle and the dark photo.
  7. In section 5, score every photo. You see AUROC on test: 0.836 and 'cost_per_day': 988.1.
  8. In section 6, count the damaged photos in each zone.

Your turn: what the review folder never shows

Section titled “Your turn: what the review folder never shows”

In the notebook, section 7, put the number of damaged test photos in the pass zone in missed_in_pass, then run the check cell. Predict first: more or fewer than in the check zone?

Show the answer
missed_in_pass = int(((test["zone"] == "pass") & test["damaged"]).sum())
print(missed_in_pass)
7

More: 7 in the pass zone, against 3 in the check zone. Their scores are from 6.60 to 8.54. Two have a different colour spot, two a damaged packet corner, and one both. One has extra wax, and one has wax melted out. These go back on sale as new, and no person sees them unless a random sample picks them.

Common mistake

New lines from the review folder only

Symptom: after a month, Priya has 900 labelled photos from the review folder. Tomás proposes to set the lines again on them alone. The new lines look excellent on those photos. The next month, damaged candles with small colour spots keep coming back from customers.

Cause: every photo in the review folder had a score between the two lines. The new lines learned only that narrow range. The damaged candles with low scores were never in the folder, so nothing could teach the lines about them.

Fix: keep a random sample of every zone in the labels, and record why each photo was picked. Choose the lines on a validation set that includes the random sample.

Consequence in AI work: a model that learns only from the cases it was unsure about becomes more and more sure about its blind spots.

Check your understanding

With lines 8.761 (check) and 10.826 (fail), a photo scores 12.0. What zone and confidence does the service give, and what does that confidence mean?

Show the answer

12.0 is above the fail line, so the zone is fail. The confidence is (12.0 − 10.826) ÷ 10.826 = 0.108. It means the score is about 11% above the fail line: a fairly close call. It is not a probability that the item is damaged.

Check your understanding

The desk's camera sends a black photo because the lens cap is on. What does the bare inspector do with it, and what does Amira's service do?

Show the answer

The bare inspector gives it a score like any other photo: 15.853, above the fail line. So the item would go to the outlet, and a good item would cost 28 dollars. The service checks the photo before the model. Its mean brightness is 8 of 255, below 25, so the fallback answers “check” with no score. A blurred copy and the reason “photo too dark” go to the review folder, and a person decides.

Check your understanding

Priya wants to set the lines again with labels from the review folder. Name the bias, and two rules that fix it.

Show the answer

Selection bias. The review folder holds only photos with scores between the lines. So damaged candles with low scores (7 of the 18 damaged test photos) are never labelled. Fixes: send a small random sample of the pass and fail zones to a person too. Record why each photo was picked (zone or random), so each group can be weighed. Hide the score from the person who labels.

Recap

  • A model always answers: the bare inspector “failed” a black photo, a white photo and a thumbnail of a good candle.
  • Amira’s service adds three rules: a confidence (the distance to the nearest line, not a probability), a fallback (“check” when it cannot judge) and a review folder with blurred copies and reasons.
  • vision serve runs it on port 8042: /health, and /inspect with a photo; a file that is not a photo gets HTTP 422. Stop it with Ctrl+C.
  • On the test photos, the service’s inspector has AUROC 0.836 and costs 988.1 dollars a day at 100 checks. At 50 checks its check line moves up: 1,092.7.
  • A person sees only the check zone. 7 of the 18 damaged test candles pass and never reach the review folder. Add a random sample of every zone, record why each photo was picked, and hide the score.

The service now sends unsure photos to a person. Some photos need care for another reason: a person leans over the table, and their face is in the photo. Next, the service blurs faces by default and keeps as little as it can.

Next: People in images