Check what a vision-language model says
- Lesson 25
- 40 min
- Module 7: Match, search and describe
Where we are
In the last lesson, the vision-language models gave fluent answers, and we checked their shape. A well-shaped answer can still be wrong. In this lesson, we measure attribute accuracy against labels, find invented details and counting errors, and compare the cost per 1,000 images with CLIP and a classifier. First, a new colleague who writes very well.
Lesson 25 of 30 · Vision AI: See, Count and InspectCheck what a vision-language model says
1. Seeing is a business problem
2. Baselines and pretrained classifiers
3. Embeddings, CLIP and few labels
4. Detect and count
6. Inspect for defects
7. Match, search and describe
8. Run vision in production
What you will be able to do
Find out how often a vision-language model is right, field by field, and where it invents details. Then decide which of its answers can be used without a person, and whether a cheaper model does the job better.
NotebookDo this lesson in a notebook
The notebook scores the recorded answers of both models against the labels. It shows every wrong answer with its photo, and has one task with a check. It needs no account, no key and no model. Its setup cells download the catalogue, the pallets and the recordings (about 21 MB). About 35 minutes.
Colab and Kaggle open your own copy. They need a free Google or Kaggle account. Stuck? See the solution notebook.
Before you startWhat to know and to have ready
What you need to know
- The two models, the recorded answers and structured output, from Vision-language models.
- Accuracy and the cost of each kind of mistake, from The return that cost twice and Set the line with the warehouse.
What you need on your computer
- The notebook route: Colab, Kaggle or Jupyter. Nothing to install.
- The project route: your
larkvisionfolder withvision matchfrom Find the same product, and22 passed. This lesson adds one command and its tests.
Tested with Python 3.12.13 on macOS, in Jupyter and in a terminal. The Windows commands, Linux, Colab and Kaggle were not tested.
The colleague who writes well
Section titled “The colleague who writes well”Omar hires a new colleague to write product descriptions. Her texts are excellent: clear, warm, no spelling mistakes. On her first day she writes 50. Before they go on the website, Omar takes 10 and compares each text with its product. Eight are perfect. One calls a white candle “green”. One describes a vase that is really a wire basket full of eggs.
Omar does not stop trusting her. He learns where she makes mistakes, and he keeps checking that kind of field until she improves. Good writing made the mistakes harder to see, not less likely.
The problem: fluent is not the same as right
Section titled “The problem: fluent is not the same as right”Here are two recorded answers of the hosted model, with each part marked against the photo:

Text description of the diagram
The first photo is labelled plant pot. It shows tulips, small pink flowers and greenery in a wire basket full of striped eggs, on a wooden table. The model wrote that “a ceramic vase with stacked, ribbed shapes in white, grey and brown” holds the flowers. The flowers, the colours and the table are right. The ceramic vase is invented: the striped eggs look like ribbed shapes, and the model turned them into a vase. So its product type (vase) and material (ceramic) are wrong too. The second photo, labelled candle, shows two white candles in a green paper wrap printed with daisies. The front one is lit, and printed words are at the bottom. The model called them “green pillar candles”. The wax is white, and the wrap is green, so “green” is open to debate. Everything else is right, but the alt text does not mention the printed words.
The first answer is the dangerous kind. Most of it is right and specific: purple tulips, small pink blossoms, a wooden table. In the middle, it describes an object that is not there. A reader who checks two details and finds them right will believe the third. A model that states an invented detail as a fact is said to hallucinate. In this course we say invented detail, because that is what the reader gets.
The second answer shows another problem. Is the main colour of a white candle in a green wrap white or green? The question did not say. Before you can call an answer wrong, you need labels, and a clear rule for each field.
The idea: score each field against labels
Section titled “The idea: score each field against labels”To measure the model, you need the right answers for some photos: labels. The course has labels for the same 50 catalogue test photos that the models described:
- Product type comes from the catalogue labels.
- Main colour and material were labelled by one person, by eye. Where two answers are fair, the label lists both, for example
yellow|orange: 18 colour labels and 5 material labels have two. Where the photo does not show the material, the label isunclear, and that field is not scored: 6 photos.
One labeller is a weak base. A second person would disagree on some photos, and you would not know which one is right. Treat these labels as good enough to compare the two models, not as the truth.
The score of a field is its accuracy: right answers divided by scored answers.
A worked example: five photos
Section titled “A worked example: five photos”| Photo | Label (type / colour / material) | Hosted answer | Type | Colour | Material |
|---|---|---|---|---|---|
| P-9a3fc835 | bowl / white / ceramic | other / white / ceramic | wrong | right | right |
| P-a2f50fc3 | bowl / black / unclear | bowl / red / glass | right | wrong | not scored |
| P-7efd7bd8 | candle / white / wax | candle / green / wax | right | wrong | right |
| P-830b42bf | candle / white / wax or glass | candle / white / wax | right | right | right |
| P-84e4c4bf | candle / yellow or orange / wax | candle / yellow / wax | right | right | right |
Count each column. Product type: 4 right of 5, which is 80%. Colour: 3 of 5, which is 60%. Material: 4 of 4, which is 100%, because one material is not scored. Five photos are far too few to trust: one more photo moves a score by 20 points. The same count on all 50 photos gives the real result.
The results on 50 photos
Section titled “The results on 50 photos”- Hosted model
- SmolVLM-256M
Text description of the chart
A column chart with three pairs of columns, on an axis from 0 to 1. Product type: hosted model 0.84, SmolVLM 0.2. Main colour: hosted 0.86, SmolVLM 0.4. Material: hosted 0.932, SmolVLM 0.25. The hosted model’s columns are three to four times higher than the small model’s.
The hosted model gets 42 of 50 product types right (84%), 43 of 50 colours (86%) and 41 of 44 materials (93%). The small model is weak on every field: 10 of 50 types, 20 of 50 colours and 11 of 44 materials. Most of its answers were broken JSON, and it calls many things a bowl.
Look at every wrong product type
Section titled “Look at every wrong product type”A number says how often. Only the photos say why. These are the hosted model’s eight wrong product types:
| Photo | Label | Said | What the photo shows (our reading, by eye) |
|---|---|---|---|
| P-9a3fc835 | bowl | other | a plate of pancakes in front; the bowl is at the edge: open to debate |
| P-3b954756 | chair | other | a wooden garden chair; the alt text says “picnic table”: the model is wrong |
| P-42ae5214 | chair | cushion | a floral cushion on a wicker lounger: open to debate |
| P-b0244e29 | lamp | other | a glowing ball in a furnace, not a lamp: the label is probably wrong |
| P-b0286c43 | lamp | other | a light bulb, close up: open to debate |
| P-a8d9684c | mug | vase | a cup with no handle: open to debate |
| P-53ba97e3 | plant pot | vase | the egg basket of the figure: the model is wrong (it invented a ceramic vase) |
| P-92630ff3 | vase | other | a painted jug: open to debate |
In five of the eight “mistakes”, a careful person could give either answer. One label is probably wrong. Only two are clear mistakes of the model. This is common: when you check a strong model, many disagreements come from the labels and the question, not the model. The fix is a clearer question and better labels, and a person for “other”.
The seven wrong colours tell a similar story. The model said “clear” for two beer glasses that the labeller called yellow, because of the beer. It said “multicoloured” for a chair photo, describing the cushion on it. Two of the seven need a closer look, because the model named a colour that the product does not have. Calling a black bowl “red” (P-a2f50fc3, next to a red bowl) is a clear mistake. Calling the white candle “green” is open to debate, as we saw above: its wrap is green.
The labels’ alternatives matter too. Score only the first alternative of each label, and the hosted model’s colour score falls from 43 to 39 of 50. Material falls from 41 to 39 of 44. How you label decides part of the score.
How sure are we?
Section titled “How sure are we?”With 50 photos, one photo is 2 points. A 95% confidence interval gives the range of true accuracies that agree with what we saw. The Wilson interval is a common one for a share, and the notebook computes it:
Text description of the chart
Three rows on an axis from 0.6 to 1, with a vertical line at 0.95. Product type: 0.84, interval 0.715 to 0.917. Main colour: 0.86, interval 0.738 to 0.93. Material: 0.932, interval 0.818 to 0.977. Every interval reaches far below the 0.95 line, and only the material interval crosses it.
Omar’s rule (an assumption): a field goes on the website without review only if it is right at least 95% of the time. No field shows that. Material is the closest, at 0.932, but its interval goes down to 0.818. To show 95% with some certainty, Omar needs a few hundred labelled photos, not 50. Until then, the model helps the labellers: it fills in a first answer, and a person accepts or corrects it. Accepting a right answer takes a few seconds, and writing one takes about a minute (an assumption).
Counting and checking returns
Section titled “Counting and checking returns”The two models also answered two other recorded questions. They counted the boxes on the 30 drawn pallet test photos. And they said whether any candle was damaged, on 40 returns test photos.

Text description of the diagram
The left picture, S-063, shows 22 cardboard boxes stacked tightly on a pallet, in four layers. The hosted model said 22, and SmolVLM said 12. The right picture, S-066, shows 51 boxes in six layers. The hosted model said 51, and SmolVLM said 4.
The hosted model counted all 30 pallets exactly: a count error of 0. The fine-tuned detector of Module 4 had an error of 2.5 boxes, and the label rule 5.23. But these pallets are clean drawings made by our code: flat colours, one white label on each box, no plastic wrap, no shadows. Real pallets in a real warehouse are much harder, and the counting case study at the end of the course uses real photos. SmolVLM’s error is 12.3 boxes, and 27 on the tight pallets: it always counts too few.
The candles are a different story. The hosted model found all 18 damaged candles, but it also called 17 of 22 good candles damaged. It sees “damage” almost everywhere. What would that cost Hamid? Suppose (an estimate) that these rates on 40 photos held for his 600 returns a day, with the test set’s share of damaged items. Then about 425 good candles a day would go to the outlet, at 28 dollars each: about 11,910 dollars a day. Sending every return back on sale with no check costs 2,229.4. The best inspector of Module 6 cost 369 on the test set. The model’s own price would be only 0.83 dollars a day. The model is cheap. Its mistakes are not.
Compare with CLIP and a trained classifier
Section titled “Compare with CLIP and a trained classifier”For the product type, Larkfield already has methods. A probe is a small classifier on top of stored embeddings, from Linear probes and few-shot. The course team ran the project’s probes on the same 50 photos:
| Method | Product type, the same 50 photos | Labelled photos needed | CPU time per 1,000 photos | Money per 1,000 photos |
|---|---|---|---|---|
| CLIP probe, balanced classes | 48 of 50 | 453 | about 8 s | none (your CPU) |
| DINOv2 probe, balanced classes | 44 of 50 | 453 | about 14 s | none (your CPU) |
| Hosted vision-language model | 42 of 50 | 0 | about 90 minutes, one after another (5.4 s each) | 1.38 dollars |
| SmolVLM-256M | 10 of 50 | 0 | about 73 minutes (4.4 s each) | none (your CPU) |
On all 160 test photos, CLIP zero-shot reaches 0.925 with no labelled photo. The ResNet-18 fine-tuned in Module 2 reaches 0.850. The CPU times are the embedding times of the comparison table: one photo in a batch of 32, with 4 threads.
For “which of eight known types is this?”, the CLIP probe wins on every column except labels. It is right more often, about 700 times faster, and costs no money per photo. The vision-language model wins only where there are no labels, or where the answer must be words. Examples: alt text, a new attribute, or a question that nobody trained a model for.
Add the check to your project
Section titled “Add the check to your project”The project gets the scoring as a command, vision vlm. It replays the recorded answers from the course’s data: no model and no key.
-
Go to your
larkvisionfolder, activate.venv, and setLARKVISION_DATAif this is a new terminal (as in Find the same product). -
Copy the command, the tests of this module and the README from the snapshot
end-of-m07:Terminal window S=../nextia-learning-labs/C42/snapshots/end-of-m07cp "$S/larkvision/commands/vlm.py" larkvision/commands/vlm.pycp "$S/tests/test_match.py" tests/test_match.pycp "$S/README.md" .Terminal window $S = "..\nextia-learning-labs\C42\snapshots\end-of-m07"Copy-Item "$S\larkvision\commands\vlm.py" larkvision\commands\vlm.pyCopy-Item "$S\tests\test_match.py" tests\test_match.pyCopy-Item "$S\README.md" . -
Run the tests. You see
24 passed:Terminal window python -m pytest -q -
Score the hosted model’s recorded answers:
Terminal window vision vlm --model hosted -
Score the small model’s recorded answers, and show its first 3 wrong answers:
Terminal window vision vlm --model local --wrong 3
Step 4 prints:
Recorded answers of gpt-6.1-sol-2026-09-29 (120 answers, replayed from a file): product_type 42/50 right (84%) main_colour 43/50 right (86%) material 41/44 right (93%) box count on 30 pallet photos: error 0.0 boxes (bias +0.0); loose 0.0, medium 0.0, tight 0.0 damaged candles: found 18 of 18; called 17 of 22 good candles damaged wrong: P-9a3fc835 product_type: said 'other', label 'bowl' wrong: P-3b954756 product_type: said 'other', label 'chair' wrong: P-42ae5214 product_type: said 'cushion', label 'chair' wrong: P-b0244e29 product_type: said 'other', label 'lamp' wrong: P-b0286c43 product_type: said 'other', label 'lamp'Step 5 prints:
Recorded answers of HuggingFaceTB/SmolVLM-256M-Instruct@7e3e67edbb (120 answers, replayed from a file): product_type 10/50 right (20%) main_colour 20/50 right (40%) material 11/44 right (25%) box count on 30 pallet photos: error 12.3 boxes (bias -12.3); loose 1.7, medium 8.2, tight 27.0 damaged candles: found 6 of 18; called 1 of 22 good candles damaged wrong: P-9e093eaa product_type: said '', label 'bowl' wrong: P-7efd7bd8 product_type: said '', label 'candle' wrong: P-830b42bf product_type: said 'bowl', label 'candle'An empty answer (said '') is a broken answer that gave no value for the field. Now your project matches the snapshot end-of-m07: vision match and vision vlm, with 24 tests. The two new tests use a tiny fake data folder, so they need no download.
Common mistake
The colours that nobody checked
Symptom: Tomás sends all 733 catalogue photos to the hosted model, and puts its colours on the website’s filters without review. A week later, a customer filters on “red bowls”, orders P-a2f50fc3, and returns it: the bowl is black.
Cause: the colour field is right about 86% of the time on the labelled photos, with an interval down to 0.738. At that rate, about 1 product in 7 has a doubtful colour. Nobody measured it before the colours went live, and nothing on the page said that a model wrote them.
Fix: measure each field on labelled photos first, as in this lesson. Publish a field without review only if its accuracy and its interval meet the rule agreed with Omar. Until then, the model fills in a first answer and a person confirms it. Keep checking a small random sample every week, because the photos change.
Consequence in AI work: a model’s errors reach customers at the model’s speed. One wrong filter value is a small cost. But a returned order costs Larkfield more than the whole month of model calls.
When to use these answers, and what they cost
Section titled “When to use these answers, and what they cost”Use a vision-language model’s answers as a first draft for a person: alt text, colours, materials, a new attribute. Measure each field on labelled photos before you trust it. Send “other” and broken answers to a person.
Do not use them as the final word where a mistake is expensive. On the candle check, the hosted model’s false alarms would cost more than no inspection at all. Do not use one for a question that a trained classifier or a probe already answers better and faster, such as the product type.
The cost per 1,000 photos is small: 1.38 dollars for the hosted model. The real cost is the labels you need to check it, and the mistakes you do not catch.
Guided practice: score the answers in the notebook
Section titled “Guided practice: score the answers in the notebook”- Open the notebook from the box Do this lesson in a notebook at the top of this page.
- Run the setup cells. The second one ends with a
Ready:line. - Run section 1. It prints the first answers next to their labels.
- Write the task of section 2, and run the check cell.
- Run section 3. Before you read the table above again, decide for each photo: model wrong, label wrong, or both fair?
- Run sections 4 to 6: the intervals, the counts and the cost per 1,000 photos.
Your turn: decide which attributes need review
Section titled “Your turn: decide which attributes need review”In section 2 of the notebook, write accuracy(answers). For each field, it returns the share of answers whose value is one of the label’s alternatives, and it skips unclear labels. Put accuracy(attrs["hosted"]) in hosted_accuracy, and run the check cell. Then score the local model with the same function. Last, decide for each field: can it go on the website without review?
Show the answer
def accuracy(answers): out = {} for f in FIELDS: right = scored = 0 for a in answers: truth = labels.loc[a["id"], f].split("|") if truth == ["unclear"]: continue scored += 1 right += norm((a["parsed"] or {}).get(f)) in [norm(t) for t in truth] out[f] = round(right / scored, 3) return out
hosted_accuracy = accuracy(attrs["hosted"])print(hosted_accuracy){'product_type': 0.84, 'main_colour': 0.86, 'material': 0.932}The local model gives {'product_type': 0.2, 'main_colour': 0.4, 'material': 0.25}. The decision: no field goes on the website without review. Each interval reaches far below 95%. For the product type, use the CLIP probe (48 of 50) and send its unsure photos to a person. For colour and material, let the hosted model fill in a first answer, and let a person confirm each one. Label a few hundred more photos, then measure again.
Knowledge check
Section titled “Knowledge check”Check your understanding
The hosted model's alt text for the egg basket was right about the tulips, the pink blossoms and the wooden table, and wrong about the 'ceramic vase'. Why is this kind of mistake more dangerous than a broken answer?
Show the answer
A broken answer fails the format check, so code catches it. This answer is valid, fluent and mostly right, so it passes every format check. A reader who checks two details and finds them right will believe the third. The invented detail looks exactly like the true ones. Only a comparison with the photo, or with a label, finds it. That is why each field needs an accuracy measured on labelled photos.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
The hosted model is 84% right on product type, and 5 of its 8 'mistakes' are open to debate. Does that mean it is really about 94% right? What would you do to know?
Show the answer
Not yet. The “open to debate” reading is one person’s reading by eye, and other people could judge some of those photos the other way. To know, write a clearer rule for each field, for example “the main colour is the colour of the product’s body”. Have two people label the same photos, and score the model again. Also label more photos: with 50, the interval of 84% runs from about 0.72 to 0.92.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
Hamid asks whether the hosted model can check his 600 returns a day: it found all 18 damaged candles, and it costs only 0.83 dollars a day. Answer him with the numbers of this lesson.
Show the answer
No. It found every damaged candle, but it also called 17 of 22 good candles damaged. If that rate held for his returns, about 425 good candles a day would go to the outlet. At 28 dollars each, that is about 11,900 dollars a day. That is more than no inspection at all (2,229) and far more than the inspector of Module 6 (369 on the test set). The model’s price is small; the cost of its false alarms is not. This is an estimate from only 40 photos, but the gap is far too large for chance to close.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Recap
- Fluent answers can hold invented details: a ceramic vase that was a basket of eggs. Only a comparison with the photo or a label finds them.
- Score each field against labels. Here, 50 photos, colours and materials labelled by one person by eye, with alternatives. The hosted model: type 84%, colour 86%, material 93%. SmolVLM-256M: 20%, 40%, 25%.
- Look at every wrong answer: many disagreements come from the labels and the question, not the model.
- With 50 photos, every 95% interval reaches far below 95%. Use the model to help labelling, not as the final word.
- The hosted model counted 30 clean drawn pallets exactly. But it called 17 of 22 good candles damaged: an estimated 11,900 dollars a day.
- For product type, the CLIP probe was right on 48 of the same 50 photos. It is about 700 times faster and costs no money per photo.
- Your project has
vision vlm, with 24 tests: the snapshotend-of-m07.
You can now measure a vision-language model and compare it with cheaper methods. But Tomás’s question is still open: when should Larkfield build any model at all? Next, you look at problems that a camera, a barcode, fixed light or a person solve more cheaply.
Next: When not to build it