Skip to content
Français

Check what a vision-language model says

  • Lesson 25
  • 40 min
  • Module 7: Match, search and describe

Where we are

In the last lesson, the vision-language models gave fluent answers, and we checked their shape. A well-shaped answer can still be wrong. In this lesson, we measure attribute accuracy against labels, find invented details and counting errors, and compare the cost per 1,000 images with CLIP and a classifier. First, a new colleague who writes very well.

Plays from YouTube. Nothing loads from YouTube until you press play. Privacy

What you will be able to do

Find out how often a vision-language model is right, field by field, and where it invents details. Then decide which of its answers can be used without a person, and whether a cheaper model does the job better.

NotebookDo this lesson in a notebookColab · Kaggle · GitHub

The notebook scores the recorded answers of both models against the labels. It shows every wrong answer with its photo, and has one task with a check. It needs no account, no key and no model. Its setup cells download the catalogue, the pallets and the recordings (about 21 MB). About 35 minutes.

Colab and Kaggle open your own copy. They need a free Google or Kaggle account. Stuck? See the solution notebook.

Before you startWhat to know and to have readyAbout 40 minutes

What you need to know

What you need on your computer

  • The notebook route: Colab, Kaggle or Jupyter. Nothing to install.
  • The project route: your larkvision folder with vision match from Find the same product, and 22 passed. This lesson adds one command and its tests.

Tested with Python 3.12.13 on macOS, in Jupyter and in a terminal. The Windows commands, Linux, Colab and Kaggle were not tested.

Omar hires a new colleague to write product descriptions. Her texts are excellent: clear, warm, no spelling mistakes. On her first day she writes 50. Before they go on the website, Omar takes 10 and compares each text with its product. Eight are perfect. One calls a white candle “green”. One describes a vase that is really a wire basket full of eggs.

Omar does not stop trusting her. He learns where she makes mistakes, and he keeps checking that kind of field until she improves. Good writing made the mistakes harder to see, not less likely.

The problem: fluent is not the same as right

Section titled “The problem: fluent is not the same as right”

Here are two recorded answers of the hosted model, with each part marked against the photo:

Two recorded answers, each with its photo. Photo 1, labelled plant pot: tulips, small pink flowers and greenery in a wire basket filled with striped white and brown eggs, on a wooden table. The alt text says: A ceramic vase with stacked, ribbed shapes in white, grey and brown holds purple tulips, small pink blossoms and trailing greenery on a wooden table. 'Ceramic vase with stacked, ribbed shapes' is marked wrong; the colours, the flowers and the wooden table are marked right. Fields: type vase, wrong, label plant pot; colour multicoloured, right; material ceramic, wrong, label metal. Note: it is a wire basket filled with striped eggs, not a ceramic vase. Photo 2, labelled candle: two white candles in a green paper wrap printed with daisies, the front one lit, and printed words at the bottom. The alt text says: Two green pillar candles decorated with white daisies sit beside pale yellow ribbon, with the front candle lit. 'Green' is marked open to debate; the rest is marked right. Fields: type candle, right; colour green, open to debate, label white; material wax, right. Note: the wax is white and the printed wrap is green. Missed: the printed words.
Two recorded answers of the hosted model, marked by the course team against the photo. Blue: right. Red: wrong. Yellow: open to debate. Photos: Nicolas Huk, GIOCONDA Centru de Frumusete, CC BY 2.0, from Open Images.
Text description of the diagram

The first photo is labelled plant pot. It shows tulips, small pink flowers and greenery in a wire basket full of striped eggs, on a wooden table. The model wrote that “a ceramic vase with stacked, ribbed shapes in white, grey and brown” holds the flowers. The flowers, the colours and the table are right. The ceramic vase is invented: the striped eggs look like ribbed shapes, and the model turned them into a vase. So its product type (vase) and material (ceramic) are wrong too. The second photo, labelled candle, shows two white candles in a green paper wrap printed with daisies. The front one is lit, and printed words are at the bottom. The model called them “green pillar candles”. The wax is white, and the wrap is green, so “green” is open to debate. Everything else is right, but the alt text does not mention the printed words.

The first answer is the dangerous kind. Most of it is right and specific: purple tulips, small pink blossoms, a wooden table. In the middle, it describes an object that is not there. A reader who checks two details and finds them right will believe the third. A model that states an invented detail as a fact is said to hallucinate. In this course we say invented detail, because that is what the reader gets.

The second answer shows another problem. Is the main colour of a white candle in a green wrap white or green? The question did not say. Before you can call an answer wrong, you need labels, and a clear rule for each field.

To measure the model, you need the right answers for some photos: labels. The course has labels for the same 50 catalogue test photos that the models described:

  • Product type comes from the catalogue labels.
  • Main colour and material were labelled by one person, by eye. Where two answers are fair, the label lists both, for example yellow|orange: 18 colour labels and 5 material labels have two. Where the photo does not show the material, the label is unclear, and that field is not scored: 6 photos.

One labeller is a weak base. A second person would disagree on some photos, and you would not know which one is right. Treat these labels as good enough to compare the two models, not as the truth.

The score of a field is its accuracy: right answers divided by scored answers.

Photo Label (type / colour / material) Hosted answer Type Colour Material
P-9a3fc835 bowl / white / ceramic other / white / ceramic wrong right right
P-a2f50fc3 bowl / black / unclear bowl / red / glass right wrong not scored
P-7efd7bd8 candle / white / wax candle / green / wax right wrong right
P-830b42bf candle / white / wax or glass candle / white / wax right right right
P-84e4c4bf candle / yellow or orange / wax candle / yellow / wax right right right

Count each column. Product type: 4 right of 5, which is 80%. Colour: 3 of 5, which is 60%. Material: 4 of 4, which is 100%, because one material is not scored. Five photos are far too few to trust: one more photo moves a score by 20 points. The same count on all 50 photos gives the real result.

  • Hosted model
  • SmolVLM-256M
The share of right answers per field (Mat.: material) on the 50 labelled catalogue photos, for the recorded answers of both models. Colour and material labels: one person, by eye.
Text description of the chart

A column chart with three pairs of columns, on an axis from 0 to 1. Product type: hosted model 0.84, SmolVLM 0.2. Main colour: hosted 0.86, SmolVLM 0.4. Material: hosted 0.932, SmolVLM 0.25. The hosted model’s columns are three to four times higher than the small model’s.

The hosted model gets 42 of 50 product types right (84%), 43 of 50 colours (86%) and 41 of 44 materials (93%). The small model is weak on every field: 10 of 50 types, 20 of 50 colours and 11 of 44 materials. Most of its answers were broken JSON, and it calls many things a bowl.

A number says how often. Only the photos say why. These are the hosted model’s eight wrong product types:

Photo Label Said What the photo shows (our reading, by eye)
P-9a3fc835 bowl other a plate of pancakes in front; the bowl is at the edge: open to debate
P-3b954756 chair other a wooden garden chair; the alt text says “picnic table”: the model is wrong
P-42ae5214 chair cushion a floral cushion on a wicker lounger: open to debate
P-b0244e29 lamp other a glowing ball in a furnace, not a lamp: the label is probably wrong
P-b0286c43 lamp other a light bulb, close up: open to debate
P-a8d9684c mug vase a cup with no handle: open to debate
P-53ba97e3 plant pot vase the egg basket of the figure: the model is wrong (it invented a ceramic vase)
P-92630ff3 vase other a painted jug: open to debate

In five of the eight “mistakes”, a careful person could give either answer. One label is probably wrong. Only two are clear mistakes of the model. This is common: when you check a strong model, many disagreements come from the labels and the question, not the model. The fix is a clearer question and better labels, and a person for “other”.

The seven wrong colours tell a similar story. The model said “clear” for two beer glasses that the labeller called yellow, because of the beer. It said “multicoloured” for a chair photo, describing the cushion on it. Two of the seven need a closer look, because the model named a colour that the product does not have. Calling a black bowl “red” (P-a2f50fc3, next to a red bowl) is a clear mistake. Calling the white candle “green” is open to debate, as we saw above: its wrap is green.

The labels’ alternatives matter too. Score only the first alternative of each label, and the hosted model’s colour score falls from 43 to 39 of 50. Material falls from 41 to 39 of 44. How you label decides part of the score.

With 50 photos, one photo is 2 points. A 95% confidence interval gives the range of true accuracies that agree with what we saw. The Wilson interval is a common one for a share, and the notebook computes it:

The hosted model's accuracy per field with its 95% Wilson interval, against a bar of 0.95 (an assumption: Omar's rule for publishing without review).
Text description of the chart

Three rows on an axis from 0.6 to 1, with a vertical line at 0.95. Product type: 0.84, interval 0.715 to 0.917. Main colour: 0.86, interval 0.738 to 0.93. Material: 0.932, interval 0.818 to 0.977. Every interval reaches far below the 0.95 line, and only the material interval crosses it.

Omar’s rule (an assumption): a field goes on the website without review only if it is right at least 95% of the time. No field shows that. Material is the closest, at 0.932, but its interval goes down to 0.818. To show 95% with some certainty, Omar needs a few hundred labelled photos, not 50. Until then, the model helps the labellers: it fills in a first answer, and a person accepts or corrects it. Accepting a right answer takes a few seconds, and writing one takes about a minute (an assumption).

The two models also answered two other recorded questions. They counted the boxes on the 30 drawn pallet test photos. And they said whether any candle was damaged, on 40 returns test photos.

Two drawn pallets of cardboard boxes, both tight. S-063: 22 boxes; the hosted model said 22, SmolVLM said 12. S-066: 51 boxes in six layers; the hosted model said 51, SmolVLM said 4.
Two tight pallet test photos (drawn by our code, CC0) and the recorded counts of both models.
Text description of the diagram

The left picture, S-063, shows 22 cardboard boxes stacked tightly on a pallet, in four layers. The hosted model said 22, and SmolVLM said 12. The right picture, S-066, shows 51 boxes in six layers. The hosted model said 51, and SmolVLM said 4.

The hosted model counted all 30 pallets exactly: a count error of 0. The fine-tuned detector of Module 4 had an error of 2.5 boxes, and the label rule 5.23. But these pallets are clean drawings made by our code: flat colours, one white label on each box, no plastic wrap, no shadows. Real pallets in a real warehouse are much harder, and the counting case study at the end of the course uses real photos. SmolVLM’s error is 12.3 boxes, and 27 on the tight pallets: it always counts too few.

The candles are a different story. The hosted model found all 18 damaged candles, but it also called 17 of 22 good candles damaged. It sees “damage” almost everywhere. What would that cost Hamid? Suppose (an estimate) that these rates on 40 photos held for his 600 returns a day, with the test set’s share of damaged items. Then about 425 good candles a day would go to the outlet, at 28 dollars each: about 11,910 dollars a day. Sending every return back on sale with no check costs 2,229.4. The best inspector of Module 6 cost 369 on the test set. The model’s own price would be only 0.83 dollars a day. The model is cheap. Its mistakes are not.

Compare with CLIP and a trained classifier

Section titled “Compare with CLIP and a trained classifier”

For the product type, Larkfield already has methods. A probe is a small classifier on top of stored embeddings, from Linear probes and few-shot. The course team ran the project’s probes on the same 50 photos:

Method Product type, the same 50 photos Labelled photos needed CPU time per 1,000 photos Money per 1,000 photos
CLIP probe, balanced classes 48 of 50 453 about 8 s none (your CPU)
DINOv2 probe, balanced classes 44 of 50 453 about 14 s none (your CPU)
Hosted vision-language model 42 of 50 0 about 90 minutes, one after another (5.4 s each) 1.38 dollars
SmolVLM-256M 10 of 50 0 about 73 minutes (4.4 s each) none (your CPU)

On all 160 test photos, CLIP zero-shot reaches 0.925 with no labelled photo. The ResNet-18 fine-tuned in Module 2 reaches 0.850. The CPU times are the embedding times of the comparison table: one photo in a batch of 32, with 4 threads.

For “which of eight known types is this?”, the CLIP probe wins on every column except labels. It is right more often, about 700 times faster, and costs no money per photo. The vision-language model wins only where there are no labels, or where the answer must be words. Examples: alt text, a new attribute, or a question that nobody trained a model for.

The project gets the scoring as a command, vision vlm. It replays the recorded answers from the course’s data: no model and no key.

  1. Go to your larkvision folder, activate .venv, and set LARKVISION_DATA if this is a new terminal (as in Find the same product).

  2. Copy the command, the tests of this module and the README from the snapshot end-of-m07:

    Terminal window
    S=../nextia-learning-labs/C42/snapshots/end-of-m07
    cp "$S/larkvision/commands/vlm.py" larkvision/commands/vlm.py
    cp "$S/tests/test_match.py" tests/test_match.py
    cp "$S/README.md" .
  3. Run the tests. You see 24 passed:

    Terminal window
    python -m pytest -q
  4. Score the hosted model’s recorded answers:

    Terminal window
    vision vlm --model hosted
  5. Score the small model’s recorded answers, and show its first 3 wrong answers:

    Terminal window
    vision vlm --model local --wrong 3

Step 4 prints:

Recorded answers of gpt-6.1-sol-2026-09-29 (120 answers, replayed from a file):
product_type 42/50 right (84%)
main_colour 43/50 right (86%)
material 41/44 right (93%)
box count on 30 pallet photos: error 0.0 boxes (bias +0.0); loose 0.0, medium 0.0, tight 0.0
damaged candles: found 18 of 18; called 17 of 22 good candles damaged
wrong: P-9a3fc835 product_type: said 'other', label 'bowl'
wrong: P-3b954756 product_type: said 'other', label 'chair'
wrong: P-42ae5214 product_type: said 'cushion', label 'chair'
wrong: P-b0244e29 product_type: said 'other', label 'lamp'
wrong: P-b0286c43 product_type: said 'other', label 'lamp'

Step 5 prints:

Recorded answers of HuggingFaceTB/SmolVLM-256M-Instruct@7e3e67edbb (120 answers, replayed from a file):
product_type 10/50 right (20%)
main_colour 20/50 right (40%)
material 11/44 right (25%)
box count on 30 pallet photos: error 12.3 boxes (bias -12.3); loose 1.7, medium 8.2, tight 27.0
damaged candles: found 6 of 18; called 1 of 22 good candles damaged
wrong: P-9e093eaa product_type: said '', label 'bowl'
wrong: P-7efd7bd8 product_type: said '', label 'candle'
wrong: P-830b42bf product_type: said 'bowl', label 'candle'

An empty answer (said '') is a broken answer that gave no value for the field. Now your project matches the snapshot end-of-m07: vision match and vision vlm, with 24 tests. The two new tests use a tiny fake data folder, so they need no download.

Common mistake

The colours that nobody checked

Symptom: Tomás sends all 733 catalogue photos to the hosted model, and puts its colours on the website’s filters without review. A week later, a customer filters on “red bowls”, orders P-a2f50fc3, and returns it: the bowl is black.

Cause: the colour field is right about 86% of the time on the labelled photos, with an interval down to 0.738. At that rate, about 1 product in 7 has a doubtful colour. Nobody measured it before the colours went live, and nothing on the page said that a model wrote them.

Fix: measure each field on labelled photos first, as in this lesson. Publish a field without review only if its accuracy and its interval meet the rule agreed with Omar. Until then, the model fills in a first answer and a person confirms it. Keep checking a small random sample every week, because the photos change.

Consequence in AI work: a model’s errors reach customers at the model’s speed. One wrong filter value is a small cost. But a returned order costs Larkfield more than the whole month of model calls.

When to use these answers, and what they cost

Section titled “When to use these answers, and what they cost”

Use a vision-language model’s answers as a first draft for a person: alt text, colours, materials, a new attribute. Measure each field on labelled photos before you trust it. Send “other” and broken answers to a person.

Do not use them as the final word where a mistake is expensive. On the candle check, the hosted model’s false alarms would cost more than no inspection at all. Do not use one for a question that a trained classifier or a probe already answers better and faster, such as the product type.

The cost per 1,000 photos is small: 1.38 dollars for the hosted model. The real cost is the labels you need to check it, and the mistakes you do not catch.

Guided practice: score the answers in the notebook

Section titled “Guided practice: score the answers in the notebook”
  1. Open the notebook from the box Do this lesson in a notebook at the top of this page.
  2. Run the setup cells. The second one ends with a Ready: line.
  3. Run section 1. It prints the first answers next to their labels.
  4. Write the task of section 2, and run the check cell.
  5. Run section 3. Before you read the table above again, decide for each photo: model wrong, label wrong, or both fair?
  6. Run sections 4 to 6: the intervals, the counts and the cost per 1,000 photos.

Your turn: decide which attributes need review

Section titled “Your turn: decide which attributes need review”

In section 2 of the notebook, write accuracy(answers). For each field, it returns the share of answers whose value is one of the label’s alternatives, and it skips unclear labels. Put accuracy(attrs["hosted"]) in hosted_accuracy, and run the check cell. Then score the local model with the same function. Last, decide for each field: can it go on the website without review?

Show the answer
def accuracy(answers):
out = {}
for f in FIELDS:
right = scored = 0
for a in answers:
truth = labels.loc[a["id"], f].split("|")
if truth == ["unclear"]:
continue
scored += 1
right += norm((a["parsed"] or {}).get(f)) in [norm(t) for t in truth]
out[f] = round(right / scored, 3)
return out
hosted_accuracy = accuracy(attrs["hosted"])
print(hosted_accuracy)
{'product_type': 0.84, 'main_colour': 0.86, 'material': 0.932}

The local model gives {'product_type': 0.2, 'main_colour': 0.4, 'material': 0.25}. The decision: no field goes on the website without review. Each interval reaches far below 95%. For the product type, use the CLIP probe (48 of 50) and send its unsure photos to a person. For colour and material, let the hosted model fill in a first answer, and let a person confirm each one. Label a few hundred more photos, then measure again.

Check your understanding

The hosted model's alt text for the egg basket was right about the tulips, the pink blossoms and the wooden table, and wrong about the 'ceramic vase'. Why is this kind of mistake more dangerous than a broken answer?

Show the answer

A broken answer fails the format check, so code catches it. This answer is valid, fluent and mostly right, so it passes every format check. A reader who checks two details and finds them right will believe the third. The invented detail looks exactly like the true ones. Only a comparison with the photo, or with a label, finds it. That is why each field needs an accuracy measured on labelled photos.

Check your understanding

The hosted model is 84% right on product type, and 5 of its 8 'mistakes' are open to debate. Does that mean it is really about 94% right? What would you do to know?

Show the answer

Not yet. The “open to debate” reading is one person’s reading by eye, and other people could judge some of those photos the other way. To know, write a clearer rule for each field, for example “the main colour is the colour of the product’s body”. Have two people label the same photos, and score the model again. Also label more photos: with 50, the interval of 84% runs from about 0.72 to 0.92.

Check your understanding

Hamid asks whether the hosted model can check his 600 returns a day: it found all 18 damaged candles, and it costs only 0.83 dollars a day. Answer him with the numbers of this lesson.

Show the answer

No. It found every damaged candle, but it also called 17 of 22 good candles damaged. If that rate held for his returns, about 425 good candles a day would go to the outlet. At 28 dollars each, that is about 11,900 dollars a day. That is more than no inspection at all (2,229) and far more than the inspector of Module 6 (369 on the test set). The model’s price is small; the cost of its false alarms is not. This is an estimate from only 40 photos, but the gap is far too large for chance to close.

Recap

  • Fluent answers can hold invented details: a ceramic vase that was a basket of eggs. Only a comparison with the photo or a label finds them.
  • Score each field against labels. Here, 50 photos, colours and materials labelled by one person by eye, with alternatives. The hosted model: type 84%, colour 86%, material 93%. SmolVLM-256M: 20%, 40%, 25%.
  • Look at every wrong answer: many disagreements come from the labels and the question, not the model.
  • With 50 photos, every 95% interval reaches far below 95%. Use the model to help labelling, not as the final word.
  • The hosted model counted 30 clean drawn pallets exactly. But it called 17 of 22 good candles damaged: an estimated 11,900 dollars a day.
  • For product type, the CLIP probe was right on 48 of the same 50 photos. It is about 700 times faster and costs no money per photo.
  • Your project has vision vlm, with 24 tests: the snapshot end-of-m07.

You can now measure a vision-language model and compare it with cheaper methods. But Tomás’s question is still open: when should Larkfield build any model at all? Next, you look at problems that a camera, a barcode, fixed light or a person solve more cheaply.

Next: When not to build it