Vision-language models
- Lesson 24
- 40 min
- Module 7: Match, search and describe
Where we are
In the last lesson, we found which catalogue product a customer’s photo shows. A match is not a description: Omar also needs words, such as alt text and the colour and material of each product. In this lesson, we ask a small local vision-language model and a recorded hosted model to describe catalogue photos. First, a phone call with a friend.
Lesson 24 of 30 · Vision AI: See, Count and InspectVision-language models
1. Seeing is a business problem
2. Baselines and pretrained classifiers
3. Embeddings, CLIP and few labels
4. Detect and count
6. Inspect for defects
7. Match, search and describe
8. Run vision in production
What you will be able to do
Get words from a photo: a caption, alt text, an answer to a question, or product attributes in a fixed shape that code can read. Then check that the answer has that shape before you use it.
NotebookDo this lesson in a notebook
The notebook replays the recorded answers of both models, checks their structured output, and has one task with a check. It needs no account and no key. Its setup cells download the catalogue (about 19 MB with the recordings). An optional last section runs the small model on your own computer (about 700 MB to download). About 30 minutes.
Colab and Kaggle open your own copy. They need a free Google or Kaggle account. Stuck? See the solution notebook.
Before you startWhat to know and to have ready
What you need to know
- Image embeddings, and CLIP’s image and text encoders, from Image embeddings and CLIP zero-shot.
- JSON: a text format for data, with keys and values, such as
{"colour": "white"}. - For how a language model writes text one token at a time, Deep Learning and Transformers Explained helps, but you do not need it.
What you need on your computer
The notebook route is enough: Colab, Kaggle or Jupyter. Your larkvision project does not change in this lesson. It gets the vision vlm command in the next lesson.
Tested with Python 3.12.13 on macOS, in Jupyter. The optional live section was run with PyTorch 2.15.0 and Transformers 5.22.2 on the course laptop’s CPU. Colab, Kaggle and Windows were not tested.
Describe it on the phone
Section titled “Describe it on the phone”A friend calls you from a shop. “I am looking at a lamp for your birthday. Should I buy it?” You cannot see it, so you ask questions. “What colour is it? Is the shade fabric or glass? Is it tall?” Your friend looks and answers in words. You never see the lamp, but after a few questions you know enough to decide.
Your friend did two jobs: she saw the lamp, and she put what she saw into words that answer your question. If she is tired, she may also guess: “I think it is brass”, when it is painted plastic. You cannot tell a guess from a fact on the phone, unless you ask her to check.
The problem: Omar needs words, not only labels
Section titled “The problem: Omar needs words, not only labels”Omar owns Larkfield’s catalogue descriptions. Each of the 733 catalogue photos needs alt text. Alt text is a short description that a screen reader reads aloud for a person who cannot see the photo. Each product also needs attributes for the website’s filters: main colour, material, and later style. Every new product needs the same.
The methods of this course so far give one label from a fixed list. A CLIP probe answers “vase” or “bowl”. To get colours, you would need labelled photos for each colour and a new classifier. For material, another one. For a new attribute such as style, another one again. And no classifier writes a sentence.
A person can write them. Suppose that one description takes a minute, at the course’s assumed 0.50 dollars a minute. Then the 733 photos cost about 367 dollars and 12 hours of work. That is affordable once, but it comes back with every new range of products.
The idea: a vision-language model
Section titled “The idea: a vision-language model”A vision-language model (VLM) does both jobs of your friend. It reads a photo and a question, and it writes an answer in text. Inside, it has two parts:
Text description of the diagram
The diagram has five boxes (one under the other on a phone). A photo of 256 × 256 pixels goes into an image encoder, which works like CLIP’s image encoder and turns the photo into image tokens. Your question, for example “Answer with JSON: product_type, main_colour, material, alt_text”, becomes text tokens. A language model reads all the tokens together, then writes the answer one token at a time. The answer is plain text, for example {"product_type": "cushion", "main_colour": "white", …}. A last, dashed box says what your code must do. It checks that the text is valid JSON with allowed values. Otherwise, it asks again or sends the photo to a person.
- An image encoder turns the photo into a list of image tokens. A token is a small piece of input that the model reads, as a word piece is for text. The encoder is like CLIP’s image encoder from Module 3.
- A language model reads the image tokens together with the tokens of your question. Then it writes the answer, one token at a time. It chooses each next token from all the tokens before it.
The same model does several jobs. You change only the question:
- Caption: “Describe this photo in one sentence.”
- Alt text: “Write one sentence for a person who cannot see the photo.”
- Visual question answering: a question about this photo, such as “How many cardboard boxes can you see?”
- Structured output: “Answer with JSON only, with these keys”, so that code can read the answer.
A worked example: one photo, one recorded answer
Section titled “A worked example: one photo, one recorded answer”The course team sent each of 50 catalogue test photos to two models with the same question, and recorded every answer once. This is the question, word for word:
This is a product photo from a home and garden shop. Answer with JSON only, with these keys:"product_type" (one of ['bowl', 'candle', 'chair', 'cushion', 'lamp', 'mug', 'plant_pot', 'vase', 'other']),"main_colour" (one of ['white', 'black', 'grey', 'brown', 'beige', 'red', 'orange', 'yellow', 'gold', 'green','blue', 'purple', 'pink', 'clear', 'silver', 'multicoloured']), "material" (one of ['ceramic', 'glass', 'metal','wood', 'fabric', 'wax', 'plastic', 'leather', 'paper', 'stone']), "alt_text" (one sentence for a person whocannot see the photo).Each key has a list of allowed values, so that the answers fit the website’s filters. Here is the first photo of the figure further down, a catalogue photo labelled cushion, and the recorded answer of the hosted model:
{"product_type":"cushion","main_colour":"white","material":"fabric","alt_text":"Two white fabric cushions are stacked with a small yellow packet partially visible between them."}Read it field by field. Product type cushion: right. Colour white and material fabric: right. The alt text sees a detail that no label has: a small yellow packet between the two cushions. The answer used 258 input tokens (the photo and the question) and 63 output tokens. It cost 0.001146 dollars and took 2.7 seconds.
Two models, the same questions
Section titled “Two models, the same questions”| SmolVLM-256M-Instruct | Hosted model | |
|---|---|---|
| Where it runs | your computer, on the CPU | the provider’s servers, through a paid API |
| Size and licence | 513 MB, Apache-2.0 | unknown; the provider’s terms |
| Time an answer (course laptop) | 4.39 s | 5.43 s, over the internet |
| Money per 1,000 photos | none (your CPU and power) | 1.38 dollars (about 236 input tokens a photo) |
| The photo leaves your computer | no | yes |
| In this course | recorded, and you can run it | recorded once; the model is Azure chat-strong (gpt-6.1-sol-2026-09-29) |
The hosted price was measured from the tokens of the 120 recorded answers (0.17 dollars in all). Prices change: check the provider’s current price page. Every hosted answer in this course is recorded: you read and replay the same answers from a file, and you need no key.
Now the small model, on the same cushion photo:
{ "product_type": "bowl", "main_colour": "white", "material": "ceramic", "alt_text": "red", "material_colour": "black", "material_text": "grey", "alt_text": "brown", "material_text": "beige", "material_text": "blue", "material_text": "purple", "material_text": "silver", "material_text": "multicoloured"}It says bowl and ceramic. Then it repeats keys: alt_text twice, material_text five times, until it reaches its limit of 120 new tokens. Small models often lose the thread of a long, strict question. You can run this model yourself in the notebook’s optional section. On the course laptop it gave the same answer, word for word. With the simpler question “Describe this photo in one sentence.”, it wrote:
In this image we can see a white color cloth and a small packet on it.That caption is fair. The small model can see the photo. It cannot follow a long question with fixed lists of values.
Structured output: check the shape first
Section titled “Structured output: check the shape first”Structured output means an answer in a fixed shape that code can read without a person. Here, it is a JSON object with four keys. Three of them must hold an allowed value; alt_text holds one sentence. A model only writes text, so nothing makes it keep the shape. Your code checks two things before it uses an answer:
- Is the whole answer one JSON object? Python’s
json.loadsmust read it, and give a dictionary. - Does each key have an allowed value?
"cushion"is allowed;"pillow"and"1.99"are not.
The 120 recorded answers of each model:
| Check | Hosted model | SmolVLM-256M |
|---|---|---|
| The whole answer is one JSON object | 120 of 120 | 4 of 120 |
| Attribute answers that pass both checks | 50 of 50 | 3 of 50 |
The small model wrote valid JSON in only 4 answers. Many others stop in the middle of a list of repeated keys, because the answer reached 120 tokens. Some invent new keys. For one beer mug it wrote a "price" of "1.99", five times.
What should a product do with an answer that fails a check? Ask once more (a retry), and if that fails too, send the photo to a person. The recording script also tried a third way. It took the first value of each "key": value pair from the broken text, and read plain sentences with simple word rules. This found values in 83 more answers. It is useful for measuring the model, as the next lesson does. In a product, a value read from a broken answer is a guess about a guess.
Alt text from the hosted model
Section titled “Alt text from the hosted model”
Text description of the diagram
Photo 1 is labelled cushion: two white pillows lie on top of each other on a bed, and a small yellow packet shows between them. Photo 2 is labelled mug: a clear glass beer mug with a large handle, full of golden beer with white foam. Photo 3 is labelled lamp: a dark metal frame with a square opening. Behind it, a bright orange ball of light glows in the dark.
| Photo | Recorded alt text (hosted model) |
|---|---|
| 1 (cushion) | Two white fabric cushions are stacked with a small yellow packet partially visible between them. |
| 2 (mug) | A clear glass beer mug with a large handle holds golden beer topped with white foam. |
| 3 (lamp) | A bright orange ball of molten glass glows inside a dark furnace. |
The first two are good alt text: short, specific, and true to the photo. The third one is interesting. The catalogue label says lamp, and the model says “molten glass in a furnace”. Look at the photo again. The model may well be right, and the label wrong: Module 1 warned that some Open Images “lamps” are not lamps. A model and a label that disagree are a question for a person, not an answer.
The model also answered “other” for this photo’s product type. The question allowed “other”, and the model used it for 5 of the 50 photos. An allowed “I am not sure” value is useful: it tells you which photos to send to a person.
A live call with your own key
Section titled “A live call with your own key”The lessons replay recorded answers. You can also send your own photos to a hosted model with your own key and account. If you do, one rule comes first: never send a photo with a person in it to a hosted model, unless the person is blurred. The photo leaves your computer and goes to the provider. The course’s catalogue photos, drawn pallets and candle photos have no people: the course team checked each set before it recorded anything. People in images in Module 8 explains how the image service blurs faces by default.
Common mistake
The JSON that parsed
Symptom: Tomás writes the first check only: if json.loads reads the answer, the attributes go to the website. The small model’s answer for the cushion photo passes. The website now shows two white cushions as a ceramic bowl, with the alt text “brown”.
Cause: valid JSON is not a right answer. This answer has alt_text twice. json.loads accepts repeated keys and keeps the last value, so "red" was lost and "brown" kept. Neither value is alt text. And bowl is an allowed value, so a check of the allowed values passes too.
Fix: check the shape strictly. Refuse an answer with repeated keys (json.loads(text, object_pairs_hook=...) can see them), and refuse values outside the lists. Then measure how often the values are right on labelled photos before any value goes to a customer. That is the next lesson.
Consequence in AI work: a format check catches broken answers, not wrong ones. Every structured-output pipeline needs both: a strict shape check on every answer, and a regular check of accuracy against labels.
When to use a vision-language model, and what it costs
Section titled “When to use a vision-language model, and what it costs”Use one when you need words about a photo. Examples: alt text, a caption, an answer to a new question with no new training, or draft attributes for a person to confirm. It needs no labelled photos to start. That is its great strength, and the reason Tomás wants it everywhere.
Its costs are real. The hosted model costs about 1.38 dollars per 1,000 photos and 5.43 seconds an answer, and each photo leaves your computer. The small local model costs no money. But on a laptop CPU it takes over 4 seconds an answer, and it could not follow our question. Both write fluent text that can be wrong, and nothing in the text says which parts are guesses. Do not use one where a cheaper method already gives a measured answer: matching a product, or classifying it into eight known classes.
Guided practice: the answers in the notebook
Section titled “Guided practice: the answers in the notebook”- Open the notebook from the box Do this lesson in a notebook at the top of this page.
- Run the setup cells. The second one ends with a
Ready:line. - Run section 1. You see 120 answers for each model: 50 attribute answers, 30 pallet counts and 40 candle checks.
- Before you run section 2, predict whether the small model’s answer will be usable JSON. Then run it.
- Run section 3 up to the task.
is_json_objectprintsTrue Falsefor its two test texts. - Run sections 4 and 5. Read the broken answer and the three alt texts.
Your turn: how many answers are JSON?
Section titled “Your turn: how many answers are JSON?”In section 3 of the notebook, count the local model’s answers (all 120) for which is_json_object is true. Put the number in n_json_local, and run the check cell. Then change one condition: run the next cell, which also checks the allowed values, and compare the two models.
Show the answer
n_json_local = sum(is_json_object(a["answer"]) for a in local)print(n_json_local, "of", len(local))4 of 120The next cell prints:
hosted usable: 50 of 50local usable: 3 of 50Only 4 local answers are one JSON object, and 3 of the 50 attribute answers pass both checks. One of those 3 is the cushion “bowl”. Passing the checks says that the answer can be read, not that it is right.
Knowledge check
Section titled “Knowledge check”Check your understanding
In your own words: what are the two parts of a vision-language model, and why can the same model write alt text, answer a question and fill in attributes?
Show the answer
An image encoder turns the photo into image tokens, as CLIP’s image encoder turns a photo into numbers. A language model reads those tokens together with the tokens of the question, and writes the answer one token at a time. The model’s only output is text, and the question decides what text. So a new job needs a new question, not a new model or new labels. Examples: “describe it”, “how many boxes?” or “answer with JSON with these keys”.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
The small model's answer for the cushion photo is valid JSON with allowed values: product type bowl, material ceramic. Why is that not enough to use it, and what would you add?
Show the answer
The two checks only show that the answer can be read. They do not show that it is right: the photo shows two cushions, not a ceramic bowl. The answer also has repeated keys, and json.loads silently kept the last alt_text. Add a strict check that refuses repeated keys. Send failed answers to a retry and then to a person. And measure accuracy against labelled photos before any value goes to the website.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
Omar wants alt text for the 733 catalogue photos. Compare a person writing them with the hosted model, using the numbers of this lesson. What would you recommend?
Show the answer
A person: about one minute a photo, so about 12 hours and 367 dollars at the course’s assumed rate. The hosted model: about 1.38 dollars per 1,000 photos, so about 1 dollar, and 5 seconds a photo. The model’s alt texts were mostly specific and true, but some can be wrong (the “furnace” lamp). Recommend the model for a first draft, sent only for photos without people, and a person who reads and corrects each text before publication. Reading takes less time than writing.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Recap
- A vision-language model has an image encoder that makes image tokens, and a language model that reads them with the question and writes text.
- One model does captions, alt text, visual questions and structured output: you change the question, not the model.
- The hosted model wrote one valid JSON object in all 120 recorded answers, for about 1.38 dollars per 1,000 photos. SmolVLM-256M runs on your CPU for free, but wrote valid JSON in only 4 of 120.
- Check every answer: one JSON object, no repeated keys, allowed values; otherwise retry, then a person.
- Never send a photo with a person in it to a hosted model. The course’s answers are recorded, so you need no key.
You can now ask a model for words about a photo and check the shape of its answer. But a well-shaped answer can still be wrong, and the cushion “bowl” passed every check. Next, you measure how often each model is right, and where it invents details.