Five kinds of answer
- Lesson 2
- 30 min
- Module 1: Seeing is a business problem
Where we are
We have seen that a wrong decision on a returned item costs money in two different ways. But "use AI on the photos" still does not say what answer we want. In this lesson, we meet five kinds of answer: a label, a box, a mask, a match and a text. We start with five questions that a friend could ask about one kitchen shelf.
Lesson 2 of 30 · Vision AI: See, Count and InspectFive kinds of answer
1. Seeing is a business problem
2. Baselines and pretrained classifiers
3. Embeddings, CLIP and few labels
4. Detect and count
6. Inspect for defects
7. Match, search and describe
8. Run vision in production
What you will be able to do
Name the kind of answer that a vision request needs before you choose a method. The kind of answer decides what you label, which methods can work and what the work costs.
Before you startWhat to know and to have ready
What you need to know
- Hamid’s returns, the two mistakes and their costs, from The return that cost twice.
- What a label is in supervised learning, from Supervised learning in Machine Learning: From Problem to Reliable Model.
What you need on your computer
Nothing. This lesson has no code. Every photo and answer on this page comes from the course’s image files, which you open in Meet the Larkfield images.
Five questions about one shelf
Section titled “Five questions about one shelf”A friend calls you while you stand in your kitchen. She asks five questions about the same shelf:
- “What is that white thing on the left?” You answer with one word: “a mug”.
- “How many jars are there, and where?” You point at each jar and count: “four, here, here, here and here”.
- “Exactly where is the crack in the big bowl?” You trace the crack with your finger, along its whole length.
- “Is that the vase that I gave you?” You compare it with the vase that you remember: “yes, that one”.
- “Describe the shelf to me.” You talk: “Three white mugs, a brown vase with dry flowers, four glass jars.”
You looked at the same shelf five times. Each time, the question asked for a different kind of answer: a word, places, an outline, a comparison and a description. You also did different work. Saying “a mug” took a second. Tracing the crack took care. Comparing the vase needed a memory of another vase.
A vision system works the same way. Before anyone builds one, someone must say which kind of answer it must give. “Use AI on the photos” does not say it.
Five answers from Larkfield’s photos
Section titled “Five answers from Larkfield’s photos”Larkfield has photos of four kinds: product photos for the catalogue, photos of returned items, photos of boxes on pallets, and photos that customers send. Here is one photo for each kind of answer, with the real answer from the course’s files.

Text description of the diagram
The figure has five panels. Label: a photo of a white mug full of black coffee. The answer is one word, “mug”. Box: a pallet with brown boxes in six columns and four layers, drawn by our code. A red rectangle surrounds each box, and the answer is 24 boxes. Mask: a close view of one tea light. A red area covers the part where wax melted out over the edge. The answer is a set of 1,456 pixels. Match: on the left, a dark and tilted phone photo of a white bowl. An arrow leads to the catalogue photo of the same bowl, with the answer “P-9d48a6c9, rank 1”. Text: a photo of a round brown vase with dry flowers. The answer is “vase, brown, ceramic”, with a sentence of description.
Label: what is it?
Section titled “Label: what is it?”A label is one answer for the whole photo, chosen from a fixed list. The mug photo gets the label mug, from the catalogue’s eight categories: bowl, candle, chair, cushion, lamp, mug, plant pot and vase. The task of giving a label is called classification.
Hamid’s returns decision is a label too. The list has three answers: pass, check or fail. So “is this pack damaged?” is a classification task, even though it is about damage.
A label is cheap to make. A person looks at the photo and types or clicks one word. But a label says nothing about where something is, or how many there are.
Box: where is it, and how many?
Section titled “Box: where is it, and how many?”A box is a rectangle around one object in the photo, with the object’s name. The task of finding boxes is called detection. Counting is often done by detection: you find the boxes, then you count them.
The pallet photo is drawn by our code, so every box is known exactly. It has 24 boxes. Each one is written as 4 numbers: the left edge (x), the top edge (y), the width (w) and the height (h), in pixels. The first box of this pallet is [23.0, 276.8, 65.7, 68.2]. It starts 23 pixels from the left of the photo and 276.8 pixels from the top. It is 65.7 pixels wide and 68.2 pixels high. So the label of this one photo is 24 × 4 = 96 numbers.
Mask: exactly which pixels?
Section titled “Mask: exactly which pixels?”A mask marks every pixel that belongs to the object, so it follows the object’s exact outline. A mask is a second image of the same size, where each pixel is yes (white) or no (black). The task of making masks is called segmentation.
The tea light photo is 320 × 291 pixels: 93,120 pixels in all. Its mask says yes for 1,456 of them, the area where wax melted out. That is 1.6% of the photo. A mask answers questions that a box cannot answer: how large is the damage, and does it reach the edge?
Match: which catalogue item is this?
Section titled “Match: which catalogue item is this?”A match finds the most similar photo in a collection that you already have. The answer is not a name from a list. It is an ID from the collection, or a short list of IDs in order.
A customer sends a phone photo of a white bowl. It is darker and tilted, on a different table. The course made it from the catalogue photo with code, as you see in Meet the Larkfield images. A model turned each photo into a list of numbers that describes it, called an embedding. Image embeddings explains it. Then the program ranked all 733 catalogue photos by how close their numbers are to the customer’s photo. These are the top three, with their similarity, a score from 0 to 1, where 1 means the same:
| Rank | Catalogue photo | Similarity |
|---|---|---|
| 1 | P-9d48a6c9 (the right bowl) | 0.749 |
| 2 | P-593a4ac1 | 0.581 |
| 3 | P-1815e7a4 | 0.572 |
The right bowl is first, well ahead of the next one. A match needs no list of categories, so it also works for products that are new in the catalogue.
Text: describe it
Section titled “Text: describe it”A text answer is words or sentences about the photo: a caption, product details, or an answer to a question. Models that read a photo and write text are called vision-language models.
The course asked a hosted vision-language model to describe the vase photo, with a fixed set of fields. This is a recorded answer: the model answered once, the course saved the answer, and the lessons replay it. You need no key and no account to see it.
{"product_type": "vase", "main_colour": "brown", "material": "ceramic", "alt_text": "A round brown ceramic vase with a narrow neck and mottled rustic finish holds dried stems on a light surface."}The type, the colour and the material agree with the labels that the course made by eye. The alt_text field is a description for people who use a screen reader. A text answer is easy to read, but hard to check: the model can write a detail that is not in the photo. This one answer took 5.6 seconds, and the course measured about 1.38 US dollars per 1,000 photos.
The kind of answer sets the method, the labels and the cost
Section titled “The kind of answer sets the method, the labels and the cost”Each kind of answer needs its own labels, its own methods and its own checks. So the first decision of any vision project is the kind of answer.
Text description of the diagram
The diagram is a table with five rows and four columns. The first column is the question, the second the kind of answer, the third what one label is, and the fourth where Larkfield uses it. “What is it?” needs a label: one word for the photo, such as “mug”. Larkfield uses it to sort catalogue photos and to decide about returns. “Where is it? How many?” needs boxes: 4 numbers for each object, so 24 boxes are 96 numbers. Larkfield uses it to count boxes on a pallet. “Exactly which pixels?” needs a mask: a yes or no for each pixel. On the tea light photo, 1,456 of 93,120 pixels are yes. Larkfield uses it to measure the damage on a returned item. “Which item is this?” needs a match: the ID of the right catalogue photo. Larkfield uses it to find the product in a customer’s photo. “Describe it.” needs a text: words or sentences, such as the colour, the material and alt text. Larkfield uses it for product details and alt text.
Read the third column from top to bottom. The label grows: one word, then 96 numbers, then a decision for every pixel. A person who makes labels for a mask works much longer per photo than for a word. So a request that really needs only a label should not pay for masks.
The table below adds the methods of this course, and the place where you meet them:
| Kind of answer | Methods in this course | How a result is judged | Where |
|---|---|---|---|
| Label | Colour rules, hand-made features, pretrained models, CLIP | The share of right labels; the worst category | Modules 2, 3 and 6 |
| Box | Detectors, pretrained or trained on a few photos | How well the boxes overlap the true boxes; the count error | Module 4 |
| Mask | Segmentation models such as SAM 2 | How well the masks overlap the true masks; the area | Module 5 |
| Match | Embeddings and nearest neighbours | Is the right item in the top 5? | Module 7 |
| Text | Vision-language models, small and hosted | The share of right fields, checked against labels | Module 7 |
The last column of the course’s comparison table follows the same idea. Each row has a task, and each task has its own main number. An accuracy for labels cannot be compared with a count error for boxes.
One request can need two kinds
Section titled “One request can need two kinds”Real requests often combine kinds. Hamid’s returns need a label (pass, check or fail) for each pack. When a pack fails, the team may also want a mask, to see where the damage is. Counting boxes on a pallet needs boxes, and the count comes from the boxes. Omar’s catalogue needs labels for the category and text for the product details.
When you split a request into kinds, ask which kind the business acts on. Hamid acts on the label: sell or outlet. The mask only helps a person decide faster, so it can be rough.
Guided practice: six Larkfield requests
Section titled “Guided practice: six Larkfield requests”For each request, write four things. First, the kind of answer. Second, the cost of a miss: the system does not find what it should. Third, the cost of a false alarm: it finds something that is not there. Fourth, what a person does with the result. Then open the answers.
- Omar: “Put each new product photo in the right category of the online shop.”
- The warehouse: “Before a truck leaves, count the boxes on the pallet.”
- Hamid: “For a damaged candle, how large is the damage? Small damage can still go to the outlet.”
- Customer service: “A customer sends a photo of a broken lamp. Which of our lamps is it?”
- Omar: “Write alt text for every product photo, for customers who use a screen reader.”
- Hamid: “Is this returned pack damaged?”
Show the answers
| Request | Kind | Cost of a miss | Cost of a false alarm | What a person does |
|---|---|---|---|---|
| 1. Category | Label | The product is in the wrong category, and customers do not find it | The same mistake from the other side: another category gets a wrong product | Checks the photos where the model is unsure |
| 2. Count boxes | Box | A box is not counted: the delivery looks short, and someone searches for it | An extra box is counted: the shop pays for a box that is not there | Counts by hand when the count does not match the order |
| 3. Size of damage | Mask | Damage too small: a bad candle goes to the outlet and comes back | Damage too large: a candle that could sell is thrown away | Looks at the mask on the photo before deciding |
| 4. Which lamp | Match | The right lamp is not in the top 5: the agent asks the customer for more photos | A wrong lamp ranks first: the customer gets the wrong spare part | Chooses from the top 5 photos |
| 5. Alt text | Text | A detail is missing: the description is less useful | The model writes a detail that is not in the photo: the description is wrong | Reads and corrects the text before it goes live |
| 6. Damaged pack | Label | A damaged pack is sold as new: 45 dollars | A good pack goes to the outlet: 28 dollars | Checks the packs in the “check” zone, up to 100 a day |
Request 6 is the only one with costs from the course. For the others, the costs are clear in words but not yet in dollars. Writing them down is the first step of every project in this course.
Common mistake
A box where a mask was needed
Symptom: Hamid wants to know how much wax melted out of the tea light in the mask panel. Tomás draws a box around the damage, because a box is faster: 51 × 57 = 2,907 pixels. He reports that the damage covers 2,907 pixels.
Cause: a box covers the damage and everything else inside the rectangle. The mask of the same damage has 1,456 pixels. The box says that the damage is twice as large as it is. Damage of an odd shape, like melted wax, fills only part of its box.
Fix: ask the question first. “Where is the damage?” needs a box. “How large is it?” needs a mask. If masks are too slow to draw by hand, Module 5 shows a model that turns a box into a mask.
Consequence in AI work: the wrong kind of answer gives numbers that look exact and are wrong. Hamid would send good candles to the outlet because their damage “looks” twice as large.
Knowledge check
Section titled “Knowledge check”Check your understanding
Hamid asks: “Is this returned pack damaged?” A colleague answers: “Then we need a detector that draws a box around each damage.” Which kind of answer does Hamid need, and what does a box add?
Show the answer
Hamid needs a label for each pack: pass, check or fail. He acts on that label: sell as new, ask a person, or send to the outlet. A box adds where the damage is. That can help a person check faster, but Hamid does not need it to decide. Boxes also cost more to label: 4 numbers for each damage, instead of one word for each photo. Start with the label, and add boxes or masks only if they change a decision.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
Predict: the customer's photo of the bowl is matched against all 733 catalogue photos. Why does a match work for a product that was added to the catalogue yesterday, when a label classifier would not?
Show the answer
A label classifier chooses from a fixed list of categories that it learned. A new product is not in its training photos, so it can only guess the closest old category. A match compares the customer’s photo with every photo in the collection, by how close their numbers are. When the new product’s photo is in the catalogue, it can rank first with no new training. The bowl’s right photo ranked first with a similarity of 0.749, well ahead of the second (0.581).
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
The pallet in the box panel has 24 boxes. How many numbers does its box label have, and what do the four numbers of a box mean?
Show the answer
Each box has 4 numbers, so the label has 24 × 4 = 96 numbers. The numbers are the left edge (x), the top edge (y), the width (w) and the height (h), in pixels. The first box is [23.0, 276.8, 65.7, 68.2]. It starts 23 pixels from the left and 276.8 pixels from the top, and it is 65.7 pixels wide and 68.2 pixels high.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Recap
- A vision system gives five kinds of answer. A label says what it is, and a box says where and how many. A mask marks the pixels, a match finds the item in a collection, and a text describes the photo.
- Hamid’s pass, check or fail is a label. Counting boxes on a pallet uses boxes: 24 boxes are 96 numbers. A mask of melted wax has 1,456 of 93,120 pixels.
- A match ranks a collection: the customer’s bowl found its catalogue photo first (similarity 0.749). A text answer from a hosted model was recorded once and is replayed, so you need no key.
- The kind of answer sets the labels, the methods and the way a result is judged. A label is one word; a mask is a decision for every pixel.
- A box measured the melted wax as 2,907 pixels; its mask has 1,456. Choose the kind of answer that the business acts on.
You can now say what a vision system must answer. But a computer does not see a mug or a melted candle: it gets only numbers. Next, you look at those numbers, and see why the same mug gives other numbers under another light.
Next: Images are numbers