Skip to content
Français

Five kinds of answer

  • Lesson 2
  • 30 min
  • Module 1: Seeing is a business problem

Where we are

We have seen that a wrong decision on a returned item costs money in two different ways. But "use AI on the photos" still does not say what answer we want. In this lesson, we meet five kinds of answer: a label, a box, a mask, a match and a text. We start with five questions that a friend could ask about one kitchen shelf.

Plays from YouTube. Nothing loads from YouTube until you press play. Privacy

What you will be able to do

Name the kind of answer that a vision request needs before you choose a method. The kind of answer decides what you label, which methods can work and what the work costs.

Before you startWhat to know and to have readyAbout 30 minutes

What you need to know

What you need on your computer

Nothing. This lesson has no code. Every photo and answer on this page comes from the course’s image files, which you open in Meet the Larkfield images.

A friend calls you while you stand in your kitchen. She asks five questions about the same shelf:

  1. “What is that white thing on the left?” You answer with one word: “a mug”.
  2. “How many jars are there, and where?” You point at each jar and count: “four, here, here, here and here”.
  3. “Exactly where is the crack in the big bowl?” You trace the crack with your finger, along its whole length.
  4. “Is that the vase that I gave you?” You compare it with the vase that you remember: “yes, that one”.
  5. “Describe the shelf to me.” You talk: “Three white mugs, a brown vase with dry flowers, four glass jars.”

You looked at the same shelf five times. Each time, the question asked for a different kind of answer: a word, places, an outline, a comparison and a description. You also did different work. Saying “a mug” took a second. Tracing the crack took care. Comparing the vase needed a memory of another vase.

A vision system works the same way. Before anyone builds one, someone must say which kind of answer it must give. “Use AI on the photos” does not say it.

Larkfield has photos of four kinds: product photos for the catalogue, photos of returned items, photos of boxes on pallets, and photos that customers send. Here is one photo for each kind of answer, with the real answer from the course’s files.

Five panels. Label: a photo of a white mug of coffee, with the answer mug. Box: a drawing of a pallet with brown boxes in six columns and four layers; a red rectangle surrounds each of the 24 boxes. Mask: a close view of a tea light; a red area covers the part where wax melted out, 1,456 pixels. Match: a dark, tilted phone photo of a white bowl, with an arrow to the catalogue photo of the same bowl, P-9d48a6c9, rank 1. Text: a photo of a round brown vase with dry flowers, with the answer vase, brown, ceramic.
One Larkfield photo for each kind of answer. Photos: mug by Joris de Vrede, bowl by 準建築人手札網站 Forgemind ArchiMedia, vase by Horacio Maria (Open Images, CC BY 2.0); tea light from the VisA dataset (Amazon, CC BY 4.0). The pallet is drawn by our code, and the customer photo is made from the catalogue photo by our code (both synthetic).
Text description of the diagram

The figure has five panels. Label: a photo of a white mug full of black coffee. The answer is one word, “mug”. Box: a pallet with brown boxes in six columns and four layers, drawn by our code. A red rectangle surrounds each box, and the answer is 24 boxes. Mask: a close view of one tea light. A red area covers the part where wax melted out over the edge. The answer is a set of 1,456 pixels. Match: on the left, a dark and tilted phone photo of a white bowl. An arrow leads to the catalogue photo of the same bowl, with the answer “P-9d48a6c9, rank 1”. Text: a photo of a round brown vase with dry flowers. The answer is “vase, brown, ceramic”, with a sentence of description.

A label is one answer for the whole photo, chosen from a fixed list. The mug photo gets the label mug, from the catalogue’s eight categories: bowl, candle, chair, cushion, lamp, mug, plant pot and vase. The task of giving a label is called classification.

Hamid’s returns decision is a label too. The list has three answers: pass, check or fail. So “is this pack damaged?” is a classification task, even though it is about damage.

A label is cheap to make. A person looks at the photo and types or clicks one word. But a label says nothing about where something is, or how many there are.

A box is a rectangle around one object in the photo, with the object’s name. The task of finding boxes is called detection. Counting is often done by detection: you find the boxes, then you count them.

The pallet photo is drawn by our code, so every box is known exactly. It has 24 boxes. Each one is written as 4 numbers: the left edge (x), the top edge (y), the width (w) and the height (h), in pixels. The first box of this pallet is [23.0, 276.8, 65.7, 68.2]. It starts 23 pixels from the left of the photo and 276.8 pixels from the top. It is 65.7 pixels wide and 68.2 pixels high. So the label of this one photo is 24 × 4 = 96 numbers.

A mask marks every pixel that belongs to the object, so it follows the object’s exact outline. A mask is a second image of the same size, where each pixel is yes (white) or no (black). The task of making masks is called segmentation.

The tea light photo is 320 × 291 pixels: 93,120 pixels in all. Its mask says yes for 1,456 of them, the area where wax melted out. That is 1.6% of the photo. A mask answers questions that a box cannot answer: how large is the damage, and does it reach the edge?

A match finds the most similar photo in a collection that you already have. The answer is not a name from a list. It is an ID from the collection, or a short list of IDs in order.

A customer sends a phone photo of a white bowl. It is darker and tilted, on a different table. The course made it from the catalogue photo with code, as you see in Meet the Larkfield images. A model turned each photo into a list of numbers that describes it, called an embedding. Image embeddings explains it. Then the program ranked all 733 catalogue photos by how close their numbers are to the customer’s photo. These are the top three, with their similarity, a score from 0 to 1, where 1 means the same:

Rank Catalogue photo Similarity
1 P-9d48a6c9 (the right bowl) 0.749
2 P-593a4ac1 0.581
3 P-1815e7a4 0.572

The right bowl is first, well ahead of the next one. A match needs no list of categories, so it also works for products that are new in the catalogue.

A text answer is words or sentences about the photo: a caption, product details, or an answer to a question. Models that read a photo and write text are called vision-language models.

The course asked a hosted vision-language model to describe the vase photo, with a fixed set of fields. This is a recorded answer: the model answered once, the course saved the answer, and the lessons replay it. You need no key and no account to see it.

{"product_type": "vase", "main_colour": "brown", "material": "ceramic",
"alt_text": "A round brown ceramic vase with a narrow neck and mottled rustic finish holds dried stems on a light surface."}

The type, the colour and the material agree with the labels that the course made by eye. The alt_text field is a description for people who use a screen reader. A text answer is easy to read, but hard to check: the model can write a detail that is not in the photo. This one answer took 5.6 seconds, and the course measured about 1.38 US dollars per 1,000 photos.

The kind of answer sets the method, the labels and the cost

Section titled “The kind of answer sets the method, the labels and the cost”

Each kind of answer needs its own labels, its own methods and its own checks. So the first decision of any vision project is the kind of answer.

A table of five rows with four columns: the question, the kind of answer, one label is, and at Larkfield. What is it: label; one word for the photo, mug; sort catalogue photos, good or damaged returns. Where is it and how many: box; 4 numbers for each object, 24 boxes are 96 numbers; count the boxes on a pallet. Exactly which pixels: mask; a yes or no for each pixel, 1,456 of 93,120 pixels are yes; measure the damage on a returned item. Which item is this: match; the ID of the right catalogue photo; find the product in a customer's photo. Describe it: text; words or sentences, colour, material, alt text; write product details and alt text.
The question decides the kind of answer, and the kind of answer decides what one label is. The label examples are the real labels of the photos above.
Text description of the diagram

The diagram is a table with five rows and four columns. The first column is the question, the second the kind of answer, the third what one label is, and the fourth where Larkfield uses it. “What is it?” needs a label: one word for the photo, such as “mug”. Larkfield uses it to sort catalogue photos and to decide about returns. “Where is it? How many?” needs boxes: 4 numbers for each object, so 24 boxes are 96 numbers. Larkfield uses it to count boxes on a pallet. “Exactly which pixels?” needs a mask: a yes or no for each pixel. On the tea light photo, 1,456 of 93,120 pixels are yes. Larkfield uses it to measure the damage on a returned item. “Which item is this?” needs a match: the ID of the right catalogue photo. Larkfield uses it to find the product in a customer’s photo. “Describe it.” needs a text: words or sentences, such as the colour, the material and alt text. Larkfield uses it for product details and alt text.

Read the third column from top to bottom. The label grows: one word, then 96 numbers, then a decision for every pixel. A person who makes labels for a mask works much longer per photo than for a word. So a request that really needs only a label should not pay for masks.

The table below adds the methods of this course, and the place where you meet them:

Kind of answer Methods in this course How a result is judged Where
Label Colour rules, hand-made features, pretrained models, CLIP The share of right labels; the worst category Modules 2, 3 and 6
Box Detectors, pretrained or trained on a few photos How well the boxes overlap the true boxes; the count error Module 4
Mask Segmentation models such as SAM 2 How well the masks overlap the true masks; the area Module 5
Match Embeddings and nearest neighbours Is the right item in the top 5? Module 7
Text Vision-language models, small and hosted The share of right fields, checked against labels Module 7

The last column of the course’s comparison table follows the same idea. Each row has a task, and each task has its own main number. An accuracy for labels cannot be compared with a count error for boxes.

Real requests often combine kinds. Hamid’s returns need a label (pass, check or fail) for each pack. When a pack fails, the team may also want a mask, to see where the damage is. Counting boxes on a pallet needs boxes, and the count comes from the boxes. Omar’s catalogue needs labels for the category and text for the product details.

When you split a request into kinds, ask which kind the business acts on. Hamid acts on the label: sell or outlet. The mask only helps a person decide faster, so it can be rough.

For each request, write four things. First, the kind of answer. Second, the cost of a miss: the system does not find what it should. Third, the cost of a false alarm: it finds something that is not there. Fourth, what a person does with the result. Then open the answers.

  1. Omar: “Put each new product photo in the right category of the online shop.”
  2. The warehouse: “Before a truck leaves, count the boxes on the pallet.”
  3. Hamid: “For a damaged candle, how large is the damage? Small damage can still go to the outlet.”
  4. Customer service: “A customer sends a photo of a broken lamp. Which of our lamps is it?”
  5. Omar: “Write alt text for every product photo, for customers who use a screen reader.”
  6. Hamid: “Is this returned pack damaged?”
Show the answers
Request Kind Cost of a miss Cost of a false alarm What a person does
1. Category Label The product is in the wrong category, and customers do not find it The same mistake from the other side: another category gets a wrong product Checks the photos where the model is unsure
2. Count boxes Box A box is not counted: the delivery looks short, and someone searches for it An extra box is counted: the shop pays for a box that is not there Counts by hand when the count does not match the order
3. Size of damage Mask Damage too small: a bad candle goes to the outlet and comes back Damage too large: a candle that could sell is thrown away Looks at the mask on the photo before deciding
4. Which lamp Match The right lamp is not in the top 5: the agent asks the customer for more photos A wrong lamp ranks first: the customer gets the wrong spare part Chooses from the top 5 photos
5. Alt text Text A detail is missing: the description is less useful The model writes a detail that is not in the photo: the description is wrong Reads and corrects the text before it goes live
6. Damaged pack Label A damaged pack is sold as new: 45 dollars A good pack goes to the outlet: 28 dollars Checks the packs in the “check” zone, up to 100 a day

Request 6 is the only one with costs from the course. For the others, the costs are clear in words but not yet in dollars. Writing them down is the first step of every project in this course.

Common mistake

A box where a mask was needed

Symptom: Hamid wants to know how much wax melted out of the tea light in the mask panel. Tomás draws a box around the damage, because a box is faster: 51 × 57 = 2,907 pixels. He reports that the damage covers 2,907 pixels.

Cause: a box covers the damage and everything else inside the rectangle. The mask of the same damage has 1,456 pixels. The box says that the damage is twice as large as it is. Damage of an odd shape, like melted wax, fills only part of its box.

Fix: ask the question first. “Where is the damage?” needs a box. “How large is it?” needs a mask. If masks are too slow to draw by hand, Module 5 shows a model that turns a box into a mask.

Consequence in AI work: the wrong kind of answer gives numbers that look exact and are wrong. Hamid would send good candles to the outlet because their damage “looks” twice as large.

Check your understanding

Hamid asks: “Is this returned pack damaged?” A colleague answers: “Then we need a detector that draws a box around each damage.” Which kind of answer does Hamid need, and what does a box add?

Show the answer

Hamid needs a label for each pack: pass, check or fail. He acts on that label: sell as new, ask a person, or send to the outlet. A box adds where the damage is. That can help a person check faster, but Hamid does not need it to decide. Boxes also cost more to label: 4 numbers for each damage, instead of one word for each photo. Start with the label, and add boxes or masks only if they change a decision.

Check your understanding

Predict: the customer's photo of the bowl is matched against all 733 catalogue photos. Why does a match work for a product that was added to the catalogue yesterday, when a label classifier would not?

Show the answer

A label classifier chooses from a fixed list of categories that it learned. A new product is not in its training photos, so it can only guess the closest old category. A match compares the customer’s photo with every photo in the collection, by how close their numbers are. When the new product’s photo is in the catalogue, it can rank first with no new training. The bowl’s right photo ranked first with a similarity of 0.749, well ahead of the second (0.581).

Check your understanding

The pallet in the box panel has 24 boxes. How many numbers does its box label have, and what do the four numbers of a box mean?

Show the answer

Each box has 4 numbers, so the label has 24 × 4 = 96 numbers. The numbers are the left edge (x), the top edge (y), the width (w) and the height (h), in pixels. The first box is [23.0, 276.8, 65.7, 68.2]. It starts 23 pixels from the left and 276.8 pixels from the top, and it is 65.7 pixels wide and 68.2 pixels high.

Recap

  • A vision system gives five kinds of answer. A label says what it is, and a box says where and how many. A mask marks the pixels, a match finds the item in a collection, and a text describes the photo.
  • Hamid’s pass, check or fail is a label. Counting boxes on a pallet uses boxes: 24 boxes are 96 numbers. A mask of melted wax has 1,456 of 93,120 pixels.
  • A match ranks a collection: the customer’s bowl found its catalogue photo first (similarity 0.749). A text answer from a hosted model was recorded once and is replayed, so you need no key.
  • The kind of answer sets the labels, the methods and the way a result is judged. A label is one word; a mask is a decision for every pixel.
  • A box measured the melted wax as 2,907 pixels; its mask has 1,456. Choose the kind of answer that the business acts on.

You can now say what a vision system must answer. But a computer does not see a mug or a melted candle: it gets only numbers. Next, you look at those numbers, and see why the same mug gives other numbers under another light.

Next: Images are numbers