Skip to content
Français

Semantic and instance segmentation

  • Lesson 17
  • 30 min
  • Module 5: Segment

Where we are

Boxes told us where things are and how many there are. A box does not say which pixels belong to an object. In this lesson, we meet semantic and instance segmentation, and we see when a mask is worth its extra cost.

Plays from YouTube. Nothing loads from YouTube until you press play. Privacy

What you will be able to do

See the difference between a box, a semantic mask and instance masks. Learn which question each one answers, and why a semantic mask cannot count objects that touch. Then decide when Larkfield should pay for masks instead of boxes.

NotebookDo this lesson in a notebookColab · Kaggle · GitHub

The notebook turns the outlines of the pallet boxes into masks. It builds a semantic mask and instance masks for one pallet, and counts the separate regions of the semantic mask. Its setup cells download the pallet photos (about 3 MB, no model and no key), so it needs no local setup. About 30 minutes; the cells run in a few seconds.

Colab and Kaggle open your own copy. They need a free Google or Kaggle account. Stuck? See the solution notebook.

Before you startWhat to know and to have readyAbout 30 minutes

What you need to know

What you need on your computer

  • For the notebook: nothing. It runs in Colab or Kaggle.
  • This lesson adds nothing to the larkvision project. The project gets a mask command at the end of this module.

Tested with Python 3.12 on macOS.

Hamid plans a new layout for the returns warehouse. He prints the floor plan and takes two highlighter pens.

First, he colours every shelf in yellow. Now he can see how much of the floor the shelves cover. But where one shelf touches the next, the yellow runs together. He cannot count the shelves from the yellow.

Then he takes a fresh copy and gives each shelf its own colour: shelf 1 red, shelf 2 blue, shelf 3 green. Now he can count them, and he can measure each one.

Computers colour photos in the same two ways. This lesson gives each way its name.

In Module 4, every object got a box: a rectangle around it. A box is cheap and good for counting. But a box is not the object. It also holds the pixels around the object.

For a cardboard box on a pallet, this does not matter much: the object is almost a rectangle. For a damage on a candle, it matters a lot. Look at one damaged tea light from the returns:

Two zoomed views of the edge of a tea light. Left: a blue rectangle around a damaged strip at the edge, titled Box: 1012 pixels. Right: the same strip coloured red, titled Mask: 408 pixels.
The damage on the returned tea light R-c3b7e506 ("damaged corner of packaging"), zoomed 6 times. The box around it holds 1,012 pixels; the damage itself holds 408 pixels. VisA candle photo and damage mask (CC BY 4.0, Amazon).
Text description of the diagram

Two views of the same corner of a tea light, zoomed so that each pixel is a small square. The damage is a thin curved strip along the bottom edge of the candle. Left: a blue rectangle drawn around the strip. It also covers wax above the strip and dark background below it: 1,012 pixels. Right: only the strip is coloured red: 408 pixels.

The damage is a thin curved strip. Its box is 1,012 pixels, but only 408 of them are damage. Suppose Hamid’s rule is “more than 500 damaged pixels: send the candle to the outlet”. The box sends this candle away. The true outline does not.

A box answers “where, roughly?”. Some questions need “which pixels, exactly?”.

A mask is a grid of yes-or-no values, one for each pixel of the photo. “Yes” means the pixel belongs to the object. “No” means it does not. You can draw a mask as a black-and-white image of the same size as the photo.

Giving masks to the objects in a photo is called segmentation: we cut the photo into segments. There are two main kinds.

  • Semantic segmentation says, for every pixel, which class it is: “box”, “candle”, “floor”. It does not separate two objects of the same class. This is Hamid’s yellow pen.
  • Instance segmentation gives each object its own mask: box 1, box 2, box 3. Two touching boxes stay apart. This is the pen with a new colour for each shelf.

A third name you may meet is panoptic segmentation: both at once. Every pixel gets a class, and every countable object also gets its own number.

A constructed photo of 6 × 10 = 60 pixels holds two boxes that touch. Box A has 16 pixels; box B has 12 pixels.

Three grids of 6 by 10 cells. The first shows two brown boxes that touch. The second marks all 28 box pixels with 1 and the rest with 0. The third marks box A's 16 pixels with 1 and box B's 12 pixels with 2.
A constructed photo of 60 pixels. The semantic grid has 28 ones in one connected region, so it cannot say how many boxes there are. The instance grid has 16 pixels of box 1 and 12 pixels of box 2.
Text description of the diagram

Three grids of 6 rows and 10 columns. Grid 1, the photo: box A covers rows 2 to 5 and columns 2 to 5 (16 pixels). Box B covers rows 3 to 5 and columns 6 to 9 (12 pixels). B touches A on its right side. Grid 2, the semantic mask: the 28 box pixels hold 1 and the other 32 pixels hold 0; the ones form one region. Grid 3, the instance masks: box A’s pixels hold 1, box B’s pixels hold 2, the rest 0.

Question Semantic grid Instance grid
How many pixels are “box”? 28 16 + 12 = 28
How many boxes? One region: it looks like 1 2
How big is box B? It cannot say 12 pixels

Both grids agree on the area of “box”. Only the instance grid can count, and only the instance grid can measure one object.

Our code draws Larkfield’s pallet photos, so we know the exact outline of every box. The outline is a polygon: the list of corners of the faces you can see. A polygon becomes a mask when we fill it in.

Here is the same medium pallet, S-062, with the three kinds of answer:

The same pallet photo three times. Left: 24 blue rectangles around the boxes. Middle: every box pixel coloured the same blue. Right: each of the 24 boxes in its own colour.
The pallet S-062 (24 boxes, medium crowding) with its true boxes, its semantic mask and its instance masks, all from the exact outlines. The pallet photos are drawn by our code (synthetic, CC0).
Text description of the diagram

Three copies of a synthetic pallet photo with 24 cardboard boxes in 6 columns and 4 layers. Panel 1: a thin blue rectangle around each box. Panel 2: every pixel of every box has the same blue, on a darker background. In a column, you cannot see where one box ends and the next begins. Panel 3: each box has its own colour, so each of the 24 boxes stands apart from its neighbours.

On this pallet, 52% of the pixels are “box”.

The boxes in each column of S-062 stand on each other. There is no gap between them. Can we still count the boxes in the semantic mask? A simple method finds the connected regions: groups of “yes” pixels that touch each other.

The semantic mask of the pallet S-062, with six coloured columns numbered 1 to 6. The title says: 24 boxes, but only 6 separate regions.
The connected regions of the semantic mask of S-062: one region for each column of four boxes. 24 boxes, 6 regions.
Text description of the diagram

The synthetic pallet S-062 with its semantic mask. The mask falls into six coloured regions, numbered 1 to 6 from left to right. Each region is one column of four boxes stacked on each other. The gaps between columns separate the regions; the boxes within a column touch, so they join.

The semantic mask has only 6 regions for 24 boxes. The 8-pixel gaps between the columns keep the columns apart. The boxes in a column touch, so they join. On a tight pallet, where every box touches its neighbours, the whole pallet can become one region.

Common mistake

Counting regions in a semantic mask

Symptom: a team counts boxes by counting the connected regions of a semantic mask. On loose pallets the count is right. On S-062 it says 6 for 24 boxes.

Cause: a semantic mask has no idea of “one object”. Two objects of the same class that touch become one region.

Fix: count from instance masks, or from boxes, as in Module 4. Use a semantic mask for areas, not for counts.

Consequence in AI work: a mask can be right in every pixel and still give the wrong answer to your question. Choose the kind of answer from the question.

Masks cost more than boxes, in three ways:

  • Labels: a person draws a box with two clicks, and an outline with many. Assume 5 seconds for a box, as in Module 4, and 30 seconds for an outline. Both are assumptions: time your own tool. The 817 boxes of the 40 training pallets then take about 68 minutes as boxes, and about 6.8 hours as outlines. At 30 dollars an hour, that is about 34 dollars against about 204 dollars.
  • Models: a model that draws masks is larger and slower than a detector of the same family.
  • Checks: a person checks a box at a glance. Checking an outline takes longer.

So pay for a mask only when the box gives a different answer. How different is a box from the true outline? Treat the box as a mask, and compute its mask IoU with the true mask: the pixels in both, divided by the pixels in either.

How well a box stands for the true outline. Pallet boxes (the 619 boxes of the 30 test pallets, mean by crowding): the box is almost the mask. Damage on the 18 damaged test candles (median): the box holds about twice the damage's pixels.
Text description of the chart

A bar chart with four bars, from 0 to 1. Pallet boxes: loose 0.97, medium 0.96, tight 0.95. Damage on the candles: 0.56, the median of 18 photos.

For pallet boxes, the box and the outline share 95% to 97% of their pixels. A mask adds almost nothing. For damage on a candle, the median IoU is 0.56: the box holds about twice as many pixels as the damage. Damage is thin, curved or in several small pieces.

Use boxes when the question is “how many?” or “where, roughly?”, and the objects are about rectangular. Counting boxes on a pallet is the example.

Use instance masks when the question is about the exact pixels of each object:

  • How big is it? The area of a damage on a candle, so that Hamid can set a rule by size. This is the last lesson of this module.
  • Cut it out. The catalogue team wants each product on a plain background. This is the next lesson.
  • What touches what? For example, whether a stain crosses the edge of a label.

Use a semantic mask when you need the area of a class, and not the objects. Examples: how much of a shelf is empty, or how much of a cushion cover has stains. A semantic mask cannot count objects that touch.

Priya looks at the three teams and their questions:

Team Question Answer kind
Warehouse (pallets) How many boxes? Boxes are enough: box and outline agree at 0.95 or more
Returns (Hamid) How much of the candle is damaged? An instance mask for each damage
Catalogue (Omar) The product without its background An instance mask for each product

The returns photos already have damage masks: people drew them for the VisA image set. The catalogue photos have only boxes. Drawing thousands of product outlines by hand would cost many hours. In the next lesson, a model draws them from a single box or click.

  1. Open the notebook from the box Do this lesson in a notebook at the top of this page.
  2. Run the setup cells. They download the pallet photos and their outlines.
  3. In section 1, run the cell. It fills each polygon into a mask. S-062 has 24 masks of 512 × 384 pixels.
  4. In section 2, run the cell. It draws the photo, the semantic mask and the instance masks side by side.
  5. Predict: how many separate regions does the semantic mask have? Then run the next cell. You see boxes: 24 | separate regions in the semantic mask: 6.
  6. Go to section 3 for your task.

Your turn: how much of a tight pallet is boxes?

Section titled “Your turn: how much of a tight pallet is boxes?”

Section 3 of the notebook has one task. For each of the 10 tight test pallets, compute the share of pixels inside a box. This share is the mean of the photo’s semantic mask. Put the mean of the 10 shares in tight_share. The check cell prints Check passed when your answer is right.

Before you run it, predict: is the share higher or lower than on S-062 (52%)?

Show the answer
shares = []
for im in pallets["images"]:
if im["crowding"] == "tight":
ms = [mask(a["segmentation"][0], im["width"], im["height"])
for a in pallets["annotations"] if a["image_id"] == im["id"]]
shares.append(np.any(ms, axis=0).mean())
tight_share = float(np.mean(shares))

The answer is 0.377: on average, 38% of a tight pallet photo is box. That is lower than on S-062, although the boxes touch. Tight is about the gaps between boxes, not about how much of the photo the pallet fills. This is a semantic question: you need no instance masks for it.

Check your understanding

A photo shows three mugs. Two of them touch. A semantic mask marks every mug pixel. How many connected regions do you expect, and why can the mask still be useful?

Show the answer

Two regions: the two touching mugs join into one, and the third mug is its own region. So you cannot count the mugs from it. It is still useful for questions about the class as a whole. For example: how many pixels are “mug”?

Check your understanding

On the pallets, a box and the true outline share 95% to 97% of their pixels. On candle damage, the median is 0.56. What does this tell you about where Larkfield should pay for masks?

Show the answer

On the pallets, a box is almost the outline, so masks add little to counting and are not worth their labelling cost. On the candles, the box holds about twice the damage’s pixels, so an area from a box would be about twice too large. Larkfield should pay for masks where the size or shape of the object drives the decision, such as damage area, and keep boxes for counting.

Check your understanding

Omar wants each catalogue product on a white background. Hamid wants the share of each returned candle's top that is damaged. Which kind of segmentation does each need: semantic or instance? Give a reason for each.

Show the answer

Omar needs an instance mask for each product, because he must cut out one object at a time. A class mask would be enough only when each photo has one product, and it would fail as soon as two products touch. Hamid needs an instance mask for each candle, to know which candle the damage is on and the size of that candle’s top. With a damage mask for that candle, he divides the damage pixels by the candle’s pixels.

Recap

  • A mask says yes or no for every pixel. Semantic segmentation gives each pixel a class; instance segmentation gives each object its own mask; panoptic does both.
  • A semantic mask cannot count objects that touch: the medium pallet S-062 has 24 boxes and only 6 regions.
  • Masks cost more than boxes: in labels (an assumed 6.8 hours against 68 minutes for 817 boxes), in models and in checks.
  • Pay for masks when the exact pixels drive the decision. For a pallet box, the box is almost the outline (IoU 0.95 to 0.97). For damage on a candle, the box is about twice too big (median 0.56).

You can now say which kind of mask a question needs. But masks are expensive to draw by hand. Next, a model draws a mask from one click or one box, with no training, and we check how often it is right.

Next: Promptable segmentation with SAM 2