Skip to content
Français

Promptable segmentation with SAM 2

  • Lesson 18
  • 40 min
  • Module 5: Segment

Where we are

Masks are expensive to draw by hand. A promptable model can draw one from a single click. In this lesson, we use SAM 2 to segment from a point or a box with no training, cut out product photos, and count how many need a fix.

Plays from YouTube. Nothing loads from YouTube until you press play. Privacy

What you will be able to do

Get a mask from a click or a box with SAM 2, with no training. Measure how good the masks are on pallets where we know every true mask. Then cut Larkfield’s products out of their photos, and catch the bad cut-outs with a simple rule. Last, see what a model like this costs.

NotebookDo this lesson in a notebookColab · Kaggle · GitHub

The notebook loads SAM 2.1 tiny and segments the tight pallet S-063 from box prompts and from clicks. It scores every mask against the true mask, cuts four catalogue products out of their photos, and fixes the one that fails the check. Its setup cells install PyTorch and Transformers, and download the catalogue and pallet photos (about 18 MB) and the model (about 150 MB). About 40 minutes; the cells run in about a minute after the setup.

Colab and Kaggle open your own copy. They need a free Google or Kaggle account. Stuck? See the solution notebook.

Before you startWhat to know and to have readyAbout 40 minutes

What you need to know

What you need on your computer

  • For the notebook: nothing. It runs in Colab or Kaggle.
  • This lesson adds nothing to the larkvision project.

Tested with Python 3.12, PyTorch 2.15.0 and Transformers 5.22.2 on macOS.

On many phones, you can press and hold a dog in a photo, and the phone lifts the dog out of the background. You do not draw an outline. You only say which object you mean. The phone finds where it ends.

That is the idea of this lesson. A person says “this one”, with a click or a rough box. A model draws the exact outline.

In the last lesson, Larkfield found two places where masks pay: the area of damage on returned candles, and products cut out for the catalogue. The candle photos came with damage masks, drawn by people for the VisA image set. The catalogue photos have only boxes.

The catalogue has 733 public photos. At an assumed 30 seconds an outline, outlines for all of them take about 6 hours. Every new product photo needs one more. And a model trained to draw masks of Larkfield’s products would need many such outlines first.

Is there a model that draws a mask for an object it was never trained on?

A promptable segmentation model draws a mask for the object that you show it. You show the object with a prompt:

  • a point: one click on the object;
  • a box: a rough rectangle around it.

The model returns a mask and a score: how sure it is that the mask is good. It needs no training on your photos.

The best-known family is SAM, the Segment Anything Model, from Meta. Its makers report that they trained the first version on about 11 million photos with more than a billion masks. This is a vendor claim, not measured here. So it has seen the edges of almost every kind of object. This course uses SAM 2.1 tiny: the smallest size of the second version. It has 31.4 million numbers (parameters), takes about 150 MB, and its code and weights are under the Apache-2.0 licence. We load it through the Transformers library. SAM 2 can also follow an object through the frames of a video; this course uses it on single photos.

A flow: Photo, then Image encoder; a Prompt; both go into the Mask decoder; out comes a Mask with a score.
How a promptable model works. The heavy part, the image encoder, reads the photo once. The light part, the mask decoder, turns each prompt into a mask, so many prompts on one photo cost little more than one.
Text description of the diagram

Five boxes with arrows. The photo goes into the image encoder, which reads it once. A prompt, a point you click or a box you draw, goes in separately. Both go into the mask decoder, which is fast and runs again for every prompt. Out comes a mask, true or false for every pixel, with a score. A note says that on a CPU, SAM 2.1 tiny took about 0.9 seconds for a pallet photo with 21 boxes.

This split matters for cost. On a laptop CPU, SAM 2.1 tiny took about 0.89 seconds for a pallet photo with about 21 box prompts. Most of that time is the encoder.

The pallet photos are drawn by our code, so we know every true mask. That makes them a fair test. We take the tight pallet S-063: 22 boxes that touch. We give SAM the 22 true boxes as prompts, one mask each. Then we try one click per box instead: the centre of the box, a little below the middle.

We score each mask with mask IoU: the pixels in both masks, divided by the pixels in either. It is the IoU of Boxes and IoU, counted in pixels.

The tight pallet S-063 twice. Left, box prompts: 22 blue outlines that follow the boxes; title: mean IoU 0.929, 0 below 0.8. Right, point prompts with a green dot on each box: 20 blue outlines and 2 orange ones in the top layer, marked IoU 0.73 and IoU 0.79; title: mean IoU 0.881, 2 below 0.8.
SAM 2.1 tiny on the tight pallet S-063 (22 boxes). Left: one box prompt per box (dashed white). Right: one click per box (green dot). Blue: mask IoU of at least 0.8 with the true mask; orange: below 0.8. Small blue rings inside a box are holes in its mask, mostly the white label. The pallet photos are drawn by our code (synthetic, CC0).
Text description of the diagram

Two copies of a synthetic pallet photo with 22 touching boxes in four layers. Left panel, box prompts: each box has a dashed white prompt box and a blue mask outline. The outline follows the box’s front face and lid. The mean mask IoU is 0.929, and no mask is below 0.8. Right panel, point prompts: each box has a green dot on its front face. Most outlines are blue. Two boxes in the top layer have orange outlines that cover only the front face, not the lid on top, with IoU 0.73 and 0.79. The mean is 0.881.

Prompt Mean mask IoU Masks below 0.8 Worst mask
22 boxes 0.929 0 0.88
22 clicks 0.881 2 0.73

Common mistake

One click, half a box

Symptom: with one click on each box, two masks in the top layer of S-063 are poor: IoU 0.73 and 0.79. Each covers the front face of its box, and stops at the lid.

Cause: a click says “this object”, but not where it ends. The front face, the lid and the whole box are all fair answers to a click on the front face. SAM picks one, and here it picks the face.

Fix: give a box when you can. A box says where the object ends. With box prompts, both masks score above 0.9. With a click, add a second click on the part that is missing. SAM accepts several points for one object, and also “not this” points.

Consequence in AI work: a prompt is part of the input. An unclear prompt gives a mask that is right for some other question. Test your prompts as you test your model.

One photo is a small test. The course ran SAM 2.1 tiny on all 30 test pallets: 619 boxes, with a box prompt and with one click each.

  • Box prompt
  • Point prompt
Mean mask IoU of SAM 2.1 tiny on the 619 boxes of the 30 test pallets, by crowding. The axis starts at 0.80. Box prompts stay above 0.91 everywhere; one click per box falls on the tight pallets.
Text description of the chart

A chart with loose, medium and tight across and the mean mask IoU from 0.80 to 1.00 up. Box prompts: 0.915, 0.929 and 0.911. Point prompts: 0.887, 0.902 and 0.844. The gap between the two lines is largest on the tight pallets.

Prompt Mean mask IoU Mean Dice Masks below 0.8 (of 619)
Box 0.916 0.955 0
One click 0.865 not measured 98

Dice is a second score for masks, close to IoU; the last lesson of this module explains it. Two results stand out:

  • A box is a safe prompt. Not one of the 619 masks fell below 0.8.
  • One click is cheaper, but riskier where objects touch. On tight pallets, the mean falls to 0.844. Where boxes touch, the edge between them is less clear. A click does not say on which side the edge belongs.

The pallets are clean drawings. On real photos, with shadows, glare and clutter, expect lower scores.

Omar’s catalogue team wants each product on a plain background. Every catalogue photo was cut from an Open Images photo around a box that a person drew. So we already have a box prompt for every product, for free.

The course cut out 20 catalogue test photos with SAM 2.1 tiny, at about 0.45 seconds a photo.

A bad cut-out raises no error. The mask is just wrong. So we need a cheap check that runs on every photo. The course uses this rule: the box around the mask should match the labelled box. When their IoU is under 0.8, the cut-out goes to a person. This is our rule, not a standard. It catches masks that leak outside the box, and masks that cover only part of the object.

Eight product cut-outs on a grey checked background: a bowl, a burning candle, an armchair, a cushion, a lamp, a red mug, a broken plant pot and a basket of eggs and flowers labelled plant pot. Each tile gives the check value, from 0.93 to 0.99; the last tile has an orange frame and says check 0.71: fix.
Products cut out by SAM 2.1 tiny from their labelled boxes, with the box check of each. Seven pass; the basket (labelled "plant pot") needs a fix. The mug is not one of the 20 photos: the three mugs among them are beer glasses, so the figure shows one more test photo, with the same prompt and rule. Photos from Open Images (CC BY 2.0), cut and masked by our code: P-9a3fc835 Micah Sittig; P-773b2e17 Peter Trimming; P-3ac9eaf3 Tom Byrne; P-9ac69b65 sushi♥ina; P-b0244e29 Dave Hogg; P-b188bd04 Martha W McQuade; P-533ffe5f mk97007; P-53ba97e3 Nicolas Huk. The authors' pages are in the catalogue manifest.
Text description of the diagram

Eight square tiles in two rows. Each shows one product, cut out by its mask, on a light grey checked background. The checks: bowl 0.97, burning candle 0.93, armchair 0.97, white cushion 0.96. Then lamp 0.94, red mug 0.99, broken plant pot 0.95. The last tile, a basket of eggs with flowers labelled plant pot, has an orange frame and the text check 0.71: fix.

Of the 20 cut-outs, 1 needs a fix: P-53ba97e3. The catalogue labels are noisy too: the three “mugs” among the 20 are beer glasses, as in earlier modules. They all pass the check.

Three views of a basket with eggs and flowers. 1: the photo with a green labelled box. 2: SAM's mask in orange, with its own box reaching the top of the photo; check 0.71. 3: the cut-out with only the largest piece kept; its box matches the labelled box; check 0.92.
The cut-out that needs a fix, P-53ba97e3 (photo: Nicolas Huk, CC BY 2.0, from Open Images). SAM's mask is in 141 pieces; a small piece at the top edge makes the mask's box too tall. Keeping only the largest piece raises the check from 0.715 to 0.919.
Text description of the diagram

Panel 1: a wire basket with eggs, tulips and small flowers, on a wooden table. A green box goes around the basket and flowers, from about a quarter of the way down to the bottom. Panel 2: the same photo with SAM’s mask outlined in orange. The mask follows the basket and most flowers, but small pieces of leaves and petals lie outside, one at the very top edge. The mask’s own orange box therefore reaches the top of the photo. Panel 3: the cut-out with only the largest piece kept, on a checked background. Its blue box almost matches the green labelled box.

The mask is in 141 separate pieces. Most are tiny: single petals and leaves. The labelled box starts 64 pixels down. One small piece at the top edge stretches the mask’s box up to the top of the photo.

Two fixes, tried in order:

Fix Model calls Box check
None 1 0.715
One more click, in the centre of the box 2 0.733
Keep only the largest piece of the mask 1 0.919

The extra click barely helped: SAM still added the stray piece. Keeping the largest piece needs no model at all, and passes the check. This is also a rule. It would be wrong for a product made of separate parts, such as a pair of cups. That is why a person still looks at every cut-out that failed the check.

What is “the product” in this photo? The labelled box covers the basket and the flowers. A person might want only the basket. A model cannot answer that question; Omar’s team must.

SAM also speeds up the labelling of masks. In a labelling tool, a person draws a rough box or clicks. SAM draws the outline. The person accepts it, or adds a click to fix it. Some labelling tools, such as CVAT and Label Studio, can connect a SAM model for this.

A rough sum, with assumed times: a box takes 5 seconds, and a quick check of SAM’s mask takes 5 more. An outline by hand takes 30 seconds. Then 817 pallet outlines take about 2.3 hours instead of 6.8. Measure your own times before you plan with these numbers.

Tomás likes the idea: “SAM segments anything. We can drop the detectors.” Priya asks him: “Where do the prompts come from?”

SAM draws the mask of the object you show it. It does not decide which objects matter. It has an automatic mode that tries a grid of points over the whole photo. That mode returns hundreds of segments: boxes, labels, tape and shadows, with no names. Something must still say which segment is a box.

Use SAM when:

  • you have a box or a point for each object: from a detector, from existing labels, or from a person;
  • you need outlines and have no mask labels;
  • you label masks, and want to label faster.

Do not use SAM alone when:

  • you need to find or count objects. Use a detector, as in Module 4, and give SAM its boxes.
  • the “object” has no clear edge, such as a faint colour spot on a candle. SAM looks for object edges, and a spot of a few pixels hardly has one.
  • you need speed on a small device: 0.89 seconds a photo on a laptop CPU is slow for a camera stream.

What it costs: no labels and no training. About 150 MB of model, PyTorch and Transformers. About 0.45 seconds a catalogue photo and 0.89 seconds a pallet photo on a laptop CPU. Plus a check rule, and a person for the photos that fail it.

Guided practice: prompts on a tight pallet

Section titled “Guided practice: prompts on a tight pallet”
  1. Open the notebook from the box Do this lesson in a notebook at the top of this page.
  2. Run the setup cells. The first one installs PyTorch and Transformers, which can take 2 minutes the first time.
  3. In section 1, run the cell. It downloads SAM 2.1 tiny and prints SAM 2.1 tiny: 31.4 million numbers.
  4. In section 2, run the cell. You see mean mask IoU 0.929; masks below 0.8: 0.
  5. In section 3, predict which prompt gives better masks. Then run the cell. You see a mean of 0.881, with 2 masks below 0.8, and the worst mask next to the true one.
  6. In section 4, run the two cells. Three cut-outs pass. P-53ba97e3 has a check of 0.715, and its mask is in about 140 pieces.
  7. Go to section 5 for your task.

Keep only the largest piece of the mask of P-53ba97e3. Compute the box check of the new mask, and put it in fixed_check. The check cell prints Check passed when your answer is right.

Show the answer
sizes = ndimage.sum(bad, pieces, range(1, n + 1))
largest = pieces == (int(np.argmax(sizes)) + 1)
fixed_check = box_check(largest, BOXES["P-53ba97e3"]["box"])

The answer is 0.919. ndimage.label gives each separate piece its own number, and ndimage.sum adds up the mask’s pixels for each number. The piece with the most pixels is the basket and its flowers. The cut-out now passes the rule.

Check your understanding

On the 619 pallet boxes, box prompts gave a mean mask IoU of 0.916 with no mask below 0.8, and one click per box gave 0.865 with 98 below 0.8. Explain the difference, and say which prompt you would use for Larkfield's pallets.

Show the answer

A box says where the object ends; a click says only which object. When boxes touch, a click can give one face only, or spill into the neighbour. So the masks get worse on tight pallets (0.844). For the pallets, use box prompts. A detector already gives a box for every box on the pallet, so the safer prompt costs nothing extra.

Check your understanding

The cut-out of P-53ba97e3 failed the box check (0.715). A colleague says: 'The model is wrong, so we need a bigger model.' What else could explain the failure, and what fixed it here?

Show the answer

The mask was mostly right: it followed the basket and the flowers. But it was in about 140 pieces, and one small stray piece at the top of the photo stretched the mask’s box. Keeping only the largest piece raised the check to 0.919, with no new model call. The photo itself is also unclear. The labelled “plant pot” is a basket of eggs and flowers, and a person must decide what the product is. A bigger model would not settle that question.

Check your understanding

Tomás wants to use SAM to count the boxes on a pallet, with no detector. What is missing, and how would you combine SAM with a detector instead?

Show the answer

SAM needs a prompt for each object: it does not find or name objects. Its automatic mode returns many unnamed segments (boxes, labels, tape, shadows), so the count is not the number of boxes. Combine them. The fine-tuned detector finds and counts the boxes. When you need the exact outline, SAM turns each detected box into a mask.

Recap

  • A promptable segmentation model draws a mask for the object you show it with a point or a box, with no training. SAM 2.1 tiny: Apache-2.0, about 150 MB, 31.4 million parameters.
  • It reads the photo once and answers each prompt quickly. On a laptop CPU: about 0.89 seconds a pallet photo with about 21 boxes, and 0.45 seconds a catalogue photo.
  • Box prompts are safer than clicks: mean mask IoU 0.916 with 0 of 619 masks below 0.8, against 0.865 with 98 below 0.8. Clicks are worst where objects touch.
  • A bad mask raises no error. A cheap check, the mask’s box against the labelled box, sent 1 of 20 cut-outs to a person. Keeping the largest piece fixed it: 0.715 to 0.919.
  • SAM does not find or count objects: pair it with a detector or a person.

You can now get masks with a click or a box, and check them. A mask is also a measurement. Next, we measure the area of the damage on Larkfield’s candles in square millimetres, and compare masks with IoU and Dice.

Next: Measure from masks