Skip to content
Français

Measure from masks

  • Lesson 19
  • 35 min
  • Module 5: Segment

Where we are

We can now get masks with a click. A mask is also a measurement. In this lesson, we measure the area of damage on a candle from its mask, compare masks with IoU and Dice, and use a reference for scale.

What you will be able to do

Turn a mask into a measurement that Hamid can use: an area in square millimetres, and the share of a candle that is damaged. Use an object of known size for scale, and see how much the answer depends on it. Then compare two masks with IoU and Dice, and see why a one-pixel mistake ruins the score of a tiny mask.

NotebookDo this lesson in a notebookColab · Kaggle · GitHub

The notebook measures the damage on the 18 damaged test candles from their masks. It counts pixels, finds the scale from the tea light’s width, and gives square millimetres. Then it computes mask IoU and Dice on a small grid, and on masks that are one pixel too big. Its setup cells download the returns photos and their masks (about 15 MB, no model and no key). About 35 minutes; the cells run in a few seconds.

Colab and Kaggle open your own copy. They need a free Google or Kaggle account. Stuck? See the solution notebook.

Before you startWhat to know and to have readyAbout 35 minutes

What you need to know

What you need on your computer

  • For the notebook: nothing. It runs in Colab or Kaggle.
  • For the project steps: your larkvision project at the end of Module 4, with its virtual environment, and the practice files in ../nextia-learning-labs. If your project is broken, copy the snapshot end-of-m04 again, as in Meet the Larkfield images.

Tested with Python 3.12 on macOS. The Windows commands are written out but were not run.

When you send a photo of a scratch on your car to an insurer, they often ask you to put a coin next to it. The coin is not for them to keep. Everybody knows how big a coin is. So the photo now says how big the scratch is, and not only how many pixels it covers.

Larkfield’s returns photos have a “coin” of their own: every photo shows four tea lights.

In the last two lessons, we got masks: from people who drew them, or from SAM. A mask gives an area at once: count the “yes” pixels. The damage on the returned tea light R-d17d264e covers 1,262 pixels.

But Hamid cannot use “1,262 pixels”. A pixel is not a size:

  • A photo taken from closer makes everything bigger. The same damage then covers more pixels.
  • A camera with more pixels gives more pixels for the same damage.
  • A resized photo changes every count. The VisA photos are 1,284 pixels wide at the source; Larkfield’s copies are 320 wide.

Hamid’s team thinks in millimetres: “a dent smaller than a fingernail is fine”. So we need a scale.

A reference object is something in the photo whose real size you know. You measure it in pixels, and you get the scale: millimetres per pixel.

An area needs the scale twice: once for the width, once for the height. If 1 pixel is 0.372 mm wide, one square pixel is 0.372 × 0.372 = 0.139 square millimetres (mm²).

Every returns photo shows four tea lights. A standard tea light is about 38 mm wide. That is an assumption: tea lights vary, so check the ones in your own photos.

Step Value
The tea light’s width in the photo 102.1 pixels
Scale: 38 mm ÷ 102.1 pixels 0.372 mm a pixel
One square pixel: 0.372 × 0.372 0.139 mm²
The damage: 1,262 pixels × 0.1386 mm² 175.0 mm²
A photo of four tea lights on a dark cloth. The bottom-left one has a red area on its left side: the damage mask. A green line across it reads 102.1 px = 38 mm. A zoomed view shows the red area as square pixels, titled 1,262 pixels times 0.139 square millimetres, about 175 square millimetres.
The returned tea light R-d17d264e ("wax melted out"; VisA writes "melded"), with its damage mask in red. Left: the tea light's width, 102.1 pixels, is the scale for 38 mm (an assumption). Right: the damage zoomed 8 times, 1,262 pixels, about 175 mm². VisA candle photo and mask (CC BY 4.0, Amazon).
Text description of the diagram

Left: four white tea lights on a dark cloth, seen from above. The bottom-left tea light has a red patch on its left side, where wax melted out. A green line across this tea light marks its width: 102.1 pixels, which stands for 38 mm. Right: the damaged area zoomed 8 times; the red patch is made of square pixels. The title gives the sum: 1,262 pixels times 0.139 mm² is about 175 mm².

How does the code find the tea light’s width? The tea lights are the brightest things in the photo. The code keeps the pixels brighter than 150 (out of 255), finds the four largest bright regions, and takes each region’s area in pixels. A circle’s area is π × r × r, so its width is 2 × √(area ÷ π). The median of the four widths is the scale. On these photos, it is between 101.9 and 102.4 pixels.

Coverage is the share of a surface that the mask covers. The top of a 38 mm tea light is a circle of about π × 19 × 19 = 1,134 mm². The damage of R-d17d264e covers 175.0 ÷ 1,134 = 15% of it.

Coverage needs no scale at all when both areas are in pixels: damage pixels ÷ candle pixels. A scale is needed only to compare with a size in the real world, such as “smaller than a fingernail”.

The project gets a new command, vision masks. For each damaged photo, it reads the damage mask, finds the tea light’s width, and prints the area in mm². It uses the true masks, drawn by people for VisA, so it measures what a perfect segmentation would give.

  1. Open a terminal in your larkvision folder and activate the virtual environment. If this is a new terminal, set LARKVISION_DATA again. An environment variable lives only in the terminal where you set it.

    Terminal window
    . .venv/bin/activate
    export LARKVISION_DATA="$(cd ../nextia-learning-labs/C42/data && pwd)"
  2. Copy this module’s command, its tests and the README from the snapshot end-of-m05:

    Terminal window
    S=../nextia-learning-labs/C42/snapshots/end-of-m05
    cp "$S/larkvision/commands/masks.py" larkvision/commands/
    cp "$S/tests/test_masks.py" tests/
    cp "$S/README.md" .

    Your project now matches the snapshot end-of-m05. The command needs no new package: SciPy is already installed.

  3. Run the tests. Two new tests check the scale and the area on a tiny fake photo:

    Terminal window
    python -m pytest -q
    ................... [100%]
    19 passed in 1.43s

    You had 17 tests at the end of Module 4. Your time will differ.

  4. Measure the damage on the test photos:

    Terminal window
    vision masks
    18 damaged test photos; scale: a tea light is 38 mm wide
    item pixels share mm2 damage
    R-de7e819e 4 0.0000 0.6 different colour spot
    R-8cb58274 5 0.0001 0.7 damaged corner of packaging; different colour spot; other
    R-15b594e6 7 0.0001 1.0 different colour spot
    R-344c8b55 10 0.0001 1.4 different colour spot
    R-74669eb6 68 0.0007 9.4 damaged corner of packaging
    R-aa4a48cb 80 0.0009 11.1 foreign particals on candle
    R-5cf1ce29 110 0.0012 15.2 foreign particals on candle; wax melded out of the candle
    R-a9048b86 173 0.0019 23.9 damaged corner of packaging; different colour spot
    R-5918b873 208 0.0022 28.8 chunk of wax missing
    R-0d4c03e5 223 0.0024 31.0 wax melded out of the candle
    R-31ebdb39 345 0.0037 48.0 wax melded out of the candle
    R-a796ccf6 351 0.0038 48.5 extra wax in candle
    R-c3b7e506 408 0.0044 56.3 damaged corner of packaging
    R-1cc8cc10 419 0.0045 58.0 weird candle wick
    R-ef0b6f29 492 0.0053 68.0 extra wax in candle
    R-800400d7 499 0.0054 69.0 wax melded out of the candle
    R-33640732 1106 0.0119 153.3 extra wax in candle
    R-d17d264e 1262 0.0136 175.0 wax melded out of the candle
    Median area 29.9 mm2

    share is the share of the whole photo. The damage names are VisA’s own, with their spelling (“melded”, “particals”).

If vision masks says no data folder found, the terminal does not have LARKVISION_DATA: set it as in step 1.

The 18 areas spread over almost three powers of ten:

The damage areas of the 18 damaged test candles, in groups of square millimetres (from vision masks, with a 38 mm tea light). Four damages are under 2 mm²: colour spots of 4 to 10 pixels.
Text description of the chart

Five bars by damage area: under 2 mm², 4 photos; 2 to 20 mm², 3 photos; 20 to 50 mm², 5 photos; 50 to 100 mm², 4 photos; over 100 mm², 2 photos.

The median of the 18 test damages is 29.9 mm², about the size of a fingernail. The four colour spots are under 2 mm². The course also measured the 12 damages of the validation photos: median 16.1 mm². Across all 30 validation and test photos, the median is 22.1 mm², from 0.6 to 175.0.

Common mistake

The scale is an assumption, and it is squared

Symptom: Hamid checks a box of Larkfield’s tea lights with a ruler: they are 40 mm wide, not 38. Every area in the table is wrong.

Cause: the scale grows with the reference’s width, and the area grows with the scale twice. A tea light 40 ÷ 38 = 1.05 times wider makes every area 1.05 × 1.05 = 1.11 times larger. Run it:

Terminal window
vision masks --diameter-mm 40
R-33640732 1106 0.0119 169.9 extra wax in candle
R-d17d264e 1262 0.0136 193.9 wax melded out of the candle
Median area 33.1 mm2

(The last three lines of the output.) The largest damage moves from 175.0 to 193.9 mm², and the median from 29.9 to 33.1.

Fix: measure the reference object, and write its size in the code as a named setting, not as a hidden number. vision masks has --diameter-mm for this reason. With a fixed camera, put a ruler in one photo and measure the scale once. Check it again when the camera moves.

Consequence in AI work: a measurement from a photo has errors from the mask and from the scale. A 5% error in the scale is a 10% error in every area. Say which assumption your numbers depend on.

Two more conditions hide in the scale. The camera must look straight down, so that a millimetre is the same number of pixels everywhere in the photo. And the reference must lie at the same height as the damage. Both are true for these photos; check them for yours.

In real use, nobody draws the mask for Hamid. A model draws it. So we must say how close a model’s mask is to the true mask. There are two common scores.

  • Mask IoU is the IoU of Boxes and IoU, counted in pixels: the pixels in both masks, divided by the pixels in either mask.
  • The Dice coefficient (Dice) is twice the pixels in both masks, divided by the two masks’ sizes added together.

Both are 1 for a perfect mask and 0 when the masks do not touch.

A 6 by 6 grid. 8 cells of the true mask T and 6 cells of the predicted mask P; 5 cells are in both. Next to it: union 9, IoU 5 divided by 9 is 0.56, Dice 2 times 5 divided by 14 is 0.71.
Mask IoU and Dice on a constructed grid. The true mask has 8 pixels, the predicted mask 6, and 5 pixels are in both. IoU = 5 ÷ 9 = 0.56; Dice = 10 ÷ 14 = 0.71.
Text description of the diagram

A grid of 6 by 6 cells. Blue cells marked T belong to the true mask only. Orange cells marked P belong to the predicted mask only. Green cells with a check mark belong to both. The true mask has 8 pixels, the predicted mask 6, and 5 are in both. Next to the grid: the union is 8 plus 6 minus 5, which is 9 pixels. IoU is 5 divided by 9, which is 0.56. Dice is 2 times 5 divided by 8 plus 6, which is 10 divided by 14, or 0.71.

Score Sum Value
Mask IoU both ÷ either = 5 ÷ (8 + 6 − 5) 5 ÷ 9 = 0.56
Dice 2 × both ÷ (size 1 + size 2) = 2 × 5 ÷ (8 + 6) 10 ÷ 14 = 0.71

Dice is always at least as high as IoU for the same two masks: Dice = 2 × IoU ÷ (1 + IoU). Here 2 × 0.556 ÷ 1.556 = 0.71. The two scores rank masks in the same order; they are two scales for the same thing. So never compare a Dice from one report with an IoU from another. In the last lesson, SAM’s box prompts on the pallets scored a mean IoU of 0.916 and a mean Dice of 0.955.

A model’s mask is often a little too big or too small at the edges. What does a one-pixel mistake do to the score? We take each true damage mask, grow it by one pixel on every side, and score the grown mask against the truth.

  • Damage under 20 pixels
  • Larger damage
Each of the 18 damaged test candles: the true mask grown by one pixel all round, scored against the true mask. The same small mistake costs a large damage almost nothing (0.91) and a 4-pixel colour spot two thirds of its score (0.33).
Text description of the chart

A scatter plot with the size of the true mask from 0 to 1,300 pixels across, and the IoU from 0 to 1 up. The four colour spots of 4 to 10 pixels sit at the far left, with IoU between 0.29 and 0.41. Damages of 68 to 110 pixels score 0.41 to 0.60. Damages of 173 to 499 pixels score 0.72 to 0.87. The two largest, 1,106 and 1,262 pixels, score 0.90 and 0.91.

Why? A one-pixel border is small next to a large area, but large next to a tiny one. The 4-pixel colour spot R-de7e819e grows to 12 pixels: 8 of the 12 are wrong. Its IoU is 4 ÷ 12 = 0.33, although the mask is in the right place.

So report a mask score with the size of the object. For tiny objects, also ask a simpler question: does the predicted mask touch the true one? The scores of large objects tell you about the outline. The scores of tiny objects mostly tell you about one pixel.

The same mistake also moves the area. The 4-pixel spot measures 0.6 mm²; its grown mask measures 1.7 mm², almost three times as much. On small damage, an area from a model’s mask is a rough number.

Use a measured area when the decision depends on size. Examples: “send damage over 50 mm² to the outlet”, or “a scratch longer than 10 mm needs a person”. Hamid’s team then argues about one number, not about a photo.

Do not use it for a damage of a few pixels. A one-pixel mistake in the mask can triple its area. Do not use it when the camera moves from photo to photo, unless every photo has a reference object.

What it costs: almost nothing once you have the mask. Counting pixels and finding four bright circles takes milliseconds. The cost is in the mask itself: a person’s drawing, or a model such as SAM that needs a prompt for each damage. In the next module, the model must first find the damage. Its heat maps point to the right area, but their outlines are rough. Compared with the true masks, they reach a mean Dice of only 0.175.

Guided practice: from pixels to square millimetres

Section titled “Guided practice: from pixels to square millimetres”
  1. Open the notebook from the box Do this lesson in a notebook at the top of this page.
  2. Run the setup cells. They download the returns photos and their damage masks.
  3. In section 1, run the cell. The damage of R-d17d264e covers 1262 pixels.
  4. In section 2, run the cell. You see a tea light is 102.1 pixels wide and 175.0 mm2.
  5. In section 3, run the cell. It prints the same table as vision masks, with a median of 29.9 mm².
  6. In section 4, run the first cell: IoU 0.556 | Dice 0.714. Predict which damage loses more from a one-pixel mistake. Then run the next cell.
  7. Go to section 5 for your task.

Section 5 of the notebook has one task. The table scores holds the IoU and Dice of each grown mask. Put the median Dice of the 18 grown masks in median_dice. The check cell prints Check passed when your answer is right.

Before you run it, predict: will the median Dice be higher or lower than the median IoU?

Show the answer
median_dice = float(scores["dice"].median())

The answer is 0.857. It is higher than the median IoU (0.75), as Dice always is. With Dice = 2 × IoU ÷ (1 + IoU), an IoU of 0.75 gives about 0.86. Neither number says “a good mask” by itself: every one of these masks is only one pixel off.

Check your understanding

A damage mask covers 500 pixels. A tea light of 38 mm is 95 pixels wide in the photo. What is the damage's area in mm²? If the tea light is really 36 mm wide, does the area go up or down, and by how much?

Show the answer

The scale is 38 ÷ 95 = 0.4 mm a pixel. One square pixel is 0.4 × 0.4 = 0.16 mm². The area is 500 × 0.16 = 80 mm². With 36 mm, the scale is 36 ÷ 95 = 0.379 mm a pixel. One square pixel is then 0.1436 mm², and the area is about 71.8 mm². The area goes down by a factor of (36 ÷ 38)² = 0.90, about 10%. The scale counts twice in an area.

Check your understanding

A true mask has 10 pixels and a predicted mask has 15 pixels; 10 pixels are in both. Compute the mask IoU and the Dice. Then say why both scores are lower than you might expect for a mask that covers the whole damage.

Show the answer

Both: 10 pixels. Either: 10 + 15 − 10 = 15 pixels. IoU = 10 ÷ 15 = 0.67. Dice = 2 × 10 ÷ (10 + 15) = 20 ÷ 25 = 0.80. The predicted mask covers every damaged pixel, but it adds 5 pixels of good wax. On a damage of 10 pixels, 5 extra pixels are half again its size, so both scores fall a lot. On a large damage, the same 5 pixels would hardly count.

Check your understanding

Hamid wants a rule: damage of more than 50 mm² goes to the outlet; smaller damage is sold at a discount. Using the 18 test photos, how many go to the outlet? Name two things that could make the rule's result wrong in production.

Show the answer

Six photos have more than 50 mm²: 56.3, 58.0, 68.0, 69.0, 153.3 and 175.0. Two things can make the result wrong in production. First, a model draws the mask, not a person. Small mistakes at the edges change the area, most of all on small damage. Second, the scale is an assumption. If the tea lights are 40 mm wide and not 38, every area is 11% larger, and damages near 50 mm² cross the line. A damage of 48.5 mm² becomes about 53.7 mm². This is a business rule for Hamid to set; the numbers only show what it would do.

Recap

  • A mask gives an area in pixels at once. A reference object of known size turns pixels into millimetres; an area needs the scale twice.
  • With a 38 mm tea light (an assumption), the median damage of the 18 damaged test candles is 29.9 mm². The areas go from 0.6 to 175.0. With 40 mm, every area is 11% larger.
  • Coverage is the share of a surface under the mask: 15% of the top of R-d17d264e. It needs no scale.
  • Mask IoU (both ÷ either) and Dice (2 × both ÷ sizes added) score a mask against the truth; Dice = 2 × IoU ÷ (1 + IoU). Never compare one with the other.
  • A one-pixel mistake costs a large mask almost nothing (IoU 0.91) and a 4-pixel spot two thirds of its score (0.33). Report mask scores with the object’s size.
  • Your project has vision masks and 19 tests, as the snapshot end-of-m05.

You can now measure damage that someone has outlined. But in the warehouse, nobody outlines the damage first: a model must find it on its own. Next, we compare two ways to find damage on Larkfield’s returns: learn from labelled damage, or learn only what “good” looks like.

Next: Two ways to find damage