Measure from masks
- Lesson 19
- 35 min
- Module 5: Segment
Where we are
We can now get masks with a click. A mask is also a measurement. In this lesson, we measure the area of damage on a candle from its mask, compare masks with IoU and Dice, and use a reference for scale.
Lesson 19 of 30 · Vision AI: See, Count and InspectMeasure from masks
1. Seeing is a business problem
2. Baselines and pretrained classifiers
3. Embeddings, CLIP and few labels
4. Detect and count
6. Inspect for defects
7. Match, search and describe
8. Run vision in production
What you will be able to do
Turn a mask into a measurement that Hamid can use: an area in square millimetres, and the share of a candle that is damaged. Use an object of known size for scale, and see how much the answer depends on it. Then compare two masks with IoU and Dice, and see why a one-pixel mistake ruins the score of a tiny mask.
NotebookDo this lesson in a notebook
The notebook measures the damage on the 18 damaged test candles from their masks. It counts pixels, finds the scale from the tea light’s width, and gives square millimetres. Then it computes mask IoU and Dice on a small grid, and on masks that are one pixel too big. Its setup cells download the returns photos and their masks (about 15 MB, no model and no key). About 35 minutes; the cells run in a few seconds.
Colab and Kaggle open your own copy. They need a free Google or Kaggle account. Stuck? See the solution notebook.
Before you startWhat to know and to have ready
What you need to know
- Masks, from Semantic and instance segmentation.
- IoU, from Boxes and IoU.
What you need on your computer
- For the notebook: nothing. It runs in Colab or Kaggle.
- For the project steps: your
larkvisionproject at the end of Module 4, with its virtual environment, and the practice files in../nextia-learning-labs. If your project is broken, copy the snapshotend-of-m04again, as in Meet the Larkfield images.
Tested with Python 3.12 on macOS. The Windows commands are written out but were not run.
A coin next to the scratch
Section titled “A coin next to the scratch”When you send a photo of a scratch on your car to an insurer, they often ask you to put a coin next to it. The coin is not for them to keep. Everybody knows how big a coin is. So the photo now says how big the scratch is, and not only how many pixels it covers.
Larkfield’s returns photos have a “coin” of their own: every photo shows four tea lights.
The problem: a pixel is not a size
Section titled “The problem: a pixel is not a size”In the last two lessons, we got masks: from people who drew them, or from SAM. A mask gives an area at once: count the “yes” pixels. The damage on the returned tea light R-d17d264e covers 1,262 pixels.
But Hamid cannot use “1,262 pixels”. A pixel is not a size:
- A photo taken from closer makes everything bigger. The same damage then covers more pixels.
- A camera with more pixels gives more pixels for the same damage.
- A resized photo changes every count. The VisA photos are 1,284 pixels wide at the source; Larkfield’s copies are 320 wide.
Hamid’s team thinks in millimetres: “a dent smaller than a fingernail is fine”. So we need a scale.
The idea: a reference object for scale
Section titled “The idea: a reference object for scale”A reference object is something in the photo whose real size you know. You measure it in pixels, and you get the scale: millimetres per pixel.
An area needs the scale twice: once for the width, once for the height. If 1 pixel is 0.372 mm wide, one square pixel is 0.372 × 0.372 = 0.139 square millimetres (mm²).
A tiny example
Section titled “A tiny example”Every returns photo shows four tea lights. A standard tea light is about 38 mm wide. That is an assumption: tea lights vary, so check the ones in your own photos.
| Step | Value |
|---|---|
| The tea light’s width in the photo | 102.1 pixels |
| Scale: 38 mm ÷ 102.1 pixels | 0.372 mm a pixel |
| One square pixel: 0.372 × 0.372 | 0.139 mm² |
| The damage: 1,262 pixels × 0.1386 mm² | 175.0 mm² |

Text description of the diagram
Left: four white tea lights on a dark cloth, seen from above. The bottom-left tea light has a red patch on its left side, where wax melted out. A green line across this tea light marks its width: 102.1 pixels, which stands for 38 mm. Right: the damaged area zoomed 8 times; the red patch is made of square pixels. The title gives the sum: 1,262 pixels times 0.139 mm² is about 175 mm².
How does the code find the tea light’s width? The tea lights are the brightest things in the photo. The code keeps the pixels brighter than 150 (out of 255), finds the four largest bright regions, and takes each region’s area in pixels. A circle’s area is π × r × r, so its width is 2 × √(area ÷ π). The median of the four widths is the scale. On these photos, it is between 101.9 and 102.4 pixels.
From area to coverage
Section titled “From area to coverage”Coverage is the share of a surface that the mask covers. The top of a 38 mm tea light is a circle of about π × 19 × 19 = 1,134 mm². The damage of R-d17d264e covers 175.0 ÷ 1,134 = 15% of it.
Coverage needs no scale at all when both areas are in pixels: damage pixels ÷ candle pixels. A scale is needed only to compare with a size in the real world, such as “smaller than a fingernail”.
Larkfield: the damage area in the project
Section titled “Larkfield: the damage area in the project”The project gets a new command, vision masks. For each damaged photo, it reads the damage mask, finds the tea light’s width, and prints the area in mm². It uses the true masks, drawn by people for VisA, so it measures what a perfect segmentation would give.
-
Open a terminal in your
larkvisionfolder and activate the virtual environment. If this is a new terminal, setLARKVISION_DATAagain. An environment variable lives only in the terminal where you set it.Terminal window . .venv/bin/activateexport LARKVISION_DATA="$(cd ../nextia-learning-labs/C42/data && pwd)"Terminal window .venv\Scripts\Activate.ps1$env:LARKVISION_DATA = (Resolve-Path ..\nextia-learning-labs\C42\data).Path -
Copy this module’s command, its tests and the README from the snapshot
end-of-m05:Terminal window S=../nextia-learning-labs/C42/snapshots/end-of-m05cp "$S/larkvision/commands/masks.py" larkvision/commands/cp "$S/tests/test_masks.py" tests/cp "$S/README.md" .Terminal window $S = "..\nextia-learning-labs\C42\snapshots\end-of-m05"Copy-Item "$S\larkvision\commands\masks.py" larkvision\commands\Copy-Item "$S\tests\test_masks.py" tests\Copy-Item "$S\README.md" .Your project now matches the snapshot
end-of-m05. The command needs no new package: SciPy is already installed. -
Run the tests. Two new tests check the scale and the area on a tiny fake photo:
Terminal window python -m pytest -q................... [100%]19 passed in 1.43sYou had 17 tests at the end of Module 4. Your time will differ.
-
Measure the damage on the test photos:
Terminal window vision masks18 damaged test photos; scale: a tea light is 38 mm wideitem pixels share mm2 damageR-de7e819e 4 0.0000 0.6 different colour spotR-8cb58274 5 0.0001 0.7 damaged corner of packaging; different colour spot; otherR-15b594e6 7 0.0001 1.0 different colour spotR-344c8b55 10 0.0001 1.4 different colour spotR-74669eb6 68 0.0007 9.4 damaged corner of packagingR-aa4a48cb 80 0.0009 11.1 foreign particals on candleR-5cf1ce29 110 0.0012 15.2 foreign particals on candle; wax melded out of the candleR-a9048b86 173 0.0019 23.9 damaged corner of packaging; different colour spotR-5918b873 208 0.0022 28.8 chunk of wax missingR-0d4c03e5 223 0.0024 31.0 wax melded out of the candleR-31ebdb39 345 0.0037 48.0 wax melded out of the candleR-a796ccf6 351 0.0038 48.5 extra wax in candleR-c3b7e506 408 0.0044 56.3 damaged corner of packagingR-1cc8cc10 419 0.0045 58.0 weird candle wickR-ef0b6f29 492 0.0053 68.0 extra wax in candleR-800400d7 499 0.0054 69.0 wax melded out of the candleR-33640732 1106 0.0119 153.3 extra wax in candleR-d17d264e 1262 0.0136 175.0 wax melded out of the candleMedian area 29.9 mm2shareis the share of the whole photo. The damage names are VisA’s own, with their spelling (“melded”, “particals”).
If vision masks says no data folder found, the terminal does not have LARKVISION_DATA: set it as in step 1.
The 18 areas spread over almost three powers of ten:
Text description of the chart
Five bars by damage area: under 2 mm², 4 photos; 2 to 20 mm², 3 photos; 20 to 50 mm², 5 photos; 50 to 100 mm², 4 photos; over 100 mm², 2 photos.
The median of the 18 test damages is 29.9 mm², about the size of a fingernail. The four colour spots are under 2 mm². The course also measured the 12 damages of the validation photos: median 16.1 mm². Across all 30 validation and test photos, the median is 22.1 mm², from 0.6 to 175.0.
Common mistake
The scale is an assumption, and it is squared
Symptom: Hamid checks a box of Larkfield’s tea lights with a ruler: they are 40 mm wide, not 38. Every area in the table is wrong.
Cause: the scale grows with the reference’s width, and the area grows with the scale twice. A tea light 40 ÷ 38 = 1.05 times wider makes every area 1.05 × 1.05 = 1.11 times larger. Run it:
vision masks --diameter-mm 40 R-33640732 1106 0.0119 169.9 extra wax in candle R-d17d264e 1262 0.0136 193.9 wax melded out of the candleMedian area 33.1 mm2(The last three lines of the output.) The largest damage moves from 175.0 to 193.9 mm², and the median from 29.9 to 33.1.
Fix: measure the reference object, and write its size in the code as a named setting, not as a hidden number. vision masks has --diameter-mm for this reason. With a fixed camera, put a ruler in one photo and measure the scale once. Check it again when the camera moves.
Consequence in AI work: a measurement from a photo has errors from the mask and from the scale. A 5% error in the scale is a 10% error in every area. Say which assumption your numbers depend on.
Two more conditions hide in the scale. The camera must look straight down, so that a millimetre is the same number of pixels everywhere in the photo. And the reference must lie at the same height as the damage. Both are true for these photos; check them for yours.
Compare two masks: IoU and Dice
Section titled “Compare two masks: IoU and Dice”In real use, nobody draws the mask for Hamid. A model draws it. So we must say how close a model’s mask is to the true mask. There are two common scores.
- Mask IoU is the IoU of Boxes and IoU, counted in pixels: the pixels in both masks, divided by the pixels in either mask.
- The Dice coefficient (Dice) is twice the pixels in both masks, divided by the two masks’ sizes added together.
Both are 1 for a perfect mask and 0 when the masks do not touch.
A tiny example
Section titled “A tiny example”Text description of the diagram
A grid of 6 by 6 cells. Blue cells marked T belong to the true mask only. Orange cells marked P belong to the predicted mask only. Green cells with a check mark belong to both. The true mask has 8 pixels, the predicted mask 6, and 5 are in both. Next to the grid: the union is 8 plus 6 minus 5, which is 9 pixels. IoU is 5 divided by 9, which is 0.56. Dice is 2 times 5 divided by 8 plus 6, which is 10 divided by 14, or 0.71.
| Score | Sum | Value |
|---|---|---|
| Mask IoU | both ÷ either = 5 ÷ (8 + 6 − 5) | 5 ÷ 9 = 0.56 |
| Dice | 2 × both ÷ (size 1 + size 2) = 2 × 5 ÷ (8 + 6) | 10 ÷ 14 = 0.71 |
Dice is always at least as high as IoU for the same two masks: Dice = 2 × IoU ÷ (1 + IoU). Here 2 × 0.556 ÷ 1.556 = 0.71. The two scores rank masks in the same order; they are two scales for the same thing. So never compare a Dice from one report with an IoU from another. In the last lesson, SAM’s box prompts on the pallets scored a mean IoU of 0.916 and a mean Dice of 0.955.
One pixel too many, all round
Section titled “One pixel too many, all round”A model’s mask is often a little too big or too small at the edges. What does a one-pixel mistake do to the score? We take each true damage mask, grow it by one pixel on every side, and score the grown mask against the truth.
- Damage under 20 pixels
- Larger damage
Text description of the chart
A scatter plot with the size of the true mask from 0 to 1,300 pixels across, and the IoU from 0 to 1 up. The four colour spots of 4 to 10 pixels sit at the far left, with IoU between 0.29 and 0.41. Damages of 68 to 110 pixels score 0.41 to 0.60. Damages of 173 to 499 pixels score 0.72 to 0.87. The two largest, 1,106 and 1,262 pixels, score 0.90 and 0.91.
Why? A one-pixel border is small next to a large area, but large next to a tiny one. The 4-pixel colour spot R-de7e819e grows to 12 pixels: 8 of the 12 are wrong. Its IoU is 4 ÷ 12 = 0.33, although the mask is in the right place.
So report a mask score with the size of the object. For tiny objects, also ask a simpler question: does the predicted mask touch the true one? The scores of large objects tell you about the outline. The scores of tiny objects mostly tell you about one pixel.
The same mistake also moves the area. The 4-pixel spot measures 0.6 mm²; its grown mask measures 1.7 mm², almost three times as much. On small damage, an area from a model’s mask is a rough number.
When to measure from masks
Section titled “When to measure from masks”Use a measured area when the decision depends on size. Examples: “send damage over 50 mm² to the outlet”, or “a scratch longer than 10 mm needs a person”. Hamid’s team then argues about one number, not about a photo.
Do not use it for a damage of a few pixels. A one-pixel mistake in the mask can triple its area. Do not use it when the camera moves from photo to photo, unless every photo has a reference object.
What it costs: almost nothing once you have the mask. Counting pixels and finding four bright circles takes milliseconds. The cost is in the mask itself: a person’s drawing, or a model such as SAM that needs a prompt for each damage. In the next module, the model must first find the damage. Its heat maps point to the right area, but their outlines are rough. Compared with the true masks, they reach a mean Dice of only 0.175.
Guided practice: from pixels to square millimetres
Section titled “Guided practice: from pixels to square millimetres”- Open the notebook from the box Do this lesson in a notebook at the top of this page.
- Run the setup cells. They download the returns photos and their damage masks.
- In section 1, run the cell. The damage of R-d17d264e covers 1262 pixels.
- In section 2, run the cell. You see
a tea light is 102.1 pixels wideand175.0 mm2. - In section 3, run the cell. It prints the same table as
vision masks, with a median of 29.9 mm². - In section 4, run the first cell:
IoU 0.556 | Dice 0.714. Predict which damage loses more from a one-pixel mistake. Then run the next cell. - Go to section 5 for your task.
Your turn: Dice of the grown masks
Section titled “Your turn: Dice of the grown masks”Section 5 of the notebook has one task. The table scores holds the IoU and Dice of each grown mask. Put the median Dice of the 18 grown masks in median_dice. The check cell prints Check passed when your answer is right.
Before you run it, predict: will the median Dice be higher or lower than the median IoU?
Show the answer
median_dice = float(scores["dice"].median())The answer is 0.857. It is higher than the median IoU (0.75), as Dice always is. With Dice = 2 × IoU ÷ (1 + IoU), an IoU of 0.75 gives about 0.86. Neither number says “a good mask” by itself: every one of these masks is only one pixel off.
Knowledge check
Section titled “Knowledge check”Check your understanding
A damage mask covers 500 pixels. A tea light of 38 mm is 95 pixels wide in the photo. What is the damage's area in mm²? If the tea light is really 36 mm wide, does the area go up or down, and by how much?
Show the answer
The scale is 38 ÷ 95 = 0.4 mm a pixel. One square pixel is 0.4 × 0.4 = 0.16 mm². The area is 500 × 0.16 = 80 mm². With 36 mm, the scale is 36 ÷ 95 = 0.379 mm a pixel. One square pixel is then 0.1436 mm², and the area is about 71.8 mm². The area goes down by a factor of (36 ÷ 38)² = 0.90, about 10%. The scale counts twice in an area.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
A true mask has 10 pixels and a predicted mask has 15 pixels; 10 pixels are in both. Compute the mask IoU and the Dice. Then say why both scores are lower than you might expect for a mask that covers the whole damage.
Show the answer
Both: 10 pixels. Either: 10 + 15 − 10 = 15 pixels. IoU = 10 ÷ 15 = 0.67. Dice = 2 × 10 ÷ (10 + 15) = 20 ÷ 25 = 0.80. The predicted mask covers every damaged pixel, but it adds 5 pixels of good wax. On a damage of 10 pixels, 5 extra pixels are half again its size, so both scores fall a lot. On a large damage, the same 5 pixels would hardly count.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
Hamid wants a rule: damage of more than 50 mm² goes to the outlet; smaller damage is sold at a discount. Using the 18 test photos, how many go to the outlet? Name two things that could make the rule's result wrong in production.
Show the answer
Six photos have more than 50 mm²: 56.3, 58.0, 68.0, 69.0, 153.3 and 175.0. Two things can make the result wrong in production. First, a model draws the mask, not a person. Small mistakes at the edges change the area, most of all on small damage. Second, the scale is an assumption. If the tea lights are 40 mm wide and not 38, every area is 11% larger, and damages near 50 mm² cross the line. A damage of 48.5 mm² becomes about 53.7 mm². This is a business rule for Hamid to set; the numbers only show what it would do.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Recap
- A mask gives an area in pixels at once. A reference object of known size turns pixels into millimetres; an area needs the scale twice.
- With a 38 mm tea light (an assumption), the median damage of the 18 damaged test candles is 29.9 mm². The areas go from 0.6 to 175.0. With 40 mm, every area is 11% larger.
- Coverage is the share of a surface under the mask: 15% of the top of R-d17d264e. It needs no scale.
- Mask IoU (both ÷ either) and Dice (2 × both ÷ sizes added) score a mask against the truth; Dice = 2 × IoU ÷ (1 + IoU). Never compare one with the other.
- A one-pixel mistake costs a large mask almost nothing (IoU 0.91) and a 4-pixel spot two thirds of its score (0.33). Report mask scores with the object’s size.
- Your project has
vision masksand 19 tests, as the snapshotend-of-m05.
You can now measure damage that someone has outlined. But in the warehouse, nobody outlines the damage first: a model must find it on its own. Next, we compare two ways to find damage on Larkfield’s returns: learn from labelled damage, or learn only what “good” looks like.
Next: Two ways to find damage