Images are numbers
- Lesson 3
- 35 min
- Module 1: Seeing is a business problem
Where we are
We know the five kinds of answer a vision system can give. We have not yet looked at what a computer gets when it opens a photo. In this lesson, we see that an image is a grid of numbers, and why light and image size change those numbers. We start with a mosaic floor, seen from very close.
Lesson 3 of 30 · Vision AI: See, Count and InspectImages are numbers
1. Seeing is a business problem
2. Baselines and pretrained classifiers
3. Embeddings, CLIP and few labels
4. Detect and count
6. Inspect for defects
7. Match, search and describe
8. Run vision in production
What you will be able to do
See what a computer really gets from a photo: a grid of numbers. Then see why a change of light or of size changes those numbers, so a method that worked yesterday can fail today.
Before you startWhat to know and to have ready
What you need to know
- The five kinds of answer, from Five kinds of answer.
- What a mean (an average) is. If you need a refresher, read Describe a dataset in Essential Mathematics and Statistics for AI.
What you need on your computer
Nothing. This lesson has no code: every number on this page was read from the course’s photos. You read the same numbers yourself in the setup notebook of Meet the Larkfield images.
A mosaic seen from close
Section titled “A mosaic seen from close”Some old buildings have mosaic floors. From the door, you see a picture: a bird, a tree, a boat. When you kneel down, the picture disappears. You see only small square tiles, and each tile has one colour. The picture is only in your head. It appears when your eyes put thousands of tiles together.
A computer never stands at the door. It gets only the tiles, and for each tile, numbers that say its colour. It has no idea of a bird or a boat. Every vision method in this course starts from these numbers and must build the “picture” from them. So it pays to know exactly what the numbers are, and what changes them.
A photo is a grid of numbers
Section titled “A photo is a grid of numbers”A digital photo is a grid of small squares called pixels (from “picture element”). Each pixel has one colour. The catalogue photos of the course are 256 pixels wide and 256 pixels high, so each one has 256 × 256 = 65,536 pixels.

Text description of the diagram
On the left is a catalogue photo of a ceramic bowl, purple and red, on a white table. A small red square marks a part of the bowl’s rim, 6 pixels wide and 6 pixels high. On the right, the inside of that square is enlarged into a grid of 36 squares. Each square is filled with the colour of its pixel and shows three numbers. The top three rows are the dark brown rim, for example red 104, green 79 and blue 49. The fourth row is lighter, for example red 178, green 150 and blue 128. The bottom two rows are a very light part of the photo next to the rim, for example red 255, green 254 and blue 239.
Channels: red, green and blue
Section titled “Channels: red, green and blue”Each pixel is three numbers, one for each channel: red (R), green (G) and blue (B). A screen makes every colour by mixing red, green and blue light. Each number goes from 0 (none of that light) to 255 (all of it). So (0, 0, 0) is black, (255, 255, 255) is white, and (255, 0, 0) is pure red.
Read the grid above with this in mind. The rim pixel (104, 79, 49) has more red than blue, so it is brown, and the numbers are low, so it is dark. The light pixel (255, 254, 239) has all three channels near 255, so it is almost white. The blue channel is a little lower, so the white is a little warm.
So a catalogue photo is 256 × 256 × 3 = 196,608 numbers. In Python, it is an array with the shape (256, 256, 3): rows, columns and channels. You see this shape in the setup notebook.
Resolution: how many pixels
Section titled “Resolution: how many pixels”The resolution of a photo is its number of pixels, written as width × height. The course’s photos have several resolutions:
| Photos | Resolution | Numbers in one photo |
|---|---|---|
| Catalogue (cut from larger photos) | 256 × 256 | 196,608 |
| Returns (the course’s copies) | 320 × 291 | 279,360 |
| Returns (the original photos) | 1,284 × 1,168 | 4,499,136 |
| Pallets (drawn by our code) | 512 × 384 | 589,824 |
More pixels show smaller details. They also cost more: every method has to read and compute more numbers. That is why many pretrained models shrink every photo to 224 × 224 first. You see below what that can cost.
The same mug under another light
Section titled “The same mug under another light”Light changes every number in a photo. A photo studio can change its lamps, or a warehouse can get a new camera. People hardly notice: we still see a white mug. The numbers change a lot.
Larkfield’s catalogue team moved to a new studio with warmer, brighter light. To test this, the course made a second copy of the 160 test photos with the new light. Our code made it with one fixed rule, so it is synthetic, and it is a problem we planted. The rule multiplies red by 1.16, green by 1.04 and blue by 0.80, then makes the photo 6% brighter and adds 6 to every number.

Text description of the diagram
The figure shows the same photo twice: a white mug full of black coffee, with a second mug behind it. A blue circle marks the same point on the side of the mug in both photos. On the left, under the studio light, the mug is white with a cool, slightly blue tint. On the right, under the new studio light, the whole photo is warmer and more yellow, and the mug looks cream.
Here is the pixel in the blue circle, read from the two files:
| Light | R | G | B | Hue | Saturation | Value |
|---|---|---|---|---|---|---|
| Studio | 187 | 214 | 225 | 197° (blue) | 17% | 88% |
| New studio | 235 | 241 | 197 | 68° (yellow-green) | 18% | 95% |
Check the rule on the red channel: 187 × 1.16 × 1.06 + 6 = 235.9. The file stores 235, because a pixel holds only whole numbers. Blue moved the other way, from 225 to 197. Your eyes still say “a white mug”. The numbers say “a blue pixel became a yellow one”.
The whole photo moves too. The chart shows the mean of each channel over all 65,536 pixels:
- Studio light
- New studio light (made by our code)
Text description of the chart
A column chart with six columns, on an axis from 0 to 255. Under the studio light, the mean red is 145.9, the mean green 155.1 and the mean blue 152.4. Under the new studio light, the mean red is 184.8, the mean green 176.5 and the mean blue 134.6. Red rises the most, green rises a little, and blue falls.
Why this breaks some methods and not others
Section titled “Why this breaks some methods and not others”A method that decides by colour reads exactly these numbers. In the course’s comparison table, a colour histogram method sorts catalogue photos by their mix of colours. (A histogram counts how many pixels fall in each range of colour.) It is right on 26.3% of the test photos. Under the new studio light, it is right on only 22.5%. A method based on shapes and edges, called HOG, ignores colour: it stays at 35.6% under both lights. Module 2 builds both methods.
Strong pretrained models change less. A classifier built on CLIP, a large pretrained model that Module 3 explains, stays near 90% under both lights. But the new light still moves its numbers, and Module 8 shows how to find such a change before it costs money.
Colour spaces: another way to write a colour
Section titled “Colour spaces: another way to write a colour”RGB is not the only way to write a colour. A colour space is a system of numbers for colours. A common one is HSV:
- Hue is the kind of colour, as an angle on a colour wheel: 0° is red, 120° is green, 240° is blue.
- Saturation is how strong the colour is: 0% is grey, 100% is pure colour.
- Value is how bright it is: 0% is black.
The table above gives the mug pixel in HSV too. Its hue jumped from 197° to 68°, but its saturation stayed low, about 17%. Low saturation means “almost white”, whatever the hue. So HSV makes one thing easier. A rule like “white = low saturation and high value” survives this light change better than a rule on R, G and B. HSV does not remove the light; it only separates “which colour” from “how strong” and “how bright”.
Shrinking can erase a scratch
Section titled “Shrinking can erase a scratch”Remember the 224 × 224 that many pretrained models use. To get there, a program resizes the photo: it computes each new pixel as a mix of several old pixels. Large shapes survive. A small spot gets mixed with the wax around it.
The returns photos are taken at 1,284 × 1,168 pixels. One training photo has a spot of another colour on a tea light. At full size, the spot covers 70 pixels. The figure shows the same small square of the photo at full size and after shrinking the whole photo to 224 pixels wide:

Text description of the diagram
The figure has two enlarged squares of the same tea light. The left square is 96 × 96 pixels at full size: pale orange wax with a small, sharp, dark red spot in the middle. The right square shows the same area after the whole photo was shrunk to 224 pixels wide. That area is now only 16 × 16 pixels, enlarged into large blocks. The spot has become two or three blocks of a paler orange-red, which fade into the wax around them.
A tiny calculation: where the spot goes
Section titled “A tiny calculation: where the spot goes”Shrinking a photo from 1,284 to 224 pixels wide divides each side by 1,284 ÷ 224 = 5.7. So it divides the area by 5.7 × 5.7 = 32.9. A spot of 70 pixels becomes 70 ÷ 32.9 = 2.1 pixels. The colour also fades, because each new pixel mixes the spot with the wax:
| Size | Pixels of the spot | Darkest pixel of the spot (R, G, B) | The wax around it (R, G, B) |
|---|---|---|---|
| 1,284 wide (full size) | 70 | 223, 100, 58 | 255, 207, 151 |
| 320 wide (the course’s copies) | 4.3 | ||
| 224 wide | 2.1 | 248, 123, 86 | 255, 207, 151 |
At full size, the spot’s green is 107 lower than the wax. At 224 wide, the difference falls to 84, and the spot is two pixels among 45,696. This spot is still there. Some are not. The smallest damage in the course’s returns is a spot of 30 pixels at full size. Its photo is in the hidden test, which the course keeps. At 224 wide, it covers 0.91 pixels: less than one pixel. It is mixed into one pixel with the wax, and no model can find it there. The median damage of the course is larger: 63.9 pixels at 224 wide. (Half of the damages are smaller than the median, and half are larger.)
When to shrink, and when not
Section titled “When to shrink, and when not”Shrinking is not wrong. A smaller photo is faster: 224 × 204 pixels are 33 times fewer numbers than 1,284 × 1,168. For the catalogue, where a mug is a mug at any size, shrinking costs almost nothing. For damage, the size of the smallest thing that matters decides. Before you shrink, ask: “How many pixels will the smallest damage have?” If the answer is below a few pixels, keep more resolution. You can also cut the photo into smaller tiles, or look at small parts of it. Module 6 uses a method that compares small parts of each photo.
File formats: JPEG and PNG
Section titled “File formats: JPEG and PNG”A photo file does not store the grid as it is. It compresses it: it writes the numbers in fewer bytes. The two common formats do this in two different ways.
- JPEG (
.jpg) throws away small details that eyes hardly see. The files are small, but the numbers change a little each time you save. The catalogue photo of the bowl is 8,056 bytes as a JPEG. The raw grid is 196,608 numbers, one byte each. - PNG (
.png) keeps every number exactly. The same bowl photo is 42,450 bytes as a PNG: five times larger.
How much does JPEG change? The course opened the bowl photo and saved it again as a JPEG, at quality 75. Then 63.9% of its pixels had at least one changed number. You would not see the difference. A program that compares numbers does.
So the course stores photos as JPEG, because small changes do not matter for a photo. It stores masks as PNG, because a mask must say exactly yes or no for each pixel.
Common mistake
A mask saved as JPEG
Symptom: you save the mask of the melted candle from Five kinds of answer as a JPEG, to save space. Later, you count its damaged pixels with “pixel equals 255”, and you get 1,111. The PNG mask said 1,456.
Cause: a mask has only two values, 0 (no) and 255 (yes). JPEG smooths the sharp edge between them. After one save, 788 pixels have grey values between 0 and 255, and the mask has 26 different values instead of 2. The test “equals 255” misses the grey pixels along the edge.
Fix: store masks and other exact labels as PNG. If you get a mask that was saved as JPEG, turn it back into yes and no with a middle line. Use “yes when the pixel is 128 or more”. For this mask, that gives back exactly 1,456 pixels.
Consequence in AI work: a wrong file format changes your labels without an error. Here the damage area is almost a quarter too small, and every score computed from it is wrong. Nobody sees why.
Guided practice: read and change pixels by hand
Section titled “Guided practice: read and change pixels by hand”You need a pen, or a calculator.
- Take the light pixel
(255, 254, 239)from the grid. Name the channel with the lowest number. - Count the numbers in one returns photo of the course: 320 × 291 pixels, with 3 channels.
- Apply the new studio rule to the pixel
(100, 120, 140). Multiply R by 1.16, G by 1.04 and B by 0.80. - Multiply each result by 1.06, and add 6.
- Drop the decimals, as a file does.
- Compare the new pixel with the old one. Write which channel rose and which fell.
Show the answers
- Blue (239). The pixel is almost white, and a little warm.
- 320 × 291 × 3 = 279,360 numbers.
- to 5. Red: 100 × 1.16 × 1.06 + 6 = 128.96, so 128. Green: 120 × 1.04 × 1.06 + 6 = 138.29, so 138. Blue: 140 × 0.80 × 1.06 + 6 = 124.72, so 124.
- The new pixel is
(128, 138, 124). Red rose by 28 and green by 18. Blue fell by 16. The pixel was blue-grey, and it became a little green-grey.
Your turn: a smaller spot
Section titled “Your turn: a smaller spot”Change one thing. Another damage covers 140 pixels at full size, 1,284 wide. Before you calculate, predict: at 224 wide, will it be more or less than 5 pixels? Then calculate it. Use the area factor of 32.9 from above.
Show the answer
140 ÷ 32.9 = 4.3 pixels. Less than 5. A spot twice as large as the one in the figure still covers only about four pixels at 224 wide. That is why the returns need care with the size: a few pixels of a slightly different colour are easy to lose.
Knowledge check
Section titled “Knowledge check”Check your understanding
A catalogue photo has the shape (256, 256, 3). What do the three numbers mean, and how many numbers does the photo hold?
Show the answer
The photo has 256 rows and 256 columns of pixels, and 3 channels for each pixel: red, green and blue. Each channel is a number from 0 to 255. So the photo holds 256 × 256 × 3 = 196,608 numbers. The computer gets only these numbers. It must build “a mug” or “a bowl” from them.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
Predict and explain: under the new studio light, a colour histogram method falls from 26.3% to 22.5%, but the HOG method stays at 35.6%. Why does one change and not the other?
Show the answer
The new light changes the colour numbers of every pixel: red rises and blue falls. A colour histogram counts pixels by their colour, so the same mug now gives another histogram, and the method makes more mistakes. HOG looks only at shapes and edges, where the brightness changes, and it ignores colour. A warmer light does not move the edges of a mug, so HOG gives the same answers. A method is only as stable as the numbers it reads.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
Hamid's camera takes returns photos at 1,284 pixels wide. Tomás wants to shrink them to 224 pixels wide to make the model faster. What do you check first, and why?
Show the answer
Check how many pixels the smallest damage that matters will have after shrinking. Shrinking from 1,284 to 224 pixels wide divides the area by about 32.9. A spot of 70 pixels becomes about 2 pixels, and its colour fades into the wax. The smallest damage in the course’s photos, 30 pixels, becomes 0.91 pixels: less than one pixel, so no model can find it. If the smallest damage falls below a few pixels, keep more resolution or look at smaller parts of the photo. Speed does not help when the damage is gone from the numbers.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Recap
- A photo is a grid of pixels. Each pixel has three channels, red, green and blue, from 0 to 255. A catalogue photo is
(256, 256, 3): 196,608 numbers. - Light changes every number. The planted new studio light moved one mug pixel from
(187, 214, 225)to(235, 241, 197). A colour histogram method fell from 26.3% to 22.5%; HOG, which ignores colour, stayed at 35.6%. - A colour space such as HSV writes a colour as hue, saturation and value. It separates “which colour” from “how strong” and “how bright”.
- Shrinking to 224 pixels wide divides the area by about 32.9. A spot of 70 pixels becomes about 2; the smallest damage in the course becomes 0.91 pixels.
- JPEG changes numbers a little at each save; PNG keeps them exactly. Store masks as PNG: a JPEG save gave one mask 26 values instead of 2.
You can now read a photo as numbers and say what changes them. But so far, you looked at single photos chosen for you. Next, you open all of Larkfield’s images, check their licences, and meet the comparison table that the whole course fills.