Convolution and patches, in one lesson
- Lesson 7
- 30 min
- Module 2: Baselines and pretrained classifiers
Where we are
Hand-made features describe edges in a fixed way. A neural network learns its own features. In this lesson, we see, in one lesson, how a CNN looks at small windows and how a vision transformer looks at patches, so that we can read a model card. First, a worker checks a large rug through a small window.
Lesson 7 of 30 · Vision AI: See, Count and InspectConvolution and patches, in one lesson
1. Seeing is a business problem
2. Baselines and pretrained classifiers
3. Embeddings, CLIP and few labels
4. Detect and count
6. Inspect for defects
7. Match, search and describe
8. Run vision in production
What you will be able to do
Understand what a convolutional network and a vision transformer do with a photo, well enough to choose a pretrained model and read its model card. How these networks learn is the job of another course; here you see what they look at.
NotebookDo this lesson in a notebook
The notebook slides a filter by hand, runs two edge filters on a real photo, and cuts a photo into patches. It uses NumPy only, and downloads the catalogue photos (about 9 MB). About 15 minutes.
Colab and Kaggle open your own copy. They need a free Google or Kaggle account. Stuck? See the solution notebook.
Before you startWhat to know and to have ready
What you need to know
- Edges, gradients and HOG, from Hand-made features.
- Helpful, not required: neurons, layers and training, from Deep Learning and Transformers Explained. This lesson explains the few ideas it needs.
What you need on your computer
- The notebook (in Colab, Kaggle or VS Code). This lesson adds nothing to the
larkvisionproject.
Tested with Python 3.12.13 on macOS.
A small window over a large rug
Section titled “A small window over a large rug”A returned rug arrives in Hamid’s warehouse. A worker must find small stains on it. She takes a piece of cardboard with a square hole, the size of her hand. She puts it on the top-left corner, and looks only through the hole. Then she slides it a little to the right, and looks again. Row by row, she covers the whole rug. On a paper map of the rug, she marks each place where she saw a stain.
She asks one question at every place: “Is there a stain in this window?” Her map of answers is smaller than the rug, and it shows where the stains are. This is exactly what a convolution does with a photo.
Why a network needs its own filters
Section titled “Why a network needs its own filters”In the last lesson, HOG used one fixed rule to find edges: right minus left, below minus above. A person chose it. It finds edges well, and then stops at 0.487 with colour. Nobody wrote a rule for a rim, a handle or a flame.
A network does not get its rules from a person. It starts with many small grids of random numbers. It slides each one over thousands of photos, and it changes the numbers a little whenever its answer is wrong. After training, the grids find the patterns that help most. How it changes the numbers is the topic of The training loop in the deep learning course. Here we look at what the grids do.
Convolution: slide, multiply, add
Section titled “Convolution: slide, multiply, add”A filter (also called a kernel) is a small grid of numbers, for example 3 × 3. To apply it, put it on the photo’s top-left corner. Multiply each filter number by the pixel under it, and add the nine products. That sum is one number of the output. Then slide the filter one pixel to the right, and do it again. This sliding sum is a convolution, and its output is a feature map: a map of how strongly each window matches the filter.
A tiny worked example
Section titled “A tiny worked example”The image is 6 × 6 grey pixels, made up for this example: dark (10) on the left, bright (200) on the right. The filter subtracts the window’s left column from its right column, as HOG did.

Text description of the diagram
Three grids in a row. The image grid has 6 rows of 6 grey values. The first four rows are 10, 10, 10, 200, 200, 200. The last two rows are 10, 10, 10, 10, 200, 200: there, the edge moves one column to the right. A blue square marks the window that covers rows 1 to 3 and columns 2 to 4. The filter grid has three rows, each minus 1, 0, 1. The output grid has 4 rows of 4 numbers. Rows 1 and 2: 0, 570, 570, 0. Row 3: 0, 380, 570, 190. Row 4: 0, 190, 570, 380. The first 570 is marked: it comes from the marked window.
The marked window holds 10, 10, 200 in each of its three rows. Each row gives −1 × 10 + 0 × 10 + 1 × 200 = 190. Three rows give 3 × 190 = 570. A window on one colour only gives 0: the −10 and the +10 cancel. So the output is large exactly where the window covers the edge.
Look at the lower rows of the output: 380 and 190. There, the window covers the edge only in part, because the edge moves. The output is a map of where the pattern is, and how much of it.
The stride is the size of each step of the window: one pixel here. A stride of 2 makes the output half as wide. The output is also smaller than the input: 4 × 4 instead of 6 × 6. A 3 × 3 window fits only 4 times along a side of 6.
The same filter on a real photo
Section titled “The same filter on a real photo”The notebook (section 2) runs this filter, and the same filter turned on its side, on a catalogue photo of wire chairs. The two learned filters on the right come from a real trained network, as the next section explains:

Text description of the diagram
Five pictures in a row. The first is the photo: two white wire chairs on a sunny stone terrace, with a dark lawn at the top. The second shows the response of the hand-made vertical filter: the chair legs and the upright wires of the seat are dark lines. The third shows the hand-made horizontal filter: the long edge between the terrace and the lawn, and the flat wires, are dark. The fourth is learned filter number 31 of ResNet-18: its response looks like the vertical one, with dark chair legs. The fifth is learned filter number 32: its strongest line is the edge of the terrace, like the horizontal filter.
The vertical filter answers to the chair legs and the upright wires. The horizontal filter answers to the edge of the terrace and the flat wires. Each filter is a question, and each map is the photo’s answer.
What a network learns
Section titled “What a network learns”A convolutional neural network, or CNN, is many layers of filters. ResNet-18, a CNN that we use in the next lesson, has 64 filters in its first layer, each 7 × 7 pixels × 3 colours. The course team drew them from the trained model’s weights:

Text description of the diagram
A grid of 64 small coloured squares, in 4 rows of 16 (8 rows of 8 on a phone). Many squares show stripes, light on one side and dark on the other, at many angles: these are edge filters, like the ones of HOG. Others show colour patches: green against purple, or orange against blue. A few show a single bright dot on grey. Two squares are marked in blue: filter 31, with upright stripes, and filter 32, with flat stripes. These are the two learned filters of the photo above.
Many learned filters look like edge filters at many angles, as HOG uses. Others answer to colours, or to a colour next to another colour. Nobody designed them. Training found that these 64 questions help most for the first layer.
Then the network repeats the idea. The second layer slides its filters over the feature maps of the first layer, not over the photo. So it finds patterns of edges: a corner, a curve, a stripe pattern. Between layers, a stride of 2 or a pooling step halves the map. Max pooling keeps the largest number of each 2 × 2 block. Each later filter therefore covers a larger part of the photo. Deep layers answer to whole parts: the rim of a mug, a chair leg, a flame.
The first row of this diagram follows a 224 × 224 photo through ResNet-18. The second row is the other design, the subject of the next section:
Text description of the diagram
Two rows of six boxes each, joined by arrows. First row, a CNN (ResNet-18), whose view grows layer by layer. Box 1: the photo, 224 by 224 pixels by 3 colours. Box 2: the first layer, 64 filters of 7 by 7, gives 64 maps of 112 by 112, for edges and colours. Box 3: more layers, filters of filters, give 128 maps of 28 by 28, for corners and textures. Box 4: the last block gives 512 maps of 7 by 7, for parts such as rims, handles and legs. Box 5: the average of each of the 512 maps gives 512 numbers. Box 6: the last layer gives one score per class. Second row, a vision transformer, where every patch can use any other from the start. Box 1: the photo. Box 2: cut into 196 patches of 16 by 16 pixels. Box 3: each patch becomes a list of numbers, a token. Box 4: transformer blocks, where every patch looks at every other patch. Box 5: one summary of the photo. Box 6: the last layer gives one score per class.
Look at box 5 of the first row. After the last block, ResNet-18 takes the average of each of its 512 maps. Every photo becomes 512 numbers that describe which parts it holds. The last layer turns them into scores for ImageNet’s 1,000 classes. In the next lesson, we replace that last layer with our own.
Patches: how a vision transformer reads a photo
Section titled “Patches: how a vision transformer reads a photo”A vision transformer, or ViT, comes from the transformers that read text. A text transformer reads words. So the ViT turns a photo into “words”: it cuts the photo into squares of 16 × 16 pixels, called patches. A 224 × 224 photo has 14 × 14 = 196 patches. Each patch holds 16 × 16 pixels × 3 colours = 768 numbers.

Text description of the diagram
On the left, a photo of a white mug of coffee, cut by white lines into a grid of 14 by 14 squares. The seventh row of squares, through the coffee, is marked in blue. On the right, the 14 squares of that row are shown larger, in a line from left to right: two of light background, then the dark edge of the rim, five of black coffee, then the white side of the mug and the background. The model reads all 196 squares of the photo this way.
Each patch becomes a list of numbers, a token. The order matters, so each token also gets its place in the grid. Then transformer blocks work on all the tokens. Their main tool is attention: each patch takes information from the other patches that matter most to it. A patch of the mug’s handle can use a patch of its rim in the very first block. In a CNN, two such far parts meet only in deep layers.
Attention step by step, in the deep learning course, shows how attention works with words. Patches work the same way.
The ViT’s design knows less about photos than a CNN’s. A CNN’s sliding filter assumes that nearby pixels belong together, and that a pattern means the same thing anywhere. A ViT must learn this from data. So ViTs usually need more training photos, or a long training on a very large set, before they beat a CNN. Both designs are in use, and newer CNNs borrow ideas from transformers. ConvNeXt, which you meet in the next lesson, is such a CNN.
Read a model card
Section titled “Read a model card”A model card is the page that describes a pretrained model. It says what the model is, what it learned from, what input it expects, and its licence. You read it before you use a model. The names of the models on the hub of timm, the library of image models that this course uses, already hold much of it:
| Model name | Read it as | Parameters | Input |
|---|---|---|---|
resnet18.a1_in1k |
ResNet with 18 layers; training recipe “a1”; learned on ImageNet-1k | 11.7 million | 224 × 224 |
resnet50.a1_in1k |
ResNet with 50 layers; same recipe and data | 25.6 million | 224 × 224 |
efficientnet_b0.ra_in1k |
EfficientNet, its smallest size “B0”; recipe “ra”; ImageNet-1k | 5.3 million | 224 × 224 |
convnext_atto.d2_in1k |
ConvNeXt, its tiny size “atto”; recipe “d2”; ImageNet-1k | 3.7 million | 224 × 224 |
vit_small_patch16_224.augreg_in21k_ft_in1k |
A small ViT with 16-pixel patches at 224 pixels; learned on ImageNet-21k, then fine-tuned on ImageNet-1k | 22.1 million | 224 × 224 |
The course team counted the parameters with timm. A parameter is one number that the network learned, for example one value in a filter. ImageNet-1k is a public set of about 1.3 million photos in 1,000 classes. ImageNet-21k is a larger one, with about 21,000 classes.
When you read a card, look for five things:
- The input: the photo size, and how to prepare the photo. That is a resize, a crop, and a colour normalisation: subtract a mean and divide by a standard deviation, per channel.
- The training data: what the model has seen. A model trained on ImageNet has seen many mugs and chairs, and few pallets.
- The size: parameters and file size. They decide the memory and the time on your CPU.
- The results: the card’s own scores are on its own test set, not on your photos. Treat them as a vendor claim, and measure on your data.
- The licence: of the code, of the weights, and of the training data. A later module looks at what ImageNet’s own terms mean for a product.
Common mistake
The photo is not prepared the way the model card says
Symptom: you load ResNet-18 and feed it photos turned into numbers from 0 to 1. Nothing fails. The model gives an answer for every photo. But many answers are different from the answers with the card’s preparation.
Cause: the card says: resize, crop to 224 × 224, then normalise each channel. The mean is (0.485, 0.456, 0.406) and the standard deviation (0.229, 0.224, 0.225), from ImageNet. The model learned on photos prepared this way. The course team ran both ways on the 160 catalogue test photos. Without the normalising step, the model’s top answer changed for 49 of them.
Fix: let the library build the preparation from the card. In timm: config = timm.data.resolve_data_config({}, model=model), then transform = timm.data.create_transform(**config). The next lesson’s notebook does this.
Consequence in AI work: a wrong preparation gives no error message, only worse answers. Check it whenever you change a model, and whenever a new camera sends photos of another size.
When to choose which design
Section titled “When to choose which design”- A CNN (ResNet, EfficientNet, ConvNeXt) is a good default for a small set of photos and a CPU. Its design already assumes that nearby pixels belong together, so it needs fewer photos to adapt.
- A ViT is a good choice when it was trained on a very large set and you can afford its size. It also helps when far parts of the photo must be compared. Many recent models that link photos and text use ViTs, as you see in the next module.
- The cost: more parameters mean more memory and more milliseconds per photo. Whether more parameters also mean better answers on Larkfield’s photos is a question to measure, not to assume.
Your turn: two small changes
Section titled “Your turn: two small changes”Section 5 of the notebook has two tasks and two check cells.
- Write the horizontal edge filter
horizontal. It subtracts the top row of a 3 × 3 window from its bottom row. - Put
conv2d(grid, horizontal)inout_h. - A model cuts a 224 × 224 photo into patches of 32 × 32 pixels. Put the number of patches in
n_patches_32. - Run the two check cells.
Show the answer
horizontal = np.array([[-1, -1, -1], [0, 0, 0], [1, 1, 1]])out_h = conv2d(grid, horizontal)n_patches_32 = (224 // 32) ** 2print(out_h.astype(int)); print(n_patches_32)[[ 0 0 0 0] [ 0 0 0 0] [ 0 -190 -190 -190] [ 0 -190 -190 -190]]49The horizontal filter answers only where something changes from top to bottom. In the grid, that is only where the edge moves one column, in the lower rows. The value is negative because the change goes from bright (above) to dark (below). With patches of 32, a side holds 7 patches, so the photo has 7 × 7 = 49 patches. That is 4 times fewer tokens, each with 4 times more pixels.
Knowledge check
Section titled “Knowledge check”Check your understanding
A 3 × 3 window holds the rows (10, 10, 200), (10, 10, 200) and (10, 200, 200). What does the filter with rows (−1, 0, 1) give? Show the steps.
Show the answer
Multiply and add row by row. Row 1: −10 + 0 + 200 = 190. Row 2: 190. Row 3: −10 + 0 + 200 = 190. The total is 570. The filter subtracts the left column from the right column. So it gives a large number wherever the window is darker on the left than on the right.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
ResNet-18's first-layer filters look like edge and colour filters. Why can a deep layer of the same network answer to a whole mug handle, when every filter is small?
Show the answer
Each layer works on the feature maps of the layer before it, not on the photo. So a second-layer filter combines edges into corners and curves, and a later one combines those into parts. Between layers, strides and pooling halve the maps. A small filter on a small map therefore covers a large part of the photo. Deep layers see parts like a handle or a rim.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
Tomás reads `vit_small_patch16_224` and `convnext_atto` on two model cards. He says: 'The ViT has 22 million parameters and the ConvNeXt 3.7 million, so the ViT will be better on our catalogue.' What do you check before you agree?
Show the answer
Parameters say how large a model is, not how good it is on Larkfield’s photos. Check what each model learned from and the input it expects (both 224 × 224 here). Check how it prepares the photos, its licence, and its time per photo on our CPU. Then measure both on the same catalogue photos with the comparison table. A larger model costs more memory and time; it is better only if the table shows it.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Recap
- A convolution slides a small filter over the photo, multiplies and adds. Its feature map is large where the window matches the filter: 570 at the edge of the worked example.
- A CNN learns its filters from photos. ResNet-18’s first 64 filters answer to edges and colours. Deep layers answer to parts, because each layer works on the maps of the layer before.
- A vision transformer cuts a 224 × 224 photo into 196 patches of 16 × 16. Through attention, every patch can use every other one.
- A model card gives the input, its preparation, the training data, the size and the licence. ResNet-18 has 11.7 million parameters; ConvNeXt-Atto has 3.7 million.
- Without the card’s normalising step, ResNet-18’s top answer changed on 49 of 160 photos, with no error.
You can now read what a pretrained model looks at and what it expects. But ResNet-18 answers with ImageNet’s 1,000 classes, not Larkfield’s 8. Next, you keep its learned filters and train only a new last layer on the catalogue.
Next: Transfer learning