Hand-made features
- Lesson 6
- 35 min
- Module 2: Baselines and pretrained classifiers
Where we are
Colour histograms gave us the first rows of the table. They know colour, but they know nothing about shape. In this lesson, we describe shapes with hand-made features, HOG and edges, and train a linear model on them. First, a worker looks at a screen with no colour at all.
Lesson 6 of 30 · Vision AI: See, Count and InspectHand-made features
1. Seeing is a business problem
2. Baselines and pretrained classifiers
3. Embeddings, CLIP and few labels
4. Detect and count
6. Inspect for defects
7. Match, search and describe
8. Run vision in production
What you will be able to do
Describe the shapes in a photo with a hand-made feature, HOG, train a linear model on it, and measure what shape adds to colour. It is the classical method of the 2000s: fast on any CPU, and blind in ways that explain why later methods exist.
NotebookDo this lesson in a notebook
The notebook has every code cell, task and check of this lesson. Its setup cells install the packages and download the catalogue photos (about 9 MB). It needs no account and no key. About 20 minutes.
Colab and Kaggle open your own copy. They need a free Google or Kaggle account. Stuck? See the solution notebook.
Before you startWhat to know and to have ready
What you need to know
- Colour histograms, k-NN and the comparison table, from Rules and colour histograms.
- What a logistic regression does with a list of numbers, from Classification intuition in Machine Learning: From Problem to Reliable Model. This lesson uses it as a tool.
What you need on your computer
- The notebook, or your
larkvisionproject at the snapshotend-of-m02, from the last lesson.
Tested with Python 3.12.13 on macOS.
A screen with no colour
Section titled “A screen with no colour”At night, Hamid’s warehouse records the loading area with an old black-and-white camera. In the morning, a worker watches the video on a small screen. There is no colour at all. Still, she says: “That is a chair, and that is a box of mugs.” She sees the outlines: the straight legs of a chair, the round rim of a mug and its handle.
The colour histogram of the last lesson cannot do this. It saw a brown vase and found two brown chairs. In this lesson, we give the computer the worker’s view: no colour, only edges and their directions.
Edges are changes in brightness
Section titled “Edges are changes in brightness”An edge is a place where the brightness changes quickly: the dark leg of a chair on a light floor. To find one, compare each pixel’s neighbours. Subtract the left neighbour from the right one, and the pixel above from the pixel below. A large difference means an edge.
A tiny worked example
Section titled “A tiny worked example”Here are 3 × 3 real pixels from the armchair photo below, in grey values from 0 (black) to 255 (white). The pixel in the middle is at the edge of the orange chair against the white wall:
| Left | Middle | Right | |
|---|---|---|---|
| Above | 243 | 209 | 100 |
| Middle row | 233 | 130 | 55 |
| Below | 218 | 98 | 60 |
For the middle pixel:
- Left to right: right − left = 55 − 233 = −178. The photo gets much darker to the right.
- Top to bottom: below − above = 98 − 209 = −111. It also gets darker downwards.
- Edge strength: √(178² + 111²) = 209.8. This is the length of the change, as on a map: 178 steps across and 111 steps down.
- Direction: the angle of that change is 31.9°. HOG does not care if the change goes from dark to light or from light to dark. So it keeps angles from 0° to 180°.
HOG puts each direction into one of 9 bins of 20°. An angle of 31.9° goes into the bin 20–40°. Every pixel of the photo gets the same treatment.
Count directions in each cell
Section titled “Count directions in each cell”One pixel says little. So HOG cuts the photo into cells of 16 × 16 pixels. In each cell, it builds a small histogram of directions: each pixel adds its edge strength to its direction’s bin. Strong edges count more than weak ones. The chart shows the cell around our pixel:
- Votes
- The bin of the worked example
Text description of the chart
A column chart with nine columns, one per range of 20 degrees. 0 to 20: 2,259. 20 to 40: 583, the bin of the worked example. 40 to 60: 345. 60 to 80: 1,055. 80 to 100: 485. 100 to 120: 653. 120 to 140: 511. 140 to 160: 161. 160 to 180: 2,622. The two largest columns are at the two ends, near 0 and 180 degrees.
The cell sits on the upright side of the chair. Brightness changes from left to right, so the two end bins (near 0° and near 180°) are the largest. In other words, the cell holds an upright edge. The name of the method says all of this: HOG is a Histogram of Oriented Gradients. A gradient is the change in brightness that we computed, with its strength and direction.
HOG adds one more step. It groups the cells in blocks of 2 × 2 cells and scales each block’s numbers to the same size. A shadow makes all edges in a block weaker, and this step undoes much of it.
From a photo to 1,764 numbers
Section titled “From a photo to 1,764 numbers”The course’s code makes the photo grey and 128 × 128 pixels. That gives 8 × 8 = 64 cells, and 7 × 7 = 49 blocks of 2 × 2 cells. Each block holds 4 cells × 9 bins = 36 numbers. So each photo becomes 49 × 36 = 1,764 numbers. The picture shows each step on two catalogue photos:

Text description of the diagram
Two rows of four pictures. In the first row, an orange armchair in front of a window. In the second row, a white mug of coffee. The first picture of each row is the photo. The second is the same photo in grey, 128 by 128 pixels. The third shows the edge strength: the outline of the chair, its legs and the window frame are dark lines; for the mug, the rim and the dark coffee are the strongest edges. The fourth is the HOG picture: in each cell, short lines point along the main edges. The chair’s cells show many upright lines; the mug’s cells show curved, mostly flat lines around the rim.
The HOG picture keeps the chair’s upright sides and the mug’s round rim. It drops the colour, the orange and the coffee. This is what we want: a description of shape.
A linear model on HOG
Section titled “A linear model on HOG”1,764 numbers per photo are too many to compare by distance, as k-NN did. Many of them are noise. So we use a model that learns which numbers matter for each class: a logistic regression. It gives each number a weight per class, adds the weighted numbers up, and picks the class with the highest sum. A model that adds weighted numbers this way is called linear. Training it takes about 2 seconds.
Section 3 of the notebook runs it:
model = make_pipeline(StandardScaler(), LogisticRegression(C=0.5, max_iter=3000)).fit(Htr, train["label"])catalogue test accuracy 0.356 worst candle 0.1new studio accuracy 0.356 worst candle 0.1HOG features: 1.1 ms per photoStandardScaler puts every number on the same scale first, so that no number wins only because it is large. C=0.5 keeps the weights small, so that the model does not learn the training photos by heart.
HOG reaches 0.356: better than the rule (0.269) and the histogram (0.263), and far from good. Candle is again the worst class (0.10). A candle’s outline changes a lot. A tea light from above is a circle, and a tall candle is a thin upright line. A glass jar adds its own edges.
The light does not move HOG
Section titled “The light does not move HOG”The new studio photos have warmer light, added by code (a problem we planted). Predict before you look: does HOG lose accuracy, as the histogram did?
- Rules
- Colour histogram
- HOG
Text description of the chart
Three lines from the studio light to the warmer studio light. Rules: 0.269 to 0.244. Histogram: 0.263 to 0.225. HOG: 0.356 to 0.356, a flat line.
HOG does not move at all: 0.356 under both lights. It works on grey values, and the blocks are scaled to the same size. So a warmer or brighter light changes almost nothing. This is HOG’s strong point.
Shape and colour together
Section titled “Shape and colour together”HOG knows shape and not colour. The histogram knows colour and not shape. What if the model gets both? Your turn in the notebook (section 5) puts the two lists side by side, 1,764 + 128 numbers per photo, and trains the same model.
- Last lesson
- HOG
- Both features
Text description of the chart
A column chart with four columns, on an axis from 0 to 1. Rules 0.269, colour histogram 0.263, HOG 0.356, HOG and colour together 0.487.
Together they reach 0.487. The two features know different things, so each one fixes some mistakes of the other. But the model is still wrong on about half of the photos. The candle recall stays at 0.10. More hand-made features would help a little more. People built such features for years. Each one needed an expert, and each one saw only what its author thought of.
Add HOG to your project
Section titled “Add HOG to your project”Your project already has HOG: larkvision/features.py holds hog_features, the same code as the notebook. Run it from your larkvision folder, with the virtual environment active and LARKVISION_DATA set. In a new terminal, set it again as in Meet the Larkfield images.
-
Run the HOG baseline. It takes about 2 seconds:
Terminal window vision baseline --method hog -
Show only your own rows of the comparison table:
Terminal window vision compare --task catalogue --mine
The output of step 1:
HOG + logistic regression, catalogue test: accuracy 0.356; worst class candle (recall 0.10); 1.2 ms per photoHOG + logistic regression, new studio: accuracy 0.356; worst class candle (recall 0.10); 1.2 ms per photoAdded to outputs/table.json. See it with: vision compare --task catalogueStep 2 prints your rows, with “(yours)” after each method. If you ran the histogram in the last lesson, you see four rows:
catalogue: accuracymethod setting value worst worst_value ms_per_image model_mb labelled_images licencecolour histogram + 5-NN (yours) catalogue test 0.263 vase 0.050 1.030 0 453 none (our code)colour histogram + 5-NN (yours) new studio 0.225 cushion 0.000 0.940 0 453 none (our code)HOG + logistic regression (yours) catalogue test 0.356 candle 0.100 1.200 0 453 none (our code)HOG + logistic regression (yours) new studio 0.356 candle 0.100 1.190 0 453 none (our code)Your milliseconds differ. Both methods need no model file (model_mb 0) and about 1 ms per photo on a laptop CPU. Keep these two columns in mind for the next lessons.
What HOG cannot see
Section titled “What HOG cannot see”HOG describes edges at fixed places. Cell 12 is always the same square of the photo. This brings three limits:
- Position and size. Photograph the same mug a little to the left, or from further away, and its edges fall into other cells. The 1,764 numbers change, although the mug did not.
- Turning. Turn the mug, and its handle’s edges change direction. The bins change too.
- Meaning. HOG sees that a cell has upright edges. It does not know if they belong to a chair leg, a candle or a window frame. The linear model must guess the meaning from where the edges are.
These limits explain the 0.356. The next lesson shows how a network learns its own edge filters, and then combines them into larger shapes, so that the position matters less.
When to use hand-made features
Section titled “When to use hand-made features”- Use them when the camera is fixed and the objects are always in about the same place and size. A part on a conveyor belt under a fixed camera is a good case. Also use them when the light changes, because HOG does not depend on colour. And use them when a CPU and 1 ms a photo is all you have.
- Do not use them for photos from many cameras, angles and distances, like the catalogue. Avoid them also when colour matters: HOG cannot tell a red mug from a white one.
- What they cost: about 1 ms per photo, no model file, and seconds to train. The real cost is the expert’s time to design the features, and a low ceiling: here, 0.487 with colour added.
Common mistake
Photos of different sizes
Symptom: you compute HOG on Hamid’s returns photos next to catalogue photos, without the resize to 128 × 128. Python stops when it puts the lists together:
ValueError: all input arrays must have the same shapeCause: the number of HOG values depends on the photo’s size. A catalogue photo of 256 × 256 gives 8,100 numbers. A returns photo of 320 × 291 gives 11,628. A linear model needs the same list length for every photo.
Fix: resize every photo to one size before HOG, as hog_features does (128 × 128). Check the shape of the first result: (1764,).
Consequence in AI work: the resize is part of the method. If a new camera gives photos of another shape, a resize to a square also squeezes the object, and HOG’s directions change. Measure again on photos from the new camera.
Your turn: shape and colour together
Section titled “Your turn: shape and colour together”Section 5 of the notebook has one task and a check cell.
- Compute the colour histogram of every training photo and every test photo.
- Put HOG first and the histogram second, with
np.hstack, intoXtr_bothandXte_both. - Train the same pipeline:
StandardScalerandLogisticRegression(C=0.5, max_iter=3000). - Put the test accuracy in
acc_both, and run the check cell.
Show the answer
Ctr = np.stack([colour_histogram("data/" + f) for f in train["file"]])Cte = np.stack([colour_histogram("data/" + f) for f in test["file"]])Xtr_both = np.hstack([Htr, Ctr])Xte_both = np.hstack([Hte, Cte])both = make_pipeline(StandardScaler(), LogisticRegression(C=0.5, max_iter=3000)).fit(Xtr_both, train["label"])acc_both = cp.classification_scores(list(test["label"]), list(both.predict(Xte_both)), CLASSES)["accuracy"]print(acc_both)0.487The check prints Check passed: acc_both is 0.487. The two features know different things, so together they beat each one alone. The candle recall stays at 0.10: neither feature describes a candle well.
Knowledge check
Section titled “Knowledge check”Check your understanding
In the worked example, the change from left to right is −178 and from top to bottom is −111. What is the edge strength, and into which of HOG's 9 bins does the direction go?
Show the answer
The edge strength is √(178² + 111²) ≈ 209.8. The direction of the change is about 31.9°, after HOG folds all angles into 0–180°. So it goes into the bin 20–40°, the second of the nine bins of 20°.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
Under the warmer studio light, the histogram falls from 0.263 to 0.225, but HOG stays at 0.356. Why?
Show the answer
HOG works on the grey photo, and it only looks at changes in brightness between neighbours. It also scales each block of cells to the same size. A warmer or brighter light changes the colours and the overall brightness, but not much the places and directions of edges. The histogram counts colours, so it changes with the light.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
Omar asks: 'HOG and colour together reach 0.487. Can we add more hand-made features until we reach 0.9?' What do you answer?
Show the answer
More features would help a little, but each one needs an expert to design it, and each sees only what its author thought of. HOG describes edges at fixed places, so a product photographed from another angle or distance gives other numbers. The catalogue has photos from thousands of cameras and rooms. A method that learns its own features from many photos, as the next lessons show, is a better use of the time.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Recap
- An edge is a quick change in brightness. Its strength and direction come from the differences between a pixel’s neighbours.
- HOG counts edge directions in 9 bins per cell of 16 × 16 pixels, and scales blocks of 2 × 2 cells. A 128 × 128 photo becomes 1,764 numbers.
- HOG with a linear model reaches 0.356, and does not move under the warmer light. With the colour histogram, it reaches 0.487. Both run in about 1 ms a photo, with no model file.
vision baseline --method hogadds HOG to your comparison table.- HOG describes edges at fixed places, so position, size and turning change its numbers, and it cannot know what an edge belongs to.
You can now describe shape with hand-made features and measure what they add. But a person had to design them, and they stop at 0.487. Next, you see how a network learns its own filters from photos. You also see how a vision transformer reads a photo in patches.