Skip to content
Français

Hand-made features

  • Lesson 6
  • 35 min
  • Module 2: Baselines and pretrained classifiers

Where we are

Colour histograms gave us the first rows of the table. They know colour, but they know nothing about shape. In this lesson, we describe shapes with hand-made features, HOG and edges, and train a linear model on them. First, a worker looks at a screen with no colour at all.

What you will be able to do

Describe the shapes in a photo with a hand-made feature, HOG, train a linear model on it, and measure what shape adds to colour. It is the classical method of the 2000s: fast on any CPU, and blind in ways that explain why later methods exist.

NotebookDo this lesson in a notebookColab · Kaggle · GitHub

The notebook has every code cell, task and check of this lesson. Its setup cells install the packages and download the catalogue photos (about 9 MB). It needs no account and no key. About 20 minutes.

Colab and Kaggle open your own copy. They need a free Google or Kaggle account. Stuck? See the solution notebook.

Before you startWhat to know and to have readyAbout 35 minutes

What you need to know

What you need on your computer

  • The notebook, or your larkvision project at the snapshot end-of-m02, from the last lesson.

Tested with Python 3.12.13 on macOS.

At night, Hamid’s warehouse records the loading area with an old black-and-white camera. In the morning, a worker watches the video on a small screen. There is no colour at all. Still, she says: “That is a chair, and that is a box of mugs.” She sees the outlines: the straight legs of a chair, the round rim of a mug and its handle.

The colour histogram of the last lesson cannot do this. It saw a brown vase and found two brown chairs. In this lesson, we give the computer the worker’s view: no colour, only edges and their directions.

An edge is a place where the brightness changes quickly: the dark leg of a chair on a light floor. To find one, compare each pixel’s neighbours. Subtract the left neighbour from the right one, and the pixel above from the pixel below. A large difference means an edge.

Here are 3 × 3 real pixels from the armchair photo below, in grey values from 0 (black) to 255 (white). The pixel in the middle is at the edge of the orange chair against the white wall:

Left Middle Right
Above 243 209 100
Middle row 233 130 55
Below 218 98 60

For the middle pixel:

  • Left to right: right − left = 55 − 233 = −178. The photo gets much darker to the right.
  • Top to bottom: below − above = 98 − 209 = −111. It also gets darker downwards.
  • Edge strength: √(178² + 111²) = 209.8. This is the length of the change, as on a map: 178 steps across and 111 steps down.
  • Direction: the angle of that change is 31.9°. HOG does not care if the change goes from dark to light or from light to dark. So it keeps angles from 0° to 180°.

HOG puts each direction into one of 9 bins of 20°. An angle of 31.9° goes into the bin 20–40°. Every pixel of the photo gets the same treatment.

One pixel says little. So HOG cuts the photo into cells of 16 × 16 pixels. In each cell, it builds a small histogram of directions: each pixel adds its edge strength to its direction’s bin. Strong edges count more than weak ones. The chart shows the cell around our pixel:

  • Votes
  • The bin of the worked example
The direction histogram of one 16 × 16 cell of the armchair photo (simple votes). The two largest bins are near 0° and 180°: the changes go from left to right, so the edge itself is upright.
Text description of the chart

A column chart with nine columns, one per range of 20 degrees. 0 to 20: 2,259. 20 to 40: 583, the bin of the worked example. 40 to 60: 345. 60 to 80: 1,055. 80 to 100: 485. 100 to 120: 653. 120 to 140: 511. 140 to 160: 161. 160 to 180: 2,622. The two largest columns are at the two ends, near 0 and 180 degrees.

The cell sits on the upright side of the chair. Brightness changes from left to right, so the two end bins (near 0° and near 180°) are the largest. In other words, the cell holds an upright edge. The name of the method says all of this: HOG is a Histogram of Oriented Gradients. A gradient is the change in brightness that we computed, with its strength and direction.

HOG adds one more step. It groups the cells in blocks of 2 × 2 cells and scales each block’s numbers to the same size. A shadow makes all edges in a block weaker, and this step undoes much of it.

The course’s code makes the photo grey and 128 × 128 pixels. That gives 8 × 8 = 64 cells, and 7 × 7 = 49 blocks of 2 × 2 cells. Each block holds 4 cells × 9 bins = 36 numbers. So each photo becomes 49 × 36 = 1,764 numbers. The picture shows each step on two catalogue photos:

Two rows of four pictures: an orange armchair and a white mug. For each: the photo, the photo in grey at 128 by 128 pixels, the edge strength as dark lines on white, and the HOG picture with short lines in each cell showing the main edge directions.
From a photo to HOG: the grey photo, the edge strength at each pixel (dark = strong), and the main directions in each 16 × 16 cell. Photos: Heritage Hotels & Resorts and Joris de Vrede, CC BY 2.0, from Open Images.
Text description of the diagram

Two rows of four pictures. In the first row, an orange armchair in front of a window. In the second row, a white mug of coffee. The first picture of each row is the photo. The second is the same photo in grey, 128 by 128 pixels. The third shows the edge strength: the outline of the chair, its legs and the window frame are dark lines; for the mug, the rim and the dark coffee are the strongest edges. The fourth is the HOG picture: in each cell, short lines point along the main edges. The chair’s cells show many upright lines; the mug’s cells show curved, mostly flat lines around the rim.

The HOG picture keeps the chair’s upright sides and the mug’s round rim. It drops the colour, the orange and the coffee. This is what we want: a description of shape.

1,764 numbers per photo are too many to compare by distance, as k-NN did. Many of them are noise. So we use a model that learns which numbers matter for each class: a logistic regression. It gives each number a weight per class, adds the weighted numbers up, and picks the class with the highest sum. A model that adds weighted numbers this way is called linear. Training it takes about 2 seconds.

Section 3 of the notebook runs it:

model = make_pipeline(StandardScaler(), LogisticRegression(C=0.5, max_iter=3000)).fit(Htr, train["label"])
catalogue test accuracy 0.356 worst candle 0.1
new studio accuracy 0.356 worst candle 0.1
HOG features: 1.1 ms per photo

StandardScaler puts every number on the same scale first, so that no number wins only because it is large. C=0.5 keeps the weights small, so that the model does not learn the training photos by heart.

HOG reaches 0.356: better than the rule (0.269) and the histogram (0.263), and far from good. Candle is again the worst class (0.10). A candle’s outline changes a lot. A tea light from above is a circle, and a tall candle is a thin upright line. A glass jar adds its own edges.

The new studio photos have warmer light, added by code (a problem we planted). Predict before you look: does HOG lose accuracy, as the histogram did?

  • Rules
  • Colour histogram
  • HOG
Accuracy on the same 160 products under the studio light and under the warmer light of the new studio. HOG has no colour, so its line is flat.
Text description of the chart

Three lines from the studio light to the warmer studio light. Rules: 0.269 to 0.244. Histogram: 0.263 to 0.225. HOG: 0.356 to 0.356, a flat line.

HOG does not move at all: 0.356 under both lights. It works on grey values, and the blocks are scaled to the same size. So a warmer or brighter light changes almost nothing. This is HOG’s strong point.

HOG knows shape and not colour. The histogram knows colour and not shape. What if the model gets both? Your turn in the notebook (section 5) puts the two lists side by side, 1,764 + 128 numbers per photo, and trains the same model.

  • Last lesson
  • HOG
  • Both features
Accuracy on the 160 catalogue test photos. Shape and colour together reach 0.487: better than each one alone, and still wrong on half the photos.
Text description of the chart

A column chart with four columns, on an axis from 0 to 1. Rules 0.269, colour histogram 0.263, HOG 0.356, HOG and colour together 0.487.

Together they reach 0.487. The two features know different things, so each one fixes some mistakes of the other. But the model is still wrong on about half of the photos. The candle recall stays at 0.10. More hand-made features would help a little more. People built such features for years. Each one needed an expert, and each one saw only what its author thought of.

Your project already has HOG: larkvision/features.py holds hog_features, the same code as the notebook. Run it from your larkvision folder, with the virtual environment active and LARKVISION_DATA set. In a new terminal, set it again as in Meet the Larkfield images.

  1. Run the HOG baseline. It takes about 2 seconds:

    Terminal window
    vision baseline --method hog
  2. Show only your own rows of the comparison table:

    Terminal window
    vision compare --task catalogue --mine

The output of step 1:

HOG + logistic regression, catalogue test: accuracy 0.356; worst class candle (recall 0.10); 1.2 ms per photo
HOG + logistic regression, new studio: accuracy 0.356; worst class candle (recall 0.10); 1.2 ms per photo
Added to outputs/table.json. See it with: vision compare --task catalogue

Step 2 prints your rows, with “(yours)” after each method. If you ran the histogram in the last lesson, you see four rows:

catalogue: accuracy
method setting value worst worst_value ms_per_image model_mb labelled_images licence
colour histogram + 5-NN (yours) catalogue test 0.263 vase 0.050 1.030 0 453 none (our code)
colour histogram + 5-NN (yours) new studio 0.225 cushion 0.000 0.940 0 453 none (our code)
HOG + logistic regression (yours) catalogue test 0.356 candle 0.100 1.200 0 453 none (our code)
HOG + logistic regression (yours) new studio 0.356 candle 0.100 1.190 0 453 none (our code)

Your milliseconds differ. Both methods need no model file (model_mb 0) and about 1 ms per photo on a laptop CPU. Keep these two columns in mind for the next lessons.

HOG describes edges at fixed places. Cell 12 is always the same square of the photo. This brings three limits:

  • Position and size. Photograph the same mug a little to the left, or from further away, and its edges fall into other cells. The 1,764 numbers change, although the mug did not.
  • Turning. Turn the mug, and its handle’s edges change direction. The bins change too.
  • Meaning. HOG sees that a cell has upright edges. It does not know if they belong to a chair leg, a candle or a window frame. The linear model must guess the meaning from where the edges are.

These limits explain the 0.356. The next lesson shows how a network learns its own edge filters, and then combines them into larger shapes, so that the position matters less.

  • Use them when the camera is fixed and the objects are always in about the same place and size. A part on a conveyor belt under a fixed camera is a good case. Also use them when the light changes, because HOG does not depend on colour. And use them when a CPU and 1 ms a photo is all you have.
  • Do not use them for photos from many cameras, angles and distances, like the catalogue. Avoid them also when colour matters: HOG cannot tell a red mug from a white one.
  • What they cost: about 1 ms per photo, no model file, and seconds to train. The real cost is the expert’s time to design the features, and a low ceiling: here, 0.487 with colour added.

Common mistake

Photos of different sizes

Symptom: you compute HOG on Hamid’s returns photos next to catalogue photos, without the resize to 128 × 128. Python stops when it puts the lists together:

ValueError: all input arrays must have the same shape

Cause: the number of HOG values depends on the photo’s size. A catalogue photo of 256 × 256 gives 8,100 numbers. A returns photo of 320 × 291 gives 11,628. A linear model needs the same list length for every photo.

Fix: resize every photo to one size before HOG, as hog_features does (128 × 128). Check the shape of the first result: (1764,).

Consequence in AI work: the resize is part of the method. If a new camera gives photos of another shape, a resize to a square also squeezes the object, and HOG’s directions change. Measure again on photos from the new camera.

Section 5 of the notebook has one task and a check cell.

  1. Compute the colour histogram of every training photo and every test photo.
  2. Put HOG first and the histogram second, with np.hstack, into Xtr_both and Xte_both.
  3. Train the same pipeline: StandardScaler and LogisticRegression(C=0.5, max_iter=3000).
  4. Put the test accuracy in acc_both, and run the check cell.
Show the answer
Ctr = np.stack([colour_histogram("data/" + f) for f in train["file"]])
Cte = np.stack([colour_histogram("data/" + f) for f in test["file"]])
Xtr_both = np.hstack([Htr, Ctr])
Xte_both = np.hstack([Hte, Cte])
both = make_pipeline(StandardScaler(), LogisticRegression(C=0.5, max_iter=3000)).fit(Xtr_both, train["label"])
acc_both = cp.classification_scores(list(test["label"]), list(both.predict(Xte_both)), CLASSES)["accuracy"]
print(acc_both)
0.487

The check prints Check passed: acc_both is 0.487. The two features know different things, so together they beat each one alone. The candle recall stays at 0.10: neither feature describes a candle well.

Check your understanding

In the worked example, the change from left to right is −178 and from top to bottom is −111. What is the edge strength, and into which of HOG's 9 bins does the direction go?

Show the answer

The edge strength is √(178² + 111²) ≈ 209.8. The direction of the change is about 31.9°, after HOG folds all angles into 0–180°. So it goes into the bin 20–40°, the second of the nine bins of 20°.

Check your understanding

Under the warmer studio light, the histogram falls from 0.263 to 0.225, but HOG stays at 0.356. Why?

Show the answer

HOG works on the grey photo, and it only looks at changes in brightness between neighbours. It also scales each block of cells to the same size. A warmer or brighter light changes the colours and the overall brightness, but not much the places and directions of edges. The histogram counts colours, so it changes with the light.

Check your understanding

Omar asks: 'HOG and colour together reach 0.487. Can we add more hand-made features until we reach 0.9?' What do you answer?

Show the answer

More features would help a little, but each one needs an expert to design it, and each sees only what its author thought of. HOG describes edges at fixed places, so a product photographed from another angle or distance gives other numbers. The catalogue has photos from thousands of cameras and rooms. A method that learns its own features from many photos, as the next lessons show, is a better use of the time.

Recap

  • An edge is a quick change in brightness. Its strength and direction come from the differences between a pixel’s neighbours.
  • HOG counts edge directions in 9 bins per cell of 16 × 16 pixels, and scales blocks of 2 × 2 cells. A 128 × 128 photo becomes 1,764 numbers.
  • HOG with a linear model reaches 0.356, and does not move under the warmer light. With the colour histogram, it reaches 0.487. Both run in about 1 ms a photo, with no model file.
  • vision baseline --method hog adds HOG to your comparison table.
  • HOG describes edges at fixed places, so position, size and turning change its numbers, and it cannot know what an edge belongs to.

You can now describe shape with hand-made features and measure what they add. But a person had to design them, and they stop at 0.487. Next, you see how a network learns its own filters from photos. You also see how a vision transformer reads a photo in patches.

Next: Convolution and patches, in one lesson