Skip to content
Français

Image embeddings

  • Lesson 9
  • 35 min
  • Module 3: Embeddings, CLIP and few labels

Where we are

In Module 2, transfer learning trained a new last layer on top of a pretrained model. The layer before it already holds a useful list of numbers for each photo. In this lesson, we use that list, the embedding. We find similar photos, draw the catalogue on a map and find the classes that the models confuse. First, a warehouse worker who sorts by likeness.

Plays from YouTube. Nothing loads from YouTube until you press play. Privacy

What you will be able to do

Turn each photo into a list of numbers that describes what it shows, so that similar photos get close numbers. With these lists, you can find similar photos, draw the whole catalogue on one map and see which classes a model mixes up.

NotebookDo this lesson in a notebookColab · Kaggle · GitHub

The notebook has every code cell, task and check of this lesson. Its setup cells install the packages, with PyTorch and timm, and download the catalogue photos as zip files. It downloads ResNet-18 (47 MB) once. It needs no account and no key. About 30 minutes.

Colab and Kaggle open your own copy. They need a free Google or Kaggle account. Stuck? See the solution notebook.

Before you startWhat to know and to have readyAbout 35 minutes

What you need to know

What you need on your computer

Nothing for the notebook route: Colab or Kaggle is enough. Your larkvision project does not change in this lesson. It gets new commands in the next two lessons.

Tested with Python 3.12.13 on macOS, in Jupyter. Colab, Kaggle and Windows were not tested.

Hamid’s warehouse has a new worker. On her first day, a box of returns arrives with no paperwork. She must put each item on the right shelf: bowls, candles, mugs, vases. Nobody gave her a list of rules. Still, she sorts quickly. She looks at an item and thinks: “This looks like the things on the vase shelf.” She compares it with what she has seen before.

She does not compare the items pixel by pixel, or colour by colour. A green vase and a clear glass vase look very different in colour. She still puts them together, because she sees the same things: a tall shape, an open top, flowers in water. In her head, each item has a short description, and similar items have similar descriptions.

A computer can do this too, if it has a good description of each photo. This lesson is about that description.

The problem: pixels do not describe a product

Section titled “The problem: pixels do not describe a product”

A catalogue photo of 256 × 256 pixels has 196,608 numbers: three colours for each pixel. Two photos of the same vase, taken a little to the left, have very different pixel numbers. Two photos of different products on the same white table can have very close pixel numbers. So the distance between pixel numbers does not measure whether two photos show the same kind of thing.

The colour histogram of Rules and colour histograms was a first fix. It counts colours, so the position of the product does not matter. But it knows only colour. Its nearest photos were often from the wrong class, and it classified only 0.263 of the test photos correctly. A green vase is close to a green plant, and a white mug is close to a white candle.

We need a description that knows about shapes, parts and objects, not only colours.

In Transfer learning, you used a model that learned from more than a million ImageNet photos. Its last layer chooses one of 1,000 ImageNet classes. The layer before it is more useful for us. It holds 512 numbers for each photo. These numbers describe what the model sees: round or square, glass or fabric, a handle or no handle. The model learned these descriptions because it needed them to tell its 1,000 classes apart.

This list of numbers is called an embedding. The word means that each photo is placed, or “embedded”, as one point in a space with 512 directions. Photos that look alike become points that are close together. The model is not trained again: we only read the numbers. So we can use the same model for the catalogue, for customer photos, or for photos of a new product.

Five boxes from left to right. A photo: 256 by 256 pixels times 3 colours equals 196,608 numbers. A pretrained model: ResNet-18, CLIP or DINOv2 without its last layer, no training here; a note says it was trained on millions of other photos. The embedding: 512 numbers, for example 0.019, 0.100, 0.109, scaled to length 1. Compare: cosine similarity with other photos, 1 means the same direction. Use it: nearest photos, a map of the catalogue, a small classifier.
From a photo to an embedding. The pretrained model is only used, not trained. The embedding is the input of the methods of this module.
Text description of the diagram

The diagram has five boxes in a row (one under the other on a phone). The first box is a photo: 256 × 256 pixels and 3 colours, so 196,608 numbers. An arrow leads to a pretrained model: ResNet-18, CLIP or DINOv2, without its last layer. It is not trained here; it was trained before, on millions of other photos. The next box is the embedding: 512 numbers, such as 0.019, 0.100 and 0.109, scaled so that the list has length 1. Then the photos are compared with the cosine similarity, where 1 means the same direction. The last box lists the uses: the nearest photos, a map of the catalogue, and a small classifier.

This course uses three models that make embeddings. ResNet-18 is the network of Module 2. CLIP ViT-B/32 learned from photos with their captions; the next lesson uses its text part. DINOv2 small learned from photos alone, by comparing parts of the same photo. The course team ran all three on every catalogue photo once and stored the results in data/embeddings/. So most cells of this module run in seconds, with no model download.

A tiny worked example: the cosine similarity

Section titled “A tiny worked example: the cosine similarity”

How do we measure “close”? Take two lists with only two numbers each, made up for this example: a = (0.6, 0.8) and b = (0.8, 0.6). Each list has length 1, because 0.6² + 0.8² = 0.36 + 0.64 = 1. Multiply the numbers in the same place, and add:

a · b = 0.6 × 0.8 + 0.8 × 0.6 = 0.48 + 0.48 = 0.96

This sum is the dot product. For two lists of length 1, it is the cosine similarity: the cosine of the angle between the two arrows. It is 1 for the same direction, 0 for a right angle, and −1 for opposite directions. Here 0.96 means “almost the same direction”: very similar. We scale every embedding to length 1 first, so that only the direction counts, not the length.

Now three real catalogue photos, with their CLIP embeddings:

Three photos side by side. A: a glass vase with flowers on a wooden table, a test photo. B: a round glass bowl vase with stems in water, a training photo. C: a black office chair, a training photo.
The three photos of the worked example: A is a test photo, B and C are training photos. Photos: Zeborah, Paul-in-London, TedsBlog, CC BY 2.0, from Open Images.
Text description of the diagram

Photo A is a small glass vase with a few flowers, on a wooden table. Photo B is a round glass vase with green stems in clear water. Photo C is a black office chair with arms.

Each embedding has 512 numbers. Here are the first four of each, and the length of the whole list:

Photo First 4 of 512 numbers Length
A (vase, test) 0.019, 0.1, 0.109, 0.046, … 1.0
B (vase, train) 0.028, 0.047, 0.015, 0.033, … 1.0
C (chair, train) 0.064, -0.134, 0.113, -0.011, … 1.0

The cosine is the same multiply-and-add, over all 512 places:

A (vase) B (vase) C (chair)
A (vase) 1.000 0.832 0.447
B (vase) 0.832 1.000 0.478
C (chair) 0.447 0.478 1.000

The two vases have a cosine of 0.832, and each vase has less than 0.5 with the chair. The two vases have different shapes and different flowers, and their colours are not the same. The embedding still puts them close, because both show flowers in water in a clear glass. No single number of the 512 means “vase”. The meaning is in the whole list.

With the cosine, you can ask: which training photos are the most similar to this test photo? Section 2 of the notebook does it with ResNet-18. The figure below does it with CLIP for two test photos:

Two rows of six photos. Row 1: the test photo, a vase with flowers, framed in blue. Then five training photos, all vases with flowers in clear water, cosines 0.83, 0.81, 0.81, 0.78 and 0.78. Row 2: the test photo, sunflowers in a white bucket labelled plant pot, framed in blue. Then five training photos, all labelled vase, cosines 0.68, 0.68, 0.66, 0.64 and 0.64, in red.
Each row: a test photo (framed) and its five most similar training photos by CLIP embedding, with the cosine. Red: a label different from the test photo's. Photos, row by row: Zeborah, Paul-in-London, jessicahtam, Jake Spurlock, MattCooper, Design Lively, mk97007, storebukkebruse, Lee Coursey, Design Lively, kaythaney, ProFlowers, CC BY 2.0, from Open Images.
Text description of the diagram

The figure has two rows. In the first row, the test photo is a vase with flowers. Its five nearest training photos are all vases with flowers in clear water, with cosines from 0.83 down to 0.78. In the second row, the test photo shows sunflowers in a white bucket; its label is plant pot. Its five nearest training photos are all labelled vase, with cosines from 0.68 down to 0.64, lower than in the first row. They are bunches of cut flowers in vases.

The first row works as the new worker would: every neighbour is a vase. The second row is more interesting. The test photo is labelled plant pot, but it shows cut sunflowers in a bucket. To CLIP, this looks like “flowers in a container”, and so do the vases. Its cosines are also lower: no training photo is really close to it. Keep this photo in mind. It comes back in the next lesson.

Classify Larkfield’s photos by the nearest photo

Section titled “Classify Larkfield’s photos by the nearest photo”

The simplest classifier with embeddings copies the worker: give each test photo the label of its single most similar training photo. Section 5 of the notebook asks you to write it. The results on the 160 test photos:

Description of each photo Nearest-photo accuracy
Colour histogram (128 numbers) 0.275
ResNet-18 embedding (512 numbers, the notebook) 0.831
CLIP ViT-B/32 embedding (512 numbers) 0.856

The same rule with a better description goes from about one photo in four to more than four in five. No model was trained: we only changed the numbers that describe each photo. This is why embeddings are the base of this whole module.

512 numbers are too many to draw. To see the whole catalogue at once, we squeeze each embedding down to two numbers, an x and a y. Then we draw one dot per photo. Two common ways do this:

  • PCA (principal component analysis) keeps the two directions in which the photos differ most. It is a straight projection, like the shadow of a 3-D object on a wall. The notebook uses it, because it is fast and simple.
  • UMAP (uniform manifold approximation and projection) tries to keep each photo’s neighbours close on the map. It bends and stretches the space to do so. Groups come out clearer than with PCA.

The course team drew this map with UMAP from the CLIP embeddings of all 733 public photos:

A scatter plot of 733 dots in eight colours, one per class. Cushion and chair form one group at the top left, with the cushions above the chairs. Lamp is a separate group at the top, candle a small group at the top right. On the right, bowl and mug form a long band that touches, with a group of vases next to the bowls. At the bottom, the vases and the plant pots form one long band, and they mix where they meet.
The 733 public catalogue photos on a UMAP map of their CLIP embeddings. Close dots are similar photos. The axes have no unit, and the distance between two far groups means little.
Text description of the diagram

The map has one dot for each of the 733 photos, coloured by label, with the class name near each group. Cushions and chairs form one group at the top left: the cushions sit above the chairs and touch them. Lamps form a separate group at the top, and candles a small, tight group at the top right. On the right, bowls and mugs form a long band, and the bowls touch the mugs. A group of vases sits next to the bowls. At the bottom, vases and plant pots form one long band. The vases are at its top end, the plant pots at its bottom end, and the two classes mix where they meet.

Read the map for neighbours, not for distances. Candles and lamps form clean, separate groups: a model will rarely confuse them. Cushions touch chairs, bowls touch mugs, and plant pots run into vases. These are the places where a classifier will make mistakes. Do not read more than that. UMAP can move two far groups closer or further apart, and the axes mean nothing on their own.

Some photos sit inside another class’s group, for example a dot of another colour among the plant pots. Such a photo is either an unusual product or a wrong label. Labels: how many, and how good uses this idea to find label errors.

The map shows the confusions with your eyes. A number makes them precise. The centre of a class is the mean of its embeddings, scaled back to length 1. The cosine between two centres says how close the two classes are on average. Section 4 of the notebook computes all 28 pairs for CLIP. These are the four closest pairs:

The four pairs of classes whose CLIP centres are closest (Pot: plant pot, cush.: cushion). The axis starts at 0.7, so the differences look larger than they are.
Text description of the chart

A column chart with four columns, on an axis from 0.7 to 0.9. Plant pot and vase: 0.864. Bowl and vase: 0.863. Chair and cushion: 0.833. Bowl and mug: 0.822. The plant pot and vase column is highlighted.

Plant pot and vase are the closest pair, with bowl and vase just behind them. Both are containers, often with plants or flowers in them, and often on a table. Chair and cushion come next: many chair photos have a cushion on the chair. These pairs are where Larkfield’s classifier will fail first, and where the labellers will disagree most.

When to use embeddings, and what they cost

Section titled “When to use embeddings, and what they cost”

Use embeddings to compare photos: find similar products, find the same product twice, group photos, or train a small classifier with few labels. They are the start of the next three lessons and of Find the same product in Module 7.

They have limits and costs:

  • The model decides what “similar” means. ImageNet and CLIP never learned Larkfield’s line between a plant pot and a vase. If your classes depend on a detail that the model does not care about, close embeddings will not separate them.
  • Small details get lost. One embedding describes the whole photo. A small chip on a candle changes it very little. Module 6 shows that damage needs a method that looks at small parts of the photo.
  • Computing them costs time once. On a laptop CPU, in batches of 32, ResNet-18 (47 MB) needs about 13 ms a photo. CLIP ViT-B/32 (605 MB) needs about 8 ms. After that, comparing two stored embeddings takes microseconds.
  • Licences come with the model. Each model’s weights have their own licence and training data. Module 7 discusses what that means for a product.

Guided practice: embeddings in the notebook

Section titled “Guided practice: embeddings in the notebook”
  1. Open the notebook from the box Do this lesson in a notebook at the top of this page.
  2. Run the setup cells. The second one ends with a Ready: line.
  3. Run section 1. It downloads ResNet-18 once and embeds the training and test photos. You see (453, 512) (160, 512).
  4. In section 2, predict whether all five neighbours of the test photo have its label. Then run the cell and look at the photos and their cosines.
  5. Run section 3. Compare the PCA maps of ResNet-18 and CLIP. Find the plant pots and the vases on each map.
  6. Run section 4. In the table, find the highest cosine that is not 1.000. It is plant pot and vase, 0.864.

In section 5 of the notebook, give each test photo the label of its single most similar training photo (ResNet-18 embeddings Xtr and Xte). Put the share of right labels in nn_accuracy, then run the check cell.

Show the answer
nearest = (Xte @ Xtr.T).argmax(1)
nn_accuracy = float(np.mean(train["label"].values[nearest] == test["label"].values))
print(round(nn_accuracy, 3))
0.831

Xte @ Xtr.T is a table of 160 × 453 cosines: each test photo against each training photo. argmax(1) takes the most similar training photo in each row. The result, 0.831, is about the same as the ResNet-18 model of Module 2 with its new last layer (0.812). One nearest photo is a strong rule when the description is good. With 160 test photos, a difference of 2 points is about 3 photos, so treat the two as equal.

Common mistake

Two models, one table

Symptom: Tomás builds a quick product search. He stores the CLIP embeddings of the 453 training photos. Later, he embeds the new test photos with ResNet-18, because it is already loaded in his notebook. Both give 512 numbers, so the code runs with no error. The nearest-photo accuracy is 0.087: worse than guessing one of eight classes (0.125).

Cause: the 512 numbers of two models mean different things. Number 37 in CLIP and number 37 in ResNet-18 have nothing in common. A cosine between them compares unrelated lists. The shapes match, so nothing stops you.

Fix: embed every photo you compare with the same model, at the same revision, with the same preparation of the image. Store the model’s name with the embeddings. The course stores each model in its own file (data/embeddings/clip.npz, resnet18.npz), and the larkvision project loads them by name. With ResNet-18 for both sides, the accuracy is back to 0.831.

Consequence in AI work: when a team updates its embedding model, every stored embedding must be computed again. If not, the old and new embeddings are mixed, and searches and classifiers fail quietly.

Check your understanding

In your own words: what is an image embedding, and why do two photos of vases with different colours get close embeddings, while their colour histograms are far apart?

Show the answer

An image embedding is a list of numbers, here 512, that a pretrained model computes for a photo. The model learned these numbers to tell many kinds of objects apart. So they describe shapes, parts and materials: tall, glass, flowers in water. Two vases share these, even when their colours differ. So their embeddings point in almost the same direction: cosine 0.832 in the worked example. A colour histogram counts only colours, so a green vase and a clear vase are far apart.

Check your understanding

On the UMAP map, the candles form a small group far from the other classes, and the plant pots run into the vases. What do you predict for a classifier on these embeddings? And what should you not conclude from the map?

Show the answer

Predict few mistakes for candles and lamps, and the most mistakes between plant pots and vases, then bowls and mugs, then chairs and cushions. The class centres agree: plant pot and vase are the closest pair (0.864). Do not read the distance between two far groups, or the axes: UMAP keeps neighbours close but can move far groups. Also, a small group on the map does not mean that a classifier finds every candle. A class with only 15 training photos can still be missed, as the next lessons show.

Check your understanding

Omar wants to find 'products that look like this one' for the website. Would you start with colour histograms or with embeddings? What is one limit you would tell him about?

Show the answer

Start with embeddings. With the same nearest-photo rule, they find the right class for 0.831 (ResNet-18) to 0.856 (CLIP) of the test photos. Colour histograms reach 0.275. One limit: the pretrained model decides what “similar” means. It may put a bucket of cut sunflowers next to vases, while Larkfield calls it a plant pot. Another limit: one embedding describes the whole photo, so small details such as a chip or a scratch change it very little.

Recap

  • A pretrained model without its last layer turns a photo into an embedding: here 512 numbers. Similar photos get embeddings that point in similar directions.
  • The cosine similarity of two embeddings of length 1 is their dot product: 1 is the same direction. Two vases: 0.832; a vase and a chair: under 0.5.
  • The nearest training photo gives the right class for 0.831 of the test photos with ResNet-18 and 0.856 with CLIP. Colour histograms give 0.275. No model was trained.
  • A UMAP map of the CLIP embeddings shows the groups and where they touch: plant pot and vase, bowl and mug, chair and cushion. Read it for neighbours, not for distances.
  • Compare only embeddings of the same model. Mixed models give 0.087, with no error.

You can now describe every photo with an embedding and find its neighbours. But every method so far needs labelled training photos to compare with. Next, you classify photos with no labelled photo at all, by comparing them with a sentence.

Next: CLIP zero-shot