Image embeddings
- Lesson 9
- 35 min
- Module 3: Embeddings, CLIP and few labels
Where we are
In Module 2, transfer learning trained a new last layer on top of a pretrained model. The layer before it already holds a useful list of numbers for each photo. In this lesson, we use that list, the embedding. We find similar photos, draw the catalogue on a map and find the classes that the models confuse. First, a warehouse worker who sorts by likeness.
Lesson 9 of 30 · Vision AI: See, Count and InspectImage embeddings
1. Seeing is a business problem
2. Baselines and pretrained classifiers
3. Embeddings, CLIP and few labels
4. Detect and count
6. Inspect for defects
7. Match, search and describe
8. Run vision in production
What you will be able to do
Turn each photo into a list of numbers that describes what it shows, so that similar photos get close numbers. With these lists, you can find similar photos, draw the whole catalogue on one map and see which classes a model mixes up.
NotebookDo this lesson in a notebook
The notebook has every code cell, task and check of this lesson. Its setup cells install the packages, with PyTorch and timm, and download the catalogue photos as zip files. It downloads ResNet-18 (47 MB) once. It needs no account and no key. About 30 minutes.
Colab and Kaggle open your own copy. They need a free Google or Kaggle account. Stuck? See the solution notebook.
Before you startWhat to know and to have ready
What you need to know
- The catalogue: 453 training, 120 validation and 160 test photos in eight classes, with only 15 training candles, from Meet the Larkfield images.
- A colour histogram with nearest neighbours, and why it reached only 0.263, from Rules and colour histograms.
- A pretrained model with a new last layer, from Transfer learning.
- For the idea of an embedding in text, Tokens and embeddings in the deep learning course helps, but you do not need it.
What you need on your computer
Nothing for the notebook route: Colab or Kaggle is enough. Your larkvision project does not change in this lesson. It gets new commands in the next two lessons.
Tested with Python 3.12.13 on macOS, in Jupyter. Colab, Kaggle and Windows were not tested.
Sorting by likeness
Section titled “Sorting by likeness”Hamid’s warehouse has a new worker. On her first day, a box of returns arrives with no paperwork. She must put each item on the right shelf: bowls, candles, mugs, vases. Nobody gave her a list of rules. Still, she sorts quickly. She looks at an item and thinks: “This looks like the things on the vase shelf.” She compares it with what she has seen before.
She does not compare the items pixel by pixel, or colour by colour. A green vase and a clear glass vase look very different in colour. She still puts them together, because she sees the same things: a tall shape, an open top, flowers in water. In her head, each item has a short description, and similar items have similar descriptions.
A computer can do this too, if it has a good description of each photo. This lesson is about that description.
The problem: pixels do not describe a product
Section titled “The problem: pixels do not describe a product”A catalogue photo of 256 × 256 pixels has 196,608 numbers: three colours for each pixel. Two photos of the same vase, taken a little to the left, have very different pixel numbers. Two photos of different products on the same white table can have very close pixel numbers. So the distance between pixel numbers does not measure whether two photos show the same kind of thing.
The colour histogram of Rules and colour histograms was a first fix. It counts colours, so the position of the product does not matter. But it knows only colour. Its nearest photos were often from the wrong class, and it classified only 0.263 of the test photos correctly. A green vase is close to a green plant, and a white mug is close to a white candle.
We need a description that knows about shapes, parts and objects, not only colours.
The idea: an embedding
Section titled “The idea: an embedding”In Transfer learning, you used a model that learned from more than a million ImageNet photos. Its last layer chooses one of 1,000 ImageNet classes. The layer before it is more useful for us. It holds 512 numbers for each photo. These numbers describe what the model sees: round or square, glass or fabric, a handle or no handle. The model learned these descriptions because it needed them to tell its 1,000 classes apart.
This list of numbers is called an embedding. The word means that each photo is placed, or “embedded”, as one point in a space with 512 directions. Photos that look alike become points that are close together. The model is not trained again: we only read the numbers. So we can use the same model for the catalogue, for customer photos, or for photos of a new product.
Text description of the diagram
The diagram has five boxes in a row (one under the other on a phone). The first box is a photo: 256 × 256 pixels and 3 colours, so 196,608 numbers. An arrow leads to a pretrained model: ResNet-18, CLIP or DINOv2, without its last layer. It is not trained here; it was trained before, on millions of other photos. The next box is the embedding: 512 numbers, such as 0.019, 0.100 and 0.109, scaled so that the list has length 1. Then the photos are compared with the cosine similarity, where 1 means the same direction. The last box lists the uses: the nearest photos, a map of the catalogue, and a small classifier.
This course uses three models that make embeddings. ResNet-18 is the network of Module 2. CLIP ViT-B/32 learned from photos with their captions; the next lesson uses its text part. DINOv2 small learned from photos alone, by comparing parts of the same photo. The course team ran all three on every catalogue photo once and stored the results in data/embeddings/. So most cells of this module run in seconds, with no model download.
A tiny worked example: the cosine similarity
Section titled “A tiny worked example: the cosine similarity”How do we measure “close”? Take two lists with only two numbers each, made up for this example: a = (0.6, 0.8) and b = (0.8, 0.6). Each list has length 1, because 0.6² + 0.8² = 0.36 + 0.64 = 1. Multiply the numbers in the same place, and add:
a · b = 0.6 × 0.8 + 0.8 × 0.6 = 0.48 + 0.48 = 0.96
This sum is the dot product. For two lists of length 1, it is the cosine similarity: the cosine of the angle between the two arrows. It is 1 for the same direction, 0 for a right angle, and −1 for opposite directions. Here 0.96 means “almost the same direction”: very similar. We scale every embedding to length 1 first, so that only the direction counts, not the length.
Now three real catalogue photos, with their CLIP embeddings:

Text description of the diagram
Photo A is a small glass vase with a few flowers, on a wooden table. Photo B is a round glass vase with green stems in clear water. Photo C is a black office chair with arms.
Each embedding has 512 numbers. Here are the first four of each, and the length of the whole list:
| Photo | First 4 of 512 numbers | Length |
|---|---|---|
| A (vase, test) | 0.019, 0.1, 0.109, 0.046, … | 1.0 |
| B (vase, train) | 0.028, 0.047, 0.015, 0.033, … | 1.0 |
| C (chair, train) | 0.064, -0.134, 0.113, -0.011, … | 1.0 |
The cosine is the same multiply-and-add, over all 512 places:
| A (vase) | B (vase) | C (chair) | |
|---|---|---|---|
| A (vase) | 1.000 | 0.832 | 0.447 |
| B (vase) | 0.832 | 1.000 | 0.478 |
| C (chair) | 0.447 | 0.478 | 1.000 |
The two vases have a cosine of 0.832, and each vase has less than 0.5 with the chair. The two vases have different shapes and different flowers, and their colours are not the same. The embedding still puts them close, because both show flowers in water in a clear glass. No single number of the 512 means “vase”. The meaning is in the whole list.
The nearest photos
Section titled “The nearest photos”With the cosine, you can ask: which training photos are the most similar to this test photo? Section 2 of the notebook does it with ResNet-18. The figure below does it with CLIP for two test photos:

Text description of the diagram
The figure has two rows. In the first row, the test photo is a vase with flowers. Its five nearest training photos are all vases with flowers in clear water, with cosines from 0.83 down to 0.78. In the second row, the test photo shows sunflowers in a white bucket; its label is plant pot. Its five nearest training photos are all labelled vase, with cosines from 0.68 down to 0.64, lower than in the first row. They are bunches of cut flowers in vases.
The first row works as the new worker would: every neighbour is a vase. The second row is more interesting. The test photo is labelled plant pot, but it shows cut sunflowers in a bucket. To CLIP, this looks like “flowers in a container”, and so do the vases. Its cosines are also lower: no training photo is really close to it. Keep this photo in mind. It comes back in the next lesson.
Classify Larkfield’s photos by the nearest photo
Section titled “Classify Larkfield’s photos by the nearest photo”The simplest classifier with embeddings copies the worker: give each test photo the label of its single most similar training photo. Section 5 of the notebook asks you to write it. The results on the 160 test photos:
| Description of each photo | Nearest-photo accuracy |
|---|---|
| Colour histogram (128 numbers) | 0.275 |
| ResNet-18 embedding (512 numbers, the notebook) | 0.831 |
| CLIP ViT-B/32 embedding (512 numbers) | 0.856 |
The same rule with a better description goes from about one photo in four to more than four in five. No model was trained: we only changed the numbers that describe each photo. This is why embeddings are the base of this whole module.
A map of the catalogue
Section titled “A map of the catalogue”512 numbers are too many to draw. To see the whole catalogue at once, we squeeze each embedding down to two numbers, an x and a y. Then we draw one dot per photo. Two common ways do this:
- PCA (principal component analysis) keeps the two directions in which the photos differ most. It is a straight projection, like the shadow of a 3-D object on a wall. The notebook uses it, because it is fast and simple.
- UMAP (uniform manifold approximation and projection) tries to keep each photo’s neighbours close on the map. It bends and stretches the space to do so. Groups come out clearer than with PCA.
The course team drew this map with UMAP from the CLIP embeddings of all 733 public photos:

Text description of the diagram
The map has one dot for each of the 733 photos, coloured by label, with the class name near each group. Cushions and chairs form one group at the top left: the cushions sit above the chairs and touch them. Lamps form a separate group at the top, and candles a small, tight group at the top right. On the right, bowls and mugs form a long band, and the bowls touch the mugs. A group of vases sits next to the bowls. At the bottom, vases and plant pots form one long band. The vases are at its top end, the plant pots at its bottom end, and the two classes mix where they meet.
Read the map for neighbours, not for distances. Candles and lamps form clean, separate groups: a model will rarely confuse them. Cushions touch chairs, bowls touch mugs, and plant pots run into vases. These are the places where a classifier will make mistakes. Do not read more than that. UMAP can move two far groups closer or further apart, and the axes mean nothing on their own.
Some photos sit inside another class’s group, for example a dot of another colour among the plant pots. Such a photo is either an unusual product or a wrong label. Labels: how many, and how good uses this idea to find label errors.
Which classes does the model confuse?
Section titled “Which classes does the model confuse?”The map shows the confusions with your eyes. A number makes them precise. The centre of a class is the mean of its embeddings, scaled back to length 1. The cosine between two centres says how close the two classes are on average. Section 4 of the notebook computes all 28 pairs for CLIP. These are the four closest pairs:
Text description of the chart
A column chart with four columns, on an axis from 0.7 to 0.9. Plant pot and vase: 0.864. Bowl and vase: 0.863. Chair and cushion: 0.833. Bowl and mug: 0.822. The plant pot and vase column is highlighted.
Plant pot and vase are the closest pair, with bowl and vase just behind them. Both are containers, often with plants or flowers in them, and often on a table. Chair and cushion come next: many chair photos have a cushion on the chair. These pairs are where Larkfield’s classifier will fail first, and where the labellers will disagree most.
When to use embeddings, and what they cost
Section titled “When to use embeddings, and what they cost”Use embeddings to compare photos: find similar products, find the same product twice, group photos, or train a small classifier with few labels. They are the start of the next three lessons and of Find the same product in Module 7.
They have limits and costs:
- The model decides what “similar” means. ImageNet and CLIP never learned Larkfield’s line between a plant pot and a vase. If your classes depend on a detail that the model does not care about, close embeddings will not separate them.
- Small details get lost. One embedding describes the whole photo. A small chip on a candle changes it very little. Module 6 shows that damage needs a method that looks at small parts of the photo.
- Computing them costs time once. On a laptop CPU, in batches of 32, ResNet-18 (47 MB) needs about 13 ms a photo. CLIP ViT-B/32 (605 MB) needs about 8 ms. After that, comparing two stored embeddings takes microseconds.
- Licences come with the model. Each model’s weights have their own licence and training data. Module 7 discusses what that means for a product.
Guided practice: embeddings in the notebook
Section titled “Guided practice: embeddings in the notebook”- Open the notebook from the box Do this lesson in a notebook at the top of this page.
- Run the setup cells. The second one ends with a
Ready:line. - Run section 1. It downloads ResNet-18 once and embeds the training and test photos. You see
(453, 512) (160, 512). - In section 2, predict whether all five neighbours of the test photo have its label. Then run the cell and look at the photos and their cosines.
- Run section 3. Compare the PCA maps of ResNet-18 and CLIP. Find the plant pots and the vases on each map.
- Run section 4. In the table, find the highest cosine that is not 1.000. It is plant pot and vase, 0.864.
Your turn: classify by the nearest photo
Section titled “Your turn: classify by the nearest photo”In section 5 of the notebook, give each test photo the label of its single most similar training photo (ResNet-18 embeddings Xtr and Xte). Put the share of right labels in nn_accuracy, then run the check cell.
Show the answer
nearest = (Xte @ Xtr.T).argmax(1)nn_accuracy = float(np.mean(train["label"].values[nearest] == test["label"].values))print(round(nn_accuracy, 3))0.831Xte @ Xtr.T is a table of 160 × 453 cosines: each test photo against each training photo. argmax(1) takes the most similar training photo in each row. The result, 0.831, is about the same as the ResNet-18 model of Module 2 with its new last layer (0.812). One nearest photo is a strong rule when the description is good. With 160 test photos, a difference of 2 points is about 3 photos, so treat the two as equal.
Common mistake
Two models, one table
Symptom: Tomás builds a quick product search. He stores the CLIP embeddings of the 453 training photos. Later, he embeds the new test photos with ResNet-18, because it is already loaded in his notebook. Both give 512 numbers, so the code runs with no error. The nearest-photo accuracy is 0.087: worse than guessing one of eight classes (0.125).
Cause: the 512 numbers of two models mean different things. Number 37 in CLIP and number 37 in ResNet-18 have nothing in common. A cosine between them compares unrelated lists. The shapes match, so nothing stops you.
Fix: embed every photo you compare with the same model, at the same revision, with the same preparation of the image. Store the model’s name with the embeddings. The course stores each model in its own file (data/embeddings/clip.npz, resnet18.npz), and the larkvision project loads them by name. With ResNet-18 for both sides, the accuracy is back to 0.831.
Consequence in AI work: when a team updates its embedding model, every stored embedding must be computed again. If not, the old and new embeddings are mixed, and searches and classifiers fail quietly.
Knowledge check
Section titled “Knowledge check”Check your understanding
In your own words: what is an image embedding, and why do two photos of vases with different colours get close embeddings, while their colour histograms are far apart?
Show the answer
An image embedding is a list of numbers, here 512, that a pretrained model computes for a photo. The model learned these numbers to tell many kinds of objects apart. So they describe shapes, parts and materials: tall, glass, flowers in water. Two vases share these, even when their colours differ. So their embeddings point in almost the same direction: cosine 0.832 in the worked example. A colour histogram counts only colours, so a green vase and a clear vase are far apart.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
On the UMAP map, the candles form a small group far from the other classes, and the plant pots run into the vases. What do you predict for a classifier on these embeddings? And what should you not conclude from the map?
Show the answer
Predict few mistakes for candles and lamps, and the most mistakes between plant pots and vases, then bowls and mugs, then chairs and cushions. The class centres agree: plant pot and vase are the closest pair (0.864). Do not read the distance between two far groups, or the axes: UMAP keeps neighbours close but can move far groups. Also, a small group on the map does not mean that a classifier finds every candle. A class with only 15 training photos can still be missed, as the next lessons show.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
Omar wants to find 'products that look like this one' for the website. Would you start with colour histograms or with embeddings? What is one limit you would tell him about?
Show the answer
Start with embeddings. With the same nearest-photo rule, they find the right class for 0.831 (ResNet-18) to 0.856 (CLIP) of the test photos. Colour histograms reach 0.275. One limit: the pretrained model decides what “similar” means. It may put a bucket of cut sunflowers next to vases, while Larkfield calls it a plant pot. Another limit: one embedding describes the whole photo, so small details such as a chip or a scratch change it very little.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Recap
- A pretrained model without its last layer turns a photo into an embedding: here 512 numbers. Similar photos get embeddings that point in similar directions.
- The cosine similarity of two embeddings of length 1 is their dot product: 1 is the same direction. Two vases: 0.832; a vase and a chair: under 0.5.
- The nearest training photo gives the right class for 0.831 of the test photos with ResNet-18 and 0.856 with CLIP. Colour histograms give 0.275. No model was trained.
- A UMAP map of the CLIP embeddings shows the groups and where they touch: plant pot and vase, bowl and mug, chair and cushion. Read it for neighbours, not for distances.
- Compare only embeddings of the same model. Mixed models give 0.087, with no error.
You can now describe every photo with an embedding and find its neighbours. But every method so far needs labelled training photos to compare with. Next, you classify photos with no labelled photo at all, by comparing them with a sentence.
Next: CLIP zero-shot