Find the same product
- Lesson 23
- 40 min
- Module 7: Match, search and describe
Where we are
In Module 3, embeddings helped us find classes and label errors. They told us what kind of product a photo shows. Now a customer sends a photo, and we must find the one product it shows. In this lesson, we match phone photos to the catalogue with nearest neighbours, measure recall at k, and find near-duplicate photos. First, a helper at a busy shop counter.
Lesson 23 of 30 · Vision AI: See, Count and InspectFind the same product
1. Seeing is a business problem
2. Baselines and pretrained classifiers
3. Embeddings, CLIP and few labels
4. Detect and count
6. Inspect for defects
7. Match, search and describe
8. Run vision in production
What you will be able to do
Find which catalogue product a customer’s photo shows, and find catalogue photos that were uploaded twice. Both are searches by likeness, and both are measured by how often the right answer is near the top.
NotebookDo this lesson in a notebook
The notebook has the search, a miss to look at, and one task with a check. Its setup cells install the packages and download the catalogue and the customer photos as zip files. It needs no account, no key and no model download: the embeddings are in the data. About 25 minutes.
Colab and Kaggle open your own copy. They need a free Google or Kaggle account. Stuck? See the solution notebook.
Before you startWhat to know and to have ready
What you need to know
- Image embeddings and the cosine similarity, from Image embeddings. This lesson uses them as a search.
- The planted second uploads in the catalogue, from Labels: how many, and how good.
What you need on your computer
- The notebook route: Colab, Kaggle or Jupyter. Nothing to install.
- The project route: your
larkvisionfolder besidenextia-learning-labs, at the snapshotend-of-m06, with its.venv, as set up in Set the line with the warehouse.python -m pytest -qprints22 passed. This lesson adds one file.
Tested with Python 3.12.13 on macOS, in Jupyter and in a terminal. The Windows commands, Linux, Colab and Kaggle were not tested.
The helper at the counter
Section titled “The helper at the counter”A customer comes to a shop counter with a phone. “I bought this bowl here last year. I want two more.” She shows a photo, taken at home, in yellow kitchen light, from above. The helper does not read the bowl’s name anywhere. She looks at the photo, then walks along the shelves. She stops at the three bowls that look most like it, and she holds the phone next to each one. After a minute, she finds it.
She did two things. First, she searched by likeness: she did not check every item, only the few that looked closest. Second, she checked: she compared the photo with each candidate before she answered. A good search does the first part fast and leaves the second part to a person.
The problem: the same product, a different photo
Section titled “The problem: the same product, a different photo”Larkfield gets such photos every day. A customer writes, “this arrived broken”, or “I want the same one”, and attaches a phone photo. Someone must find the product among the 733 catalogue photos. The course’s 160 customer photos stand in for them. Our code made each one from a catalogue test photo. It changed the angle, made the product smaller on a table, and added poor light, a colour cast, blur and noise. They are synthetic, and each one has exactly one right catalogue photo.
The first idea is to compare the photos directly. Two simple methods do this:
- The colour histogram of Rules and colour histograms counts the colours of each photo.
- A perceptual hash turns each photo into 64 bits. This lesson explains it in its second half.
Both fail. The colour histogram finds the right photo in its first five answers for 0.087 of the customer photos: about 1 in 11. The perceptual hash does so for 0.019: 3 photos in 160. The customer’s yellow kitchen light changes every colour, and the new angle moves every part of the product. A description of the pixels cannot find the same product in a new photo.
The idea: search with embeddings
Section titled “The idea: search with embeddings”You already have a better description. An embedding describes what a photo shows, not its pixels, and similar photos get embeddings that point in a similar direction. So the search works like the helper at the counter:
- Store the embedding of every catalogue photo once.
- When a customer photo arrives, compute its embedding with the same model.
- Compute the cosine similarity with every stored embedding.
- Show the catalogue photos with the highest cosines, best first.
This is called nearest-neighbour search: the answer is the stored items nearest to the question. The list of the first k answers is the top k. For 733 photos, the cosines take less than a millisecond.
Here is one customer photo of a white bowl with a black spoon, and the top 5 of two models:

Text description of the diagram
The figure has two rows, one for each model. Each row starts with the same customer photo: a small white bowl with a black spoon, on a grey card, a little blurred and tilted. In the first row, DINOv2 small puts the right catalogue photo first, with a cosine of 0.75. The next four answers are other bowls and a cup, with cosines from 0.58 down to 0.47: far below the first one. In the second row, CLIP returns five white bowls and cups with cosines from 0.65 to 0.61. They are all close to each other, and none of them is the right bowl. CLIP ranks the right bowl sixth.
Both models found white bowls. Only DINOv2 found this white bowl. Look at the cosines too. DINOv2’s first answer is far ahead of its second (0.75 against 0.58). CLIP’s five answers are almost equal. A clear gap between the first and the second answer is a useful sign that the match is real.
A tiny worked example: recall at k
Section titled “A tiny worked example: recall at k”How do we score a search? For each customer photo, find the rank of the right catalogue photo: 1 if it is first, 2 if it is second, and so on. Recall at k is the share of customer photos whose right photo has a rank of k or better. Here are the first four customer photos with CLIP:
| Customer photo | Product | Rank of the right photo (CLIP) | In the top 1? | In the top 5? |
|---|---|---|---|---|
| C-9cb24dd8 | bowl | 1 | yes | yes |
| C-e8d1feae | bowl (the figure) | 6 | no | no |
| C-e4d89d21 | bowl | 1 | yes | yes |
| C-2218b07f | bowl | 1 | yes | yes |
Three of the four right photos are first, so the recall at 1 is 3 / 4 = 0.75. The recall at 5 is also 3 / 4 = 0.75, because rank 6 is outside the top 5. The recall at 10 is 4 / 4 = 1.0. A larger k always gives the same recall or a higher one. Choose k from the way people use the result. If a person looks at 5 suggestions, measure recall at 5.
The results on 160 customer photos
Section titled “The results on 160 customer photos”The course team ran five methods on the same 160 customer photos and the same 733 catalogue photos:
Text description of the chart
A column chart with five columns on an axis from 0 to 1. Colour histogram: 0.087. Perceptual hash: 0.019. ResNet-18 embeddings: 0.756. CLIP embeddings: 0.756. DINOv2 small embeddings: 0.988, highlighted. The two pixel methods are near zero, and DINOv2 is far above the other two embeddings.
| Method | Recall at 1 | Recall at 5 | Worst product (at 5) | Model size | CPU time a photo |
|---|---|---|---|---|---|
| Colour histogram | 0.019 | 0.087 | bowl (0.05) | none | 0.86 ms |
| Perceptual hash | 0.013 | 0.019 | bowl (0.00) | none | 2.94 ms |
| ResNet-18 | 0.581 | 0.756 | plant pot (0.45) | 47 MB | 13.7 ms |
| CLIP ViT-B/32 | 0.562 | 0.756 | chair (0.65) | 605 MB | 7.67 ms |
| DINOv2 small | 0.944 | 0.988 | cushion (0.95) | 88 MB | 13.64 ms |
The times are for one photo in a batch of 32, with 4 threads, on the course laptop. With DINOv2, the right photo is first for 151 of the 160 customer photos. It is in places 2 to 5 for 7, and in places 6 to 10 for 2. With CLIP, it is first for only 90, and 22 right photos are not even in the top 10.
Why the smaller model wins
Section titled “Why the smaller model wins”Tomás is surprised. “CLIP has 605 MB, and DINOv2 small only 88 MB. A bigger model should see better.” Priya explains that the two models learned different things.
CLIP learned from photos with their captions. A caption says “a white bowl on a table”, not which white bowl. So CLIP learned to tell the kind of thing apart, and it puts all white bowls close together.
DINOv2 learned from photos alone. It compared different crops of the same photo, and learned to recognise that they show the same object. So it keeps the details of one object: the shape of the rim, the angle of the spoon. That is exactly what finding the same product needs.
So the question decides the model. For “what kind of product is this?”, choose CLIP. For “which product is this?”, choose DINOv2. The size of the model did not decide it.
Add the search to your project
Section titled “Add the search to your project”The project gets the search as a command, vision match. It reads the stored embeddings of the course’s data, so it needs no model download. Check the place first: you must be in your larkvision folder, with the virtual environment active.
-
Open a terminal and go to your
larkvisionfolder. Activate the virtual environment:source .venv/bin/activateon macOS and Linux,.venv\Scripts\Activate.ps1on Windows. -
If this is a new terminal, set
LARKVISION_DATAagain. It lives only in the terminal where you set it:Terminal window export LARKVISION_DATA="$(cd ../nextia-learning-labs/C42/data && pwd)"Terminal window $env:LARKVISION_DATA = (Resolve-Path ..\nextia-learning-labs\C42\data).Path -
Copy the file of this lesson from the snapshot
end-of-m07:Terminal window S=../nextia-learning-labs/C42/snapshots/end-of-m07cp "$S/larkvision/commands/match.py" larkvision/commands/match.pyTerminal window $S = "..\nextia-learning-labs\C42\snapshots\end-of-m07"Copy-Item "$S\larkvision\commands\match.py" larkvision\commands\match.py -
Run the tests. You still see
22 passed: the test of the new command comes with Check what a vision-language model says.Terminal window python -m pytest -q -
Search with DINOv2, and show the top 5 of the first three customer photos:
Terminal window vision match --embeddings dinov2 --show 3 -
Search with CLIP:
Terminal window vision match --embeddings clip --show 3
Step 5 prints:
DINOv2 small: 160 customer photos against 733 catalogue photos; recall at 1 0.944, at 5 0.988 C-9cb24dd8 (bowl): *P-9a3fc835, P-558af3a2, P-75598db9, P-60f9ab1c, P-aef88230 C-e8d1feae (bowl): *P-9d48a6c9, P-593a4ac1, P-1815e7a4, P-080e2d04, P-9650ea1d C-e4d89d21 (bowl): *P-9e093eaa, P-8fff6af0, P-0782f68e, P-3dd314fa, P-a8d9684cStep 6 prints:
CLIP ViT-B/32 (OpenCLIP, laion2b_s34b_b79k): 160 customer photos against 733 catalogue photos; recall at 1 0.562, at 5 0.756 C-9cb24dd8 (bowl): *P-9a3fc835, P-7308e8ee, P-99a52778, P-a0ca8a5a, P-1c7efe8e C-e8d1feae (bowl): P-9e093eaa, P-9650ea1d, P-614eb253, P-78abb816, P-080e2d04 C-e4d89d21 (bowl): *P-9e093eaa, P-7308e8ee, P-3dd314fa, P-0152f9f0, P-4ef55c7eA star marks the right catalogue photo. The second line of each output is the customer photo of the figure above. Each run adds a row to your own table: vision compare --task match shows it at the end, marked “(yours)”. vision match --k 10 measures recall at 10. larkvision/commands/match.py holds the function rank: the same four steps as above, in four lines of NumPy.
Photos uploaded twice: perceptual hashes
Section titled “Photos uploaded twice: perceptual hashes”Matching has a second use inside the catalogue. In Labels: how many, and how good, you met six training photos that are in the catalogue twice, cropped a little and brighter. This is a problem we planted. A photo in the catalogue twice can end up in both the training set and the test set. Then the test score is too good.
For near-duplicates like these, a smaller tool than an embedding is often enough: a perceptual hash. The pHash method does four things:
- Shrink the photo to 32 × 32 pixels in grey.
- Describe it as a sum of smooth patterns, from very smooth to fine. (The maths is the discrete cosine transform, also used in JPEG files.)
- Keep the 8 × 8 smoothest patterns: the rough layout of light and dark.
- Write 1 for each pattern above the median value, and 0 for the others. The result is 64 bits.
Two hashes are compared by the Hamming distance: the number of bits that differ. Take two short hashes of 8 bits, made up for this example: 10110010 and 10010011. They differ in the third bit and in the last bit, so the distance is 2. Same photo: about 0. Two unrelated photos: about half of the bits, so about 32 of 64.

Text description of the diagram
The figure has two rows. The top row shows the two uploads of the same photo of a black armchair on a carpet. The second upload is cropped a little and brighter. Next to them are their two hashes, each an 8 × 8 grid of black and white squares. The grids are almost the same: only 6 of the 64 squares differ, outlined in red. The bottom row shows the customer photo of the white bowl and its own catalogue photo. Their grids differ in 30 of the 64 squares, which is what two unrelated photos usually give.
The planted second uploads differ by 6 to 18 bits. So the search for duplicates compares every pair of the 453 training photos and keeps the pairs under a line:
| Keep pairs with at most | Pairs found | Planted second uploads among them |
|---|---|---|
| 4 bits | 0 | 0 of 6 |
| 8 bits | 2 | 2 of 6 |
| 12 bits | 6 | 3 of 6 |
| 16 bits | 45 | 5 of 6 |
At 8 bits, both pairs found are planted ones: no false pair, but only 2 of 6. At 16 bits, 5 of 6 are found among 45 pairs, so a person must look at 45 pairs. This is the usual trade: a looser line finds more and asks for more checking. In Module 3, the embeddings found 5 or 6 of the 6 planted pairs among their 10 most similar pairs.
Now the bottom row of the figure. A customer photo and its own catalogue photo differ by a median of 30 bits. Unrelated photos differ by 32. Not one of the 160 customer photos is within 16 bits of its own photo. A pHash survives a small crop and a change of brightness. It does not survive a new angle or a new photo. That is why it scored 0.019 above.
So a pHash finds copies of the same photo: it is fast and needs no model. An embedding finds the same product in a new photo. For pairs and groups of records that may be the same thing, the record matching course goes further. It is not on the site yet.
Common mistake
The product that is not in the catalogue
Symptom: Omar tests the search with a photo of a teapot. Larkfield does not sell teapots. The search still returns five products, with a confident first answer. The support team sends the customer a link to a vase.
Cause: a nearest-neighbour search always returns the nearest items, even when none of them is right. It never says “none of these”. The course team measured how bad this is: for each customer photo, it removed the right photo from the catalogue and searched again. With DINOv2, the median cosine of the right photo is 0.673. When the right photo is missing, the best remaining one still has a median of 0.515.
Fix: add a “no match” line: below a cosine, the answer is “not found, ask a person”. A line at 0.469 keeps 95% of the right matches. But 64% of the searches for a missing product still pass it. For CLIP, 99.4% pass. So a line alone is weak. Show the top 5 to a person, and keep a “none of these” button for them.
Consequence in AI work: every search and every classifier answers, whatever you give it. Before you use the answer, ask what the system does with an input that has no right answer. Measure it, as here, by hiding the right answer on purpose.
When to use a search, and what it costs
Section titled “When to use a search, and what it costs”Use an embedding search when a person needs a short list of candidates. Examples: “which product is this?” or “show similar products”. It needs no labels: DINOv2 found the right product first for 0.944 of the customer photos with no training at all. The cost is one embedding per photo, about 14 ms on a laptop CPU, and a few kilobytes to store each one.
Do not use it as the final decision. A match is a suggestion. Two products can look almost the same and differ in size or price, and the customer’s product may not be in the catalogue. The candle is DINOv2’s weakest product at 1: 85% of the candle photos find the right candle first, because many candles look alike. When an order number or a barcode is available, use it first. When not to build it comes back to this.
Guided practice: search in the notebook
Section titled “Guided practice: search in the notebook”- Open the notebook from the box Do this lesson in a notebook at the top of this page.
- Run the setup cells. The second one ends with a
Ready:line. - Run section 1. It shows three customer photos next to their catalogue photos.
- Before you run section 2, predict which of the three models finds the right photo first most often.
- Run section 2. Compare its three lines with the table above.
- Run section 3. It shows a photo that CLIP misses. Look at its first three answers: what do they have in common with the customer photo?
Your turn: recall at 5 for DINOv2
Section titled “Your turn: recall at 5 for DINOv2”In section 4 of the notebook, compute the recall at 5 of the DINOv2 embeddings with cp.recall_at_k. Put it in dinov2_r5, and run the check cell. Then change one condition: compute the recall at 1 and at 10 for ResNet-18. Which k would you show to the support team?
Show the answer
dinov2_r5 = cp.recall_at_k(ranked("dinov2"), truth, 5)print(dinov2_r5)0.9875The check rounds it to 0.988: 158 of the 160 right photos are in DINOv2’s top 5. For ResNet-18, cp.recall_at_k(ranked("resnet18"), truth, 1) gives 0.581 and k=10 gives 0.831. With ResNet-18, a person would need about 10 suggestions to find most products. With DINOv2, 5 are enough, and the first one is right 94.4% of the time.
Knowledge check
Section titled “Knowledge check”Check your understanding
Recall at 1 is 0.562 for CLIP and 0.944 for DINOv2 on the same customer photos. CLIP was the best model for classes in Module 3. Explain the difference in your own words.
Show the answer
The two models answer different questions. CLIP learned from photos with captions, so its embedding describes the kind of thing: “a white bowl”. All white bowls get close embeddings, and the search returns white bowls, not this bowl. DINOv2 learned from photos alone, by matching crops of the same photo, so its embedding keeps the details of one object. Finding the same product needs those details. Classes need the kind. The model’s size (605 MB against 88 MB) did not decide it.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
Your search shows 5 suggestions to the support team. Which number do you report: recall at 1, at 5 or at 10? And why does a recall at 5 of 0.988 not mean that the search is right 98.8% of the time?
Show the answer
Report recall at 5, because a person looks at 5 suggestions. Recall at 1 also matters if the first answer is used alone. A recall at 5 of 0.988 means that the right product is somewhere in the 5 suggestions for 98.8% of these customer photos. A person must still choose it. It also says nothing about photos of products that are not in the catalogue: the search returns 5 suggestions for them too.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
Larkfield wants to find photos that a supplier uploaded twice, and also to match customer photos. Which tool would you use for each, and why?
Show the answer
For photos uploaded twice, a perceptual hash. The same photo, cropped a little or brighter, keeps almost the same 64 bits: the planted pairs differ by 6 to 18. It is fast and needs no model. A person checks the pairs under a line, for example 16 bits (45 pairs, 5 of the 6 planted). For customer photos, an embedding search with DINOv2. A new photo of the same product changes about half of the hash bits (median 30 of 64), so pHash cannot find it.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Recap
- A nearest-neighbour search compares one embedding with every stored embedding, and returns the top k by cosine. Use the same model for both sides.
- Recall at k is the share of questions whose right answer is in the top k. Choose k from what a person looks at.
- On 160 customer photos, DINOv2 small finds the right product first for 0.944, and in the top 5 for 0.988. CLIP gives 0.562 and 0.756: it finds the kind of product, not the product. Pixel methods are near zero.
- A perceptual hash (64 bits, Hamming distance) finds copies of the same photo: the planted pairs differ by 6 to 18 bits. It cannot find the same product in a new photo.
- A search always answers. Test it with a product that is missing, and keep a person and a “none of these” choice.
- Your project has
vision match, and still 22 tests.
You can now find which product a photo shows. But Omar also needs words about each product: a description, its colour and its material, for 733 photos and every new one. Next, you ask models that read a photo and write text.
Next: Vision-language models