Skip to content
Français

Find the same product

  • Lesson 23
  • 40 min
  • Module 7: Match, search and describe

Where we are

In Module 3, embeddings helped us find classes and label errors. They told us what kind of product a photo shows. Now a customer sends a photo, and we must find the one product it shows. In this lesson, we match phone photos to the catalogue with nearest neighbours, measure recall at k, and find near-duplicate photos. First, a helper at a busy shop counter.

Plays from YouTube. Nothing loads from YouTube until you press play. Privacy

What you will be able to do

Find which catalogue product a customer’s photo shows, and find catalogue photos that were uploaded twice. Both are searches by likeness, and both are measured by how often the right answer is near the top.

NotebookDo this lesson in a notebookColab · Kaggle · GitHub

The notebook has the search, a miss to look at, and one task with a check. Its setup cells install the packages and download the catalogue and the customer photos as zip files. It needs no account, no key and no model download: the embeddings are in the data. About 25 minutes.

Colab and Kaggle open your own copy. They need a free Google or Kaggle account. Stuck? See the solution notebook.

Before you startWhat to know and to have readyAbout 40 minutes

What you need to know

What you need on your computer

  • The notebook route: Colab, Kaggle or Jupyter. Nothing to install.
  • The project route: your larkvision folder beside nextia-learning-labs, at the snapshot end-of-m06, with its .venv, as set up in Set the line with the warehouse. python -m pytest -q prints 22 passed. This lesson adds one file.

Tested with Python 3.12.13 on macOS, in Jupyter and in a terminal. The Windows commands, Linux, Colab and Kaggle were not tested.

A customer comes to a shop counter with a phone. “I bought this bowl here last year. I want two more.” She shows a photo, taken at home, in yellow kitchen light, from above. The helper does not read the bowl’s name anywhere. She looks at the photo, then walks along the shelves. She stops at the three bowls that look most like it, and she holds the phone next to each one. After a minute, she finds it.

She did two things. First, she searched by likeness: she did not check every item, only the few that looked closest. Second, she checked: she compared the photo with each candidate before she answered. A good search does the first part fast and leaves the second part to a person.

The problem: the same product, a different photo

Section titled “The problem: the same product, a different photo”

Larkfield gets such photos every day. A customer writes, “this arrived broken”, or “I want the same one”, and attaches a phone photo. Someone must find the product among the 733 catalogue photos. The course’s 160 customer photos stand in for them. Our code made each one from a catalogue test photo. It changed the angle, made the product smaller on a table, and added poor light, a colour cast, blur and noise. They are synthetic, and each one has exactly one right catalogue photo.

The first idea is to compare the photos directly. Two simple methods do this:

  • The colour histogram of Rules and colour histograms counts the colours of each photo.
  • A perceptual hash turns each photo into 64 bits. This lesson explains it in its second half.

Both fail. The colour histogram finds the right photo in its first five answers for 0.087 of the customer photos: about 1 in 11. The perceptual hash does so for 0.019: 3 photos in 160. The customer’s yellow kitchen light changes every colour, and the new angle moves every part of the product. A description of the pixels cannot find the same product in a new photo.

You already have a better description. An embedding describes what a photo shows, not its pixels, and similar photos get embeddings that point in a similar direction. So the search works like the helper at the counter:

  1. Store the embedding of every catalogue photo once.
  2. When a customer photo arrives, compute its embedding with the same model.
  3. Compute the cosine similarity with every stored embedding.
  4. Show the catalogue photos with the highest cosines, best first.

This is called nearest-neighbour search: the answer is the stored items nearest to the question. The list of the first k answers is the top k. For 733 photos, the cosines take less than a millisecond.

Here is one customer photo of a white bowl with a black spoon, and the top 5 of two models:

Two rows of photos. Row 1, DINOv2 small, 88 MB: the customer photo, a small white bowl with a black spoon on a grey card, framed in blue. Then its top 5: first the same white bowl with a black spoon, cosine 0.75, framed in blue and ticked; then a bowl with a green pattern, 0.58; a gold metal bowl, 0.57; a white cup with a handle, 0.48; a white bowl, 0.47. Row 2, CLIP ViT-B/32, 605 MB: the same customer photo, then five white bowls and cups, cosines 0.65, 0.63, 0.62, 0.62 and 0.61. The right bowl is not among them.
One customer photo (synthetic, made by our code from a catalogue photo) and its top 5 catalogue photos with two embedding models. Blue frame and tick: the right product. Photos: 準建築人手札網站 Forgemind ArchiMedia, homestilo, Rosmarie Voegtli, Hash Milhan, Bit Boy, Chris Mear, Ratha Grimes, See-ming Lee, CC BY 2.0, from Open Images.
Text description of the diagram

The figure has two rows, one for each model. Each row starts with the same customer photo: a small white bowl with a black spoon, on a grey card, a little blurred and tilted. In the first row, DINOv2 small puts the right catalogue photo first, with a cosine of 0.75. The next four answers are other bowls and a cup, with cosines from 0.58 down to 0.47: far below the first one. In the second row, CLIP returns five white bowls and cups with cosines from 0.65 to 0.61. They are all close to each other, and none of them is the right bowl. CLIP ranks the right bowl sixth.

Both models found white bowls. Only DINOv2 found this white bowl. Look at the cosines too. DINOv2’s first answer is far ahead of its second (0.75 against 0.58). CLIP’s five answers are almost equal. A clear gap between the first and the second answer is a useful sign that the match is real.

How do we score a search? For each customer photo, find the rank of the right catalogue photo: 1 if it is first, 2 if it is second, and so on. Recall at k is the share of customer photos whose right photo has a rank of k or better. Here are the first four customer photos with CLIP:

Customer photo Product Rank of the right photo (CLIP) In the top 1? In the top 5?
C-9cb24dd8 bowl 1 yes yes
C-e8d1feae bowl (the figure) 6 no no
C-e4d89d21 bowl 1 yes yes
C-2218b07f bowl 1 yes yes

Three of the four right photos are first, so the recall at 1 is 3 / 4 = 0.75. The recall at 5 is also 3 / 4 = 0.75, because rank 6 is outside the top 5. The recall at 10 is 4 / 4 = 1.0. A larger k always gives the same recall or a higher one. Choose k from the way people use the result. If a person looks at 5 suggestions, measure recall at 5.

The course team ran five methods on the same 160 customer photos and the same 733 catalogue photos:

Recall at 5 of five methods on the 160 customer photos (RN-18: ResNet-18). A higher bar finds the right product in the top 5 more often.
Text description of the chart

A column chart with five columns on an axis from 0 to 1. Colour histogram: 0.087. Perceptual hash: 0.019. ResNet-18 embeddings: 0.756. CLIP embeddings: 0.756. DINOv2 small embeddings: 0.988, highlighted. The two pixel methods are near zero, and DINOv2 is far above the other two embeddings.

Method Recall at 1 Recall at 5 Worst product (at 5) Model size CPU time a photo
Colour histogram 0.019 0.087 bowl (0.05) none 0.86 ms
Perceptual hash 0.013 0.019 bowl (0.00) none 2.94 ms
ResNet-18 0.581 0.756 plant pot (0.45) 47 MB 13.7 ms
CLIP ViT-B/32 0.562 0.756 chair (0.65) 605 MB 7.67 ms
DINOv2 small 0.944 0.988 cushion (0.95) 88 MB 13.64 ms

The times are for one photo in a batch of 32, with 4 threads, on the course laptop. With DINOv2, the right photo is first for 151 of the 160 customer photos. It is in places 2 to 5 for 7, and in places 6 to 10 for 2. With CLIP, it is first for only 90, and 22 right photos are not even in the top 10.

Tomás is surprised. “CLIP has 605 MB, and DINOv2 small only 88 MB. A bigger model should see better.” Priya explains that the two models learned different things.

CLIP learned from photos with their captions. A caption says “a white bowl on a table”, not which white bowl. So CLIP learned to tell the kind of thing apart, and it puts all white bowls close together.

DINOv2 learned from photos alone. It compared different crops of the same photo, and learned to recognise that they show the same object. So it keeps the details of one object: the shape of the rim, the angle of the spoon. That is exactly what finding the same product needs.

So the question decides the model. For “what kind of product is this?”, choose CLIP. For “which product is this?”, choose DINOv2. The size of the model did not decide it.

The project gets the search as a command, vision match. It reads the stored embeddings of the course’s data, so it needs no model download. Check the place first: you must be in your larkvision folder, with the virtual environment active.

  1. Open a terminal and go to your larkvision folder. Activate the virtual environment: source .venv/bin/activate on macOS and Linux, .venv\Scripts\Activate.ps1 on Windows.

  2. If this is a new terminal, set LARKVISION_DATA again. It lives only in the terminal where you set it:

    Terminal window
    export LARKVISION_DATA="$(cd ../nextia-learning-labs/C42/data && pwd)"
  3. Copy the file of this lesson from the snapshot end-of-m07:

    Terminal window
    S=../nextia-learning-labs/C42/snapshots/end-of-m07
    cp "$S/larkvision/commands/match.py" larkvision/commands/match.py
  4. Run the tests. You still see 22 passed: the test of the new command comes with Check what a vision-language model says.

    Terminal window
    python -m pytest -q
  5. Search with DINOv2, and show the top 5 of the first three customer photos:

    Terminal window
    vision match --embeddings dinov2 --show 3
  6. Search with CLIP:

    Terminal window
    vision match --embeddings clip --show 3

Step 5 prints:

DINOv2 small: 160 customer photos against 733 catalogue photos; recall at 1 0.944, at 5 0.988
C-9cb24dd8 (bowl): *P-9a3fc835, P-558af3a2, P-75598db9, P-60f9ab1c, P-aef88230
C-e8d1feae (bowl): *P-9d48a6c9, P-593a4ac1, P-1815e7a4, P-080e2d04, P-9650ea1d
C-e4d89d21 (bowl): *P-9e093eaa, P-8fff6af0, P-0782f68e, P-3dd314fa, P-a8d9684c

Step 6 prints:

CLIP ViT-B/32 (OpenCLIP, laion2b_s34b_b79k): 160 customer photos against 733 catalogue photos; recall at 1 0.562, at 5 0.756
C-9cb24dd8 (bowl): *P-9a3fc835, P-7308e8ee, P-99a52778, P-a0ca8a5a, P-1c7efe8e
C-e8d1feae (bowl): P-9e093eaa, P-9650ea1d, P-614eb253, P-78abb816, P-080e2d04
C-e4d89d21 (bowl): *P-9e093eaa, P-7308e8ee, P-3dd314fa, P-0152f9f0, P-4ef55c7e

A star marks the right catalogue photo. The second line of each output is the customer photo of the figure above. Each run adds a row to your own table: vision compare --task match shows it at the end, marked “(yours)”. vision match --k 10 measures recall at 10. larkvision/commands/match.py holds the function rank: the same four steps as above, in four lines of NumPy.

Matching has a second use inside the catalogue. In Labels: how many, and how good, you met six training photos that are in the catalogue twice, cropped a little and brighter. This is a problem we planted. A photo in the catalogue twice can end up in both the training set and the test set. Then the test score is too good.

For near-duplicates like these, a smaller tool than an embedding is often enough: a perceptual hash. The pHash method does four things:

  1. Shrink the photo to 32 × 32 pixels in grey.
  2. Describe it as a sum of smooth patterns, from very smooth to fine. (The maths is the discrete cosine transform, also used in JPEG files.)
  3. Keep the 8 × 8 smoothest patterns: the rough layout of light and dark.
  4. Write 1 for each pattern above the median value, and 0 for the others. The result is 64 bits.

Two hashes are compared by the Hamming distance: the number of bits that differ. Take two short hashes of 8 bits, made up for this example: 10110010 and 10010011. They differ in the third bit and in the last bit, so the distance is 2. Same photo: about 0. Two unrelated photos: about half of the bits, so about 32 of 64.

Two rows. Row 1: a black armchair on a patterned carpet, a photo uploaded twice; the two uploads look the same, the second is a little brighter. Their 64-bit hashes are drawn as 8 by 8 grids of black and white squares; 6 squares are outlined in red. Row 2: the synthetic customer photo of a white bowl with a black spoon, and its own catalogue photo. Their hash grids differ in 30 squares, outlined in red.
The pHash of a planted second upload (top) and of a customer photo with its own catalogue photo (bottom). Each hash is 64 bits, drawn as 8 × 8 squares: black is 1, white is 0. Photos: John Philip Green, 準建築人手札網站 Forgemind ArchiMedia, CC BY 2.0, from Open Images; the customer photo is synthetic, made by our code from the second one.
Text description of the diagram

The figure has two rows. The top row shows the two uploads of the same photo of a black armchair on a carpet. The second upload is cropped a little and brighter. Next to them are their two hashes, each an 8 × 8 grid of black and white squares. The grids are almost the same: only 6 of the 64 squares differ, outlined in red. The bottom row shows the customer photo of the white bowl and its own catalogue photo. Their grids differ in 30 of the 64 squares, which is what two unrelated photos usually give.

The planted second uploads differ by 6 to 18 bits. So the search for duplicates compares every pair of the 453 training photos and keeps the pairs under a line:

Keep pairs with at most Pairs found Planted second uploads among them
4 bits 0 0 of 6
8 bits 2 2 of 6
12 bits 6 3 of 6
16 bits 45 5 of 6

At 8 bits, both pairs found are planted ones: no false pair, but only 2 of 6. At 16 bits, 5 of 6 are found among 45 pairs, so a person must look at 45 pairs. This is the usual trade: a looser line finds more and asks for more checking. In Module 3, the embeddings found 5 or 6 of the 6 planted pairs among their 10 most similar pairs.

Now the bottom row of the figure. A customer photo and its own catalogue photo differ by a median of 30 bits. Unrelated photos differ by 32. Not one of the 160 customer photos is within 16 bits of its own photo. A pHash survives a small crop and a change of brightness. It does not survive a new angle or a new photo. That is why it scored 0.019 above.

So a pHash finds copies of the same photo: it is fast and needs no model. An embedding finds the same product in a new photo. For pairs and groups of records that may be the same thing, the record matching course goes further. It is not on the site yet.

Common mistake

The product that is not in the catalogue

Symptom: Omar tests the search with a photo of a teapot. Larkfield does not sell teapots. The search still returns five products, with a confident first answer. The support team sends the customer a link to a vase.

Cause: a nearest-neighbour search always returns the nearest items, even when none of them is right. It never says “none of these”. The course team measured how bad this is: for each customer photo, it removed the right photo from the catalogue and searched again. With DINOv2, the median cosine of the right photo is 0.673. When the right photo is missing, the best remaining one still has a median of 0.515.

Fix: add a “no match” line: below a cosine, the answer is “not found, ask a person”. A line at 0.469 keeps 95% of the right matches. But 64% of the searches for a missing product still pass it. For CLIP, 99.4% pass. So a line alone is weak. Show the top 5 to a person, and keep a “none of these” button for them.

Consequence in AI work: every search and every classifier answers, whatever you give it. Before you use the answer, ask what the system does with an input that has no right answer. Measure it, as here, by hiding the right answer on purpose.

Use an embedding search when a person needs a short list of candidates. Examples: “which product is this?” or “show similar products”. It needs no labels: DINOv2 found the right product first for 0.944 of the customer photos with no training at all. The cost is one embedding per photo, about 14 ms on a laptop CPU, and a few kilobytes to store each one.

Do not use it as the final decision. A match is a suggestion. Two products can look almost the same and differ in size or price, and the customer’s product may not be in the catalogue. The candle is DINOv2’s weakest product at 1: 85% of the candle photos find the right candle first, because many candles look alike. When an order number or a barcode is available, use it first. When not to build it comes back to this.

  1. Open the notebook from the box Do this lesson in a notebook at the top of this page.
  2. Run the setup cells. The second one ends with a Ready: line.
  3. Run section 1. It shows three customer photos next to their catalogue photos.
  4. Before you run section 2, predict which of the three models finds the right photo first most often.
  5. Run section 2. Compare its three lines with the table above.
  6. Run section 3. It shows a photo that CLIP misses. Look at its first three answers: what do they have in common with the customer photo?

In section 4 of the notebook, compute the recall at 5 of the DINOv2 embeddings with cp.recall_at_k. Put it in dinov2_r5, and run the check cell. Then change one condition: compute the recall at 1 and at 10 for ResNet-18. Which k would you show to the support team?

Show the answer
dinov2_r5 = cp.recall_at_k(ranked("dinov2"), truth, 5)
print(dinov2_r5)
0.9875

The check rounds it to 0.988: 158 of the 160 right photos are in DINOv2’s top 5. For ResNet-18, cp.recall_at_k(ranked("resnet18"), truth, 1) gives 0.581 and k=10 gives 0.831. With ResNet-18, a person would need about 10 suggestions to find most products. With DINOv2, 5 are enough, and the first one is right 94.4% of the time.

Check your understanding

Recall at 1 is 0.562 for CLIP and 0.944 for DINOv2 on the same customer photos. CLIP was the best model for classes in Module 3. Explain the difference in your own words.

Show the answer

The two models answer different questions. CLIP learned from photos with captions, so its embedding describes the kind of thing: “a white bowl”. All white bowls get close embeddings, and the search returns white bowls, not this bowl. DINOv2 learned from photos alone, by matching crops of the same photo, so its embedding keeps the details of one object. Finding the same product needs those details. Classes need the kind. The model’s size (605 MB against 88 MB) did not decide it.

Check your understanding

Your search shows 5 suggestions to the support team. Which number do you report: recall at 1, at 5 or at 10? And why does a recall at 5 of 0.988 not mean that the search is right 98.8% of the time?

Show the answer

Report recall at 5, because a person looks at 5 suggestions. Recall at 1 also matters if the first answer is used alone. A recall at 5 of 0.988 means that the right product is somewhere in the 5 suggestions for 98.8% of these customer photos. A person must still choose it. It also says nothing about photos of products that are not in the catalogue: the search returns 5 suggestions for them too.

Check your understanding

Larkfield wants to find photos that a supplier uploaded twice, and also to match customer photos. Which tool would you use for each, and why?

Show the answer

For photos uploaded twice, a perceptual hash. The same photo, cropped a little or brighter, keeps almost the same 64 bits: the planted pairs differ by 6 to 18. It is fast and needs no model. A person checks the pairs under a line, for example 16 bits (45 pairs, 5 of the 6 planted). For customer photos, an embedding search with DINOv2. A new photo of the same product changes about half of the hash bits (median 30 of 64), so pHash cannot find it.

Recap

  • A nearest-neighbour search compares one embedding with every stored embedding, and returns the top k by cosine. Use the same model for both sides.
  • Recall at k is the share of questions whose right answer is in the top k. Choose k from what a person looks at.
  • On 160 customer photos, DINOv2 small finds the right product first for 0.944, and in the top 5 for 0.988. CLIP gives 0.562 and 0.756: it finds the kind of product, not the product. Pixel methods are near zero.
  • A perceptual hash (64 bits, Hamming distance) finds copies of the same photo: the planted pairs differ by 6 to 18 bits. It cannot find the same product in a new photo.
  • A search always answers. Test it with a product that is missing, and keep a person and a “none of these” choice.
  • Your project has vision match, and still 22 tests.

You can now find which product a photo shows. But Omar also needs words about each product: a description, its colour and its material, for 733 photos and every new one. Next, you ask models that read a photo and write text.

Next: Vision-language models