Skip to content
Français

Search by meaning

  • Lesson 15
  • 40 min
  • Module 5: Dense and hybrid retrieval

Where we are

Lexical search needs the same words in the query and in the product. People do not always use the product's words. In this lesson, we turn queries and products into embeddings, and we search by meaning.

Plays from YouTube. Nothing loads from YouTube until you press play. Privacy

What you will be able to do

Find products that match what a person means, even when the query and the product share no useful word.

NotebookSearch by meaning in a notebookColab · Kaggle · GitHub

The notebook loads the product and query vectors, scores every product with one matrix product, and measures dense search on the English validation queries. A last, optional part downloads the model and encodes a new query. Its setup cells download the data, so it needs no local setup. About 35 minutes.

Colab and Kaggle open your own copy. They need a free Google or Kaggle account. Stuck? See the solution notebook.

Before you startWhat to know and to have readyAbout 40 minutes

What you need to know

What you need on your computer

  • The project larkfield-search with its virtual environment active, and the labs repository next to it, as in the earlier modules. This lesson adds four files from the snapshot end-of-m05, updates requirements.txt and the README, and installs two large packages (about 760 MB on disk).

A customer who does not know the product’s name

Section titled “A customer who does not know the product’s name”

A customer walks into a garden shop. She says: “I need a seat for the garden.” No label in the shop says “seat for the garden”. The labels say “teak bench”, “folding stool” and “patio chair”.

A good shop assistant does not look for the words “seat for the garden”. She thinks about what the customer means: something to sit on, that can stay outside. Then she walks to the benches and the stools. She does not walk to the garden statues, although they are “for the garden” too.

This is search by meaning. The assistant compares ideas, not words.

The problem: the same idea in different words

Section titled “The problem: the same idea in different words”

BM25 compares words. A product scores only on the query words that appear in its text. So when a person uses other words than the catalogue, BM25 can only use the words that happen to match.

Try the best lexical run so far:

Terminal window
search query "seat for the garden" --run bm25-norm+spell+syn -k 5
bm25-norm+spell+syn: "seat for the garden" -> 5 results
1. LK-80545 smile for me solar garden gnome statue Garden Statues 531.99
2. LK-30203 francine , the fairy gardener cast garden statue Garden Statues 630.99
3. LK-19627 the secret garden fairy statue Garden Statues 513.99
4. LK-98696 bulldoze and the garden gnome statue Garden Statues 248.99
5. LK-98373 the enchanted garden fairies statue Garden Statues 281.99

Five garden statues. “garden” is in their short titles, so BM25 gives them high scores. “seat” is a common word in long descriptions, so it adds little. Nothing on this page is a seat.

This query is not in the course’s query set, so it has no judgments. We wrote it to show the problem.

An embedding is a list of numbers that stands for the meaning of a text. A model reads the text and writes the list. Texts with similar meanings get similar lists. Texts with different meanings get different lists.

A list of numbers is a vector. So we can compare two texts by comparing their vectors. The usual comparison is the cosine similarity: 1 when two vectors point the same way, 0 when they have nothing in common. Similarity and distance in the mathematics course explains it with pictures.

Search with embeddings is called dense retrieval, or dense search. “Dense” means that every number of the vector is used. BM25 is the opposite: its vectors have one place per word of the vocabulary, and almost all of them are zero.

The model we use is a bi-encoder. “Bi” means two: it encodes the query and the product separately, with the same model. The two never meet inside the model. This has one big advantage. We can encode all the products once, before anyone searches. At search time, only the query is new.

Top row, done once: 40,066 products (title, class and the first 40 words of the description) go into the bi-encoder granite-embedding-97m-multilingual-r2, which writes a table of 40,066 rows by 384 numbers, 30.8 MB in float16. The course team did this; it took 163 seconds. Bottom rows, for each search, about 8 ms in all: the query seat for the garden goes into the same bi-encoder (5.5 ms) and becomes one vector of 384 numbers. One matrix product with the product table gives the cosine with every product (2.1 ms). The 5 best scores: 0.877 noreen owl garden stool, 0.874 gaspar bollen girl picking flowers statue, 0.873 owl garden figurine, 0.871 haussen fairy garden stake, 0.870 fairy garden figurine.
A bi-encoder encodes the products once and the query at search time. The search itself is one matrix product.
Text description of the diagram

The top row happens once, before any search. The 40,066 products go into the bi-encoder. For each product, the model reads the title, the product class and the first 40 words of the description. It writes a table with 40,066 rows of 384 numbers. Each number is stored as a float16, a format of 2 bytes, so the table takes 30.8 MB in memory. The course team did this for you: 42,066 products (with the 2,000 held back for the final assignment) took 163 seconds. The lower rows happen for each search, in about 8 ms. The query “seat for the garden” goes into the same model (5.5 ms) and becomes one vector of 384 numbers. One matrix product with the table gives the cosine with every product (2.1 ms). The 5 best scores are 0.877 for a noreen owl garden stool, then 0.874, 0.873, 0.871 and 0.870 for four garden statues and ornaments.

Our model is IBM’s granite-embedding-97m-multilingual-r2. It writes 384 numbers for each text, and it reads many languages, including French. Its licence is Apache-2.0. Its model card says that every training set has a permissive licence. That is the vendor’s claim; we did not check the training data ourselves.

Real embeddings have 384 numbers, and the numbers have no names. To see the arithmetic, here is a toy with three numbers. We made these vectors up. Say the three numbers mean “for sitting”, “for the garden” and “for decoration”.

Text sitting garden decoration
query “seat for the garden” 0.8 0.6 0.0
garden bench 0.6 0.8 0.0
office chair 1.0 0.0 0.0
garden hose 0.0 1.0 0.0
garden statue 0.0 0.6 0.8

Each vector has length 1: for example, 0.6 × 0.6 + 0.8 × 0.8 = 1. For vectors of length 1, the cosine is the dot product: multiply the numbers place by place, then add.

Product Dot product with the query Cosine
garden bench 0.8 × 0.6 + 0.6 × 0.8 + 0 × 0 0.96
office chair 0.8 × 1.0 + 0.6 × 0 + 0 × 0 0.80
garden hose 0.8 × 0 + 0.6 × 1.0 + 0 × 0 0.60
garden statue 0.8 × 0 + 0.6 × 0.6 + 0 × 0.8 0.36

The bench comes first, because it is close on both numbers that the query has. The statue comes last: it shares “garden” only, and it has a number that the query does not have.

A real model works the same way, with 384 numbers. But it learned its numbers from data, and it does not always agree with our toy. We encoded the same short texts with the real model:

Text pair Cosine (real model)
“seat for the garden” and “garden statue” 0.887
“seat for the garden” and “garden bench” 0.853
“seat for the garden” and “office chair” 0.796

The real model puts the statue a little above the bench. “garden” weighs a lot in its numbers. Also see the range: every cosine is between 0.79 and 0.89. Only the order matters, not the size. A cosine of 0.85 does not mean “85% relevant”.

For one query, we need the cosine with all 40,066 products. Put the product vectors in a table, one row per product. Then one matrix product of the table and the query vector gives all 40,066 dot products at once. NumPy does it in about 2 ms on a laptop.

This is a flat index: no clever structure, every product is scored. At 40,066 products, it is fast enough. Very large catalogues use an approximate index that scores only some products. We return to this in Module 8.

The course team encoded every product and every query of the course once, and shipped the vectors in data/embeddings/. So the lessons do not wait for the model. A query that is not in the course’s set is encoded on your computer, and the model is downloaded the first time.

  1. Copy this module’s files from the snapshot end-of-m05. Four files are new: the text that the model reads (embedtext.py), dense search (dense.py), the runs of this module (runs/dense.py) and their tests. requirements.txt and the README are updated:

    Terminal window
    S=../nextia-learning-labs/C38/snapshots/end-of-m05
    for f in requirements.txt larkfield_search/embedtext.py larkfield_search/dense.py larkfield_search/runs/dense.py tests/test_dense.py README.md; do
    cp "$S/$f" "$f"
    done
  2. Install the two new packages, sentence-transformers and PyTorch. On Linux, install the CPU version of PyTorch first, so pip does not download the large GPU version:

    Terminal window
    python -m pip install torch==2.15.0 --index-url https://download.pytorch.org/whl/cpu # Linux only
    python -m pip install -r requirements.txt

    The download is large. The virtual environment grows from about 240 MB to about 1 GB.

  3. Run the tests. They use a tiny fake catalogue, so they need no model:

    Terminal window
    python -m pytest -q
    ........................... [100%]
    27 passed in 7.17s

    Three tests are new. One of them checks reciprocal rank fusion by hand; you meet it in Hybrid search.

  4. Search by meaning:

    Terminal window
    search query "seat for the garden" --run dense -k 5
    dense: "seat for the garden" -> 5 results
    1. LK-87094 noreen owl garden stool Accent Stools 59.99
    2. LK-24871 gaspar bollen girl picking flowers statue Garden Statues 576.99
    3. LK-14244 owl garden figurine Garden Statues 176.99
    4. LK-67532 haussen fairy garden stake Garden Accents 243.99
    5. LK-95148 fairy garden figurine Garden Statues 611.99

    The first time, the command downloads the model (about 200 MB) and takes about 15 seconds. Each time it loads the model, it can print a few lines first. They are a note about redirects (on macOS), a progress bar “Loading weights”, and a warning about a missing Hugging Face token. You can ignore them.

A stool comes first. The word “seat” is not in its title, and BM25 did not find it. Dense search found it because a stool is a seat. But the next four results are garden ornaments. The model understood “seat” a little, and “garden” a lot. Search by meaning is not magic: it is a learned guess about which texts are close.

One query proves nothing. Measure the run on the 243 test queries, as every run of the course:

Terminal window
search eval --run dense
dense on 243 test queries: NDCG@10 0.519 (95% interval 0.471 to 0.566)
recall@100 0.453, MRR 0.510, zero results 0.0%, judged in the top 10 57.4%; 1.2 ms a query (p50), 1.4 ms (p95)

The time here is short because the test queries already have their vectors. Encoding a new query live adds about 5.5 ms. The full dense search takes 8.0 ms (p50) and 9.8 ms (p95) on one thread.

NDCG@10 on the test queries (mean of 241), with 95% intervals. Dense search is far above Larkfield's search today, and clearly below the best lexical run.
Text description of the chart

The chart shows five runs, each with NDCG@10 and its 95% interval on the test queries (mean of 241 of the 243). current, Larkfield’s search today: 0.195 (0.155 to 0.237). bm25: 0.429 (0.381 to 0.476). bm25-fields+codes: 0.506 (0.458 to 0.552). bm25-norm+spell+syn, the best lexical run of Module 4: 0.620 (0.574 to 0.662). dense, highlighted: 0.519 (0.471 to 0.566).

Dense search alone scores 0.519. That is much better than Larkfield’s search today (0.195). But it is clearly worse than the best lexical run, 0.620. The two intervals do not even touch.

Two details matter for later. First, the run dense keeps the code rule of Exact matches and rules. Every run since Module 4 does. Without it, dense search would never find a code. Second, only 57.4% of dense’s top 10 products have a judgment. This share is printed as “judged in the top 10”, and the course writes it judged@10. The judges graded the products that the older searches found. An unjudged product counts as irrelevant, so dense search is a little underrated. Judgments from a language model measured this with the model’s labels for the unjudged products. With them, dense moves from 0.494 to 0.514 on the English and French test queries. That is a small change, not the gap of 0.1.

Tomás reads the numbers and frowns. “Vector search is supposed to replace keyword search.” The next lesson finds out where dense search wins, where it loses, and why.

When to use dense search, and what it costs

Section titled “When to use dense search, and what it costs”

Use it when people describe products in their own words, or when the query and the catalogue use different languages. It also works as a second opinion beside a lexical search. It never returns an empty page: every product has a cosine.

Do not use it alone when exact words matter: names, codes, sizes and rare brands. The next lesson measures these cases.

What it costs:

Cost Value at Larkfield Label
Encode the catalogue once 163 s for 42,066 products (257.8 a second, 4 threads) measured
Product vectors 30.8 MB in memory in float16; float32 (4 bytes a number) is twice the size measured
float16 instead of float32 99.3% of the same top 10 on 200 queries measured
Encode one query 5.5 ms (p50, 1 thread) measured
Score every product 2.1 ms (p50) measured
Packages about 760 MB more in the virtual environment measured
The model 195 MB, downloaded once measured

A new product needs its vector before dense search can find it. So the encoding step belongs in the pipeline that adds products to the index. Module 8 returns to this.

Common mistake

Two models, one index

Symptom: after a model upgrade, dense search returns nonsense for every query. Nothing crashed, and every query still gets ten results.

Cause: the products were encoded with the old model, and the queries with the new one. Each model has its own numbers. A cosine between vectors of two models means nothing.

Diagnosis: check which model and version wrote each table. The project stores this in data/embeddings/model.json and loads the model at that exact revision.

Fix: pin the model’s revision. When you change the model, encode the whole catalogue again, and switch the query side and the product side at the same time.

Consequence in AI work: an embedding is only meaningful next to embeddings of the same model. Store the model name and revision with every vector table, in search and in retrieval for a language model.

Search by meaning with your own words. Predict first, then run:

Terminal window
search query "something to sit on outside" --run dense -k 5

What we got: an outdoor seating group, a patio sofa, a rocking patio set and two more outdoor seating groups. None of these titles contains “sit” or “outside”. Then try the same query with --run bm25-norm+spell+syn, and compare.

For more about embeddings in retrieval for a language model, with chunks of documents instead of products, see Embeddings for search in the RAG course.

Check your understanding

Why can a bi-encoder search 40,066 products in a few milliseconds, although the model needs 163 seconds to read them all?

Show the answer

A bi-encoder encodes the query and each product separately. So all the products are encoded once, before any search, and the vectors are stored. At search time, the model reads only the query (about 5.5 ms). Then one matrix product of the stored table and the query vector gives the cosine with every product (about 2 ms). The 163 seconds are paid once, not per search.

Check your understanding

In the toy example, the query is (0.8, 0.6, 0.0) and a product is (0.6, 0.0, 0.8). Both have length 1. What is the cosine, and is the product close to the query?

Show the answer

The dot product is 0.8 × 0.6 + 0.6 × 0.0 + 0.0 × 0.8 = 0.48. For vectors of length 1, this is the cosine: 0.48. It is below the bench (0.96) and the office chair (0.80). It is above the statue (0.36). In a real model, only the order of the cosines matters, not their size.

Check your understanding

Dense search scores 0.519 on the test queries, and the best lexical run 0.620. Only 57.4% of dense's top 10 products are judged. Omar asks: “Is dense search really worse, or are the judgments unfair?” What do you answer?

Show the answer

Both are partly true, but the gap is real. Unjudged products count as irrelevant, so dense search is underrated a little. The model’s labels for those products moved dense from 0.494 to 0.514 on the English and French test queries: about 0.02. The gap to the lexical run is about 0.1, and the two intervals do not touch. So dense search alone is worse here. To be fair, report judged@10 beside NDCG, and judge more of dense’s products before a decision.

Recap

  • An embedding is a vector of numbers for the meaning of a text. A bi-encoder encodes the query and the products separately, with the same model.
  • The products are encoded once (163 s here); at search time, only the query is encoded (5.5 ms). One matrix product gives the cosine similarity with every product (2.1 ms).
  • For vectors of length 1, the cosine is the dot product. Only the order of the cosines matters.
  • “seat for the garden” finds a garden stool first, which BM25 missed, and then garden statues. Meaning is a learned guess.
  • Dense search scores NDCG@10 0.519 (0.471 to 0.566). That is far above Larkfield’s search today, and clearly below the best lexical run (0.620). Part of the gap is unjudged products, but only about 0.02.

Dense search is better on some queries and worse on others. An average hides which. Next, you measure dense search against BM25 by query type, and you find the queries where it fails.

Next: Where dense search fails