Skip to content
Français

Where dense search fails

  • Lesson 16
  • 40 min
  • Module 5: Dense and hybrid retrieval

Where we are

Dense search found products that share no word with the query. Tomás thinks it can replace keyword search. In this lesson, we measure dense search against BM25 by query type, and we find where dense search fails.

Plays from YouTube. Nothing loads from YouTube until you press play. Privacy

What you will be able to do

Find the kinds of queries where search by meaning fails, so you know what a lexical search must still do.

NotebookFind the failures of dense search in a notebookColab · Kaggle · GitHub

The notebook measures BM25 and dense search query by query on the validation queries. You compute the paired difference by source and by kind of query, and find the worst queries. Then you test dense search on product codes without the code rule. Its setup cells download the data, so it needs no local setup. About 30 minutes.

Colab and Kaggle open your own copy. They need a free Google or Kaggle account. Stuck? See the solution notebook.

Before you startWhat to know and to have readyAbout 40 minutes

What you need to know

What you need on your computer

  • The project larkfield-search with the files of the last lesson. This lesson adds no file.

An assistant who understands ideas but not names

Section titled “An assistant who understands ideas but not names”

Think again of the shop assistant who understands what customers mean. A customer says: “I want the Kelly Clarkson light fixtures.” The assistant has never heard of Kelly Clarkson. She hears “light fixtures”, and she walks to the lights. Any lights.

A second customer reads from a note: “cushion replacemnet insert”, with a spelling mistake. The assistant does not know the word “replacemnet”. She guesses from “cushion” and shows pillow covers.

The assistant is good with ideas. She is weak with names, codes and odd spellings, because she never learned them. Dense search has the same weakness, for the same reason.

The problem: an average that hides failures

Section titled “The problem: an average that hides failures”

Tomás has an argument. “Dense search is modern. BM25 is from the 1990s. Let us replace BM25.” The average says no: 0.519 against 0.620. But an average mixes many kinds of queries. Dense search could be better on some and much worse on others. To decide what to build, we need to know which.

Start with three queries from the test set, side by side:

Three queries, each with the top 5 of BM25 and of dense search and the judges' grades. kelly clarkson light fixtures, known-item: BM25 NDCG@10 0.943, four exact lights and one partial (a clarkson chandelier); dense 0.113, vanity lights: three not judged and two partial, one of them a kelly lantern. cushion replacemnet insert, specific: BM25 1.000, five partial replacement cushions; dense 0.095, pillow covers, a cushen comforter set and a cushion, four not judged and one partial. verden callan, known-item: BM25 1.000, the verden callan comforter set first (exact), then four partial; dense 0.229, a task chair and a daybed first, the comforter set third.
BM25 and dense search on three test queries: a brand name, a misspelling and a rare product name. The grades are the WANDS judges'.
Text description of the diagram

Three test queries, each with the top 5 of BM25 (the run bm25-norm+spell+syn) and of dense search. “kelly clarkson light fixtures” is a known-item query. BM25 scores NDCG@10 0.943: four exact lights and one partial, a clarkson chandelier. Dense scores 0.113: vanity lights; three are not judged and two partial, and a kelly lantern is fifth. “cushion replacemnet insert” is a specific query with a misspelling. BM25 scores 1.000 with five partial replacement cushions. Dense scores 0.095: pillow covers, a cushen comforter set and a cushion; four are not judged. “verden callan” is a known-item query. BM25 scores 1.000 with the verden callan comforter set first. Dense scores 0.229: a task chair and a daybed come first, and the comforter set is third.

Each failure has a clear cause:

  • A brand name. “Kelly Clarkson” is a brand of home products. Dense search reads “light fixtures” and returns vanity lights. BM25 matches the words “kelly” and “clarkson” in the product text.
  • A misspelling. “replacemnet” is not a word. BM25’s run corrects it first, with the spelling correction of Fix spelling. Dense search reads the query as it is. The model splits the odd word into pieces and finds “cushen” and pillow covers.
  • A rare product name. “verden callan” is the name of one comforter set. The model has no idea what it means, so the vector points at nothing in particular. BM25 finds both rare words in one product.

The idea: measure by slice, not only on average

Section titled “The idea: measure by slice, not only on average”

You met slices in Kinds of search: groups of queries with the same kind, source or language. Measuring by slice shows where a method wins and where it loses. The paired difference with its interval tells us if a difference is real.

Why does dense search fail on these slices? A bi-encoder squeezes a whole text into 384 numbers. It keeps what it saw often in training: common ideas, like “a light” or “a chair”. It loses what it saw rarely: a brand, a product name, a code, a misspelt word. BM25 keeps every word exactly, and rare words get the highest IDF. So the two methods are strong in opposite places.

Here is one slice by hand: the three queries of the diagram, plus one where dense search wins. The numbers are NDCG@10 from the runs.

Query Kind BM25 Dense Dense minus BM25
kelly clarkson light fixtures known-item 0.943 0.113 −0.830
cushion replacemnet insert specific, misspelt 1.000 0.095 −0.905
verden callan known-item 1.000 0.229 −0.771
palram harmony greenhouses known-item 0.358 0.915 +0.557

The mean difference of these four is (−0.830 − 0.905 − 0.771 + 0.557) ÷ 4 = −0.487. One query goes the other way. Palram is a brand too, but BM25 put three greenhouse accessories first: an anchor kit, plant hangers and a heater. Dense search put real greenhouses first. So “brand names fail” is a tendency, not a law. That is why we measure many queries, not four.

Larkfield: dense against BM25, slice by slice

Section titled “Larkfield: dense against BM25, slice by slice”

search compare gives the paired difference on every test query, and --by splits it. First by kind of query:

Terminal window
search compare bm25-norm+spell+syn dense --by qtype
dense minus bm25-norm+spell+syn on 241 test queries: ndcg@10 -0.101 (95% interval -0.139 to -0.062); better on 64, worse on 119, the same on 58
qtype queries bm25-norm+spell+syn dense difference low high worse
broad 25 0.703 0.535 -0.168 -0.307 -0.022 15
known-item 80 0.729 0.659 -0.071 -0.137 -0.012 25
specific 136 0.540 0.434 -0.106 -0.158 -0.055 79

Then by source of the query:

Terminal window
search compare bm25-norm+spell+syn dense --by source
dense minus bm25-norm+spell+syn on 241 test queries: ndcg@10 -0.101 (95% interval -0.139 to -0.062); better on 64, worse on 119, the same on 58
source queries bm25-norm+spell+syn dense difference low high worse
code 20 1.000 1.000 0.000 0.000 0.000 0
planted 3 0.515 0.333 -0.182 -0.546 0.188 2
translated 59 0.301 0.216 -0.085 -0.186 0.007 25
typo 40 0.659 0.403 -0.256 -0.357 -0.158 29
wands 119 0.703 0.632 -0.071 -0.111 -0.027 63

The compare uses 241 queries, not 243: two test queries have only irrelevant judgments, so their NDCG has no value.

Dense minus BM25 (bm25-norm+spell+syn), NDCG@10 on the test queries, by kind and by source, with 95% intervals. Dense search is worse on every kind of query, and much worse on misspellings and broad queries.
Text description of the chart

Two panels. The axis is the change in NDCG@10, dense minus BM25, from −0.6 to +0.3; 0 means no change. By kind of query: broad (25 queries) −0.168 (−0.307 to −0.022), highlighted; known-item (80) −0.071 (−0.137 to −0.012); specific (136) −0.106 (−0.158 to −0.055). By source: English WANDS queries (119) −0.071 (−0.111 to −0.027); French (59) −0.085 (−0.186 to +0.007); misspelt (40) −0.256 (−0.357 to −0.158), highlighted; product codes (20) 0.000, because both runs keep the code rule; planted broad (3) −0.182 (−0.546 to +0.188).

Read the chart from the worst slice:

  • Misspelt queries: −0.256. The largest loss. BM25’s run corrects spelling first; dense search does not.
  • Broad queries: −0.168. One- or two-word queries, like “couch” or “industrial”. A short query gives the model little meaning to work with.
  • Specific queries: −0.106; known-item queries: −0.071. Worse, but less. Known-item queries often contain a name, which BM25 matches exactly.
  • English WANDS queries: −0.071. Even on clean English queries, dense search loses.
  • French: −0.085, and the interval reaches +0.007. Not clear. The last lesson of this module looks at French closely.

Dense search is still better on 64 queries. It is not useless. It is a different tool, and its failures fall on different queries.

The broad query “couch” shows how bad a short query can be:

Terminal window
search query "couch" --run dense -k 5
dense: "couch" -> 5 results (query P-002: judged results are marked)
1. LK-69717 sectional microfibra box cushion chaise lounge slipcover Slipcovers 102.99 [irrelevant]
2. LK-71532 desks with hutch 56.99 [not judged]
3. LK-43423 cushion Furniture Cushions 13.99 [irrelevant]
4. LK-37928 cora chick fuzzy faux throw Blankets And Throws 17.99 [out of stock] [not judged]
5. LK-29009 crock-pot 2.5 qt . casserole slow cooker 62.99 [not judged]

A slipcover, a desk, a cushion, a throw and a slow cooker. NDCG@10 is 0.000. “couch” is a planted query, a practice problem we made on purpose. Its judgments come from a rule on the product class. The model places the word near many things that go with a couch, but not near couches. BM25 finds products with “couch” in the title, and two sofas in its top 5 are exact.

Every run since Module 4 keeps the code rule. So the code slice shows no difference: both runs score 1.000. What does dense search do alone? We ran it on the 20 code queries with the rule removed. NDCG@10 was 0.000. The right product was in the top 10 for 0 of 20 queries. A code like “lk 68270” has no meaning that a model could learn. Keep the rule in front of every method.

Many people expect dense search to fail on sizes, such as “4 inch” or “18x18”. The test set has 10 queries with a number that are not codes. On them, BM25 scores 0.378 and dense search 0.450. Dense search did better here. In “door pulls bronze matte 4 inch”, it put a 4-inch bronze pull first, while BM25 put an iron pull first. Ten queries are too few for a firm conclusion. But it is a useful warning: test your expectations on your own data.

What about “no”, “not” and “without”? The query set has no such query, so we wrote one. It is not in the course’s query set:

Terminal window
search query "chair with no arms" --run dense -k 5
dense: "chair with no arms" -> 5 results
1. LK-58749 chair 67.99
2. LK-40140 leather arm chair in cognac 239.99
3. LK-26323 callie armchair Accent Chairs 78.99
4. LK-27914 armchair Reception Seating Chairs 383.99
5. LK-53018 abarca 33 '' wide armchair Accent Chairs 74.99

Four armchairs in the top 5: the opposite of what the person asked. The cosines explain it. “chair with no arms” and “armless chair” have a cosine of 0.906. “chair with no arms” and “armchair” have 0.904. For the model, the word “no” almost does not change the vector. BM25 does not understand “no” either. Negation needs a rule or a filter, such as “armless” as a product attribute.

Do not use dense search alone for:

Kind of query What happens What to do instead
Product codes never finds the product (0.000 without the rule) an exact-match rule
Brand and product names returns the general kind of product BM25, which keeps rare words
Misspellings reads the misspelt word as it is (−0.256) spelling correction before any search
Short, broad queries lands near related things, not the thing (−0.168) BM25 on the title and the product class
Negation ignores “no” and “without” a filter on an attribute

Use dense search where these do not apply: descriptions in the customer’s own words, and as a second list beside BM25. On 64 test queries it is better. The next lesson combines the two lists, so each method can cover the other’s failures.

What this measurement costs. Nothing new: search compare --by reuses the runs. The real cost is attention. A team that reports only the average would have seen “dense is 0.1 worse” and stopped. The slices tell you what to fix.

Common mistake

A model upgrade that wins on average and loses a slice

Symptom: a new embedding model raises NDCG@10 by 0.02 on the test queries. After the release, support calls about “I typed the exact name and it is not there” go up.

Cause: the new model is better on descriptions and worse on rare names. Descriptions are many of the test queries, so the average rises. Known-item queries fall, and those customers know exactly what they want.

Diagnosis: search compare old new --by qtype and --by source. Look for a slice with an interval below zero, even when the overall difference is positive.

Fix: keep BM25 for exact words (the next lesson), or block a release when an important slice loses. Decide which slices are important before you measure, not after.

Consequence in AI work: every model change needs a check by slice. A model that is better on average can be worse for a group of users, and the average will not show it.

Pick a brand from the catalogue and test it. Predict first: will dense search find it?

Terminal window
search query "palram harmony greenhouses" --run dense -k 3
search query "palram harmony greenhouses" --run bm25-norm+spell+syn -k 3

What we got: dense search put three greenhouses first (partial). BM25 put three greenhouse accessories first (irrelevant), and one of them is a Palram heater. So here, the brand name led BM25 to the wrong product. Then run search compare bm25-norm+spell+syn dense --by band. Is dense search worse on the head queries (the most searched) or on the tail? What we got: head −0.212 (16 queries, interval −0.377 to −0.065), torso −0.043, tail −0.104. The most searched queries lose the most, and they are often short and broad.

Check your understanding

Dense search is worse than BM25 by 0.101 on average. On which slice is the loss largest, and what in the BM25 run explains it?

Show the answer

The misspelt queries: −0.256 (interval −0.357 to −0.158) on 40 queries. The BM25 run bm25-norm+spell+syn corrects spelling before it searches, so “replacemnet” becomes “replacement”. Dense search reads the query as it is, and the model maps the odd word to unrelated products. Spelling correction before dense search would remove part of this loss.

Check your understanding

Tomás says: “The code slice shows 0.000 difference, so dense search handles product codes as well as BM25.” Is he right?

Show the answer

No. Both runs keep the exact-match rule for codes, and the rule puts the right product first. So both score 1.000, and the difference is zero. Without the rule, dense search scored 0.000 on the 20 code queries and never had the right product in its top 10. A code has no meaning that a model can learn. The slice measured the rule, not dense search.

Check your understanding

A colleague says: “Dense search understands meaning, so it understands ‘chair with no arms’.” What did the run show, and what would you build instead?

Show the answer

The run put four armchairs in the top 5. “chair with no arms” has a cosine of 0.906 with “armless chair” and 0.904 with “armchair”. The word “no” almost does not move the vector. Neither dense search nor BM25 understands negation. Build a filter on a product attribute (“armless”, “without arms”). Then detect the negation in the query with a rule, as Detect what the query asks for detected prices.

Recap

  • An average hides where a method fails. Measure by slice: by kind, by source, by language, with the paired difference and its interval.
  • Dense minus BM25 is −0.101 on average, and worse on every kind of query: broad −0.168, specific −0.106, known-item −0.071.
  • The largest loss is on misspellings (−0.256): the BM25 run corrects spelling first, dense search does not.
  • Dense search fails on brand and product names, codes (0.000 without the rule) and short broad queries (“couch”), and it ignores negation.
  • It is better on 64 test queries, and it did better on the 10 queries with sizes. Test expectations on your own data.

Dense search and BM25 fail on different queries. That suggests using both. Next, you combine the two result lists in two ways, and you choose the method and its setting on the validation queries.

Next: Hybrid search