Where dense search fails
- Lesson 16
- 40 min
- Module 5: Dense and hybrid retrieval
Where we are
Dense search found products that share no word with the query. Tomás thinks it can replace keyword search. In this lesson, we measure dense search against BM25 by query type, and we find where dense search fails.
Lesson 16 of 30 · Search and Ranking: Find the Best MatchWhere dense search fails
1. The search problem
2. Measure relevance before you change anything
3. Lexical search: the baseline to beat
4. Understand the query
5. Dense and hybrid retrieval
6. Re-rank the top results
7. The parts of search that people see
8. Run search in production
What you will be able to do
Find the kinds of queries where search by meaning fails, so you know what a lexical search must still do.
NotebookFind the failures of dense search in a notebook
The notebook measures BM25 and dense search query by query on the validation queries. You compute the paired difference by source and by kind of query, and find the worst queries. Then you test dense search on product codes without the code rule. Its setup cells download the data, so it needs no local setup. About 30 minutes.
Colab and Kaggle open your own copy. They need a free Google or Kaggle account. Stuck? See the solution notebook.
Before you startWhat to know and to have ready
What you need to know
- Dense search and the run
dense, from Search by meaning. search compareand slices with--by, from Build a test set that represents your users.- The kinds of query (broad, specific, known-item), from Kinds of search.
What you need on your computer
- The project
larkfield-searchwith the files of the last lesson. This lesson adds no file.
An assistant who understands ideas but not names
Section titled “An assistant who understands ideas but not names”Think again of the shop assistant who understands what customers mean. A customer says: “I want the Kelly Clarkson light fixtures.” The assistant has never heard of Kelly Clarkson. She hears “light fixtures”, and she walks to the lights. Any lights.
A second customer reads from a note: “cushion replacemnet insert”, with a spelling mistake. The assistant does not know the word “replacemnet”. She guesses from “cushion” and shows pillow covers.
The assistant is good with ideas. She is weak with names, codes and odd spellings, because she never learned them. Dense search has the same weakness, for the same reason.
The problem: an average that hides failures
Section titled “The problem: an average that hides failures”Tomás has an argument. “Dense search is modern. BM25 is from the 1990s. Let us replace BM25.” The average says no: 0.519 against 0.620. But an average mixes many kinds of queries. Dense search could be better on some and much worse on others. To decide what to build, we need to know which.
Start with three queries from the test set, side by side:
Text description of the diagram
Three test queries, each with the top 5 of BM25 (the run bm25-norm+spell+syn) and of dense search. “kelly clarkson light fixtures” is a known-item query. BM25 scores NDCG@10 0.943: four exact lights and one partial, a clarkson chandelier. Dense scores 0.113: vanity lights; three are not judged and two partial, and a kelly lantern is fifth. “cushion replacemnet insert” is a specific query with a misspelling. BM25 scores 1.000 with five partial replacement cushions. Dense scores 0.095: pillow covers, a cushen comforter set and a cushion; four are not judged. “verden callan” is a known-item query. BM25 scores 1.000 with the verden callan comforter set first. Dense scores 0.229: a task chair and a daybed come first, and the comforter set is third.
Each failure has a clear cause:
- A brand name. “Kelly Clarkson” is a brand of home products. Dense search reads “light fixtures” and returns vanity lights. BM25 matches the words “kelly” and “clarkson” in the product text.
- A misspelling. “replacemnet” is not a word. BM25’s run corrects it first, with the spelling correction of Fix spelling. Dense search reads the query as it is. The model splits the odd word into pieces and finds “cushen” and pillow covers.
- A rare product name. “verden callan” is the name of one comforter set. The model has no idea what it means, so the vector points at nothing in particular. BM25 finds both rare words in one product.
The idea: measure by slice, not only on average
Section titled “The idea: measure by slice, not only on average”You met slices in Kinds of search: groups of queries with the same kind, source or language. Measuring by slice shows where a method wins and where it loses. The paired difference with its interval tells us if a difference is real.
Why does dense search fail on these slices? A bi-encoder squeezes a whole text into 384 numbers. It keeps what it saw often in training: common ideas, like “a light” or “a chair”. It loses what it saw rarely: a brand, a product name, a code, a misspelt word. BM25 keeps every word exactly, and rare words get the highest IDF. So the two methods are strong in opposite places.
A tiny example: one query, two scores
Section titled “A tiny example: one query, two scores”Here is one slice by hand: the three queries of the diagram, plus one where dense search wins. The numbers are NDCG@10 from the runs.
| Query | Kind | BM25 | Dense | Dense minus BM25 |
|---|---|---|---|---|
| kelly clarkson light fixtures | known-item | 0.943 | 0.113 | −0.830 |
| cushion replacemnet insert | specific, misspelt | 1.000 | 0.095 | −0.905 |
| verden callan | known-item | 1.000 | 0.229 | −0.771 |
| palram harmony greenhouses | known-item | 0.358 | 0.915 | +0.557 |
The mean difference of these four is (−0.830 − 0.905 − 0.771 + 0.557) ÷ 4 = −0.487. One query goes the other way. Palram is a brand too, but BM25 put three greenhouse accessories first: an anchor kit, plant hangers and a heater. Dense search put real greenhouses first. So “brand names fail” is a tendency, not a law. That is why we measure many queries, not four.
Larkfield: dense against BM25, slice by slice
Section titled “Larkfield: dense against BM25, slice by slice”search compare gives the paired difference on every test query, and --by splits it. First by kind of query:
search compare bm25-norm+spell+syn dense --by qtypedense minus bm25-norm+spell+syn on 241 test queries: ndcg@10 -0.101 (95% interval -0.139 to -0.062); better on 64, worse on 119, the same on 58
qtype queries bm25-norm+spell+syn dense difference low high worse broad 25 0.703 0.535 -0.168 -0.307 -0.022 15known-item 80 0.729 0.659 -0.071 -0.137 -0.012 25 specific 136 0.540 0.434 -0.106 -0.158 -0.055 79Then by source of the query:
search compare bm25-norm+spell+syn dense --by sourcedense minus bm25-norm+spell+syn on 241 test queries: ndcg@10 -0.101 (95% interval -0.139 to -0.062); better on 64, worse on 119, the same on 58
source queries bm25-norm+spell+syn dense difference low high worse code 20 1.000 1.000 0.000 0.000 0.000 0 planted 3 0.515 0.333 -0.182 -0.546 0.188 2translated 59 0.301 0.216 -0.085 -0.186 0.007 25 typo 40 0.659 0.403 -0.256 -0.357 -0.158 29 wands 119 0.703 0.632 -0.071 -0.111 -0.027 63The compare uses 241 queries, not 243: two test queries have only irrelevant judgments, so their NDCG has no value.
Text description of the chart
Two panels. The axis is the change in NDCG@10, dense minus BM25, from −0.6 to +0.3; 0 means no change. By kind of query: broad (25 queries) −0.168 (−0.307 to −0.022), highlighted; known-item (80) −0.071 (−0.137 to −0.012); specific (136) −0.106 (−0.158 to −0.055). By source: English WANDS queries (119) −0.071 (−0.111 to −0.027); French (59) −0.085 (−0.186 to +0.007); misspelt (40) −0.256 (−0.357 to −0.158), highlighted; product codes (20) 0.000, because both runs keep the code rule; planted broad (3) −0.182 (−0.546 to +0.188).
Read the chart from the worst slice:
- Misspelt queries: −0.256. The largest loss. BM25’s run corrects spelling first; dense search does not.
- Broad queries: −0.168. One- or two-word queries, like “couch” or “industrial”. A short query gives the model little meaning to work with.
- Specific queries: −0.106; known-item queries: −0.071. Worse, but less. Known-item queries often contain a name, which BM25 matches exactly.
- English WANDS queries: −0.071. Even on clean English queries, dense search loses.
- French: −0.085, and the interval reaches +0.007. Not clear. The last lesson of this module looks at French closely.
Dense search is still better on 64 queries. It is not useless. It is a different tool, and its failures fall on different queries.
Short, broad queries
Section titled “Short, broad queries”The broad query “couch” shows how bad a short query can be:
search query "couch" --run dense -k 5dense: "couch" -> 5 results (query P-002: judged results are marked) 1. LK-69717 sectional microfibra box cushion chaise lounge slipcover Slipcovers 102.99 [irrelevant] 2. LK-71532 desks with hutch 56.99 [not judged] 3. LK-43423 cushion Furniture Cushions 13.99 [irrelevant] 4. LK-37928 cora chick fuzzy faux throw Blankets And Throws 17.99 [out of stock] [not judged] 5. LK-29009 crock-pot 2.5 qt . casserole slow cooker 62.99 [not judged]A slipcover, a desk, a cushion, a throw and a slow cooker. NDCG@10 is 0.000. “couch” is a planted query, a practice problem we made on purpose. Its judgments come from a rule on the product class. The model places the word near many things that go with a couch, but not near couches. BM25 finds products with “couch” in the title, and two sofas in its top 5 are exact.
Product codes
Section titled “Product codes”Every run since Module 4 keeps the code rule. So the code slice shows no difference: both runs score 1.000. What does dense search do alone? We ran it on the 20 code queries with the rule removed. NDCG@10 was 0.000. The right product was in the top 10 for 0 of 20 queries. A code like “lk 68270” has no meaning that a model could learn. Keep the rule in front of every method.
Numbers: not the failure we expected
Section titled “Numbers: not the failure we expected”Many people expect dense search to fail on sizes, such as “4 inch” or “18x18”. The test set has 10 queries with a number that are not codes. On them, BM25 scores 0.378 and dense search 0.450. Dense search did better here. In “door pulls bronze matte 4 inch”, it put a 4-inch bronze pull first, while BM25 put an iron pull first. Ten queries are too few for a firm conclusion. But it is a useful warning: test your expectations on your own data.
Negation
Section titled “Negation”What about “no”, “not” and “without”? The query set has no such query, so we wrote one. It is not in the course’s query set:
search query "chair with no arms" --run dense -k 5dense: "chair with no arms" -> 5 results 1. LK-58749 chair 67.99 2. LK-40140 leather arm chair in cognac 239.99 3. LK-26323 callie armchair Accent Chairs 78.99 4. LK-27914 armchair Reception Seating Chairs 383.99 5. LK-53018 abarca 33 '' wide armchair Accent Chairs 74.99Four armchairs in the top 5: the opposite of what the person asked. The cosines explain it. “chair with no arms” and “armless chair” have a cosine of 0.906. “chair with no arms” and “armchair” have 0.904. For the model, the word “no” almost does not change the vector. BM25 does not understand “no” either. Negation needs a rule or a filter, such as “armless” as a product attribute.
When dense search is the wrong tool
Section titled “When dense search is the wrong tool”Do not use dense search alone for:
| Kind of query | What happens | What to do instead |
|---|---|---|
| Product codes | never finds the product (0.000 without the rule) | an exact-match rule |
| Brand and product names | returns the general kind of product | BM25, which keeps rare words |
| Misspellings | reads the misspelt word as it is (−0.256) | spelling correction before any search |
| Short, broad queries | lands near related things, not the thing (−0.168) | BM25 on the title and the product class |
| Negation | ignores “no” and “without” | a filter on an attribute |
Use dense search where these do not apply: descriptions in the customer’s own words, and as a second list beside BM25. On 64 test queries it is better. The next lesson combines the two lists, so each method can cover the other’s failures.
What this measurement costs. Nothing new: search compare --by reuses the runs. The real cost is attention. A team that reports only the average would have seen “dense is 0.1 worse” and stopped. The slices tell you what to fix.
Common mistake
A model upgrade that wins on average and loses a slice
Symptom: a new embedding model raises NDCG@10 by 0.02 on the test queries. After the release, support calls about “I typed the exact name and it is not there” go up.
Cause: the new model is better on descriptions and worse on rare names. Descriptions are many of the test queries, so the average rises. Known-item queries fall, and those customers know exactly what they want.
Diagnosis: search compare old new --by qtype and --by source. Look for a slice with an interval below zero, even when the overall difference is positive.
Fix: keep BM25 for exact words (the next lesson), or block a release when an important slice loses. Decide which slices are important before you measure, not after.
Consequence in AI work: every model change needs a check by slice. A model that is better on average can be worse for a group of users, and the average will not show it.
Try it yourself
Section titled “Try it yourself”Pick a brand from the catalogue and test it. Predict first: will dense search find it?
search query "palram harmony greenhouses" --run dense -k 3search query "palram harmony greenhouses" --run bm25-norm+spell+syn -k 3What we got: dense search put three greenhouses first (partial). BM25 put three greenhouse accessories first (irrelevant), and one of them is a Palram heater. So here, the brand name led BM25 to the wrong product. Then run search compare bm25-norm+spell+syn dense --by band. Is dense search worse on the head queries (the most searched) or on the tail? What we got: head −0.212 (16 queries, interval −0.377 to −0.065), torso −0.043, tail −0.104. The most searched queries lose the most, and they are often short and broad.
Knowledge check
Section titled “Knowledge check”Check your understanding
Dense search is worse than BM25 by 0.101 on average. On which slice is the loss largest, and what in the BM25 run explains it?
Show the answer
The misspelt queries: −0.256 (interval −0.357 to −0.158) on 40 queries. The BM25 run bm25-norm+spell+syn corrects spelling before it searches, so “replacemnet” becomes “replacement”. Dense search reads the query as it is, and the model maps the odd word to unrelated products. Spelling correction before dense search would remove part of this loss.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
Tomás says: “The code slice shows 0.000 difference, so dense search handles product codes as well as BM25.” Is he right?
Show the answer
No. Both runs keep the exact-match rule for codes, and the rule puts the right product first. So both score 1.000, and the difference is zero. Without the rule, dense search scored 0.000 on the 20 code queries and never had the right product in its top 10. A code has no meaning that a model can learn. The slice measured the rule, not dense search.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
A colleague says: “Dense search understands meaning, so it understands ‘chair with no arms’.” What did the run show, and what would you build instead?
Show the answer
The run put four armchairs in the top 5. “chair with no arms” has a cosine of 0.906 with “armless chair” and 0.904 with “armchair”. The word “no” almost does not move the vector. Neither dense search nor BM25 understands negation. Build a filter on a product attribute (“armless”, “without arms”). Then detect the negation in the query with a rule, as Detect what the query asks for detected prices.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Recap
- An average hides where a method fails. Measure by slice: by kind, by source, by language, with the paired difference and its interval.
- Dense minus BM25 is −0.101 on average, and worse on every kind of query: broad −0.168, specific −0.106, known-item −0.071.
- The largest loss is on misspellings (−0.256): the BM25 run corrects spelling first, dense search does not.
- Dense search fails on brand and product names, codes (0.000 without the rule) and short broad queries (“couch”), and it ignores negation.
- It is better on 64 test queries, and it did better on the 10 queries with sizes. Test expectations on your own data.
Dense search and BM25 fail on different queries. That suggests using both. Next, you combine the two result lists in two ways, and you choose the method and its setting on the validation queries.
Next: Hybrid search