Hybrid search
- Lesson 17
- 45 min
- Module 5: Dense and hybrid retrieval
Where we are
BM25 and dense search fail on different queries. We can use both. In this lesson, we combine the two lists with reciprocal rank fusion and with score blending, and we choose the method on the validation queries.
Lesson 17 of 30 · Search and Ranking: Find the Best MatchHybrid search
1. The search problem
2. Measure relevance before you change anything
3. Lexical search: the baseline to beat
4. Understand the query
5. Dense and hybrid retrieval
6. Re-rank the top results
7. The parts of search that people see
8. Run search in production
What you will be able to do
Combine a lexical list and a dense list into one ranking, and choose how by measurement on the validation queries.
NotebookBuild both kinds of hybrid search in a notebook
The notebook fuses two short lists by hand, then builds reciprocal rank fusion and score blending over BM25 and dense search. You tune k and alpha on the validation queries, and measure the chosen setting once on the test queries. Its setup cells download the data, so it needs no local setup. About 35 minutes.
Colab and Kaggle open your own copy. They need a free Google or Kaggle account. Stuck? See the solution notebook.
Before you startWhat to know and to have ready
What you need to know
- Dense search and its failures, from Search by meaning and Where dense search fails.
- Validation and test queries, and why you tune on validation only, from Build a test set that represents your users.
What you need on your computer
- The project
larkfield-searchwith the files of Search by meaning. The two hybrid runs are already inruns/dense.py. This lesson adds no file.
Two shop assistants, one list
Section titled “Two shop assistants, one list”A shop has two assistants. Ana reads the product labels word by word. She is perfect with names, codes and sizes. Leo understands what customers mean, even in odd words or in French. He is bad with names.
A customer asks a question. Each assistant writes a list of five products. The manager has to show one list. She could pick Ana’s list every time. That loses Leo’s good answers. A better way: put first the products that are high on both lists. A product that both assistants like is probably right.
This is hybrid search: one ranking from a lexical list and a dense list.
The problem: two lists, two scales
Section titled “The problem: two lists, two scales”In the last lesson, dense search was better than BM25 on 64 test queries and worse on 119. Choosing one method per query would be ideal, but we cannot know in advance which one is right.
Merging the two lists is not simple either. Look at the scores for one query, “door pulls bronze matte 4 inch”:
- BM25 scores are sums of word parts. For this query, the best product scores 12.3; scores have no fixed top.
- The cosines of dense search are between 0.847 and 0.897 for its top 100.
Adding the raw numbers would let BM25 decide everything, because its numbers are bigger. We need a way to put the two lists on the same footing. There are two common ways.
Idea 1: reciprocal rank fusion
Section titled “Idea 1: reciprocal rank fusion”Reciprocal rank fusion (RRF) ignores the scores. It uses only the rank of each product in each list. A product gets, from each list, 1 ÷ (k + rank), and the parts are added:
k is a constant, usually 60. A product that a list does not contain gets nothing from that list. A large k makes the ranks matter less; a small k gives much more to the top ranks. RRF needs no tuning and no scale. That is why it is the usual first hybrid.
A real example with every value
Section titled “A real example with every value”Here is RRF on the test query “door pulls bronze matte 4 inch”, with k 60. Each list holds its top 100. These are the top 5 after fusion:
| Product | BM25 rank | Dense rank | 1 ÷ (60 + BM25 rank) | 1 ÷ (60 + dense rank) | RRF | Judges |
|---|---|---|---|---|---|---|
| traditional bronze pull 8’’ center to center bar pull | 2 | 8 | 0.01613 | 0.01471 | 0.03083 | partial |
| traditional bronze pull 4’’ center to center arch pull | 18 | 1 | 0.01282 | 0.01639 | 0.02921 | partial |
| edge 1 1/4’’ center to center finger pull | 11 | 21 | 0.01408 | 0.01235 | 0.02643 | partial |
| 4’’ center to center arch pull | 19 | 17 | 0.01266 | 0.01299 | 0.02565 | partial |
| sliding closet door finger pull | 10 | 58 | 0.01429 | 0.00847 | 0.02276 | partial |
Check the first row: 1 ÷ 62 = 0.01613, and 1 ÷ 68 = 0.01471. The sum of the rounded parts is 0.03084; the table adds the exact parts, so it shows 0.03083. The product is second for BM25 and eighth for dense search, so it is high on both lists, and it comes first. The second row is first for dense search, but only 18th for BM25. The irrelevant iron pull that BM25 put first is not in the fused top 5. Its dense rank is low, so its sum is small.
Idea 2: score blending
Section titled “Idea 2: score blending”Score blending keeps the scores, and solves the scale problem by scaling each list to the range 0 to 1:
- BM25: divide every score by the best score of the query. The best product gets 1.
- Dense: over the top 100 cosines, the lowest becomes 0 and the highest 1. Products below the top 100 get 0.
Then mix the two with a weight alpha between 0 and 1:
With alpha 1, the blend is BM25 alone; with alpha 0, dense alone. The project uses alpha 0.8: mostly BM25, with a fifth from dense search. Here is the same query, with every value:
| Product | BM25 scaled | Dense cosine | Dense scaled | 0.8 × BM25 + 0.2 × dense | Judges |
|---|---|---|---|---|---|
| traditional bronze pull 8’’ center to center bar pull | 0.941 | 0.870 | 0.462 | 0.845 | partial |
| iron craft ii rustic 4’’ center to center bar pull | 1.000 | 0.841 | 0.000 | 0.800 | irrelevant |
| traditional bronze pull 4’’ center to center arch pull | 0.749 | 0.897 | 1.000 | 0.799 | partial |
| mission vertical 4 1/5’’ center to center ring pull | 0.940 | 0.836 | 0.000 | 0.752 | partial |
| edge 1 1/4’’ center to center finger pull | 0.807 | 0.862 | 0.308 | 0.707 | partial |
Check the third row: 0.8 × 0.749 + 0.2 × 1.000 = 0.599 + 0.200 = 0.799. The irrelevant iron pull is second: its BM25 score is the best (1.000), and that gives it 0.800 alone. Its cosine of 0.841 is below the top 100 of dense search, so dense scaled is 0. With alpha 0.8, a strong BM25 score is hard to beat.
So the two methods disagree on this query. RRF trusts the two lists equally. The blend trusts BM25 four times more. Which is better on all queries is a question for measurement.
Choose the setting on validation queries
Section titled “Choose the setting on validation queries”Both methods have a setting: k for RRF, alpha for the blend. We choose each on the 121 validation queries and keep the test queries for the final measurement, as in BM25.
| RRF k | 10 | 30 | 60 | 100 |
|---|---|---|---|---|
| NDCG@10, validation | 0.609 | 0.617 | 0.621 | 0.620 |
RRF is flat between k 30 and 100. The project keeps the usual 60.
Text description of the chart
A line chart with alpha, the weight of BM25, on the horizontal axis from 0.2 to 0.8, and NDCG@10 on the validation queries on the vertical axis. alpha 0.2: 0.559; alpha 0.3: 0.570; alpha 0.4: 0.582; alpha 0.5: 0.594; alpha 0.6: 0.606; alpha 0.7: 0.618; alpha 0.8: 0.619. The line rises steadily and flattens between 0.7 and 0.8.
The blend improves as alpha grows, and alpha 0.8 is best (0.619). The grid stops at 0.8: alpha 1 is BM25 alone, which we already have. Between 0.7 and 0.8 the difference is 0.001, which is noise on 121 queries. Both settings say the same thing: BM25 should lead.
You can reproduce two points of the grid:
search eval --run hybrid-blend --set alpha=0.5 --split validationsearch eval --run hybrid-rrf --set rrf_k=10 --split validationhybrid-blend on 121 validation queries: NDCG@10 0.594 (95% interval 0.528 to 0.657)recall@100 0.585, MRR 0.522, zero results 0.0%, judged in the top 10 67.2%; 1.5 ms a query (p50), 1.8 ms (p95)hybrid-rrf on 121 validation queries: NDCG@10 0.609 (95% interval 0.545 to 0.670)recall@100 0.563, MRR 0.581, zero results 0.0%, judged in the top 10 68.3%; 1.5 ms a query (p50), 1.8 ms (p95)Larkfield: measure both on the test queries
Section titled “Larkfield: measure both on the test queries”Now measure each chosen setting once on the test queries, against the best lexical run:
search compare bm25-norm+spell+syn hybrid-rrf --by qtypehybrid-rrf minus bm25-norm+spell+syn on 241 test queries: ndcg@10 +0.001 (95% interval -0.022 to +0.024); better on 92, worse on 78, the same on 71
qtype queries bm25-norm+spell+syn hybrid-rrf difference low high worse broad 25 0.703 0.616 -0.087 -0.168 -0.011 12known-item 80 0.729 0.744 0.015 -0.024 0.053 14 specific 136 0.540 0.549 0.009 -0.022 0.038 52search compare bm25-norm+spell+syn hybrid-blend --by qtypehybrid-blend minus bm25-norm+spell+syn on 241 test queries: ndcg@10 +0.016 (95% interval +0.007 to +0.025); better on 90, worse on 41, the same on 110
qtype queries bm25-norm+spell+syn hybrid-blend difference low high worse broad 25 0.703 0.689 -0.014 -0.043 0.012 7known-item 80 0.729 0.755 0.026 0.011 0.041 6 specific 136 0.540 0.556 0.016 0.003 0.029 28Read the first lines:
- RRF: +0.001, with an interval from −0.022 to +0.024. No real gain. It is better on 92 queries and worse on 78: it moves many results, and the wins and losses cancel.
- Blend: +0.016, with an interval from +0.007 to +0.025. A small gain, but a clear one. It is better on 90 queries and worse on only 41.
And the slices: RRF is worse than BM25 on broad queries, by 0.087 (−0.168 to −0.011). The blend is not.
- BM25 (bm25-norm+spell+syn)
- dense
- hybrid-rrf
- hybrid-blend
Text description of the chart
Three panels, one per kind of query, each with four methods and their 95% intervals. Broad (27 queries): BM25 0.703, dense 0.535, hybrid-rrf 0.616, hybrid-blend 0.689. Known-item (80): BM25 0.729, dense 0.659, hybrid-rrf 0.744, hybrid-blend 0.755. Specific (136): BM25 0.540, dense 0.434, hybrid-rrf 0.549, hybrid-blend 0.556. The intervals of the four methods overlap in every panel; the paired differences above are more precise, because they compare the same queries.
The chart counts 27 broad queries; two of them have no defined NDCG, so the compare above uses 25. The intervals of the four methods overlap in each panel. That does not mean “no difference”. The paired comparison above is more precise, because it compares each query with itself.
Four queries, three methods
Section titled “Four queries, three methods”Why does RRF hurt broad queries? Look at four test queries side by side:
Text description of the diagram
Four test queries, each with the top 5 of the three runs and the judges’ grades. “door pulls bronze matte 4 inch” (specific): BM25 scores 0.313, with an irrelevant iron pull first; dense 0.842; hybrid RRF 0.863, with five partial pulls. “kelly clarkson light fixtures” (known-item): BM25 0.943 with four exact lights; dense 0.113 with vanity lights; hybrid 0.424, with three partial vanity fixtures first and an exact sconce fourth. “couch” (broad): BM25 0.546 with two exact sofas in its top 5; dense 0.000 with a slipcover, a desk, a cushion, a throw and a slow cooker; hybrid 0.078 with a cushion cover and the slipcover first. “chaise ergonomique” (specific, French): BM25 0.032 with chaise lounges; dense 1.000 with five exact ergonomic chairs; hybrid 0.578, with ergonomic chairs and chaise lounges in turn.
The pattern is clear. When one list is good and the other is bad, RRF lands in the middle. It gives the bad list the same vote as the good one. On “door pulls” and “chaise ergonomique”, the middle is a big gain over BM25. On “kelly clarkson” and “couch”, it is a big loss. Broad queries are where dense search is weakest (it lost 0.168 there), so RRF loses there too.
The blend is more careful. With alpha 0.8, dense search can only reorder products that BM25 already scores well. On “couch”, the blend scores 0.546, the same as BM25. On “door pulls”, it gains less than RRF (0.647). It wins less often, but it rarely loses badly.
The comparison table
Section titled “The comparison table”| Run | What it does | NDCG@10 (95% interval) | recall@100 | MRR | judged@10 | p50 / p95 ms |
|---|---|---|---|---|---|---|
bm25-norm+spell+syn |
BM25 with normalising, spelling and synonyms | 0.620 (0.574 to 0.662) | 0.546 | 0.626 | 76.0% | 1.1 / 2.0 |
dense |
cosine of multilingual embeddings | 0.519 (0.471 to 0.566) | 0.453 | 0.510 | 57.4% | 8.0 / 9.8 |
hybrid-rrf |
RRF (k 60) of the two top 100 lists | 0.621 (0.576 to 0.663) | 0.540 | 0.634 | 66.9% | 9.2 / 15.1 |
hybrid-blend |
0.8 × BM25 + 0.2 × dense, each scaled 0 to 1 | 0.636 (0.590 to 0.678) | 0.560 | 0.631 | 69.7% | not measured live (about as hybrid-rrf) |
The times are live, for 40 English test queries on one thread, with the model encoding each query. Hybrid search costs one dense search more than BM25: about 8 ms. That fits easily in Amira’s budget of 200 ms.
One more thing to know: judged@10 is lower for every run that uses dense search. Some of the hybrid’s “losses” are unjudged products that may be fine. The language model labelled the unjudged products. With its labels, hybrid RRF moves from 0.585 to 0.602 on the English and French test queries (Judgments from a language model). That small change does not reverse the result.
When to use which, and what it costs
Section titled “When to use which, and what it costs”Use RRF when you cannot compare the scores and you have no judged queries to tune with. It is simple and needs no setting. But know that it gives a bad list a full vote. Check it by slice before a release.
Use score blending when you have validation judgments to choose alpha. It lets the stronger method lead. Here it gives the only clear gain of this module: +0.016.
Use neither if the gain does not pay for the cost. Hybrid search needs the model on the search server and the product vectors in memory (about 31 MB here). It also needs a new encoding step when products change. For +0.016, Omar might say no. In Module 6, the hybrid list becomes the input of a learned ranker, and there it pays off.
The RAG course uses the same two fusions for retrieval for a language model: see Lexical, dense and hybrid search.
Common mistake
A hybrid that wins on average and loses on the head
Symptom: after a switch to hybrid RRF, the overall NDCG@10 does not change. But the shop’s most common searches, like “couch”, now show cushions and throws.
Cause: RRF gives the dense list a full vote. The most searched queries are short and broad, and dense search is weakest on them. The average hides this, because the losses on broad queries cancel gains elsewhere.
Diagnosis: search compare bm25-norm+spell+syn hybrid-rrf --by qtype and --by band. A negative interval on broad queries, or on the head band, is the warning.
Fix: use a blend with a weight chosen on validation, or give the dense list less weight in the fusion. Check the head queries by eye before each release.
Consequence in AI work: combining two models is not automatically better than the better one. A fusion inherits the failures of both parts on the queries where one part is bad.
Try it yourself
Section titled “Try it yourself”Before you run anything, predict: on the validation queries, which is better, RRF with k 10 or with k 100? Then run:
search eval --run hybrid-rrf --set rrf_k=100 --split validationWe measured 0.620 with k 100, against 0.609 with k 10. A difference of 0.011 on 121 queries is close to noise. Then try a blend that trusts dense search more, on the test queries:
search compare bm25-norm+spell+syn hybrid-blend --set alpha=0.5 --by qtypeWhat we got: −0.028 overall (−0.055 to −0.002), and −0.095 on broad queries. More weight on dense search brings back the losses of RRF. This is a look at the test queries for learning only; the setting of the project stays the one chosen on validation.
Knowledge check
Section titled “Knowledge check”Check your understanding
With k 60, a product is 3rd in the BM25 list and 40th in the dense list. Another product is 1st in the dense list and not in the BM25 top 100. Which one does RRF put higher?
Show the answer
The first product gets 1 ÷ 63 + 1 ÷ 100 = 0.01587 + 0.01000 = 0.02587. The second gets only 1 ÷ 61 = 0.01639, because the BM25 list gives it nothing. So the first product is higher. RRF rewards products that both lists contain, even at a modest rank.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
Hybrid RRF is +0.001 overall (interval −0.022 to +0.024) and −0.087 on broad queries (−0.168 to −0.011). Omar asks: “Do we release it?” What do you answer?
Show the answer
No. Overall, RRF gives no measurable gain: the interval is around zero. On broad queries it is clearly worse, and broad queries are often the most searched ones (“couch”, “industrial”). So it costs a model on the server and changes many pages for no gain, with a risk on common searches. The blend (alpha 0.8) is the better choice: +0.016, with an interval above zero, and no clear loss on any kind of query.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
Why must alpha be chosen on the validation queries, and not on the test queries?
Show the answer
If you try many alphas on the test queries and keep the best, the best result is partly luck on those queries. The test score then overstates how the setting will do on new queries. Choosing on validation and measuring once on test keeps the test score fair. In BM25, the field weights tuned on validation lost on test: settings do not always transfer.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Recap
- Hybrid search combines a lexical list and a dense list. The raw scores are on different scales, so you cannot just add them.
- Reciprocal rank fusion adds 1 ÷ (k + rank) from each list (k 60). It needs no tuning, and it gives each list an equal vote.
- Score blending scales each list to 0 to 1 and mixes them with alpha. Alpha 0.8, chosen on validation, lets BM25 lead.
- On the test queries: RRF +0.001 (−0.022 to +0.024), no real gain, and −0.087 on broad queries. The blend: +0.016 (+0.007 to +0.025), small but clear. NDCG@10 0.636.
- A fusion inherits the failures of the bad list on the queries where that list is bad. Check every hybrid by slice.
Dense search was clearly better on one French query, “chaise ergonomique”. But on the French slice as a whole, every method so far is weak. Next, you measure French on its own, and you compare a multilingual model with translating the query.
Next: Many languages