Skip to content
Français

Hybrid search

  • Lesson 17
  • 45 min
  • Module 5: Dense and hybrid retrieval

Where we are

BM25 and dense search fail on different queries. We can use both. In this lesson, we combine the two lists with reciprocal rank fusion and with score blending, and we choose the method on the validation queries.

Plays from YouTube. Nothing loads from YouTube until you press play. Privacy

What you will be able to do

Combine a lexical list and a dense list into one ranking, and choose how by measurement on the validation queries.

NotebookBuild both kinds of hybrid search in a notebookColab · Kaggle · GitHub

The notebook fuses two short lists by hand, then builds reciprocal rank fusion and score blending over BM25 and dense search. You tune k and alpha on the validation queries, and measure the chosen setting once on the test queries. Its setup cells download the data, so it needs no local setup. About 35 minutes.

Colab and Kaggle open your own copy. They need a free Google or Kaggle account. Stuck? See the solution notebook.

Before you startWhat to know and to have readyAbout 45 minutes

What you need to know

What you need on your computer

  • The project larkfield-search with the files of Search by meaning. The two hybrid runs are already in runs/dense.py. This lesson adds no file.

A shop has two assistants. Ana reads the product labels word by word. She is perfect with names, codes and sizes. Leo understands what customers mean, even in odd words or in French. He is bad with names.

A customer asks a question. Each assistant writes a list of five products. The manager has to show one list. She could pick Ana’s list every time. That loses Leo’s good answers. A better way: put first the products that are high on both lists. A product that both assistants like is probably right.

This is hybrid search: one ranking from a lexical list and a dense list.

In the last lesson, dense search was better than BM25 on 64 test queries and worse on 119. Choosing one method per query would be ideal, but we cannot know in advance which one is right.

Merging the two lists is not simple either. Look at the scores for one query, “door pulls bronze matte 4 inch”:

  • BM25 scores are sums of word parts. For this query, the best product scores 12.3; scores have no fixed top.
  • The cosines of dense search are between 0.847 and 0.897 for its top 100.

Adding the raw numbers would let BM25 decide everything, because its numbers are bigger. We need a way to put the two lists on the same footing. There are two common ways.

Reciprocal rank fusion (RRF) ignores the scores. It uses only the rank of each product in each list. A product gets, from each list, 1 ÷ (k + rank), and the parts are added:

RRF(p)=∑lists1k+rank of p\text{RRF}(p) = \sum_{\text{lists}} \frac{1}{k + \text{rank of } p}

k is a constant, usually 60. A product that a list does not contain gets nothing from that list. A large k makes the ranks matter less; a small k gives much more to the top ranks. RRF needs no tuning and no scale. That is why it is the usual first hybrid.

Here is RRF on the test query “door pulls bronze matte 4 inch”, with k 60. Each list holds its top 100. These are the top 5 after fusion:

Product BM25 rank Dense rank 1 ÷ (60 + BM25 rank) 1 ÷ (60 + dense rank) RRF Judges
traditional bronze pull 8’’ center to center bar pull 2 8 0.01613 0.01471 0.03083 partial
traditional bronze pull 4’’ center to center arch pull 18 1 0.01282 0.01639 0.02921 partial
edge 1 1/4’’ center to center finger pull 11 21 0.01408 0.01235 0.02643 partial
4’’ center to center arch pull 19 17 0.01266 0.01299 0.02565 partial
sliding closet door finger pull 10 58 0.01429 0.00847 0.02276 partial

Check the first row: 1 ÷ 62 = 0.01613, and 1 ÷ 68 = 0.01471. The sum of the rounded parts is 0.03084; the table adds the exact parts, so it shows 0.03083. The product is second for BM25 and eighth for dense search, so it is high on both lists, and it comes first. The second row is first for dense search, but only 18th for BM25. The irrelevant iron pull that BM25 put first is not in the fused top 5. Its dense rank is low, so its sum is small.

Score blending keeps the scores, and solves the scale problem by scaling each list to the range 0 to 1:

  • BM25: divide every score by the best score of the query. The best product gets 1.
  • Dense: over the top 100 cosines, the lowest becomes 0 and the highest 1. Products below the top 100 get 0.

Then mix the two with a weight alpha between 0 and 1:

blend(p)=α×BM25 scaled+(1−α)×dense scaled\text{blend}(p) = \alpha \times \text{BM25 scaled} + (1 - \alpha) \times \text{dense scaled}

With alpha 1, the blend is BM25 alone; with alpha 0, dense alone. The project uses alpha 0.8: mostly BM25, with a fifth from dense search. Here is the same query, with every value:

Product BM25 scaled Dense cosine Dense scaled 0.8 × BM25 + 0.2 × dense Judges
traditional bronze pull 8’’ center to center bar pull 0.941 0.870 0.462 0.845 partial
iron craft ii rustic 4’’ center to center bar pull 1.000 0.841 0.000 0.800 irrelevant
traditional bronze pull 4’’ center to center arch pull 0.749 0.897 1.000 0.799 partial
mission vertical 4 1/5’’ center to center ring pull 0.940 0.836 0.000 0.752 partial
edge 1 1/4’’ center to center finger pull 0.807 0.862 0.308 0.707 partial

Check the third row: 0.8 × 0.749 + 0.2 × 1.000 = 0.599 + 0.200 = 0.799. The irrelevant iron pull is second: its BM25 score is the best (1.000), and that gives it 0.800 alone. Its cosine of 0.841 is below the top 100 of dense search, so dense scaled is 0. With alpha 0.8, a strong BM25 score is hard to beat.

So the two methods disagree on this query. RRF trusts the two lists equally. The blend trusts BM25 four times more. Which is better on all queries is a question for measurement.

Both methods have a setting: k for RRF, alpha for the blend. We choose each on the 121 validation queries and keep the test queries for the final measurement, as in BM25.

RRF k 10 30 60 100
NDCG@10, validation 0.609 0.617 0.621 0.620

RRF is flat between k 30 and 100. The project keeps the usual 60.

NDCG@10 of score blending on the 121 validation queries, for each alpha. More weight on BM25 is better, up to 0.8.
Text description of the chart

A line chart with alpha, the weight of BM25, on the horizontal axis from 0.2 to 0.8, and NDCG@10 on the validation queries on the vertical axis. alpha 0.2: 0.559; alpha 0.3: 0.570; alpha 0.4: 0.582; alpha 0.5: 0.594; alpha 0.6: 0.606; alpha 0.7: 0.618; alpha 0.8: 0.619. The line rises steadily and flattens between 0.7 and 0.8.

The blend improves as alpha grows, and alpha 0.8 is best (0.619). The grid stops at 0.8: alpha 1 is BM25 alone, which we already have. Between 0.7 and 0.8 the difference is 0.001, which is noise on 121 queries. Both settings say the same thing: BM25 should lead.

You can reproduce two points of the grid:

Terminal window
search eval --run hybrid-blend --set alpha=0.5 --split validation
search eval --run hybrid-rrf --set rrf_k=10 --split validation
hybrid-blend on 121 validation queries: NDCG@10 0.594 (95% interval 0.528 to 0.657)
recall@100 0.585, MRR 0.522, zero results 0.0%, judged in the top 10 67.2%; 1.5 ms a query (p50), 1.8 ms (p95)
hybrid-rrf on 121 validation queries: NDCG@10 0.609 (95% interval 0.545 to 0.670)
recall@100 0.563, MRR 0.581, zero results 0.0%, judged in the top 10 68.3%; 1.5 ms a query (p50), 1.8 ms (p95)

Larkfield: measure both on the test queries

Section titled “Larkfield: measure both on the test queries”

Now measure each chosen setting once on the test queries, against the best lexical run:

Terminal window
search compare bm25-norm+spell+syn hybrid-rrf --by qtype
hybrid-rrf minus bm25-norm+spell+syn on 241 test queries: ndcg@10 +0.001 (95% interval -0.022 to +0.024); better on 92, worse on 78, the same on 71
qtype queries bm25-norm+spell+syn hybrid-rrf difference low high worse
broad 25 0.703 0.616 -0.087 -0.168 -0.011 12
known-item 80 0.729 0.744 0.015 -0.024 0.053 14
specific 136 0.540 0.549 0.009 -0.022 0.038 52
Terminal window
search compare bm25-norm+spell+syn hybrid-blend --by qtype
hybrid-blend minus bm25-norm+spell+syn on 241 test queries: ndcg@10 +0.016 (95% interval +0.007 to +0.025); better on 90, worse on 41, the same on 110
qtype queries bm25-norm+spell+syn hybrid-blend difference low high worse
broad 25 0.703 0.689 -0.014 -0.043 0.012 7
known-item 80 0.729 0.755 0.026 0.011 0.041 6
specific 136 0.540 0.556 0.016 0.003 0.029 28

Read the first lines:

  • RRF: +0.001, with an interval from −0.022 to +0.024. No real gain. It is better on 92 queries and worse on 78: it moves many results, and the wins and losses cancel.
  • Blend: +0.016, with an interval from +0.007 to +0.025. A small gain, but a clear one. It is better on 90 queries and worse on only 41.

And the slices: RRF is worse than BM25 on broad queries, by 0.087 (−0.168 to −0.011). The blend is not.

  • BM25 (bm25-norm+spell+syn)
  • dense
  • hybrid-rrf
  • hybrid-blend
NDCG@10 by kind of query for each method, on the test queries, with 95% intervals. On broad queries, RRF falls below BM25; the blend stays close to BM25 everywhere and gains a little on known-item and specific queries.
Text description of the chart

Three panels, one per kind of query, each with four methods and their 95% intervals. Broad (27 queries): BM25 0.703, dense 0.535, hybrid-rrf 0.616, hybrid-blend 0.689. Known-item (80): BM25 0.729, dense 0.659, hybrid-rrf 0.744, hybrid-blend 0.755. Specific (136): BM25 0.540, dense 0.434, hybrid-rrf 0.549, hybrid-blend 0.556. The intervals of the four methods overlap in every panel; the paired differences above are more precise, because they compare the same queries.

The chart counts 27 broad queries; two of them have no defined NDCG, so the compare above uses 25. The intervals of the four methods overlap in each panel. That does not mean “no difference”. The paired comparison above is more precise, because it compares each query with itself.

Why does RRF hurt broad queries? Look at four test queries side by side:

Four test queries, each with the top 5 of BM25, dense and hybrid RRF, and the judges' grades. door pulls bronze matte 4 inch, specific: BM25 0.313 (an irrelevant iron pull first), dense 0.842, hybrid 0.863 with five partial pulls. kelly clarkson light fixtures, known-item: BM25 0.943 with four exact lights, dense 0.113 with vanity lights, hybrid 0.424 with three partial vanity fixtures and an exact sconce fourth. couch, broad: BM25 0.546 with two exact sofas, dense 0.000 with a slipcover, a desk, a cushion, a throw and a slow cooker, hybrid 0.078 with a cushion cover and the slipcover first. chaise ergonomique, specific, French: BM25 0.032 with chaise lounges, dense 1.000 with five exact ergonomic chairs, hybrid 0.578 with ergonomic chairs and chaise lounges in turn.
BM25, dense search and hybrid RRF on four test queries. RRF helps when the dense list is good and hurts when it is bad.
Text description of the diagram

Four test queries, each with the top 5 of the three runs and the judges’ grades. “door pulls bronze matte 4 inch” (specific): BM25 scores 0.313, with an irrelevant iron pull first; dense 0.842; hybrid RRF 0.863, with five partial pulls. “kelly clarkson light fixtures” (known-item): BM25 0.943 with four exact lights; dense 0.113 with vanity lights; hybrid 0.424, with three partial vanity fixtures first and an exact sconce fourth. “couch” (broad): BM25 0.546 with two exact sofas in its top 5; dense 0.000 with a slipcover, a desk, a cushion, a throw and a slow cooker; hybrid 0.078 with a cushion cover and the slipcover first. “chaise ergonomique” (specific, French): BM25 0.032 with chaise lounges; dense 1.000 with five exact ergonomic chairs; hybrid 0.578, with ergonomic chairs and chaise lounges in turn.

The pattern is clear. When one list is good and the other is bad, RRF lands in the middle. It gives the bad list the same vote as the good one. On “door pulls” and “chaise ergonomique”, the middle is a big gain over BM25. On “kelly clarkson” and “couch”, it is a big loss. Broad queries are where dense search is weakest (it lost 0.168 there), so RRF loses there too.

The blend is more careful. With alpha 0.8, dense search can only reorder products that BM25 already scores well. On “couch”, the blend scores 0.546, the same as BM25. On “door pulls”, it gains less than RRF (0.647). It wins less often, but it rarely loses badly.

Run What it does NDCG@10 (95% interval) recall@100 MRR judged@10 p50 / p95 ms
bm25-norm+spell+syn BM25 with normalising, spelling and synonyms 0.620 (0.574 to 0.662) 0.546 0.626 76.0% 1.1 / 2.0
dense cosine of multilingual embeddings 0.519 (0.471 to 0.566) 0.453 0.510 57.4% 8.0 / 9.8
hybrid-rrf RRF (k 60) of the two top 100 lists 0.621 (0.576 to 0.663) 0.540 0.634 66.9% 9.2 / 15.1
hybrid-blend 0.8 × BM25 + 0.2 × dense, each scaled 0 to 1 0.636 (0.590 to 0.678) 0.560 0.631 69.7% not measured live (about as hybrid-rrf)

The times are live, for 40 English test queries on one thread, with the model encoding each query. Hybrid search costs one dense search more than BM25: about 8 ms. That fits easily in Amira’s budget of 200 ms.

One more thing to know: judged@10 is lower for every run that uses dense search. Some of the hybrid’s “losses” are unjudged products that may be fine. The language model labelled the unjudged products. With its labels, hybrid RRF moves from 0.585 to 0.602 on the English and French test queries (Judgments from a language model). That small change does not reverse the result.

Use RRF when you cannot compare the scores and you have no judged queries to tune with. It is simple and needs no setting. But know that it gives a bad list a full vote. Check it by slice before a release.

Use score blending when you have validation judgments to choose alpha. It lets the stronger method lead. Here it gives the only clear gain of this module: +0.016.

Use neither if the gain does not pay for the cost. Hybrid search needs the model on the search server and the product vectors in memory (about 31 MB here). It also needs a new encoding step when products change. For +0.016, Omar might say no. In Module 6, the hybrid list becomes the input of a learned ranker, and there it pays off.

The RAG course uses the same two fusions for retrieval for a language model: see Lexical, dense and hybrid search.

Common mistake

A hybrid that wins on average and loses on the head

Symptom: after a switch to hybrid RRF, the overall NDCG@10 does not change. But the shop’s most common searches, like “couch”, now show cushions and throws.

Cause: RRF gives the dense list a full vote. The most searched queries are short and broad, and dense search is weakest on them. The average hides this, because the losses on broad queries cancel gains elsewhere.

Diagnosis: search compare bm25-norm+spell+syn hybrid-rrf --by qtype and --by band. A negative interval on broad queries, or on the head band, is the warning.

Fix: use a blend with a weight chosen on validation, or give the dense list less weight in the fusion. Check the head queries by eye before each release.

Consequence in AI work: combining two models is not automatically better than the better one. A fusion inherits the failures of both parts on the queries where one part is bad.

Before you run anything, predict: on the validation queries, which is better, RRF with k 10 or with k 100? Then run:

Terminal window
search eval --run hybrid-rrf --set rrf_k=100 --split validation

We measured 0.620 with k 100, against 0.609 with k 10. A difference of 0.011 on 121 queries is close to noise. Then try a blend that trusts dense search more, on the test queries:

Terminal window
search compare bm25-norm+spell+syn hybrid-blend --set alpha=0.5 --by qtype

What we got: −0.028 overall (−0.055 to −0.002), and −0.095 on broad queries. More weight on dense search brings back the losses of RRF. This is a look at the test queries for learning only; the setting of the project stays the one chosen on validation.

Check your understanding

With k 60, a product is 3rd in the BM25 list and 40th in the dense list. Another product is 1st in the dense list and not in the BM25 top 100. Which one does RRF put higher?

Show the answer

The first product gets 1 ÷ 63 + 1 ÷ 100 = 0.01587 + 0.01000 = 0.02587. The second gets only 1 ÷ 61 = 0.01639, because the BM25 list gives it nothing. So the first product is higher. RRF rewards products that both lists contain, even at a modest rank.

Check your understanding

Hybrid RRF is +0.001 overall (interval −0.022 to +0.024) and −0.087 on broad queries (−0.168 to −0.011). Omar asks: “Do we release it?” What do you answer?

Show the answer

No. Overall, RRF gives no measurable gain: the interval is around zero. On broad queries it is clearly worse, and broad queries are often the most searched ones (“couch”, “industrial”). So it costs a model on the server and changes many pages for no gain, with a risk on common searches. The blend (alpha 0.8) is the better choice: +0.016, with an interval above zero, and no clear loss on any kind of query.

Check your understanding

Why must alpha be chosen on the validation queries, and not on the test queries?

Show the answer

If you try many alphas on the test queries and keep the best, the best result is partly luck on those queries. The test score then overstates how the setting will do on new queries. Choosing on validation and measuring once on test keeps the test score fair. In BM25, the field weights tuned on validation lost on test: settings do not always transfer.

Recap

  • Hybrid search combines a lexical list and a dense list. The raw scores are on different scales, so you cannot just add them.
  • Reciprocal rank fusion adds 1 ÷ (k + rank) from each list (k 60). It needs no tuning, and it gives each list an equal vote.
  • Score blending scales each list to 0 to 1 and mixes them with alpha. Alpha 0.8, chosen on validation, lets BM25 lead.
  • On the test queries: RRF +0.001 (−0.022 to +0.024), no real gain, and −0.087 on broad queries. The blend: +0.016 (+0.007 to +0.025), small but clear. NDCG@10 0.636.
  • A fusion inherits the failures of the bad list on the queries where that list is bad. Check every hybrid by slice.

Dense search was clearly better on one French query, “chaise ergonomique”. But on the French slice as a whole, every method so far is weak. Next, you measure French on its own, and you compare a multilingual model with translating the query.

Next: Many languages