Skip to content
Français

Two-stage ranking

  • Lesson 19
  • 30 min
  • Module 6: Re-rank the top results

Where we are

In the last module, hybrid search found good candidates in about 10 milliseconds. But its order is not always right: for "beds that have leds", it shows light bulbs first. A slower model could order the top of the list better. In this lesson, we split ranking into two stages. We see why the first stage must find the good products, so that the second stage can rank them.

Plays from YouTube. Nothing loads from YouTube until you press play. Privacy

What you will be able to do

Put a slow, precise model where it pays: on the top few candidates of a fast search. You learn how two-stage ranking splits the work, and why the first stage decides what the second stage can find. Then you choose how many candidates the second stage reads.

NotebookDo this lesson in a notebookColab · Kaggle · GitHub

The notebook re-ranks the top candidates of hybrid search with recorded scores of a precise model. You measure NDCG@10 for 0, 10, 20, 50 and 100 re-ranked candidates on the validation queries, and choose the best number. Its setup cells download the practice data, so it needs no local setup. About 30 minutes.

Colab and Kaggle open your own copy. They need a free Google or Kaggle account. Stuck? See the solution notebook.

Before you startWhat to know and to have readyAbout 30 minutes

What you need to know

What you need on your computer

  • Nothing for this lesson: the notebook runs in Colab, Kaggle or your own Jupyter. The project gets this module’s files in the next lesson.

A customer asks a shop assistant for “a bed with lights in it”. The shop has forty thousand products. The assistant cannot read every label carefully. That would take all day.

So she works in two passes. First, a fast walk through the shop. She looks only at the signs above the shelves and picks a hundred things that might fit: beds, bed frames, lights. This pass is quick and rough. Then a slow pass. She reads the labels of the best few, one by one, against what the customer said. A box of light bulbs has “light” on it, but it is not a bed. She puts the beds with built-in lights first.

The second pass is only as good as the first. If the right bed is not among the hundred things she picked, her careful reading cannot find it.

The problem: the precise model is too slow, and the fast one makes mistakes

Section titled “The problem: the precise model is too slow, and the fast one makes mistakes”

Larkfield’s hybrid search from Hybrid search is the fast pass. It reads all 40,066 products for every search in about 10 milliseconds. But it reads words and vectors, not meaning. For the test query “beds that have leds”, its top 5 are a solar LED spot light and four LED light bulbs. The first bed is at rank 7. NDCG@10 is 0.073.

A cross-encoder reads better. It is a language model that reads the query and one product together and gives one score. You meet it properly in the next lesson. For “beds that have leds”, it puts five beds in its top 5.

But the cross-encoder is slow. On the course team’s laptop, it scored 75.7 query-product pairs a second on 4 CPU threads. To score all 40,066 products for one search, it would need 40,066 ÷ 75.7 ≈ 529 seconds: almost nine minutes. Amira’s budget for a whole search is 200 milliseconds.

So the fast search reads everything badly, and the precise model reads well but can read only a few products.

Two-stage ranking uses both. Stage one, also called retrieval, is a fast search over the whole catalogue. It returns the top 100 candidates: the products that might answer the query. Stage two, also called re-ranking, is a slow, precise model. It reads only the top n of those candidates and puts them in a new order. The page shows the top 10.

The two-stage funnel. The catalogue of 40,066 products goes to stage one, hybrid search, which is fast, reads every product and takes about 10 ms. It gives 100 candidates in hybrid order. Stage two, the cross-encoder, is slow and precise and re-orders only the top n. The page shows the top 10. A dotted arrow from stage one: a product that stage one misses never reaches stage two and is lost for this search.
Two-stage ranking: a fast stage finds candidates, a slow stage orders the top few. What stage one misses, stage two never sees.
Text description of the diagram

Five boxes from left to right: “Catalogue, 40,066 products” → “Stage one: hybrid search, fast, reads every product, about 10 ms” → “100 candidates in hybrid order” → “Stage two: cross-encoder, slow, precise, re-orders only the top n” → “The page, top 10 shown”. A dotted arrow from stage one goes to a box with a red border: “Lost for this search”, labelled “a product that stage one misses never reaches stage two”.

Why the name? The work happens in two steps, one after the other, like a funnel. Each stage is good at one thing. Stage one is good at recall: it finds most of the right products somewhere in its list. Stage two is good at precision at the top: it puts the best ones first.

Here is a small constructed example: eight products for “bed with lights”, with made-up scores. Stage one keeps its top 4. Stage two re-orders those 4 by its own score.

Product Stage one score Stage one rank Kept? Stage two score Final rank Right answer?
LED light bulb 9.1 1 yes 0.76 3 no
Solar spot light 8.7 2 yes 0.75 4 no
Bed with LED headboard 8.2 3 yes 0.88 1 yes
Bed LED kit 7.9 4 yes 0.85 2 yes
Floor lamp 6.5 5 no – – no
Storage bed with lights 6.1 6 no – – yes
Desk lamp 5.8 7 no – – no
Platform bed 4.0 8 no – – partly

Read it row by row. Stage two fixes the order of the four it reads: the two beds move above the two lights. But “Storage bed with lights” was at stage-one rank 6. Stage two never reads it, so it cannot move it up, however good it is.

Two numbers describe this example. Stage two reads 4 candidates. And of the three right answers, stage one put 2 in its top 4. That share is the recall of stage one at 4: 2 of 3, or 67%. Stage two can never do better than what stage one found.

On Larkfield’s 243 test queries, the same thing happens at scale. Count the exact products of each query, and ask how many of them hybrid search has in its top n.

Measured on Larkfield's 243 test queries: the share of each query's exact products that hybrid search has in its top n. Stage two can only order products that are in this share.
Text description of the chart

A line with four points. In its top 10, hybrid search has 27% of the exact products; in its top 20, 33%; in its top 50, 44%; in its top 100, 54%.

In its top 10, hybrid search has 27% of the exact products. In its top 100, it has 54%. The other 46% are not in the candidate list at all. No second stage can bring them back. You can see this in the comparison table. Every run of this module has the same recall@100, 0.540. Each one only re-orders the same 100 candidates.

There are two ways to give stage two more to work with. Read more candidates (a bigger n), or make stage one better. A bigger n costs time, as the explorer below shows. A better stage one was the work of Modules 3 to 5.

Larkfield: how many candidates should stage two read?

Section titled “Larkfield: how many candidates should stage two read?”

The course team ran the cross-encoder once on the top 100 candidates of every validation and test query, and recorded its scores. With these recorded scores, you can measure any n without running the model again. The team also measured the time of a live search on their laptop, with the model, for each n.

Explore

The two-stage funnel: quality against time

NDCG@10
0.621
Time per search (p50)
10 ms
Budget
200 ms

No re-ranking: stage one alone takes 10 ms per search on 4 threads. NDCG@10 is 0.621 (0.576 to 0.663). For comparison, LambdaMART reaches 0.668 in 34 ms.

  1. 40,066products in the catalogue
  2. Stage one: hybrid search100 candidates10 ms
  3. Stage two: cross-encoderswitched off
  4. The pagetop 10 shown
Two-stage ranking on Larkfield’s 243 test queries (4 threads). Rows marked “over” miss the budget.
Re-rankedExact products in stage one’s top nNDCG@10 (95% interval)Time per search, p50, 4 threads budget 200 ms
0 (stage one only)none0.621 (0.576–0.663)10 ms · fits
1027%0.631 (0.587–0.672)155 ms · fits
2033%0.641 (0.595–0.684)295 ms · over
5044%0.635 (0.589–0.679)732 ms · over
10054%0.631 (0.584–0.674)1,396 ms · over
LambdaMART insteadnone0.66834 ms · fits
Hybrid search finds 100 candidates; the cross-encoder re-orders the top n. NDCG@10 on Larkfield's test queries (mean of 241 of the 243) (recorded scores) and the live time per search (p50, the median) on the course team's laptop.
Text description of the explorer

The explorer has buttons for the number of candidates the cross-encoder re-ranks (0, 10, 20, 50 or 100) and for the number of CPU threads (1 or 4). A strip shows the stages: 40,066 products, then stage one with 100 candidates in 9.2 ms (1 thread) or 10.3 ms (4 threads), then the cross-encoder on the top n, then the top 10 shown. A table shows every choice. Re-ranked 0: NDCG@10 0.621 (0.576 to 0.663), 9.2 ms on 1 thread, 10.3 ms on 4 threads. Re-ranked 10: stage one has 27% of the exact products in its top 10; NDCG@10 0.631 (0.587 to 0.672); 194 ms on 1 thread, 155 ms on 4 threads. Re-ranked 20: 33%; 0.641 (0.595 to 0.684); 400 ms and 295 ms. Re-ranked 50: 44%; 0.635 (0.589 to 0.679); 981 ms and 732 ms. Re-ranked 100: 54%; 0.631 (0.584 to 0.674); 1,904 ms and 1,396 ms. The budget line is at 200 ms. The last row is LambdaMART: 0.668 in 36 ms on 1 thread and 34 ms on 4 threads.

What the explorer shows

Only 10 candidates fit the budget: 155 ms per search on 4 threads, 194 ms on 1 thread. Twenty candidates already take 295 ms on 4 threads. Fifty take 732 ms, more than three times the budget.

The time grows in a straight line with n, because the cross-encoder runs once for every pair. Twice the candidates means twice the model runs. More threads help a little, not much: 50 candidates take 981 ms on 1 thread and 732 ms on 4.

Look at the NDCG@10 column. Without re-ranking, it is 0.621. The best is 0.641, with 20 candidates. With all 100 candidates, it falls back to 0.631.

Two things stand out.

  1. The gains are small. All five intervals overlap. With 50 candidates, the paired difference against no re-ranking is +0.014 (95% interval −0.008 to +0.036). The interval includes zero, so this is not a clear gain on these 243 queries.
  2. More candidates is not always better. Each extra candidate is one more chance for the model to make a mistake. With 100 candidates, the model sometimes lifts a wrong product from rank 90 into the top 10.

Stage two also does not see everything that matters. The cross-encoder reads only text, and only English text, as the next lesson shows. The last row of the table, LambdaMART, is a different second stage. It learns from many signals at once and reaches 0.668 in 36 ms. You build it in Learning to rank.

The table above uses the test queries, so you must not use it to choose n. Choose on the validation queries, then measure once on test. The notebook does this. On the validation queries without the product codes, it finds:

Re-ranked 0 10 20 50 100
NDCG@10 (validation) 0.587 0.580 0.593 0.588 0.587

The best is 20 again, but the gain is only +0.006 on about 110 queries. Remember Build a test set that represents your users. There, a change of 0.037 was clear in fewer than half of the random draws of 50 queries. A change of 0.006 is noise. The honest reading: on Larkfield’s data, the cross-encoder adds little, and the budget allows only 10 candidates.

Why does the notebook leave out the product codes? The code rule of Exact matches and rules puts the right product first before any ranking. A re-ranker that ignored the rule would break all 10 code queries of validation, and they would hide everything else.

Guided practice: choose the depth in the notebook

Section titled “Guided practice: choose the depth in the notebook”

Open the notebook Two-stage ranking (the link is at the top of this lesson).

  1. Run the setup cells. The last one prints Ready: 5 files in larkfield-search.
  2. Run section 1. It loads the recorded scores: 364 queries, 36400 candidates. These are the 121 validation and 243 test queries, with 100 candidates each.
  3. Before you run section 2, write your prediction: is 100 candidates better than 20?
  4. Run section 2. Compare the table with the validation table above.
  5. In section 3, set best_n to the number with the highest NDCG@10, and run the check. It prints Check passed: best_n is 20.
  6. Run the last cell. The left chart is NDCG@10; the right chart is the measured time against the 200 ms budget line.

Use the explorer, not the notebook. Before you look, predict the answers.

  1. Amira’s budget is 200 ms, and the server has only 1 thread for each search. Which n would you choose, and what NDCG@10 do you expect on test?
  2. Omar asks: “If we allowed 500 ms, would people get a much better page?” Use 4 threads.
Answers
  1. n = 10. It takes 194 ms per search on 1 thread, just inside the budget. NDCG@10 is 0.631 (0.587 to 0.672), +0.010 over no re-ranking. That is not a clear gain. LambdaMART reaches 0.668 in 36 ms, so it is the better use of the budget.
  2. Not much. With 500 ms, n = 20 fits (295 ms on 4 threads), but n = 50 does not (732 ms). With n = 20, NDCG@10 on test is 0.641: +0.010 over n = 10. Omar would pay 140 ms more per search for a gain that the test set cannot separate from noise.

Common mistake

Re-ranking more candidates to find the missing products

Symptom: Tomás sees that 46% of the exact products are not in the top 100. He re-ranks all 100 candidates instead of 50 and expects more of them on the page:

Terminal window
search eval --run rerank-ce --set rerank=100
rerank-ce on 243 test queries: NDCG@10 0.631 (95% interval 0.584 to 0.674)
recall@100 0.540, MRR 0.661, zero results 0.0%, judged in the top 10 66.3%; 1.6 ms a query (p50), 2.1 ms (p95)

recall@100 is still 0.540, and NDCG@10 went down from 0.635 to 0.631. (The time shown, 1.6 ms, is only the evaluation with recorded scores. Live, 100 candidates take 1.4 to 1.9 seconds per search.) You get this command in the project in the next lesson.

Cause: stage two only re-orders the candidates it gets. It cannot add a product that stage one did not return. A larger n gives stage two more products to order, but also more chances to lift a wrong one.

Fix: to find more of the right products, improve stage one: better query understanding (Normalise text and the rest of that module), or hybrid search. Choose n by quality and time on validation.

Consequence in search work: a slow second stage cannot repair a weak first stage. Measure the recall of stage one before you spend milliseconds on stage two.

When to use two stages, and what they cost

Section titled “When to use two stages, and what they cost”

Use two-stage ranking when you have a model that orders products better than your search, but is too slow for the whole catalogue. This is the normal design of product search, web search and the retrieval step of a RAG system (see the RAG course). Recommendation systems use the same funnel: many candidates first, a slow model on the few (a planned course on recommendation).

Check two things before you add a second stage:

  • Recall of stage one at n. If the right products are not in the top n, stage two cannot help. On Larkfield, hybrid search has only 54% of the exact products in its top 100.
  • Time per candidate. The slow model’s time grows with n. On the course team’s laptop, the cross-encoder fits only 10 candidates into 200 ms.

Do not add a second stage only because it sounds better. On Larkfield’s test queries, the cross-encoder on 50 candidates gives +0.014, not a clear gain, for about a second per search. The next two lessons measure why, and try a faster second stage.

Check your understanding

Stage one returns 100 candidates. A query has 10 exact products; 4 of them are in the top 20 candidates, and 6 are in the top 100. The second stage re-ranks the top 20 perfectly. How many exact products can it put on the first page?

Show the answer

At most 4. Stage two reads only the top 20 candidates, and only 4 exact products are among them. The other 2 exact products in the top 100 stay below rank 20, and the remaining 4 are not candidates at all.

Check your understanding

The cross-encoder scores 75.7 pairs a second. Why can Larkfield not use it as a single-stage search over all 40,066 products?

Show the answer

One search would need 40,066 model runs: 40,066 ÷ 75.7 ≈ 529 seconds, almost nine minutes. The budget is 200 ms. A fast first stage has to cut the catalogue down to a few candidates first.

Check your understanding

Every run of this module has recall@100 = 0.540, the same as hybrid-rrf. Is that a mistake in the table?

Show the answer

No. Every second stage of this module re-orders the same 100 candidates of hybrid search. Re-ordering changes which products are at the top, so NDCG@10 and MRR change. It cannot change which products are in the top 100, so recall@100 stays the same.

Recap

  • Two-stage ranking: a fast stage one (retrieval) finds about 100 candidates; a slow, precise stage two (re-ranking) re-orders the top n.
  • Stage one limits stage two. Hybrid search has 27% of the exact products in its top 10 and 54% in its top 100. Every re-ranker of this module keeps recall@100 at 0.540.
  • On Larkfield’s test queries, re-ranking the top 10 / 20 / 50 / 100 with a cross-encoder gives NDCG@10 0.631 / 0.641 / 0.635 / 0.631, against 0.621 without. No gain is clear.
  • Time grows with n: only 10 candidates fit the 200 ms budget (155 ms on 4 threads). Choose n on validation; on Larkfield, the best is 20, by a gain of only 0.006.

You now know where a second stage fits, and what limits it. Next, you open the second stage: how a cross-encoder reads the query and the product together, where it helps and where it hurts.

Next: Cross-encoder re-ranking