Skip to content
Français

Cross-encoder re-ranking

  • Lesson 20
  • 40 min
  • Module 6: Re-rank the top results

Where we are

In the last lesson, we split ranking into a fast first stage and a slow, precise second stage. The second stage needs a model that reads better than our search. A cross-encoder reads the query and the product together. In this lesson, we add it to the project, measure where it helps and where it hurts, and count the extra milliseconds.

Plays from YouTube. Nothing loads from YouTube until you press play. Privacy

What you will be able to do

Re-rank the top candidates with a model that reads the query and the product together. You add a cross-encoder to the project, measure its gain on Larkfield’s test queries by language, and count what it costs in time.

NotebookDo this lesson in a notebookColab · Kaggle · GitHub

The notebook looks at the recorded cross-encoder scores of one query. It measures re-ranking on the validation queries by language. Then you work out how many candidates fit a 200 ms budget. Its setup cells download the practice data, so it needs no local setup and no model download. About 30 minutes.

Colab and Kaggle open your own copy. They need a free Google or Kaggle account. Stuck? See the solution notebook.

Before you startWhat to know and to have readyAbout 40 minutes

What you need to know

What you need on your computer (only for the guided practice)

  • The project larkfield-search at the end of Module 5, from Hybrid search. python -m pytest -q must say 27 passed.
  • The labs repository beside it (../nextia-learning-labs), as in Meet the catalogue and the query log.
  • On macOS, Homebrew’s OpenMP library for LightGBM, which the next lesson uses: brew install libomp.

A shop assistant who reads the whole request

Section titled “A shop assistant who reads the whole request”

A customer hands the shop assistant a note: “beds that have leds”. A new assistant looks for the words. “LEDs” is on many boxes: light bulbs, spot lights, strips. She brings a box of LED bulbs.

An experienced assistant reads the whole note at once. She sees that “beds” is the thing the customer wants, and “that have leds” describes it. She brings beds with lights built in. She did not look at the words one by one. She read how the words relate to each other.

The problem: hybrid search matches words, not the request

Section titled “The problem: hybrid search matches words, not the request”

Larkfield’s hybrid search reads words (BM25) and meanings (the bi-encoder of Search by meaning). For “beds that have leds”, here is its top 5 on the test set:

hybrid-rrf: "beds that have leds" -> 10 results (query W-014: judged results are marked)
1. LK-97303 solar powered led spot light pack Landscape Lighting 623.99 [irrelevant]
2. LK-55076 25 watt equivalent blue e12/candelabra led light bulb Light Bulbs 396.99 [irrelevant]
3. LK-44879 5 watt ( 60 watt equivalent ) , ca10 led , dimmable light bu Light Bulbs 328.99 [irrelevant]
4. LK-66358 25 watt equivalent purple e12/candelabra led light bulb Light Bulbs 227.99 [irrelevant]
5. LK-83953 25 watt equivalent red e12/candelabra led light bulb Light Bulbs 622.99 [irrelevant]

“led” is a rare word in the catalogue, so BM25 gives it a high weight. The bi-encoder made one vector for the whole query, and one for each product, separately. The query vector sits somewhere between “bed” and “LED”, and the light bulbs are close to it. Neither part of hybrid search reads the query and the product together.

A cross-encoder is a language model that reads the query and one product’s text as one input. It returns one score: how well this product answers this query. Inside the model, every word of the query can look at every word of the product, and the other way round. The words “cross” between the two texts; that is where the name comes from.

Compare it with the bi-encoder of Search by meaning. A bi-encoder encodes the query and each product separately into vectors. The product vectors are computed once, ahead of time, so a search only encodes the query and compares vectors. That is fast, but the model never sees the query and the product side by side.

Two models compared. Bi-encoder, stage one: the query goes through the model once and becomes a query vector; each product was encoded once ahead of time into 40,066 product vectors; the vectors are compared in a few milliseconds for all products. Cross-encoder, stage two: the query plus one product, for example beds that have leds plus emete low profile standard bed, go into the model together; it reads both word by word and gives one score, 0.878. One model run per pair: 50 pairs need 50 runs.
A bi-encoder encodes the query and the products apart, so products are encoded ahead of time. A cross-encoder reads each pair together: more precise, and one model run per pair.
Text description of the diagram

Two boxes, one above the other. “Bi-encoder (stage one, Module 5)”: “Query: beds that have leds” → “Model, one run per query” → “Query vector”; “Each product, encoded once, ahead of time” → “40,066 product vectors”; both vectors → “Compare vectors, a few milliseconds for all products”. “Cross-encoder (stage two)”: “Query + one product: beds that have leds + emete low profile standard bed” → “Model reads both together, word by word” → “One score: 0.878, one model run per pair, 50 pairs = 50 runs”.

So the cross-encoder trades speed for precision. It cannot prepare anything ahead of time, because every score depends on the query. For 50 candidates, it runs 50 times for every search. That is why it belongs in stage two.

The course team ran the cross-encoder on the top 100 hybrid candidates of “beds that have leds”, and recorded the scores. Here are five of them:

Product Hybrid rank Cross-encoder score Rank by this score (of 100) Judged
solar powered led spot light pack 1 0.754 70 irrelevant
25 watt equivalent blue e12/candelabra led light bulb 2 0.766 49 irrelevant
emete low profile standard bed 7 0.878 1 exact
satty bed led kit 17 0.849 5 exact
galway upholstered led storage platform bed 23 0.835 8 exact

The scores are close together, between about 0.75 and 0.88. Only their order matters. The light bulb and the spot light score lowest, around 0.75 to 0.77. The beds, and the bed LED kit, score highest. The cross-encoder read “beds” as the thing, and “leds” as a detail.

Re-ranking means: take the top n candidates of stage one, sort them by this score, and keep the rest of the list below them. Candidates below rank n keep their hybrid order.

The project uses IBM’s granite-embedding-reranker-english-r2: 149 million parameters, a 598 MB download, Apache-2.0 licence. Its model card states that all its training data has a permissive licence and that it was not trained on MS MARCO. That is a vendor claim; the course did not check the training data. Many popular cross-encoders, such as cross-encoder/ms-marco-MiniLM-L6-v2, are trained on MS MARCO. Its terms say that the datasets are “intended for non-commercial research purposes only”. For a shop, the licence of the training data matters as much as the licence of the weights.

The cost of this choice: the model is trained for English. You will see what that means for Larkfield’s French queries.

Guided practice: add the module’s files and re-rank

Section titled “Guided practice: add the module’s files and re-rank”

This module adds six Python files and one test file to the project. You copy all of them now, from the snapshot end-of-m06. This lesson explains rerank.py and the run rerank-ce. The next two lessons explain the others.

  1. In larkfield-search, with the virtual environment active, copy this module’s files from the snapshot. They are the new code, its tests, the new list of packages and the README.

    Terminal window
    S=../nextia-learning-labs/C38/snapshots/end-of-m06
    for f in requirements.txt larkfield_search/rerank.py larkfield_search/ltr.py larkfield_search/clicks.py larkfield_search/runs/rerank.py larkfield_search/commands/ltr.py larkfield_search/commands/clicks.py tests/test_rerank.py README.md; do
    cp "$S/$f" "$f"
    done
  2. Install the new package. The new requirements.txt adds LightGBM, for the next lesson:

    Terminal window
    python -m pip install -r requirements.txt

    The last line says Successfully installed lightgbm-4.7.0.

  3. Run the tests:

    Terminal window
    python -m pytest -q
    ............................... [100%]
    31 passed in 9.12s

    Four tests are new. One checks that rerank-ce uses the recorded scores.

  4. Open larkfield_search/runs/rerank.py. The run rerank-ce is seven lines:

    def rerank_ce(query, query_id=None, k=100):
    n = int(settings.get("rerank", RERANK))
    cand = hybrid_rrf(query, query_id, max(k, 100))
    head = [p for p, _ in cand[:n]]
    s = rerank.scores(query, query_id, head)
    head = sorted(zip(head, s), key=lambda x: (-x[1], x[0]))
    return with_codes(query, head + cand[n:], k)

    It takes the top 100 of hybrid-rrf (stage one), scores the first n (50 by default), sorts them by score, and keeps the rest below. The last line applies the product-code rule of Exact matches and rules. rerank.scores returns the recorded score when the course has one. For any other query or product, it downloads the model (about 600 MB) the first time, and scores the pairs on your computer.

  5. Compare the two orders for “beds that have leds”:

    Terminal window
    search query "beds that have leds" --run rerank-ce
    rerank-ce: "beds that have leds" -> 10 results (query W-014: judged results are marked)
    1. LK-61422 emete low profile standard bed Beds 680.99 [out of stock] [exact]
    2. LK-72114 abner upholstered sleigh bed Beds 967.99 [exact]
    3. LK-62518 skofte odessa upholstered platform bed Beds 720.99 [exact]
    4. LK-43564 bassik upholstered low profile platform bed Beds 1113.99 [exact]
    5. LK-21564 satty bed led kit Bed Accessories 152.99 [exact]
    6. LK-21124 adjustable bed base Adjustable Beds 650.99 [irrelevant]
    7. LK-96147 galway upholstered led storage platform bed Beds 576.99 [exact]
    8. LK-59268 cieran tufted upholstered storage platform bed Beds 699.99 [out of stock] [irrelevant]
    9. LK-29326 jovenko upholstered led storage platform bed Beds 817.99 [exact]
    10. LK-84745 sabara upholstered led storage platform bed Beds 659.99 [out of stock] [exact]

    Eight of the ten are exact. NDCG@10 for this query goes from 0.073 to 0.852. Products 2, 3 and 4 were at hybrid ranks 35, 36 and 44: stage one found them, but too low.

  6. Measure on all test queries:

    Terminal window
    search eval --run rerank-ce
    rerank-ce on 243 test queries: NDCG@10 0.635 (95% interval 0.589 to 0.679)
    recall@100 0.540, MRR 0.672, zero results 0.0%, judged in the top 10 66.7%; 1.6 ms a query (p50), 2.3 ms (p95)

    hybrid-rrf gave 0.621 (0.576 to 0.663). One query improved a lot; the average moved a little.

An average can hide two groups that move in opposite directions. Compare the two runs by language:

Terminal window
search compare hybrid-rrf rerank-ce --by language
rerank-ce minus hybrid-rrf on 241 test queries: ndcg@10 +0.014 (95% interval -0.008 to +0.036); better on 93, worse on 75, the same on 73
language queries hybrid-rrf rerank-ce difference low high worse
en 181 0.723 0.758 0.034 0.012 0.056 49
fr 60 0.311 0.264 -0.048 -0.100 0.005 26

The difference is computed on 241 queries, not 243: two test queries have no exact product, so their NDCG is not defined. The chart shows the same comparison by language and by kind of query.

The cross-encoder on the top 50, minus hybrid search, on Larkfield's test queries, with the paired 95% interval. It helps English queries and hurts French ones.
Text description of the chart

Two panels of differences, with a line at zero. By language: all queries +0.014 (−0.008 to +0.036); English +0.034 (+0.012 to +0.056); French −0.048 (−0.100 to +0.005). By kind: English WANDS +0.035 (+0.009 to +0.061); misspelt +0.043 (−0.018 to +0.109); French −0.048 (−0.102 to +0.001).

On English queries, the gain is +0.034 (95% interval +0.012 to +0.056): clear. On French queries, re-ranking makes things worse: −0.048 (−0.100 to +0.005). The interval just touches zero, so the harm is likely but not certain. Overall, the two cancel out: +0.014, not a clear gain.

The model is trained for English. It reads a French query as if it were English. You met the French test query “chaise ergonomique” (“ergonomic chair”) in Hybrid search. The multilingual bi-encoder found five exact chairs, BM25 found chaise lounges, and hybrid search mixed the two. Hybrid search has three exact office chairs in its top 5. The cross-encoder’s top 3:

rerank-ce: "chaise ergonomique" -> 10 results (query F-422: judged results are marked)
1. LK-25009 iancarlo chesterfield chaise lounge 161.99 [out of stock] [not judged]
2. LK-72973 rosemead velvet chaise lounge Indoor Chaise Lounges 269.99 [out of stock] [not judged]
3. LK-77648 anne chaise lounge Indoor Chaise Lounges 281.99 [not judged]

All ten results are chaise lounges or similar. In English, “chaise” appears in “chaise lounge”, a long seat for lying down. The model matched the English word and ignored “ergonomique”. The office chairs, which hybrid search had at ranks 1, 3 and 5, fall out of the top 10. NDCG@10 for this query falls from 0.578 to 0.032.

Remember that our French queries are machine-translated from English WANDS queries. Real French customers may write differently. But the direction is clear: an English-only model should not re-order French results. A simple guard is to skip the cross-encoder when the query is French. That is a choice to measure, not to assume.

How many candidates, and how many milliseconds?

Section titled “How many candidates, and how many milliseconds?”

Two-stage ranking showed the trade-off in its explorer. The model’s time grows with every candidate, and only 10 candidates fit Amira’s 200 ms budget. The project’s default is 50 candidates: 981 ms per search on 1 thread, for NDCG@10 0.635.

The times come from the course team’s laptop (an Apple M4 CPU), with the model running live. A second measurement split one search into its stages, on 40 English test queries with 1 thread. For the default of 50 candidates, the cross-encoder alone took 977 ms at p50 (710 ms on 4 threads). The whole search took 1,121 ms at p50 and 1,445 ms at p95. Every other stage took a few milliseconds. The two measurements were separate runs, so their numbers differ a little (981 ms above). On your computer, the times will differ, but the shape will not: the model’s time grows with each pair.

Money is not the problem. Assume that a 4-vCPU server costs US$0.17 an hour (check your provider’s current price). Then 1,000 searches with the cross-encoder on 1 thread cost about US$0.013. Time is the problem.

The notebook found 20 candidates best on validation. Before you run it, predict: with 20 candidates, does French still get worse?

Terminal window
search eval --run rerank-ce --set rerank=20 --by language
The output, and what it shows
rerank-ce on 243 test queries: NDCG@10 0.641 (95% interval 0.595 to 0.684)
recall@100 0.540, MRR 0.682, zero results 0.0%, judged in the top 10 68.5%; 1.5 ms a query (p50), 2.3 ms (p95)
language queries ndcg@10 low high zero-result rate
en 182 0.757 0.716 0.795 0.000
fr 61 0.293 0.207 0.378 0.000

With 20 candidates, NDCG@10 is 0.641 on all test queries, the best of the five depths. English stays at about 0.757. French is 0.293, still below hybrid search’s 0.311, but less harmed than with 50 candidates (0.264). Fewer candidates means fewer chances to lift a wrong chaise lounge. But 20 candidates take 295 ms per search on 4 threads: still over the budget.

Common mistake

The evaluation time is not the model's time

Symptom: Tomás reads the output of search eval --run rerank-ce:

recall@100 0.540, MRR 0.672, zero results 0.0%, judged in the top 10 66.7%; 1.6 ms a query (p50), 2.3 ms (p95)

He tells Amira: “The cross-encoder adds almost nothing: 1.6 ms a query. Let’s ship it.”

Cause: search eval uses the recorded scores. It looks up 50 numbers in a table. It never runs the model, so its time is the time of hybrid search plus a lookup. The live time, with the model, is about 1.1 seconds per search on 1 thread.

Fix: measure the time of the live system, with the model running, as Latency and cost budgets does with search bench. Report the recorded-score time and the live time apart, and say which one you quote.

Consequence in search work: recordings make experiments fast and repeatable. They also hide the cost of the thing you recorded. Before a release, measure the real path, on hardware like your servers.

When to use a cross-encoder, and what it costs

Section titled “When to use a cross-encoder, and what it costs”

Use a cross-encoder when the top of your list needs a careful reading of the query, and the time is available. It helps most on longer, descriptive queries, such as “beds that have leds”: on Larkfield’s specific queries, the gain is +0.030 (+0.002 to +0.058). On the English test queries, it gives +0.034, a clear gain.

Do not use it blindly:

  • Language. An English-only model hurts French queries: −0.048 on Larkfield’s test set. Use a multilingual re-ranker, or skip re-ranking for languages the model does not know.
  • Time. One model run per pair. On a laptop CPU, only 10 candidates fit 200 ms. A GPU would be much faster; this course did not measure one.
  • Licence. Check the training data as well as the weights.

On Larkfield’s data, the cross-encoder’s total gain is not clear, and it costs a second per search. The next lesson tries another second stage. It learns from many signals, including the text scores you already have, and it runs in about 36 ms.

Check your understanding

Why can a bi-encoder compute its product vectors ahead of time, while a cross-encoder cannot compute its scores ahead of time?

Show the answer

A bi-encoder encodes each product alone, without the query, so the product vector is the same for every search. A cross-encoder reads the query and the product together, so its score depends on the query. It can only run when the query arrives: one model run per pair.

Check your understanding

The cross-encoder gives +0.014 overall (−0.008 to +0.036), +0.034 on English and −0.048 on French. Omar asks if it is a gain. What do you tell him?

Show the answer

Not overall: the interval includes zero. It is a clear gain on English queries and probably a loss on French ones, because the model is trained for English. If Larkfield wants it, use it only for English queries, and measure that change. It also costs about a second per search with 50 candidates.

Check your understanding

search eval reports 1.6 ms a query for rerank-ce. Amira's budget is 200 ms. Can she use 50 candidates?

Show the answer

No. The 1.6 ms is with recorded scores: the model did not run. Live, 50 candidates took about 981 ms per search on 1 thread and 732 ms on 4 threads on the course team’s laptop. Only 10 candidates fit 200 ms (155 ms on 4 threads).

Recap

  • A cross-encoder reads the query and one product together and gives one score. It is more precise than a bi-encoder, but it runs once per pair, so it fits only in stage two.
  • On “beds that have leds”, it moves NDCG@10 from 0.073 to 0.852. On all of Larkfield’s test queries, re-ranking the top 50 gives 0.635 (0.589 to 0.679), +0.014 over hybrid search: not clear.
  • It helps English queries (+0.034, clear) and hurts French ones (−0.048): the model is English only, and it reads “chaise ergonomique” as “chaise lounge”.
  • Live, 50 candidates take about 1.1 s per search on 1 thread. search eval’s 1.6 ms uses recorded scores and hides the model’s time.

The cross-encoder knows only the text of the query and the product. A shop knows more: price, stock, ratings, what people bought. Next, you train a model that combines all of these signals.

Next: Learning to rank