Learning to rank
- Lesson 21
- 45 min
- Module 6: Re-rank the top results
Where we are
In the last lesson, the cross-encoder read the query and the product together. It helped English queries, hurt French ones, and took about a second per search. It also knows only the text. A shop knows more: the kind of product, the price, the stock, what people bought. In this lesson, we describe each candidate with features and train a LambdaMART model with LightGBM on our judgments.
Lesson 21 of 30 · Search and Ranking: Find the Best MatchLearning to rank
1. The search problem
2. Measure relevance before you change anything
3. Lexical search: the baseline to beat
4. Understand the query
5. Dense and hybrid retrieval
6. Re-rank the top results
7. The parts of search that people see
8. Run search in production
What you will be able to do
Combine many signals into one ranking, and let the judgments decide how much each signal counts. You describe each candidate with 19 features and train a LambdaMART model with LightGBM. Then you read what it learned, and compare it with the cross-encoder for a 200 ms budget.
NotebookDo this lesson in a notebook
The notebook works out a pairwise example by hand and builds 12 features for the recorded candidates. Then it trains LambdaMART with LightGBM on the validation queries, and measures it on the test queries. It installs LightGBM in its first cell, and its setup cells download the practice data, so it needs no local setup. About 40 minutes.
Colab and Kaggle open your own copy. They need a free Google or Kaggle account. Stuck? See the solution notebook.
Before you startWhat to know and to have ready
What you need to know
- Two-stage ranking and the cross-encoder, from Two-stage ranking and Cross-encoder re-ranking.
- BM25 per field, from BM25.
- Decision trees and gradient boosting help but are not required: the machine learning course explains them in Decision trees and Ensembles.
What you need on your computer (only for the guided practice)
- The project
larkfield-searchwith this module’s files, from the guided practice of Cross-encoder re-ranking.python -m pytest -qmust say31 passed. - On macOS, Homebrew’s OpenMP library for LightGBM:
brew install libomp.
A shop assistant who learned from experience
Section titled “A shop assistant who learned from experience”An experienced shop assistant does not follow one rule. When a customer asks for a “king size bed”, she weighs many clues at once. Does the label say “king”? Is it a bed, or a bed sheet? Is it in stock? Do people who ask for this usually buy it? Is the price close to what this customer can pay?
Nobody gave her a formula with weights. She learned them over years, from what worked: which products customers took home, and which they brought back. A new assistant has the same clues, but does not yet know how much each one counts.
The problem: each method uses one kind of signal
Section titled “The problem: each method uses one kind of signal”Each search method so far uses one kind of signal:
- BM25 counts words. BM25 weighted its fields by hand and by a grid search. The weights chosen on validation lost on test.
- Dense search compares meanings, but misses names and numbers.
- Hybrid search combines the two by rank, with one fixed rule.
- The cross-encoder reads the text better, but only English text, and it never sees price, stock or popularity.
Here is a French test query, “lit très grand format”: a bed in a very large size, from the English query “king size bed”. Hybrid search’s top 5:
hybrid-rrf: "lit très grand format" -> 10 results (query F-122: judged results are marked) 1. LK-30733 grand baroque cake knife Flatware & Silverware Se 485.99 [not judged] 2. LK-75214 legrand tv stand for tvs up to 88 '' TV Stands & Entertainmen 813.99 [not judged] 3. LK-50365 tv stand for tvs up to 88 '' TV Stands & Entertainmen 464.99 [not judged] 4. LK-97583 shiela grand dining table Dining Tables 433.99 [out of stock] [not judged] 5. LK-45765 grand hall console table Sofa & Console Tables 273.99 [out of stock] [not judged]A cake knife and TV stands. The English word “grand” matched. NDCG@10 is 0.000. Yet the information to do better exists: some products have a French title that says “lit” (bed), and BM25 on the French titles scores them. Hybrid search has no way to know that, for this query, the French title should count more than the English description. Writing such rules by hand, for every kind of query, does not work.
The idea: learning to rank
Section titled “The idea: learning to rank”Learning to rank trains a model to order candidates from examples. The examples are queries with judged products: Larkfield’s 260 training queries, with the WANDS grades (exact 2, partial 1, irrelevant 0). The model never reads text. It reads features: numbers that describe one query-product pair.
Text description of the diagram
Boxes from left to right: “One candidate for one query (hybrid top 100)” → “19 features: text: BM25 per field, dense, RRF score, hybrid rank; shop: price, stock, sale, rating, clicks, carts, age; query: words, has a number” → “LambdaMART: 300 small decision trees, each adds a little” → “A score” → “Sort the 100 candidates by score”. A dotted arrow from “Judgments of the 260 training queries: exact 2, partial 1, irrelevant 0” to LambdaMART, labelled “learns the trees”.
The 19 features
Section titled “The 19 features”The project’s file larkfield_search/ltr.py computes them for each of the 100 candidates of a query:
| Group | Features | What they say |
|---|---|---|
| Text | bm25_title, bm25_description, bm25_category, bm25_attributes, bm25_title_fr |
BM25 score of the query in each field, after spelling correction and stemming (Normalise text, Fix spelling) |
| Text | dense, rrf, hybrid_rank |
the bi-encoder’s cosine similarity, the hybrid RRF score and the rank in hybrid search |
| Kind | class_match |
1 if the product’s class is the class that the query asks for (the classifier of Detect what the query asks for) |
| Shop | price_log, in_stock, on_sale, rating, rating_count_log |
price, stock, sale flag and ratings (prices, stock and sale flags are synthetic) |
| Popularity | clicks_log, carts_log |
clicks and add-to-cart of the product in the synthetic log, on any query |
| Freshness | age_days |
days since the product was added |
| Query | query_words, query_has_number |
how long the query is, and whether it has a number |
Some features describe the pair (BM25, dense). Some describe only the product (price, popularity). Some describe only the query (its length). A tree model can combine them: “a long query with a high French-title score” is one branch.
Three ways to learn an order
Section titled “Three ways to learn an order”How do you train a model to order things? There are three ideas.
- Pointwise: predict the grade of each product alone, then sort by the prediction. Simple, but a perfect grade prediction is not needed: only the order matters.
- Pairwise: take two products of the same query, and learn which one should be above the other. This is closer to ranking.
- Listwise: look at the whole list, and improve a measure of the whole list, such as NDCG.
LambdaMART mixes the last two. It looks at pairs. For each pair in the wrong order, it asks: how much would NDCG of the whole list gain if these two swapped? Then it pushes harder on the pairs with the bigger gain. The “MART” part means multiple additive regression trees: many small decision trees, each one correcting a little of what the trees before it got wrong. This is gradient boosting, explained in the machine learning course’s lesson Ensembles. LightGBM is a fast, open-source library (MIT licence) that trains such trees; its objective lambdarank is LambdaMART.
A tiny example: which pair matters most?
Section titled “A tiny example: which pair matters most?”Three candidates for one query, with the model’s current scores and the judges’ grades. This is the notebook’s first cell:
| Rank | Product | Model score now | Grade |
|---|---|---|---|
| 1 | A | 2.1 | 0 (irrelevant) |
| 2 | B | 1.5 | 2 (exact) |
| 3 | C | 0.3 | 1 (partial) |
With gains 2^grade − 1 and discounts log2(rank + 1), as in Ranking measures: DCG now is 0 + 3 ÷ 1.585 + 1 ÷ 2 = 2.393. The ideal order (B, C, A) gives 3 + 1 ÷ 1.585 + 0 = 3.631. So NDCG now is 2.393 ÷ 3.631 = 0.659.
Two pairs are in the wrong order. A better product sits below a worse one:
| Wrong pair | NDCG if the two swap | Gain from the swap |
|---|---|---|
| B should be above A | 0.964 | +0.305 |
| C should be above A | 0.797 | +0.138 |
The pair B–A is worth more than twice the pair C–A. B is exact, and the swap moves it to rank 1, where the discount is smallest. LambdaMART uses these gains as weights. The next trees work hardest to lift B above A.
Guided practice: train and measure
Section titled “Guided practice: train and measure”-
Train the model on the judgments of the 260 training queries. The project computes the 19 features for 26,000 candidates, then trains. It took 8 seconds on the course team’s laptop.
Terminal window search ltr trainTrained LambdaMART on 260 training queries, 26,000 candidates (13,763 with a label above 0), labels: judgments. Saved outputs/ltr-judgments.txtFeature importance (share of the total gain):rrf 25.4%hybrid_rank 15.5%bm25_title 11.2%bm25_title_fr 7.3%bm25_category 5.9%dense 4.7%bm25_description 4.1%bm25_attributes 3.7%price_log 3.6%query_words 3.2%carts_log 3.1%age_days 2.9%clicks_log 2.9%rating_count_log 2.9%class_match 2.2%query_has_number 0.7%rating 0.6%on_sale 0.1%in_stock 0.1%If you see
Library not loaded: libomp.dylibon macOS, runbrew install libompand try again. We did not test a Mac without it. -
Measure it on the test queries:
Terminal window search eval --run ltrltr on 243 test queries: NDCG@10 0.668 (95% interval 0.628 to 0.708)recall@100 0.540, MRR 0.698, zero results 0.0%, judged in the top 10 73.9%; 18.6 ms a query (p50), 44.4 ms (p95)This time, the model really runs: computing the features and the 300 trees is the 18.6 ms. Live, on 1 thread, the course team measured 36 ms at p50 and 54 ms at p95.
-
Compare with hybrid search, by kind of query:
Terminal window search compare hybrid-rrf ltr --by sourceltr minus hybrid-rrf on 241 test queries: ndcg@10 +0.047 (95% interval +0.023 to +0.073); better on 101, worse on 75, the same on 65source queries hybrid-rrf ltr difference low high worsecode 20 1.000 1.000 0.000 0.000 0.000 0planted 3 0.335 0.571 0.236 0.159 0.337 0translated 59 0.315 0.474 0.158 0.101 0.216 14typo 40 0.612 0.615 0.003 -0.065 0.079 17wands 119 0.719 0.729 0.010 -0.012 0.032 44 -
Look at the French query again:
Terminal window search query "lit très grand format" --run ltrltr: "lit très grand format" -> 10 results (query F-122: judged results are marked)1. LK-57005 standard king bed 107.99 [exact]2. LK-77012 litchville king bed Beds 629.99 [exact]3. LK-90382 atalfo king standard bed Beds 581.99 [exact]4. LK-17316 full bunk bed 73.99 [not judged]5. LK-88796 california king upholstered platform bed Beds 860.99 [partial]The first five results (of ten) are beds, and three of them are exact king beds. NDCG@10 for this query goes from 0.000 to 0.715.
Where the gain comes from
Section titled “Where the gain comes from”The model beats hybrid search by +0.047 (95% interval +0.023 to +0.073): the first clear gain of this module. NDCG@10 is 0.668 (0.628 to 0.708) on Larkfield’s test queries. But the gain is not spread evenly.
Text description of the chart
Differences with a line at zero. All queries +0.047 (+0.023 to +0.073); English WANDS +0.010 (−0.012 to +0.032); misspelt +0.003 (−0.065 to +0.079); French +0.159 (+0.104 to +0.216); broad queries +0.130 (+0.062 to +0.204).
- French queries: +0.159. The model learned when to trust the BM25 score of the French titles. Hybrid search mixes all its signals with one fixed rule; the model can give the French-title score a large weight only when it is high.
- Broad queries: +0.130. For “sofa” or “couch”, many products match the words. Signals such as the product class, the rating count and popularity help to choose among them.
- Real English queries: +0.010 (−0.012 to +0.032): not clear. For these, hybrid search was already good, and the model mostly agrees with it.
The output above shows +0.158 for the 59 machine-translated queries. With the planted French query “canapé”, the 60 French queries give +0.159. Remember that the French queries are machine-translated. The large French gain may be smaller on real French customers’ queries.
Read the feature importance
Section titled “Read the feature importance”LightGBM reports how much each feature helped the trees. Gain importance is the total improvement of the training objective from all splits on that feature. Here are the ten largest shares:
Text description of the chart
One dot per feature, from 0% to 30%: rrf 25.4%, hybrid_rank 15.5%, bm25_title 11.2%, bm25_title_fr 7.3%, bm25_category 5.9%, dense 4.7%, bm25_description 4.1%, bm25_attributes 3.7%, price_log 3.6%, query_words 3.2%; the other 9 features together 15.4%.
Three things stand out.
- The model trusts stage one.
rrfandhybrid_ranktogether take 40.9% of the gain. The model starts from hybrid search’s order and corrects it. bm25_title_fris fourth, at 7.3%. This is the French gain in one number.in_stockandon_saleare almost zero, 0.1% each. Does stock not matter? It matters to a shop, but the model learns only what the labels reward. The WANDS judges graded relevance: “is this the product the person asked for?” An out-of-stock king bed is still exact. If Larkfield wants stock to count, that is a business rule, not relevance. Zero results and business rules measures what such rules cost.
Importance tells you what the model uses, not what is true. Two features that carry the same information share the gain between them, and either one alone could do the job.
Your turn: the cross-encoder or LambdaMART for 200 ms?
Section titled “Your turn: the cross-encoder or LambdaMART for 200 ms?”Amira’s budget is 200 ms per search. The model you trained includes click features (clicks_log, carts_log). Before you run anything, predict: on English queries, which one is better, the cross-encoder or LambdaMART?
search compare rerank-ce ltr --by languageThe output, and a choice
ltr minus rerank-ce on 241 test queries: ndcg@10 +0.033 (95% interval +0.005 to +0.064); better on 93, worse on 85, the same on 63
language queries rerank-ce ltr difference low high worse en 181 0.758 0.733 -0.024 -0.048 0.001 74 fr 60 0.264 0.471 0.207 0.141 0.271 11On English queries, the cross-encoder is a little better: 0.758 against 0.733 (−0.024, interval −0.048 to +0.001, just touching zero). On French queries, LambdaMART is far better: 0.471 against 0.264.
For a 200 ms budget, choose LambdaMART. It reaches 0.668 overall in 36 ms (p50, 1 thread). The cross-encoder on 50 candidates reaches 0.635 in about 1.1 s. Within 200 ms, the cross-encoder could read only 10 candidates (0.631). A later idea to measure: give LambdaMART the cross-encoder’s score of the top 10 as one more feature, if the time allows.
Common mistake
The model looks excellent on its own training queries
Symptom: Tomás measures his new model on the queries it learned from, because “they have the most judgments”:
search eval --run ltr --split trainltr on 260 train queries: NDCG@10 0.878 (95% interval 0.850 to 0.904)recall@100 0.586, MRR 0.904, zero results 0.0%, judged in the top 10 90.0%; 18.3 ms a query (p50), 34.7 ms (p95)He reports 0.878. On the test queries, the same model gives 0.668. On validation, 0.698.
Cause: the trees learned details of these 260 queries and their judged products, including details that do not repeat on new queries. This is overfitting. Every number measured on training data is too optimistic.
Fix: report the model only on queries it never saw: choose settings on validation, measure once on test. The machine learning course explains why in Train, validation and test sets.
Consequence in search work: a ranking model tuned on the queries it is judged on will disappoint in production. Keep the test queries out of training, and out of your choices.
When to use learning to rank, and what it costs
Section titled “When to use learning to rank, and what it costs”Use learning to rank when you have judged queries and several signals that each help a little. On Larkfield’s test queries, it is the best ranking so far that fits the budget. It scores 0.668, +0.047 over hybrid search, in 36 ms.
It has costs:
- Judgments. The model learned from 13,763 positive labels on 260 queries. Without judgments, there is nothing to learn from. The next lesson asks whether clicks can replace them.
- Features at query time. Every feature must be computed for every candidate on every search. Here that is about 35 ms on 1 thread: within the budget, but not free.
- Maintenance. When a feature changes (a new BM25 analyzer, a new embedding model), retrain the model and measure again.
- Money is small. About US$0.0004 per 1,000 searches at the course’s assumed CPU price.
Do not expect it to fix the candidates. Like every second stage, it keeps recall@100 at 0.540: it only re-orders what hybrid search found. And do not read its importance as the truth about your shop: it shows what predicts the judges’ labels.
Knowledge check
Section titled “Knowledge check”Check your understanding
A list has three products in this order, with grades 0, 2, 1. Which wrong pair does LambdaMART push hardest, and why?
Show the answer
The exact product (grade 2) at rank 2 below the irrelevant one at rank 1. Swapping them raises NDCG from 0.659 to 0.964 (+0.305). Swapping the partial product above the irrelevant one gives only +0.138. LambdaMART weights each pair by the NDCG gain of the swap, and the top of the list counts most.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
in_stock has 0.1% of the gain. Hana concludes that stock does not matter to Larkfield's customers. What do you tell her?
Show the answer
The model learns what predicts its labels. The labels are the WANDS judges’ relevance grades, and an out-of-stock product can still be exactly what the person asked for. So the model has no reason to use stock. Whether people want out-of-stock products lower is a business question, measured with clicks, carts or a rule (Module 7), not by this importance.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
LambdaMART gives +0.047 over hybrid search overall, +0.159 on French and +0.010 on the real English queries. What would you check before telling Omar it will help French customers this much?
Show the answer
Our French queries are machine-translated from English queries, and they share the English judgments. Real French customers may write differently. Before promising the French gain, measure on real French queries, or watch French searches online with a test (Module 8).
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Recap
- Learning to rank trains a model to order candidates from judged queries. It reads features: 19 numbers per candidate, from text scores to price, popularity and age.
- LambdaMART looks at wrong pairs and weights each by the NDCG gain of swapping them. It is gradient-boosted trees; LightGBM’s
lambdaranktrains it. - On Larkfield’s test queries,
ltrreaches 0.668 (0.628 to 0.708): +0.047 over hybrid search, mostly from French (+0.159) and broad queries (+0.130). It runs in 36 ms (p50, 1 thread). - Feature importance: the hybrid score and rank lead (40.9%), and the French-title BM25 score is fourth (7.3%). Stock and sale are near zero, because the labels are relevance grades.
- A model measured on its training queries looks too good: 0.878 on training, 0.668 on test.
The model learned from 13,763 judged labels. Judging is slow and costly, but the query log has thousands of free clicks. Next, you train on clicks, and see why a click is not a relevance label.
Next: Learn from clicks