A learned ranker
- Lesson 21
- 45 min
- Module 7: Two stages: candidates and ranking
Where we are
We have candidates from several sources. Now we must put the best ones first. In this lesson, we build features for each customer and product, and train a LambdaMART ranker with LightGBM on the next purchase.
Lesson 21 of 29 · Recommendation and Personalisation: Show Each Person What MattersA learned ranker
1. The recommendation problem
2. Data, implicit feedback and an honest split
3. Measure recommendations offline
4. Neighbours and collaborative filtering
5. Matrix factorisation for implicit feedback
6. Content, hybrids and cold start
7. Two stages: candidates and ranking
8. From offline scores to the real shop
9. Run it in production
What you will be able to do
Order each customer’s candidates with a model that learned from past orders. Describe every candidate with 26 features. Train a LightGBM LambdaMART ranker on the two weeks before the test period. Then measure what it adds over the best single source, segment by segment.
NotebookDo this lesson in a notebook
The notebook works out one small ranking by hand. Then it builds two candidate sources and six features, and trains a LightGBM LambdaMART ranker on the two weeks before the test period. You score the test customers’ candidates and compare the ranker with trending alone. Its first cell installs LightGBM, and its setup cells download about 7 MB of practice data. About 25 minutes.
Colab and Kaggle open your own copy. They need a free Google or Kaggle account. Stuck? See the solution notebook.
Before you startWhat to know and to have ready
What you need to know
- Candidates and the candidate ceiling, from Why two stages.
- NDCG, from Ranking metrics.
- Decision trees and gradient boosting, from the machine learning course’s Ensembles. This lesson explains what it needs.
What you need on your computer (only for the project steps)
- Your
larkfield-recsproject with the files of the last lesson:recs candidates --quickworks.
Tested on macOS with Python 3.12 and LightGBM 4.7.0. The Windows (PowerShell) and Linux commands were not run.
The assistant who weighs the clues
Section titled “The assistant who weighs the clues”Back to the shop assistant. She has twenty products on the counter. Now she must choose the order. She asks herself small questions:
- Did this customer look at it last week?
- Is it selling well this week?
- Does it match the style that this customer likes?
- Does the customer often buy in this category?
Each clue alone is weak. Together they are strong. And she knows how much each clue counts, because she has watched many customers buy. Her experience is a model, learned from past sales.
A learned ranker does the same. It describes each candidate with numbers, and it learns from past orders how to weigh them.
The problem: five lists, one order
Section titled “The problem: five lists, one order”The last lesson gave customer C106777 273 candidates from five sources. Here are three of them:
| Candidate | trending | item-cf | ALS |
|---|---|---|---|
| Lumi Green Nordic Coffee Table | #1 | – | #89 |
| Navy Nitrile-Grip Garden Gloves | #2 | #48 | #2 |
| Bramble Terracotta Mosaic Side Table | #4 | – | #39 |
Which one comes first? We can try simple rules:
- “Trending decides.” Then the coffee table comes first. That is the trending baseline: recall@10 0.156.
- “Add the scores with fixed weights.” That is the hybrid blend of Module 6: 0.167.
Neither rule knows the customer. For a new customer, trending is the only clue. For a loyal customer, their own history counts more. And there are many more clues than three source places. Writing the right rule by hand, for every kind of customer, is impossible.
The idea: learn the order from past orders
Section titled “The idea: learn the order from past orders”We turn the question into data. For a past period, we know the candidates and we know what each customer ordered. A model can learn which candidates were ordered.
Features: numbers that describe a candidate
Section titled “Features: numbers that describe a candidate”A feature is one number about one candidate of one customer. The project computes 26 of them, in four groups:
| Group | Examples |
|---|---|
| The sources | the place in each source’s list; the score of item-cf, ALS and content; how many sources found it |
| The product | buyers in 7 days, in 28 days and in all; the season (recent orders against usual orders); price; days on sale |
| The customer | number of orders; events in the last 30 days; days since the last visit |
| The pair | days since the customer touched it; the customer’s share of events in its category, subcategory and style; bought before; recent views and basket adds |
The pair features matter most for a personal list. “Days since the customer touched it” is a different number for every customer and product.
Labels: what was ordered next
Section titled “Labels: what was ordered next”The label of a candidate is 1 if the customer ordered it in the next two weeks, and 0 if not. The model learns to give label-1 candidates higher scores than label-0 candidates of the same customer.
That last part is important. We do not compare a candidate of one customer with a candidate of another. Each customer’s candidates form one group, and only the order inside a group matters. Learning an order inside groups is called learning to rank. The search course’s lesson Learning to rank teaches it in depth; here we use what we need.
The model: LambdaMART
Section titled “The model: LambdaMART”The project uses LightGBM, a library for gradient-boosted decision trees. A decision tree asks a few questions, such as “touched in the last 30 days?”, and gives a score. Gradient boosting adds hundreds of small trees. Each new tree corrects the mistakes of the trees before it.
With the training goal lambdarank, the trees learn an order. This combination is called LambdaMART. The idea: look at each pair inside a group where a label-0 candidate is above a label-1 candidate. Push the two apart. And push harder when the swap would change NDCG a lot, that is, near the top of the list.
A worked example: which mistake costs more?
Section titled “A worked example: which mistake costs more?”One customer has four candidates, A, B, C and D (a tiny example we made). The customer ordered one of them. NDCG gives a found product at place k the value 1 / log2(k + 1). With one ordered product, the best possible value is 1.
| The ordered product is at | Its value | After it moves up one place | Gain |
|---|---|---|---|
| place 2 | 1 / log2(3) = 0.631 | place 1: 1.000 | +0.369 |
| place 4 | 1 / log2(5) = 0.431 | place 3: 1 / log2(4) = 0.500 | +0.069 |
Both mistakes are one wrong pair. But fixing the mistake at the top is worth 0.369 / 0.069 = about 5 times more. So LambdaMART pushes that pair about 5 times harder. The ranker spends its effort where the customer looks: the first places.
Train without seeing the test period
Section titled “Train without seeing the test period”The ranker learns on the two weeks before the test period. Candidates and features come from the events before 8 June. The labels are the orders of 8 to 21 June. Then, for the test period, the project builds new candidates and features from the events before 22 June, and the trained trees score them.
Text description of the diagram
Box 1, train the ranker: the events before 8 June give the candidates and the features. The orders of 8 to 21 June give the labels (1 = ordered, 0 = not ordered). The trained trees go to box 2, use it on the test period: the events before 22 June give new candidates and features. The ranker scores them and keeps 10 suggestions. The orders of 22 June to 5 July give recall@10. They were never used for training.
So the test period stays honest. The ranker also learns from the most recent two weeks before it, which matters in a shop with seasons.
Negative sampling
Section titled “Negative sampling”Each training customer has about 200 candidates. Usually one or two of them were ordered; the rest have label 0. The project keeps every label-1 candidate and 100 random label-0 candidates per customer. This is negative sampling. It makes training faster and keeps the groups smaller. Customers with no ordered candidate are left out: their group has nothing to learn from.
On Larkfield’s test period, the number of negatives changed recall@10 only a little: 10 negatives 0.228, 50 negatives 0.230, 100 negatives 0.234, 1,000 negatives 0.226. The training window mattered more: 7 days 0.216, 14 days 0.234, 28 days 0.227. One week has too few orders to learn from. Four weeks reach back into a different part of the season.
Larkfield: what the ranker learned
Section titled “Larkfield: what the ranker learned”recs rank trains the ranker and prints its training data. It learned from 1,452 customers who ordered a candidate in the two weeks, with 146,949 candidate rows. 2,117 rows have label 1.
Feature importance says how much each feature helped. LightGBM adds up the gain: how much the splits on that feature improved the training goal, in all trees. The ten largest:
Text description of the chart
A dot chart with ten rows, from the most important feature: days since the customer touched it 9.6%; buyers in the last 7 days 8.3%; customer's share of this style 7.0%; customer's share of this category 7.0%; buyers over the whole log 6.8%; buyers in the last 28 days 6.3%; place in trending 6.0%; price 4.6%; days since the customer's last visit 4.6%; place in ALS 4.4%. The other 16 features share the rest of the gain.
The first feature is personal: how recently the customer touched the product. The second is popularity this week. Then come the customer’s taste in style and category. The ranker mixes “what this person likes” with “what sells now”. No single source does both.
Read importance with care. It says what the trees used, not what causes an order. Two features that carry the same information share their gain. The three popularity features together have 21.3% of the gain.
One customer’s list
Section titled “One customer’s list”For C106777, the ranker’s first three places and the two ordered products look like this:
| Place | Candidate | trending | ALS | Days since touched | Score | Ordered? |
|---|---|---|---|---|---|---|
| 1 | Lumi Green Nordic Coffee Table | #1 | #89 | 37 | 1.107 | no |
| 2 | Blush Rosé Wine | – | #73 | 37 | 1.089 | no |
| 3 | Oak Label Shiraz | #22 | #26 | 66 | 0.893 | no |
| 11 | Bramble Terracotta Mosaic Side Table | #4 | #39 | 381 | -0.158 | yes |
| 15 | Navy Nitrile-Grip Garden Gloves | #2 | #2 | 238 | -0.307 | yes |
The two ordered products are at places 11 and 15: just outside the list. Places 2 and 3 are wines that the customer ordered before. The ranker learned that people often buy again what they touched recently. For this customer that is wrong here, and a shop may not even be allowed to suggest wine. The next lesson’s rules change this list.
What the ranker adds, by segment
Section titled “What the ranker adds, by segment”The ranker’s recall@10 on Larkfield’s test period is 0.234 (95% interval 0.217 to 0.251). By segment: new 0.198, occasional 0.268, loyal 0.258.
The fair question is: what does it add over the best single source? That is trending (0.156). And over the best combination by hand, the hybrid blend (0.167)?
Text description of the chart
Two panels with a line at zero. Two-stage minus trending: new customers 0.029 (interval 0.015 to 0.044), occasional 0.106 (0.075 to 0.138), loyal 0.126 (0.091 to 0.159), all customers 0.077 (0.062 to 0.092). Two-stage minus the hybrid blend: new 0.039, occasional 0.114, loyal 0.064, all 0.066 (0.052 to 0.080). Every interval is above zero.
Every interval is above zero, so the gain is clear in every segment. But it is not the same size:
- Occasional and loyal customers gain the most over trending: +0.106 and +0.126. They have a history, so the pair features have something to say.
- New customers gain the least: +0.029. Most of them have no history at all: 633 of the 952 touched nothing before the period. For them, the ranker has only the product features, and those say “popular now”, as trending does.
The notebook shows where most of the gain comes from. A small ranker with only two sources (trending and recent products) and six features gets 0.227 on the same test customers. Mixing “what sells this week” with “what the customer touched recently” does most of the work. The other three sources and 20 features add a little more.
Against the hybrid blend, occasional customers gain the most (+0.114). The blend uses ALS, and ALS is weak for customers with one or two old orders. The ranker learns to trust the recent and popular signals for them instead.
Hybrids also tuned the blend’s weights on the validation period (ALS 1, content 1, trending 2): 0.193 on the test period. Compare the ranker with it: recs compare "hybrid-blend[w_content=1,w_trending=2]" two-stage --by segment. The ranker still adds +0.041 (+0.027 to +0.054). By segment: occasional customers +0.064, loyal +0.062, new +0.013.
When to use it, when not, and what it costs
Section titled “When to use it, when not, and what it costs”Use a learned ranker when you have several candidate sources and enough orders to learn from. Larkfield’s two weeks had 2,117 ordered candidates. Use it when the right weights differ from customer to customer.
Do not use it when you have few orders: a small shop may have too few label-1 rows, and a hand blend is safer. Do not use it when you cannot compute the same features at serving time as in training. A feature that differs between the two silently breaks the ranker.
Cost: fit 12.4 seconds and 841 MB at peak on one thread, against 3.8 seconds and 496 MB for ALS (our laptop; your times will differ). Lists for the 2,118 test customers took about 5 seconds. At request time, one customer takes about 10 milliseconds. Every source and every feature is one more thing to keep fresh and to monitor.
Guided practice: train the ranker
Section titled “Guided practice: train the ranker”-
Open the notebook from the box Do this lesson in a notebook at the top of this page. Run the install cell and the setup cells.
-
Run section 1. It computes the two NDCG gains of the worked example: 0.369 and 0.069.
-
Run sections 2 and 3. They build the training rows from the two weeks before the test period, and train the ranker.
-
In section 4, rank the test customers’ candidates and compute the ranker’s recall@10. The check prints
Check passed. -
In a terminal, in your
larkfield-recsfolder with the virtual environment active, copy therankcommand:Terminal window S=../nextia-learning-labs/C37/snapshots/end-of-m07cp $S/larkfield_recs/commands/rank.py larkfield_recs/commands/Terminal window $S = "..\nextia-learning-labs\C37\snapshots\end-of-m07"Copy-Item "$S\larkfield_recs\commands\rank.py" larkfield_recs\commands\ -
Train the ranker and see what it learned:
Terminal window recs rankTrained on the two weeks before 2026-06-22: 1,452 customers, 146,949 candidate rows, 2,117 ordered (label 1); candidates found 56.2% of the ordered productsFit time (both stages, two cut-offs): 11.5 sMost important features (share of the total gain):days_since_user_touch 9.6%pop_7 8.3%style_share 7.0%cat_share 7.0%pop_all 6.8%pop_28 6.2%trending_rank 6.0%log_price 4.6%user_days_since_last 4.6%als_rank 4.4% -
Score the two-stage recommender on the test period:
Terminal window recs evaluate two-stage --by segmentMethod two-stage (candidates from 5 sources, then a LightGBM LambdaMART ranker), test periodCustomers scored: 2,118; trained on events before 2026-06-22recall@10 0.234 (95% interval 0.217 to 0.251), NDCG@10 0.156, MAP@10 0.119, precision@10 0.036, hit rate 30.4%coverage 29.5% of the products on sale, novelty 5.68 bits, diversity 0.93, new products in the lists 23.2%time: fit 12.4 s, suggestions 5.4 ssegment customers recall@10 NDCG@10 95% interval (recall)new 952 0.198 0.133 0.175 to 0.221occasional 556 0.268 0.173 0.235 to 0.303loyal 610 0.258 0.177 0.229 to 0.288new products: 647 orders of new products in the period; recall@10 on them 0.329 -
Compare it with the best single source. If
outputs/runs/has no run oftrending, runrecs evaluate trendingfirst.Terminal window recs compare trending two-stage --by segmentrecall@10: trending 0.156, two-stage 0.234difference +0.077 (95% interval +0.062 to +0.092); better for 322 customers, worse for 113, the same for 1,683The difference is clear: the interval does not include 0.coverage 0.3% -> 29.5%; novelty 5.10 -> 5.68new 952 customers +0.029 (+0.015 to +0.044)occasional 556 customers +0.106 (+0.075 to +0.138)loyal 610 customers +0.126 (+0.091 to +0.159) -
Compare it with the hybrid blend. If you have no saved run, run
recs evaluate hybrid-blendfirst.Terminal window recs compare hybrid-blend two-stage --by segmentrecall@10: hybrid-blend 0.167, two-stage 0.234difference +0.066 (95% interval +0.052 to +0.080); better for 302 customers, worse for 104, the same for 1,712The difference is clear: the interval does not include 0.coverage 32.4% -> 29.5%; novelty 5.85 -> 5.68new 952 customers +0.039 (+0.025 to +0.053)occasional 556 customers +0.114 (+0.080 to +0.147)loyal 610 customers +0.064 (+0.033 to +0.095)
Your turn: fewer negatives
Section titled “Your turn: fewer negatives”Train with 20 negatives per customer instead of 100. Before you run it, predict: how many training rows will there be, and will recall@10 change clearly?
recs rank --set negatives=20recs evaluate two-stage --set negatives=20 --by segmentrecs compare two-stage "two-stage[negatives=20]-test" --by segmentThe quotes keep the square brackets safe in zsh and PowerShell.
Show the answer
recs rank --set negatives=20 prints 1,452 customers, 31,157 candidate rows, 2,117 ordered (label 1) and a fit time of 9.0 seconds. The same customers and the same label-1 rows, but about a fifth of the rows.
recall@10: two-stage 0.234, two-stage[negatives=20]-test 0.233difference -0.001 (95% interval -0.009 to +0.008); better for 103 customers, worse for 100, the same for 1,915The difference is not clear: the interval includes 0.coverage 29.5% -> 29.4%; novelty 5.68 -> 5.82 new 952 customers +0.002 (-0.008 to +0.012) occasional 556 customers -0.023 (-0.044 to -0.004) loyal 610 customers +0.015 (-0.005 to +0.037)Overall, no clear change. The occasional segment’s interval does not include 0, but with three segments, one can look clear by chance. Check it on the validation period before you believe it. With 20 negatives the ranker trains faster on a fifth of the rows, for about the same recall.
Common mistakes
Section titled “Common mistakes”Common mistake
Choosing the ranker's settings on the test period
Symptom: Tomás reads the settings table and sees that 100 trees score 0.240 on the test period, against 0.234 with 300 trees. He changes the default to 100 trees and reports 0.240.
Cause: the test period chose the setting. The reported number is now too optimistic, as with tuning in Tune without fooling yourself.
Diagnose: ask on which period each setting was chosen.
Fix: compare settings with --period validation, choose one, then report the test period once. Keep the defaults until the validation period says otherwise.
Common mistake
A feature that knows the label
Symptom: Priya adds a feature “viewed in the last 7 days”. The ranker’s training score is very high, but recall@10 on the test period falls.
Cause: in training, she counted the views up to the end of the label window. Customers view a product just before they order it, so the feature reveals the label. At test time, those views are in the future and the feature is empty.
Diagnose: for every feature, check the last day of events it uses. It must be before the first day of its labels, in training and in testing.
Fix: compute every feature from the events before the cut-off only, with the same code for training and for serving.
Knowledge check
Section titled “Knowledge check”Check your understanding
In one customer's list, an ordered product is at place 2 below an unordered product. In another customer's list, an ordered product is at place 9 below an unordered one at place 8. Which pair does LambdaMART push harder, and why?
Show the answer
The pair at places 1 and 2. Moving the ordered product from place 2 to place 1 gains 0.369 of NDCG (0.631 to 1). Moving it from place 9 to place 8 gains only 0.014 (0.301 to 0.315). LambdaMART weighs each pair by the change in NDCG, so mistakes near the top count most.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
Why does the ranker learn on the two weeks before the test period, and not on the test period itself?
Show the answer
If it learned from the test period’s orders, the test score would measure memory, not prediction: it would be leakage. The two weeks before the period give labels that a real shop would have on the day it trains the ranker. The features for those labels come from the events before 8 June, as the test features come from the events before 22 June.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
Explain what the ranker added over the best single candidate source, by segment, and why the new customers gained the least.
Show the answer
The best single source is trending (0.156). The two-stage ranker gets 0.234: +0.077 overall. New customers gain +0.029, occasional +0.106 and loyal +0.126, all with intervals above zero. Customers with a history gain most, because the pair features describe them. Most new customers touched nothing before the period. For them, the ranker has only product features, and these say about what trending says.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Recap
- A learned ranker describes each candidate with features (26 here). It learns from past orders: label 1 for an ordered candidate.
- Learning to rank compares candidates only inside one customer’s group. LambdaMART (gradient-boosted trees with
lambdarank) pushes wrong pairs apart, harder near the top. A fix at place 2 is worth about 5 times a fix at place 4. - It trains on the two weeks before the test period, with 100 negatives per customer.
- Most important features: days since the customer touched the product, buyers in the last 7 days, style share and category share.
- Recall@10 0.234 (0.217 to 0.251). Over trending: +0.077, with new customers +0.029 and loyal +0.126. Over the hybrid blend: +0.066.
The ranker orders candidates by predicted interest. It suggested wine, and products the customer already bought. Next, we add the shop’s business rules and measure what each rule costs.