Skip to content
Français

A learned ranker

  • Lesson 21
  • 45 min
  • Module 7: Two stages: candidates and ranking

Where we are

We have candidates from several sources. Now we must put the best ones first. In this lesson, we build features for each customer and product, and train a LambdaMART ranker with LightGBM on the next purchase.

What you will be able to do

Order each customer’s candidates with a model that learned from past orders. Describe every candidate with 26 features. Train a LightGBM LambdaMART ranker on the two weeks before the test period. Then measure what it adds over the best single source, segment by segment.

NotebookDo this lesson in a notebookColab · Kaggle · GitHub

The notebook works out one small ranking by hand. Then it builds two candidate sources and six features, and trains a LightGBM LambdaMART ranker on the two weeks before the test period. You score the test customers’ candidates and compare the ranker with trending alone. Its first cell installs LightGBM, and its setup cells download about 7 MB of practice data. About 25 minutes.

Colab and Kaggle open your own copy. They need a free Google or Kaggle account. Stuck? See the solution notebook.

Before you startWhat to know and to have readyAbout 45 minutes

What you need to know

  • Candidates and the candidate ceiling, from Why two stages.
  • NDCG, from Ranking metrics.
  • Decision trees and gradient boosting, from the machine learning course’s Ensembles. This lesson explains what it needs.

What you need on your computer (only for the project steps)

  • Your larkfield-recs project with the files of the last lesson: recs candidates --quick works.

Tested on macOS with Python 3.12 and LightGBM 4.7.0. The Windows (PowerShell) and Linux commands were not run.

Back to the shop assistant. She has twenty products on the counter. Now she must choose the order. She asks herself small questions:

  • Did this customer look at it last week?
  • Is it selling well this week?
  • Does it match the style that this customer likes?
  • Does the customer often buy in this category?

Each clue alone is weak. Together they are strong. And she knows how much each clue counts, because she has watched many customers buy. Her experience is a model, learned from past sales.

A learned ranker does the same. It describes each candidate with numbers, and it learns from past orders how to weigh them.

The last lesson gave customer C106777 273 candidates from five sources. Here are three of them:

Candidate trending item-cf ALS
Lumi Green Nordic Coffee Table #1 – #89
Navy Nitrile-Grip Garden Gloves #2 #48 #2
Bramble Terracotta Mosaic Side Table #4 – #39

Which one comes first? We can try simple rules:

  • “Trending decides.” Then the coffee table comes first. That is the trending baseline: recall@10 0.156.
  • “Add the scores with fixed weights.” That is the hybrid blend of Module 6: 0.167.

Neither rule knows the customer. For a new customer, trending is the only clue. For a loyal customer, their own history counts more. And there are many more clues than three source places. Writing the right rule by hand, for every kind of customer, is impossible.

The idea: learn the order from past orders

Section titled “The idea: learn the order from past orders”

We turn the question into data. For a past period, we know the candidates and we know what each customer ordered. A model can learn which candidates were ordered.

Features: numbers that describe a candidate

Section titled “Features: numbers that describe a candidate”

A feature is one number about one candidate of one customer. The project computes 26 of them, in four groups:

Group Examples
The sources the place in each source’s list; the score of item-cf, ALS and content; how many sources found it
The product buyers in 7 days, in 28 days and in all; the season (recent orders against usual orders); price; days on sale
The customer number of orders; events in the last 30 days; days since the last visit
The pair days since the customer touched it; the customer’s share of events in its category, subcategory and style; bought before; recent views and basket adds

The pair features matter most for a personal list. “Days since the customer touched it” is a different number for every customer and product.

The label of a candidate is 1 if the customer ordered it in the next two weeks, and 0 if not. The model learns to give label-1 candidates higher scores than label-0 candidates of the same customer.

That last part is important. We do not compare a candidate of one customer with a candidate of another. Each customer’s candidates form one group, and only the order inside a group matters. Learning an order inside groups is called learning to rank. The search course’s lesson Learning to rank teaches it in depth; here we use what we need.

The project uses LightGBM, a library for gradient-boosted decision trees. A decision tree asks a few questions, such as “touched in the last 30 days?”, and gives a score. Gradient boosting adds hundreds of small trees. Each new tree corrects the mistakes of the trees before it.

With the training goal lambdarank, the trees learn an order. This combination is called LambdaMART. The idea: look at each pair inside a group where a label-0 candidate is above a label-1 candidate. Push the two apart. And push harder when the swap would change NDCG a lot, that is, near the top of the list.

A worked example: which mistake costs more?

Section titled “A worked example: which mistake costs more?”

One customer has four candidates, A, B, C and D (a tiny example we made). The customer ordered one of them. NDCG gives a found product at place k the value 1 / log2(k + 1). With one ordered product, the best possible value is 1.

The ordered product is at Its value After it moves up one place Gain
place 2 1 / log2(3) = 0.631 place 1: 1.000 +0.369
place 4 1 / log2(5) = 0.431 place 3: 1 / log2(4) = 0.500 +0.069

Both mistakes are one wrong pair. But fixing the mistake at the top is worth 0.369 / 0.069 = about 5 times more. So LambdaMART pushes that pair about 5 times harder. The ranker spends its effort where the customer looks: the first places.

The ranker learns on the two weeks before the test period. Candidates and features come from the events before 8 June. The labels are the orders of 8 to 21 June. Then, for the test period, the project builds new candidates and features from the events before 22 June, and the trained trees score them.

Two boxes from left to right. Box 1, Train the ranker: events before 8 June give candidates and features; the orders of 8 to 21 June give the labels, 1 for ordered and 0 for not ordered. An arrow labelled 'the trained trees' leads to box 2, Use it on the test period: events before 22 June give candidates and features; the ranker scores them and keeps 10 suggestions; the orders of 22 June to 5 July give recall@10, and were never seen in training.
The ranker learns on the two weeks before the test period, then ranks the test period's candidates. The test orders are never used for training.
Text description of the diagram

Box 1, train the ranker: the events before 8 June give the candidates and the features. The orders of 8 to 21 June give the labels (1 = ordered, 0 = not ordered). The trained trees go to box 2, use it on the test period: the events before 22 June give new candidates and features. The ranker scores them and keeps 10 suggestions. The orders of 22 June to 5 July give recall@10. They were never used for training.

So the test period stays honest. The ranker also learns from the most recent two weeks before it, which matters in a shop with seasons.

Each training customer has about 200 candidates. Usually one or two of them were ordered; the rest have label 0. The project keeps every label-1 candidate and 100 random label-0 candidates per customer. This is negative sampling. It makes training faster and keeps the groups smaller. Customers with no ordered candidate are left out: their group has nothing to learn from.

On Larkfield’s test period, the number of negatives changed recall@10 only a little: 10 negatives 0.228, 50 negatives 0.230, 100 negatives 0.234, 1,000 negatives 0.226. The training window mattered more: 7 days 0.216, 14 days 0.234, 28 days 0.227. One week has too few orders to learn from. Four weeks reach back into a different part of the season.

recs rank trains the ranker and prints its training data. It learned from 1,452 customers who ordered a candidate in the two weeks, with 146,949 candidate rows. 2,117 rows have label 1.

Feature importance says how much each feature helped. LightGBM adds up the gain: how much the splits on that feature improved the training goal, in all trees. The ten largest:

The ten most important of the ranker's 26 features, by share of the total gain. Personal features (recency, style, category) and popularity features both matter.
Text description of the chart

A dot chart with ten rows, from the most important feature: days since the customer touched it 9.6%; buyers in the last 7 days 8.3%; customer's share of this style 7.0%; customer's share of this category 7.0%; buyers over the whole log 6.8%; buyers in the last 28 days 6.3%; place in trending 6.0%; price 4.6%; days since the customer's last visit 4.6%; place in ALS 4.4%. The other 16 features share the rest of the gain.

The first feature is personal: how recently the customer touched the product. The second is popularity this week. Then come the customer’s taste in style and category. The ranker mixes “what this person likes” with “what sells now”. No single source does both.

Read importance with care. It says what the trees used, not what causes an order. Two features that carry the same information share their gain. The three popularity features together have 21.3% of the gain.

For C106777, the ranker’s first three places and the two ordered products look like this:

Place Candidate trending ALS Days since touched Score Ordered?
1 Lumi Green Nordic Coffee Table #1 #89 37 1.107 no
2 Blush Rosé Wine – #73 37 1.089 no
3 Oak Label Shiraz #22 #26 66 0.893 no
11 Bramble Terracotta Mosaic Side Table #4 #39 381 -0.158 yes
15 Navy Nitrile-Grip Garden Gloves #2 #2 238 -0.307 yes

The two ordered products are at places 11 and 15: just outside the list. Places 2 and 3 are wines that the customer ordered before. The ranker learned that people often buy again what they touched recently. For this customer that is wrong here, and a shop may not even be allowed to suggest wine. The next lesson’s rules change this list.

The ranker’s recall@10 on Larkfield’s test period is 0.234 (95% interval 0.217 to 0.251). By segment: new 0.198, occasional 0.268, loyal 0.258.

The fair question is: what does it add over the best single source? That is trending (0.156). And over the best combination by hand, the hybrid blend (0.167)?

What the two-stage ranker adds to recall@10 on Larkfield's test period, by segment, with 95% intervals (paired bootstrap over customers). It helps every segment; new customers gain the least.
Text description of the chart

Two panels with a line at zero. Two-stage minus trending: new customers 0.029 (interval 0.015 to 0.044), occasional 0.106 (0.075 to 0.138), loyal 0.126 (0.091 to 0.159), all customers 0.077 (0.062 to 0.092). Two-stage minus the hybrid blend: new 0.039, occasional 0.114, loyal 0.064, all 0.066 (0.052 to 0.080). Every interval is above zero.

Every interval is above zero, so the gain is clear in every segment. But it is not the same size:

  • Occasional and loyal customers gain the most over trending: +0.106 and +0.126. They have a history, so the pair features have something to say.
  • New customers gain the least: +0.029. Most of them have no history at all: 633 of the 952 touched nothing before the period. For them, the ranker has only the product features, and those say “popular now”, as trending does.

The notebook shows where most of the gain comes from. A small ranker with only two sources (trending and recent products) and six features gets 0.227 on the same test customers. Mixing “what sells this week” with “what the customer touched recently” does most of the work. The other three sources and 20 features add a little more.

Against the hybrid blend, occasional customers gain the most (+0.114). The blend uses ALS, and ALS is weak for customers with one or two old orders. The ranker learns to trust the recent and popular signals for them instead.

Hybrids also tuned the blend’s weights on the validation period (ALS 1, content 1, trending 2): 0.193 on the test period. Compare the ranker with it: recs compare "hybrid-blend[w_content=1,w_trending=2]" two-stage --by segment. The ranker still adds +0.041 (+0.027 to +0.054). By segment: occasional customers +0.064, loyal +0.062, new +0.013.

When to use it, when not, and what it costs

Section titled “When to use it, when not, and what it costs”

Use a learned ranker when you have several candidate sources and enough orders to learn from. Larkfield’s two weeks had 2,117 ordered candidates. Use it when the right weights differ from customer to customer.

Do not use it when you have few orders: a small shop may have too few label-1 rows, and a hand blend is safer. Do not use it when you cannot compute the same features at serving time as in training. A feature that differs between the two silently breaks the ranker.

Cost: fit 12.4 seconds and 841 MB at peak on one thread, against 3.8 seconds and 496 MB for ALS (our laptop; your times will differ). Lists for the 2,118 test customers took about 5 seconds. At request time, one customer takes about 10 milliseconds. Every source and every feature is one more thing to keep fresh and to monitor.

  1. Open the notebook from the box Do this lesson in a notebook at the top of this page. Run the install cell and the setup cells.

  2. Run section 1. It computes the two NDCG gains of the worked example: 0.369 and 0.069.

  3. Run sections 2 and 3. They build the training rows from the two weeks before the test period, and train the ranker.

  4. In section 4, rank the test customers’ candidates and compute the ranker’s recall@10. The check prints Check passed.

  5. In a terminal, in your larkfield-recs folder with the virtual environment active, copy the rank command:

    Terminal window
    S=../nextia-learning-labs/C37/snapshots/end-of-m07
    cp $S/larkfield_recs/commands/rank.py larkfield_recs/commands/
  6. Train the ranker and see what it learned:

    Terminal window
    recs rank
    Trained on the two weeks before 2026-06-22: 1,452 customers, 146,949 candidate rows, 2,117 ordered (label 1); candidates found 56.2% of the ordered products
    Fit time (both stages, two cut-offs): 11.5 s
    Most important features (share of the total gain):
    days_since_user_touch 9.6%
    pop_7 8.3%
    style_share 7.0%
    cat_share 7.0%
    pop_all 6.8%
    pop_28 6.2%
    trending_rank 6.0%
    log_price 4.6%
    user_days_since_last 4.6%
    als_rank 4.4%
  7. Score the two-stage recommender on the test period:

    Terminal window
    recs evaluate two-stage --by segment
    Method two-stage (candidates from 5 sources, then a LightGBM LambdaMART ranker), test period
    Customers scored: 2,118; trained on events before 2026-06-22
    recall@10 0.234 (95% interval 0.217 to 0.251), NDCG@10 0.156, MAP@10 0.119, precision@10 0.036, hit rate 30.4%
    coverage 29.5% of the products on sale, novelty 5.68 bits, diversity 0.93, new products in the lists 23.2%
    time: fit 12.4 s, suggestions 5.4 s
    segment customers recall@10 NDCG@10 95% interval (recall)
    new 952 0.198 0.133 0.175 to 0.221
    occasional 556 0.268 0.173 0.235 to 0.303
    loyal 610 0.258 0.177 0.229 to 0.288
    new products: 647 orders of new products in the period; recall@10 on them 0.329
  8. Compare it with the best single source. If outputs/runs/ has no run of trending, run recs evaluate trending first.

    Terminal window
    recs compare trending two-stage --by segment
    recall@10: trending 0.156, two-stage 0.234
    difference +0.077 (95% interval +0.062 to +0.092); better for 322 customers, worse for 113, the same for 1,683
    The difference is clear: the interval does not include 0.
    coverage 0.3% -> 29.5%; novelty 5.10 -> 5.68
    new 952 customers +0.029 (+0.015 to +0.044)
    occasional 556 customers +0.106 (+0.075 to +0.138)
    loyal 610 customers +0.126 (+0.091 to +0.159)
  9. Compare it with the hybrid blend. If you have no saved run, run recs evaluate hybrid-blend first.

    Terminal window
    recs compare hybrid-blend two-stage --by segment
    recall@10: hybrid-blend 0.167, two-stage 0.234
    difference +0.066 (95% interval +0.052 to +0.080); better for 302 customers, worse for 104, the same for 1,712
    The difference is clear: the interval does not include 0.
    coverage 32.4% -> 29.5%; novelty 5.85 -> 5.68
    new 952 customers +0.039 (+0.025 to +0.053)
    occasional 556 customers +0.114 (+0.080 to +0.147)
    loyal 610 customers +0.064 (+0.033 to +0.095)

Train with 20 negatives per customer instead of 100. Before you run it, predict: how many training rows will there be, and will recall@10 change clearly?

Terminal window
recs rank --set negatives=20
recs evaluate two-stage --set negatives=20 --by segment
recs compare two-stage "two-stage[negatives=20]-test" --by segment

The quotes keep the square brackets safe in zsh and PowerShell.

Show the answer

recs rank --set negatives=20 prints 1,452 customers, 31,157 candidate rows, 2,117 ordered (label 1) and a fit time of 9.0 seconds. The same customers and the same label-1 rows, but about a fifth of the rows.

recall@10: two-stage 0.234, two-stage[negatives=20]-test 0.233
difference -0.001 (95% interval -0.009 to +0.008); better for 103 customers, worse for 100, the same for 1,915
The difference is not clear: the interval includes 0.
coverage 29.5% -> 29.4%; novelty 5.68 -> 5.82
new 952 customers +0.002 (-0.008 to +0.012)
occasional 556 customers -0.023 (-0.044 to -0.004)
loyal 610 customers +0.015 (-0.005 to +0.037)

Overall, no clear change. The occasional segment’s interval does not include 0, but with three segments, one can look clear by chance. Check it on the validation period before you believe it. With 20 negatives the ranker trains faster on a fifth of the rows, for about the same recall.

Common mistake

Choosing the ranker's settings on the test period

Symptom: Tomás reads the settings table and sees that 100 trees score 0.240 on the test period, against 0.234 with 300 trees. He changes the default to 100 trees and reports 0.240.

Cause: the test period chose the setting. The reported number is now too optimistic, as with tuning in Tune without fooling yourself.

Diagnose: ask on which period each setting was chosen.

Fix: compare settings with --period validation, choose one, then report the test period once. Keep the defaults until the validation period says otherwise.

Common mistake

A feature that knows the label

Symptom: Priya adds a feature “viewed in the last 7 days”. The ranker’s training score is very high, but recall@10 on the test period falls.

Cause: in training, she counted the views up to the end of the label window. Customers view a product just before they order it, so the feature reveals the label. At test time, those views are in the future and the feature is empty.

Diagnose: for every feature, check the last day of events it uses. It must be before the first day of its labels, in training and in testing.

Fix: compute every feature from the events before the cut-off only, with the same code for training and for serving.

Check your understanding

In one customer's list, an ordered product is at place 2 below an unordered product. In another customer's list, an ordered product is at place 9 below an unordered one at place 8. Which pair does LambdaMART push harder, and why?

Show the answer

The pair at places 1 and 2. Moving the ordered product from place 2 to place 1 gains 0.369 of NDCG (0.631 to 1). Moving it from place 9 to place 8 gains only 0.014 (0.301 to 0.315). LambdaMART weighs each pair by the change in NDCG, so mistakes near the top count most.

Check your understanding

Why does the ranker learn on the two weeks before the test period, and not on the test period itself?

Show the answer

If it learned from the test period’s orders, the test score would measure memory, not prediction: it would be leakage. The two weeks before the period give labels that a real shop would have on the day it trains the ranker. The features for those labels come from the events before 8 June, as the test features come from the events before 22 June.

Check your understanding

Explain what the ranker added over the best single candidate source, by segment, and why the new customers gained the least.

Show the answer

The best single source is trending (0.156). The two-stage ranker gets 0.234: +0.077 overall. New customers gain +0.029, occasional +0.106 and loyal +0.126, all with intervals above zero. Customers with a history gain most, because the pair features describe them. Most new customers touched nothing before the period. For them, the ranker has only product features, and these say about what trending says.

Recap

  • A learned ranker describes each candidate with features (26 here). It learns from past orders: label 1 for an ordered candidate.
  • Learning to rank compares candidates only inside one customer’s group. LambdaMART (gradient-boosted trees with lambdarank) pushes wrong pairs apart, harder near the top. A fix at place 2 is worth about 5 times a fix at place 4.
  • It trains on the two weeks before the test period, with 100 negatives per customer.
  • Most important features: days since the customer touched the product, buyers in the last 7 days, style share and category share.
  • Recall@10 0.234 (0.217 to 0.251). Over trending: +0.077, with new customers +0.029 and loyal +0.126. Over the hybrid blend: +0.066.

The ranker orders candidates by predicted interest. It suggested wine, and products the customer already bought. Next, we add the shop’s business rules and measure what each rule costs.

Next: Business rules and re-ranking