Why two stages
- Lesson 20
- 30 min
- Module 7: Two stages: candidates and ranking
Where we are
Our hybrids combine methods by hand. Large shops use a fixed pattern for this. In this lesson, we take candidates from several sources and see why the candidate step limits everything that comes after it.
Lesson 20 of 29 · Recommendation and Personalisation: Show Each Person What MattersWhy two stages
1. The recommendation problem
2. Data, implicit feedback and an honest split
3. Measure recommendations offline
4. Neighbours and collaborative filtering
5. Matrix factorisation for implicit feedback
6. Content, hybrids and cold start
7. Two stages: candidates and ranking
8. From offline scores to the real shop
9. Run it in production
What you will be able to do
See why large shops recommend in two stages. Several fast sources find a few hundred candidate products, then a careful model ranks them. Measure what each of Larkfield’s five sources finds, and the limit that the candidates put on every ranker after them.
NotebookDo this lesson in a notebook
The notebook builds two simple candidate sources: the 100 trending products, and the products each customer touched in the last 30 days. It measures the share of ordered products that each source finds. You compute the share that both sources find together. Its setup cells download about 7 MB of practice data. About 15 minutes.
Colab and Kaggle open your own copy. They need a free Google or Kaggle account. Stuck? See the solution notebook.
Before you startWhat to know and to have ready
What you need to know
- Recall@10, from Ranking metrics.
- Item-item collaborative filtering, from Item-item collaborative filtering.
- ALS, from ALS and BPR with implicit, and content-based suggestions, from Content-based recommendations.
- The hybrid blend, from Hybrids.
What you need on your computer (only for the project steps; the notebook needs none of it)
- Your
larkfield-recsproject at the end of Module 6, with its virtual environment active.recs checkprintsSetup check passed. - The practice files in
../nextia-learning-labs, beside the project folder. - On macOS, Homebrew’s OpenMP library for LightGBM:
brew install libomp.
Tested on macOS with Python 3.12 and LightGBM 4.7.0. The Windows (PowerShell) and Linux commands were not run.
The shop assistant and the full warehouse
Section titled “The shop assistant and the full warehouse”A customer asks a shop assistant: “What would I like?” The shop sells 3,000 products. The assistant does not think hard about each of the 3,000. That would take all day.
She works in two steps. First, she walks quickly past a few shelves and picks about twenty likely products. Some are bestsellers at the front. Some go with what the customer bought last time. Some look like what the customer looked at today. This step is fast and rough.
Then she looks carefully at the twenty. She thinks about the customer’s taste, the season and the price. She puts the best five on the counter.
A two-stage recommender works the same way. This lesson is about the first step, and the limit it sets.
The problem: one method cannot do both jobs
Section titled “The problem: one method cannot do both jobs”In Modules 4 to 6, each method did everything at once. It scored all 2,994 products for each customer and kept the top 10. That has two problems.
A careful model is too slow for every product
Section titled “A careful model is too slow for every product”A careful model looks at many facts about each customer and product pair. The next lesson uses 26 facts per pair. Larkfield has 40,000 customers and 2,994 products. That is 40,000 × 2,994 = 119.8 million pairs to describe and score every night.
With about 200 candidates per customer, it is 40,000 × 202 = 8.1 million pairs. That is about 15 times less work. In Batch or real time, the project ranks one customer’s 200 candidates in about 10 milliseconds. Fifteen times more is about 150 milliseconds: more than Amira’s budget of 100 milliseconds per request (an estimate, not a measurement).
Each fast method misses different products
Section titled “Each fast method misses different products”The fast methods each see a different part of the truth:
- Trending finds the in-season bestsellers, for everyone.
- Item-cf finds products bought with the customer’s past products.
- ALS finds products that match the customer’s hidden tastes.
- Content finds products with a similar text, also new products with no history.
- The customer’s own recent products find what they looked at and did not buy yet.
In Hybrids, we added three of these scores with fixed weights. That got 0.167, and 0.193 with weights tuned on the validation period. But fixed weights cannot say “for this customer, trust ALS more”. We need a better way to combine them.
The idea: retrieve, then rank
Section titled “The idea: retrieve, then rank”A two-stage recommender splits the work:
- Candidate generation (also called retrieval). Several fast sources each give a short list. Together they make a few hundred candidates per customer.
- Ranking. A careful model scores only the candidates and orders them. Then the business rules remove products the shop must not show.
This pattern is common in large shops and streaming services. YouTube’s engineers described it in 2016 for videos. The first stage must be fast and must not miss much. The second stage must be accurate on a short list.
Text description of the diagram
One customer’s history goes to five candidate sources. They are trending (100 products), item-cf (100), ALS (100), content (50) and the customer’s recent products of 30 days (up to 30). Together they give about 202 different candidates out of 2,994 products, because the lists overlap. A ranker scores each candidate with 26 features (LightGBM). Business rules remove products the shop must not show. These are out of stock, age-restricted, already bought or sensitive, or too many of one kind. Ten suggestions are left.
Candidate recall: the ceiling
Section titled “Candidate recall: the ceiling”The ranker can only put first a product that a source found. A product that no source found is lost, whatever the ranker does.
So we measure the first stage alone. Candidate recall is the share of the products ordered in the test period that are among a customer’s candidates, at any place. It is the ceiling of the second stage. No ranker can find more.
A worked example: one customer
Section titled “A worked example: one customer”Customer C106777 is a loyal customer. You met this customer in Ranking metrics. In the test period, the customer ordered three products. The five sources made 273 candidates for this customer. The table shows the place of each ordered product in each source’s list (“–” = not in that list).
| Ordered product | trending | item-cf | ALS | content | recent | A candidate? |
|---|---|---|---|---|---|---|
| Navy Nitrile-Grip Garden Gloves | #2 | #48 | #2 | – | – | yes |
| Bramble Terracotta Mosaic Side Table | #4 | – | #39 | – | – | yes |
| Hartley Blue Wall Lantern | – | – | – | – | – | no |
The gloves are in three lists, and the side table in two. The wall lantern is in no list. So, for this customer, the ceiling is 2 of 3 = 0.667. Even a perfect ranker could not show the lantern.
Look at the side table too. Trending has it at place 4, and ALS at place 39. Should it come first? That is the ranker’s question, in the next lesson.
The shares do not add up
Section titled “The shares do not add up”Each source finds a share of all the ordered products. If we add the five shares, we get 37.8% + 25.7% + 23.4% + 10.7% + 6.8% = 104.5%. That is more than all the products. The true share of all sources together is 55.5%.
The reason is overlap, as with the gloves above. Many ordered products are in two or three lists. A new source helps only with the products that the other sources miss.
Larkfield: what each source finds
Section titled “Larkfield: what each source finds”Now all 2,118 test customers. The project’s recs candidates command measures each source and all of them together on Larkfield’s test period.
Text description of the chart
A column chart with six columns. Trending (100 products) finds 37.8% of the products ordered in the test period. ALS (100) finds 25.7%. Item-cf (100) finds 23.4%. Content (50) finds 10.7%. The customer’s recent products (up to 30) find 6.8%. All five together find 55.5%.
Three things stand out:
- Trending finds the most. It is one list for everyone, and it finds 37.8%. In late June, many customers buy the same in-season bestsellers.
- The personal sources add a lot. Together with trending, they raise the share from 37.8% to 55.5%.
- Almost half is lost. 44.5% of the ordered products are not among the 202 candidates. No ranker can find them. Many are products a customer never touched, in a category they never visited.
The ceiling and the real scores
Section titled “The ceiling and the real scores”The ceiling counts a candidate at any of about 200 places. A home page shows 10. So the real recall@10 is far below the ceiling:
Text description of the chart
A column chart with four columns. Recall@10 on the test period: trending 0.156, the hybrid blend 0.167, the two-stage ranker 0.234. The fourth column is the candidate ceiling, 0.555: the share of ordered products found anywhere among the candidates.
The two-stage ranker gets 0.234 (95% interval 0.217 to 0.251) on Larkfield’s test period. That is the best score in the course so far. The next lesson explains how. But the gap to 0.555 is large. A better ranker can close part of it. A better first stage can raise the ceiling itself.
Why not more candidates?
Section titled “Why not more candidates?”More candidates raise the ceiling, but they cost something:
- Time. The ranker must describe and score every candidate.
- Noise. Place 300 of a list is rarely a good product. The ranker must push down more weak candidates.
- Memory. Every candidate row holds 26 numbers.
We did not measure other source sizes in this course. The sizes 100, 100, 100, 50 and 30 are a reasonable start, not a tuned choice. To change them, measure candidate recall on the validation period first.
When to use it, when not, and what it costs
Section titled “When to use it, when not, and what it costs”Use two stages when the catalogue is large and you have several signals that see different things. It gives one place to combine them: the ranker. It also lets each source stay simple and fast.
Do not use it when the catalogue is small. A shop with 200 products can score every product with the careful model. Do not start with it either: build the baselines first, as this course did. Without them, you cannot tell what the second stage adds.
Cost: more parts to keep fresh. Larkfield now builds item-cf, ALS and content, then trains the ranker. On our laptop, building the sources and training the ranker took about 12 seconds on one thread (your time will differ). Each source can fail on its own, so each needs a check.
Guided practice: measure the candidates
Section titled “Guided practice: measure the candidates”-
Open the notebook from the box Do this lesson in a notebook at the top of this page. Run the setup cells.
-
Run section 1. It builds two simple sources: the 100 trending products, and each customer’s products of the last 30 days.
-
Run section 2. The notebook’s simpler sources find a little more than the project’s: trending 37.9% and recent 6.9%.
-
In section 3, predict: does the union of both sources beat the best one alone, and by how much? Then compute the share. The check prints
Check passed: r_both is 0.418. -
In a terminal, go to your
larkfield-recsfolder and activate its virtual environment. Copy the new requirements file, and install LightGBM:Terminal window S=../nextia-learning-labs/C37/snapshots/end-of-m07cp $S/requirements.txt .pip install -r requirements.txtTerminal window $S = "..\nextia-learning-labs\C37\snapshots\end-of-m07"Copy-Item "$S\requirements.txt" .pip install -r requirements.txtThe file adds one line,
lightgbm==4.7.0. On macOS, LightGBM also needsbrew install libomp. -
Copy the two-stage method and the
candidatescommand:Terminal window cp $S/larkfield_recs/methods/twostage.py larkfield_recs/methods/cp $S/larkfield_recs/commands/candidates.py larkfield_recs/commands/Terminal window Copy-Item "$S\larkfield_recs\methods\twostage.py" larkfield_recs\methods\Copy-Item "$S\larkfield_recs\commands\candidates.py" larkfield_recs\commands\ -
Check that the project knows the new method:
Terminal window recs evaluate --listThe list has a new line:
two-stage candidates from 5 sources, then a LightGBM LambdaMART ranker. -
Measure the candidates on a fixed sample of 1,000 test customers:
Terminal window recs candidates --quick1,000 customers, 202 candidates per customer on averagesource size ordered products foundtrending 100 38.8%item_cf 100 24.6%als 100 26.0%content 50 10.9%recent 30 6.1%all 56.9% <- no ranker can find more than thisThe command also trains the ranker of the next lesson. It took about 14 seconds on our laptop.
Show the answer to step 4
r_both = found(lambda c: trending100 | recent_by_customer.get(c, set()))The | operator joins two sets. Each product counts once, so the overlap counts once. Together the two sources find 0.418: more than trending alone (0.379), but less than 0.379 + 0.069. Some recent products are also trending.
Your turn: the whole test period
Section titled “Your turn: the whole test period”The quick sample has 1,000 customers. Before you run it, predict: on all 2,118 test customers, will the share of all five sources be higher or lower than 56.9%? Then run recs candidates without --quick.
Show the answer
2,118 customers, 202 candidates per customer on averagesource size ordered products foundtrending 100 37.8%item_cf 100 23.4%als 100 25.7%content 50 10.7%recent 30 6.8%all 55.5% <- no ranker can find more than thisIt is a little lower: 55.5% against 56.9%. A sample of 1,000 customers is close, but not the same. Every source moves by about one point, and the order of the sources stays the same. Quote the full run when you report a number. Use --quick while you try things.
Common mistakes
Section titled “Common mistakes”Common mistake
Tuning the ranker when the candidates are the limit
Symptom: Tomás spends a week on the ranker’s settings. Recall@10 hardly moves.
Cause: 44.5% of the ordered products are not among the candidates. The ranker cannot show them. A tuned ranker moves products within the candidates only.
Diagnose: run recs candidates. Compare the ceiling (55.5%) with recall@10 (0.234). Then look at the ordered products that no source found. Which kind of source would find them?
Fix: improve the step that limits you. If the ceiling is low, add or enlarge a source. If the ceiling is high and recall@10 is low, work on the ranker.
Common mistake
A candidate source that sees the test period
Symptom: Priya adds a new source, “products the customer viewed this month”. The candidate recall jumps to over 80%.
Cause: “this month” includes the test period. The source looks at views during the two weeks it is tested on. Customers often view a product just before they order it. This is leakage, as in Split by time.
Diagnose: check the cut-off date of every source. In the project, all five sources use only events before the period.
Fix: build every source from the events before the period only. A jump that looks too good usually is.
Knowledge check
Section titled “Knowledge check”Check your understanding
Source A finds 30% of the ordered products, and source B finds 20%. Together, can they find 60%?
Show the answer
No. Together they find at most 30% + 20% = 50%, and only if they never find the same product. Usually they overlap, so the union is less. At Larkfield, the five shares add up to 104.4%, but together the sources find 55.5%.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
Why does the candidate step limit the ranker, but the ranker does not limit the candidate step?
Show the answer
The ranker only sees the candidates. It can move a candidate up or down, but it cannot add a product that no source found. So the share of ordered products among the candidates is the highest recall any ranker can reach. The candidate step does not depend on the ranker at all.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
Larkfield's candidates contain 55.5% of the ordered products, and the ranker's recall@10 is 0.234. Name one way to raise each number.
Show the answer
To raise the ceiling (55.5%), add a source that finds other products, or make a source’s list longer. Check candidate recall on the validation period. To raise recall@10 (0.234), give the ranker better features or better training data. Then more of the found products reach the top 10. The next lesson does the second.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Recap
- A two-stage recommender first retrieves a few hundred candidates from fast sources, then ranks them with a careful model. It is about 15 times less work than scoring every product.
- Candidate recall is the share of ordered products among the candidates. It is the ceiling for any ranker.
- Larkfield’s five sources (trending, item-cf, ALS, content, recent) give about 202 candidates per customer. Together they find 55.5% of the ordered products. Trending alone finds 37.8%.
- The shares overlap. They add up to 104.5%, but the union is 55.5%.
- For C106777, the ceiling is 2 of 3: no source found the wall lantern.
We have candidates from five sources. Now we must put the best ones first. Next, we describe each candidate with features and train a LambdaMART ranker with LightGBM on the next orders.
Next: A learned ranker