Skip to content
Français

Why two stages

  • Lesson 20
  • 30 min
  • Module 7: Two stages: candidates and ranking

Where we are

Our hybrids combine methods by hand. Large shops use a fixed pattern for this. In this lesson, we take candidates from several sources and see why the candidate step limits everything that comes after it.

Plays from YouTube. Nothing loads from YouTube until you press play. Privacy

What you will be able to do

See why large shops recommend in two stages. Several fast sources find a few hundred candidate products, then a careful model ranks them. Measure what each of Larkfield’s five sources finds, and the limit that the candidates put on every ranker after them.

NotebookDo this lesson in a notebookColab · Kaggle · GitHub

The notebook builds two simple candidate sources: the 100 trending products, and the products each customer touched in the last 30 days. It measures the share of ordered products that each source finds. You compute the share that both sources find together. Its setup cells download about 7 MB of practice data. About 15 minutes.

Colab and Kaggle open your own copy. They need a free Google or Kaggle account. Stuck? See the solution notebook.

Before you startWhat to know and to have readyAbout 30 minutes

What you need to know

What you need on your computer (only for the project steps; the notebook needs none of it)

  • Your larkfield-recs project at the end of Module 6, with its virtual environment active. recs check prints Setup check passed.
  • The practice files in ../nextia-learning-labs, beside the project folder.
  • On macOS, Homebrew’s OpenMP library for LightGBM: brew install libomp.

Tested on macOS with Python 3.12 and LightGBM 4.7.0. The Windows (PowerShell) and Linux commands were not run.

A customer asks a shop assistant: “What would I like?” The shop sells 3,000 products. The assistant does not think hard about each of the 3,000. That would take all day.

She works in two steps. First, she walks quickly past a few shelves and picks about twenty likely products. Some are bestsellers at the front. Some go with what the customer bought last time. Some look like what the customer looked at today. This step is fast and rough.

Then she looks carefully at the twenty. She thinks about the customer’s taste, the season and the price. She puts the best five on the counter.

A two-stage recommender works the same way. This lesson is about the first step, and the limit it sets.

The problem: one method cannot do both jobs

Section titled “The problem: one method cannot do both jobs”

In Modules 4 to 6, each method did everything at once. It scored all 2,994 products for each customer and kept the top 10. That has two problems.

A careful model is too slow for every product

Section titled “A careful model is too slow for every product”

A careful model looks at many facts about each customer and product pair. The next lesson uses 26 facts per pair. Larkfield has 40,000 customers and 2,994 products. That is 40,000 × 2,994 = 119.8 million pairs to describe and score every night.

With about 200 candidates per customer, it is 40,000 × 202 = 8.1 million pairs. That is about 15 times less work. In Batch or real time, the project ranks one customer’s 200 candidates in about 10 milliseconds. Fifteen times more is about 150 milliseconds: more than Amira’s budget of 100 milliseconds per request (an estimate, not a measurement).

Each fast method misses different products

Section titled “Each fast method misses different products”

The fast methods each see a different part of the truth:

  • Trending finds the in-season bestsellers, for everyone.
  • Item-cf finds products bought with the customer’s past products.
  • ALS finds products that match the customer’s hidden tastes.
  • Content finds products with a similar text, also new products with no history.
  • The customer’s own recent products find what they looked at and did not buy yet.

In Hybrids, we added three of these scores with fixed weights. That got 0.167, and 0.193 with weights tuned on the validation period. But fixed weights cannot say “for this customer, trust ALS more”. We need a better way to combine them.

A two-stage recommender splits the work:

  1. Candidate generation (also called retrieval). Several fast sources each give a short list. Together they make a few hundred candidates per customer.
  2. Ranking. A careful model scores only the candidates and orders them. Then the business rules remove products the shop must not show.

This pattern is common in large shops and streaming services. YouTube’s engineers described it in 2016 for videos. The first stage must be fast and must not miss much. The second stage must be accurate on a short list.

A flow from left to right. One customer's history goes to five candidate sources: trending 100 products, item-cf 100, ALS 100, content 50, and recent products of the last 30 days 30. Their lists join into about 202 different candidates, out of 2,994 products. A ranker with 26 features per candidate, built with LightGBM, orders them. Business rules for stock, age, already-bought products, health products and variety remove some. The result is 10 suggestions.
Larkfield's two-stage recommender. Five fast sources make about 202 candidates per customer; a careful ranker orders them; rules clean the list.
Text description of the diagram

One customer’s history goes to five candidate sources. They are trending (100 products), item-cf (100), ALS (100), content (50) and the customer’s recent products of 30 days (up to 30). Together they give about 202 different candidates out of 2,994 products, because the lists overlap. A ranker scores each candidate with 26 features (LightGBM). Business rules remove products the shop must not show. These are out of stock, age-restricted, already bought or sensitive, or too many of one kind. Ten suggestions are left.

The ranker can only put first a product that a source found. A product that no source found is lost, whatever the ranker does.

So we measure the first stage alone. Candidate recall is the share of the products ordered in the test period that are among a customer’s candidates, at any place. It is the ceiling of the second stage. No ranker can find more.

Customer C106777 is a loyal customer. You met this customer in Ranking metrics. In the test period, the customer ordered three products. The five sources made 273 candidates for this customer. The table shows the place of each ordered product in each source’s list (“–” = not in that list).

Ordered product trending item-cf ALS content recent A candidate?
Navy Nitrile-Grip Garden Gloves #2 #48 #2 – – yes
Bramble Terracotta Mosaic Side Table #4 – #39 – – yes
Hartley Blue Wall Lantern – – – – – no

The gloves are in three lists, and the side table in two. The wall lantern is in no list. So, for this customer, the ceiling is 2 of 3 = 0.667. Even a perfect ranker could not show the lantern.

Look at the side table too. Trending has it at place 4, and ALS at place 39. Should it come first? That is the ranker’s question, in the next lesson.

Each source finds a share of all the ordered products. If we add the five shares, we get 37.8% + 25.7% + 23.4% + 10.7% + 6.8% = 104.5%. That is more than all the products. The true share of all sources together is 55.5%.

The reason is overlap, as with the gloves above. Many ordered products are in two or three lists. A new source helps only with the products that the other sources miss.

Now all 2,118 test customers. The project’s recs candidates command measures each source and all of them together on Larkfield’s test period.

Share of the products ordered in the test period that each candidate source finds, at any place. All five together find 55.5%: the ceiling for any ranker. 2,118 customers.
Text description of the chart

A column chart with six columns. Trending (100 products) finds 37.8% of the products ordered in the test period. ALS (100) finds 25.7%. Item-cf (100) finds 23.4%. Content (50) finds 10.7%. The customer’s recent products (up to 30) find 6.8%. All five together find 55.5%.

Three things stand out:

  • Trending finds the most. It is one list for everyone, and it finds 37.8%. In late June, many customers buy the same in-season bestsellers.
  • The personal sources add a lot. Together with trending, they raise the share from 37.8% to 55.5%.
  • Almost half is lost. 44.5% of the ordered products are not among the 202 candidates. No ranker can find them. Many are products a customer never touched, in a category they never visited.

The ceiling counts a candidate at any of about 200 places. A home page shows 10. So the real recall@10 is far below the ceiling:

Recall@10 on Larkfield's test period of trending, the hybrid blend and the two-stage ranker, beside the candidate ceiling (any place among about 202 candidates).
Text description of the chart

A column chart with four columns. Recall@10 on the test period: trending 0.156, the hybrid blend 0.167, the two-stage ranker 0.234. The fourth column is the candidate ceiling, 0.555: the share of ordered products found anywhere among the candidates.

The two-stage ranker gets 0.234 (95% interval 0.217 to 0.251) on Larkfield’s test period. That is the best score in the course so far. The next lesson explains how. But the gap to 0.555 is large. A better ranker can close part of it. A better first stage can raise the ceiling itself.

More candidates raise the ceiling, but they cost something:

  • Time. The ranker must describe and score every candidate.
  • Noise. Place 300 of a list is rarely a good product. The ranker must push down more weak candidates.
  • Memory. Every candidate row holds 26 numbers.

We did not measure other source sizes in this course. The sizes 100, 100, 100, 50 and 30 are a reasonable start, not a tuned choice. To change them, measure candidate recall on the validation period first.

When to use it, when not, and what it costs

Section titled “When to use it, when not, and what it costs”

Use two stages when the catalogue is large and you have several signals that see different things. It gives one place to combine them: the ranker. It also lets each source stay simple and fast.

Do not use it when the catalogue is small. A shop with 200 products can score every product with the careful model. Do not start with it either: build the baselines first, as this course did. Without them, you cannot tell what the second stage adds.

Cost: more parts to keep fresh. Larkfield now builds item-cf, ALS and content, then trains the ranker. On our laptop, building the sources and training the ranker took about 12 seconds on one thread (your time will differ). Each source can fail on its own, so each needs a check.

  1. Open the notebook from the box Do this lesson in a notebook at the top of this page. Run the setup cells.

  2. Run section 1. It builds two simple sources: the 100 trending products, and each customer’s products of the last 30 days.

  3. Run section 2. The notebook’s simpler sources find a little more than the project’s: trending 37.9% and recent 6.9%.

  4. In section 3, predict: does the union of both sources beat the best one alone, and by how much? Then compute the share. The check prints Check passed: r_both is 0.418.

  5. In a terminal, go to your larkfield-recs folder and activate its virtual environment. Copy the new requirements file, and install LightGBM:

    Terminal window
    S=../nextia-learning-labs/C37/snapshots/end-of-m07
    cp $S/requirements.txt .
    pip install -r requirements.txt

    The file adds one line, lightgbm==4.7.0. On macOS, LightGBM also needs brew install libomp.

  6. Copy the two-stage method and the candidates command:

    Terminal window
    cp $S/larkfield_recs/methods/twostage.py larkfield_recs/methods/
    cp $S/larkfield_recs/commands/candidates.py larkfield_recs/commands/
  7. Check that the project knows the new method:

    Terminal window
    recs evaluate --list

    The list has a new line: two-stage candidates from 5 sources, then a LightGBM LambdaMART ranker.

  8. Measure the candidates on a fixed sample of 1,000 test customers:

    Terminal window
    recs candidates --quick
    1,000 customers, 202 candidates per customer on average
    source size ordered products found
    trending 100 38.8%
    item_cf 100 24.6%
    als 100 26.0%
    content 50 10.9%
    recent 30 6.1%
    all 56.9% <- no ranker can find more than this

    The command also trains the ranker of the next lesson. It took about 14 seconds on our laptop.

Show the answer to step 4
r_both = found(lambda c: trending100 | recent_by_customer.get(c, set()))

The | operator joins two sets. Each product counts once, so the overlap counts once. Together the two sources find 0.418: more than trending alone (0.379), but less than 0.379 + 0.069. Some recent products are also trending.

The quick sample has 1,000 customers. Before you run it, predict: on all 2,118 test customers, will the share of all five sources be higher or lower than 56.9%? Then run recs candidates without --quick.

Show the answer
2,118 customers, 202 candidates per customer on average
source size ordered products found
trending 100 37.8%
item_cf 100 23.4%
als 100 25.7%
content 50 10.7%
recent 30 6.8%
all 55.5% <- no ranker can find more than this

It is a little lower: 55.5% against 56.9%. A sample of 1,000 customers is close, but not the same. Every source moves by about one point, and the order of the sources stays the same. Quote the full run when you report a number. Use --quick while you try things.

Common mistake

Tuning the ranker when the candidates are the limit

Symptom: Tomás spends a week on the ranker’s settings. Recall@10 hardly moves.

Cause: 44.5% of the ordered products are not among the candidates. The ranker cannot show them. A tuned ranker moves products within the candidates only.

Diagnose: run recs candidates. Compare the ceiling (55.5%) with recall@10 (0.234). Then look at the ordered products that no source found. Which kind of source would find them?

Fix: improve the step that limits you. If the ceiling is low, add or enlarge a source. If the ceiling is high and recall@10 is low, work on the ranker.

Common mistake

A candidate source that sees the test period

Symptom: Priya adds a new source, “products the customer viewed this month”. The candidate recall jumps to over 80%.

Cause: “this month” includes the test period. The source looks at views during the two weeks it is tested on. Customers often view a product just before they order it. This is leakage, as in Split by time.

Diagnose: check the cut-off date of every source. In the project, all five sources use only events before the period.

Fix: build every source from the events before the period only. A jump that looks too good usually is.

Check your understanding

Source A finds 30% of the ordered products, and source B finds 20%. Together, can they find 60%?

Show the answer

No. Together they find at most 30% + 20% = 50%, and only if they never find the same product. Usually they overlap, so the union is less. At Larkfield, the five shares add up to 104.4%, but together the sources find 55.5%.

Check your understanding

Why does the candidate step limit the ranker, but the ranker does not limit the candidate step?

Show the answer

The ranker only sees the candidates. It can move a candidate up or down, but it cannot add a product that no source found. So the share of ordered products among the candidates is the highest recall any ranker can reach. The candidate step does not depend on the ranker at all.

Check your understanding

Larkfield's candidates contain 55.5% of the ordered products, and the ranker's recall@10 is 0.234. Name one way to raise each number.

Show the answer

To raise the ceiling (55.5%), add a source that finds other products, or make a source’s list longer. Check candidate recall on the validation period. To raise recall@10 (0.234), give the ranker better features or better training data. Then more of the found products reach the top 10. The next lesson does the second.

Recap

  • A two-stage recommender first retrieves a few hundred candidates from fast sources, then ranks them with a careful model. It is about 15 times less work than scoring every product.
  • Candidate recall is the share of ordered products among the candidates. It is the ceiling for any ranker.
  • Larkfield’s five sources (trending, item-cf, ALS, content, recent) give about 202 candidates per customer. Together they find 55.5% of the ordered products. Trending alone finds 37.8%.
  • The shares overlap. They add up to 104.5%, but the union is 55.5%.
  • For C106777, the ceiling is 2 of 3: no source found the wall lantern.

We have candidates from five sources. Now we must put the best ones first. Next, we describe each candidate with features and train a LambdaMART ranker with LightGBM on the next orders.

Next: A learned ranker