Skip to content
Français

Offline wins are not online wins

  • Lesson 24
  • 35 min
  • Module 8: From offline scores to the real shop

Where we are

Our two-stage system wins the offline comparison by a wide margin. But offline data shows only what the old shop chose to show, and how customers reacted to it. In this lesson, we see position bias and logged-data bias, and why an offline win may not be an online win.

Plays from YouTube. Nothing loads from YouTube until you press play. Privacy

What you will be able to do

Understand why the method with the best offline score is not always the best for the shop. Larkfield’s data and its simulation show three reasons. They are position bias, a log shaped by the old site, and orders that would happen anyway.

NotebookDo this lesson in a notebookColab · Kaggle · GitHub

The notebook reads the simulated online test of three home pages, one row per customer. You separate revenue from suggestions and added revenue, compute the share that customers would have spent anyway, and check a difference with an interval. Its setup cells download less than 1 MB. About 15 minutes.

Colab and Kaggle open your own copy. They need a free Google or Kaggle account. Stuck? See the solution notebook.

Before you startWhat to know and to have readyAbout 35 minutes

What you need to know

What you need on your computer

  • Nothing for the lesson. The notebook runs in Colab, Kaggle or your own Jupyter. This lesson does not change the project.

The notebook was tested on macOS with Python 3.12. Colab and Kaggle were not tested.

Think of a small shop on a rainy morning. The assistant puts the umbrellas next to the door. By lunch, the shop has sold 30 umbrellas. “The new place works,” she says.

Does it? Many of those customers came in for an umbrella. They would have found it at the back of the shop too. And the scarves that she moved to the back sold less. That does not mean that people dislike scarves. They just did not see them.

The till receipts show what sold. They do not show why. They do not show what would have sold with another display. A recommender learns from the same kind of receipts: the shop log.

The problem: the offline winner is not clearly better online

Section titled “The problem: the offline winner is not clearly better online”

In Module 7, the two-stage system won the comparison table by a wide margin. With business rules, its recall@10 on the test period is 0.215. Trending, the list of last month’s bestsellers, has 0.151. ALS has 0.136.

Omar does not pay for recall. He pays for added revenue per visit: money from suggestions that customers would not have spent anyway. The course tested the three lists in an online test. The test is simulated: it runs on our model of customer behaviour, never on real people. Each of 5,078 customers who visited in the test period saw one of the three lists, at random, for two weeks.

The same three lists, offline and in the simulated online test. Offline, the intervals of two-stage and trending do not touch. Online, they overlap almost completely.
Text description of the chart

The figure has two panels with the same three lists. The first panel shows offline recall@10 with business rules on the test period, with 95% intervals. Two-stage 0.215 (0.199 to 0.231), trending 0.151 (0.137 to 0.165), ALS 0.136 (0.123 to 0.150). Two-stage’s interval is far to the right of the others. The second panel shows simulated added revenue per visit, with 95% intervals. Two-stage 6.82 dollars (5.45 to 8.25), trending 6.64 dollars (5.33 to 8.01), ALS 3.18 dollars (2.25 to 4.10). The intervals of two-stage and trending overlap almost completely. ALS is clearly behind.

Offline, two-stage finds 42% more of the ordered products than trending. Online, it adds +0.19 dollars per visit more than trending, with a 95% interval from −1.75 to +2.05. The interval includes 0, so the online result is not clearly better. It is not a loss either. ALS is the list that is clearly worse online.

Tomás asks the natural question: “How can the best model not win?” There are three reasons. Each one is a way in which the log differs from the live shop.

Reason 1: people look at the top of a list

Section titled “Reason 1: people look at the top of a list”

Most people read a list from the top and stop early. A product in place 1 is seen by almost everybody who opens the page. A product in place 8 is seen by few. This is position bias: the place of a product changes how often it is seen, so it changes how often it is clicked.

The course’s simulation uses a simple rule: a customer looks at place k of the home page with a chance of 1 / k. That is an assumption of the simulation, not a measurement. Real shops measure their own curve.

The chance of looking at each place of a home page list of 10, as the simulation assumes (1 / place). In an email, the simulation uses 0.3 / place.
Text description of the chart

A bar chart with one bar for each place of a list of 10. Place 1: 100%. Place 2: 50%. Place 3: 33%. Place 4: 25%. Place 5: 20%. Place 6: 17%. Place 7: 14%. Place 8: 13%. Place 9: 11%. Place 10: 10%. The bars fall fast at first, then slowly. These are assumptions of the course’s simulation.

Here is a constructed example. Two products are equally good: when a customer looks at either one, the chance of a click is 10%. The old home page puts lantern A in place 1 and lantern B in place 5. 1,000 customers open the page.

Product Place Customers who look (1,000 × 1 / place) Clicks (10% of those who look) Clicks in the log
Lantern A 1 1,000 100 100
Lantern B 5 200 20 20

The log says that lantern A is five times better. In truth, the two are equal. The only difference is the place. A model that learns from these clicks learns the old page’s order, not the customers’ taste.

Reason 2: the log shows what the old site chose to show

Section titled “Reason 2: the log shows what the old site chose to show”

The second reason is bigger. A customer can only click, or buy, what the shop showed them. So the log is a record of the old site’s choices and of the customers’ reactions to them.

Larkfield’s synthetic log was made this way on purpose. In our behaviour model, about one regular visit in ten starts on the old home page. That page shows the 40 products with the highest appeal for the season, and customers look mostly at its first places. Every other product is found only when a customer browses a category.

A flow from left to right. The old home page shows 40 products, chosen by appeal and season. An arrow leads to: customers look mostly at the first places. An arrow leads to: the shop log, with views, baskets and orders. A dotted arrow from every other product, found only by browsing or not at all, also leads to the log. From the log, an arrow leads to the offline test, which asks: was the ordered product in our list? Separately, a new list leads to the online test, which asks: did our list change what the customer did?
How the shop log is made. The old site chose what customers saw, so the log is full of its choices. An offline test can only ask about the log; an online test asks about the new list.
Text description of the diagram

The picture shows how the shop log is made. The old home page shows 40 products, chosen by appeal and season. Customers look mostly at its first places. Their views, baskets and orders go into the shop log. Every other product is found only when customers browse, or not at all, so it reaches the log less often. From the log, the offline test asks one question: was the ordered product in our list? The online test asks a different question about a new list: did our list change what the customer did?

This is logged-data bias (also called exposure bias): the data only contains reactions to what was shown. It has two effects:

  • Products that the old site showed look better than they are. They had more chances to be seen and bought.
  • Products that the old site never showed have almost no data. No model can learn that people would like them. You saw a version of this with new products in Cold start for new products and new customers.

Reason 3: offline counts orders that happen anyway

Section titled “Reason 3: offline counts orders that happen anyway”

The third reason is about what an offline hit means. Recall@10 counts a hit when the list contains a product that the customer ordered in the test period. But the customer ordered that product without our list. The log was made by the old site. So a hit says “we predicted this order”. It does not say “our list caused this order”.

Online, the difference matters. An order that the customer would have made anyway adds no money. In the simulation, Larkfield can count those orders, because the behaviour model knows which orders were in the log already.

Here is an honest surprise. In this simulation, the effect is small. Only 3.3% of two-stage’s revenue from suggestions would have been spent anyway. For trending, it is 0.2%. So reason 3 does not explain much of the gap here. In a real shop, it can be much larger. That is why Omar’s measure counts added revenue, not revenue from suggestions.

So where does two-stage’s offline lead go? Split added revenue per visit into four steps that you can see in the online test. Multiply them, and you get the measure:

added revenue per visit = clicks per visit × orders per click × dollars per order × (1 − share spent anyway)

The clicks count both home page clicks and email clicks. Every number below is from the simulated test.

List Clicks per visit Orders per click Dollars per order Spent anyway Added revenue per visit
Trending 0.174 0.254 150.40 0.2% 6.64
Two-stage 0.154 0.302 152.17 3.3% 6.82
ALS 0.086 0.277 138.70 3.9% 3.18

Check one row by hand. For trending: 0.174 × 0.254 × 150.40 × (1 − 0.002) = 6.63. The table says 6.64: the small gap comes from rounding each step.

Now read the columns:

  • Two-stage gets fewer clicks than trending (0.154 against 0.174 per visit). In the simulation, a click depends on how attractive a product looks when the customer sees it. Trending shows in-season bestsellers, and they attract clicks.
  • Two-stage turns more clicks into orders (0.302 against 0.254). Its products fit the customer better.
  • The two effects almost cancel. The result is +0.19 dollars per visit, inside the noise.
  • ALS gets half the clicks of trending (0.086 per visit), so it adds about half the money.

Offline recall sees none of these steps. It only asks whether the ordered product was somewhere in the list. That is why Priya says: “Offline scores choose which methods are worth an online test. They do not replace it.”

Could we correct the log instead of running an online test? There is a family of methods for this, called counterfactual evaluation. “Counterfactual” means “what would have happened if we had shown something else”.

The simplest idea is to give rare events more weight. Take a click at place 5, where only 1 in 5 customers looks. Count it as 5 clicks. The chance of being seen is called the propensity, so this is inverse propensity weighting: divide each click by its propensity. In the lantern example, lantern A has 100 clicks ÷ 1 = 100. Lantern B has 20 clicks ÷ 0.2 = 100. After the correction, the two lanterns are equal again, as they should be.

Product Place Propensity (chance of being seen) Clicks in the log Clicks ÷ propensity
Lantern A 1 1.0 100 100
Lantern B 5 0.2 20 100

The idea has three costs:

  • You must know the propensity of every shown product. The old site must log what it showed, and where.
  • A product that the old site never showed has a propensity of 0. You cannot divide by 0, so no correction can tell you anything about it.
  • Large weights make the estimate noisy. One lucky click at place 50 can count as 50 clicks.

The course does not build counterfactual evaluation. Larkfield answers the question with an online test, in the next lesson.

Guided practice: the simulated test in the notebook

Section titled “Guided practice: the simulated test in the notebook”

The notebook of this lesson uses the same file as the project: data/online/ab_test.csv, one row per customer of the simulated test.

  1. Open the notebook from the box at the top of this lesson, in Colab, Kaggle or your own Jupyter. Run the setup cells.

  2. Run section 1. It reads 5,078 rows. Each row has the customer’s list (the arm), visits, clicks, orders, revenue from suggestions, added revenue, returns, emails and unsubscribes.

  3. Run section 2. The table shows revenue from suggestions and added revenue per visit:

    customers visits clicks from_suggestions added \
    arm
    als 1724 2723 166 9015.35 8665.37
    trending 1604 2511 308 16694.89 16663.90
    two-stage 1750 2695 306 19020.75 18391.79
    from_suggestions_per_visit added_per_visit
    arm
    als 3.311 3.182
    trending 6.649 6.636
    two-stage 7.058 6.824

    The table is wide, so pandas prints it in two parts. The \ at the end of the first line means “continued below”.

  4. In section 3, compute the share of two-stage’s revenue from suggestions that customers would have spent anyway: 1 − added ÷ from suggestions. The check prints Check passed: anyway is 0.033.

  5. Run section 4. It resamples the customers of each list 500 times:

    two-stage - trending: +0.19 dollars per visit (95% interval -1.60 to +2.24)

The project’s recs online, in the next lesson, gives −1.75 to +2.05 for the same difference. It resamples 1,000 times with another random stream. The edges of a bootstrap interval move a little from run to run. The conclusion does not move: the interval includes 0.

A dashboard shows revenue from suggestions, not added revenue. Before you compute anything, predict: on that dashboard, is two-stage ahead of trending by more or by less than +0.19 dollars per visit? Then use the table of step 3.

Show the answer

By more. Revenue from suggestions per visit is 7.058 for two-stage and 6.649 for trending: a gap of +0.41 dollars. Added revenue is 6.824 against 6.636: a gap of +0.19 dollars. The dashboard number is bigger because it also counts the 3.3% of two-stage’s revenue that customers would have spent anyway. For trending, that share is only 0.2%. A dashboard that counts every order after a click flatters the list that suggests what people buy anyway.

Offline evaluation is still the right first step. It is cheap, fast and repeatable. It costs no customer anything, and it can compare twenty methods in an afternoon. Use it to:

  • reject methods that are clearly worse, such as random lists or a stale winter coat;
  • choose a few candidates for an online test;
  • check a change before release, as a regression test.

Do not use it to decide the final winner. This matters most when the differences are small, or when the new list shows very different products from the old one. The more a new list differs from what the old site showed, the less the log can say about it.

Common mistake

'The model with the best offline score wins'

Symptom: Tomás plans to release two-stage because its recall@10 is 0.215 against 0.151.

Cause: offline recall asks whether the list contained products that customers ordered without it. It does not see position bias, the old site’s choices, clicks or added money.

Fix: treat the offline table as a shortlist. Release only after an online test of the business measure, with its interval. In the simulation, two-stage is +0.19 dollars per visit (−1.75 to +2.05): not clearly better.

Common mistake

Revenue from suggestions read as added revenue

Symptom: a report says that two-stage earns 0.41 dollars more per visit than trending.

Cause: the report uses revenue from suggestions. It counts orders that customers would have made anyway.

Fix: report added revenue per visit, the business measure. In the simulation, the gap is +0.19 dollars, and its interval includes 0.

Check your understanding

Explain to Tomás why the offline winner, two-stage, is not clearly better online than trending. Use two numbers from the simulated test.

Show the answer

Offline recall only asks whether the list contained the products that customers ordered without it. Online, a suggestion must first be seen and clicked, then ordered, and only orders that would not have happened anyway count. In the simulation, two-stage gets fewer clicks per visit than trending (0.154 against 0.174), because trending’s in-season bestsellers attract clicks. It turns more clicks into orders (0.302 against 0.254). The effects almost cancel: +0.19 dollars per visit, with a 95% interval from −1.75 to +2.05, which includes 0.

Check your understanding

The old home page showed a garden bench in place 1 and a bird bath in place 6. The log has 60 clicks for the bench and 10 for the bird bath. Using the simulation's rule (a customer looks at place k with a chance of 1 / k), which product do customers like more when they see it?

Show the answer

Divide each click count by the chance of being seen. The bench: 60 ÷ 1 = 60. The bird bath: 10 ÷ (1/6) = 60. When customers see them, they like the two products equally. The difference in the log comes from the place, not from taste: that is position bias. This correction is inverse propensity weighting, and it needs the logged place of every shown product.

Check your understanding

Why can no correction of the log tell you how customers would react to a product that the old site never showed and that nobody found by browsing?

Show the answer

The log only has reactions to what customers saw. A product that nobody saw has no views, no clicks and no orders. Its chance of being seen, the propensity, is 0, and inverse propensity weighting cannot divide by 0. Only showing it, for example in an online test or with exploration, can produce data about it.

Recap

  • Offline, two-stage has recall@10 0.215 against trending’s 0.151. In the simulated online test, it adds +0.19 dollars per visit (−1.75 to +2.05): not clearly better. ALS is clearly worse (3.18 against 6.64 dollars).
  • Position bias: places at the top are seen more, so they get more clicks. The simulation assumes a chance of 1 / place.
  • Logged-data bias: the log only has reactions to what the old site showed. Products it showed look better; products it never showed have no data.
  • Offline hits are orders that happened without the list. In this simulation, only 3.3% of two-stage’s revenue would have been spent anyway, so the main reason here is clicks: trending gets more of them.
  • Counterfactual evaluation reweights logged events by their propensity. It needs logged propensities, fails for products never shown, and is noisy.

You now know why the offline table is a shortlist. Next, you read the full simulated online test, with a main measure and guardrails, and decide which list to release.

Next: Online tests and guardrails