Online tests and guardrails
- Lesson 25
- 40 min
- Module 8: From offline scores to the real shop
Where we are
In the last lesson, we saw why offline scores can mislead: the log shows what the old site chose to show. An online test measures what customers do with the new list. In this lesson, we read simulated A/B results of three models, with a main measure and guardrails, and choose which model to release.
Lesson 25 of 29 · Recommendation and Personalisation: Show Each Person What MattersOnline tests and guardrails
1. The recommendation problem
2. Data, implicit feedback and an honest split
3. Measure recommendations offline
4. Neighbours and collaborative filtering
5. Matrix factorisation for implicit feedback
6. Content, hybrids and cold start
7. Two stages: candidates and ranking
8. From offline scores to the real shop
9. Run it in production
What you will be able to do
Read an online test the way Omar does. Decide one main measure in advance, a few guardrails that must not get clearly worse, and a decision rule. Then decide which of three home page lists Larkfield should release.
NotebookDo this lesson in a notebook
The notebook reads the simulated A/B test and computes the main measure and two guardrails with their intervals. You count the events behind each guardrail and put a price on them under the course’s assumptions. Then you estimate how much longer the test would need to run. Its setup cells download less than 1 MB. About 15 minutes.
Colab and Kaggle open your own copy. They need a free Google or Kaggle account. Stuck? See the solution notebook.
Before you startWhat to know and to have ready
What you need to know
- Added revenue per visit and the three reasons why offline and online disagree, from Offline wins are not online wins.
- What a 95% interval is, from Ranking metrics.
What you need on your computer (only for the project steps)
- Your
larkfield-recsproject at the end of Module 7, with its virtual environment active.recs checkprintsSetup check passed. - The practice files in
../nextia-learning-labs, beside the project folder. You copy one file from its snapshotend-of-m08.
Tested on macOS with Python 3.12. The Windows (PowerShell) commands were not run.
A café tries a new menu
Section titled “A café tries a new menu”Think of a café owner with a new menu. She is not sure that it is better. So for two weeks, she flips a coin for each new table. Heads, the table gets the new menu. Tails, it gets the old one.
Before she starts, she writes down one number that decides: money per table. She also writes down two things that must not get worse. Plates sent back to the kitchen, and the time that people wait for their food. A menu that earns more but makes people wait an hour is not a better menu.
After two weeks, she compares the two groups of tables. The coin made the two groups alike in every other way: the same days, the same weather, the same kinds of customers. So a difference between them comes from the menu, or from chance.
The problem: one number is not enough
Section titled “The problem: one number is not enough”Larkfield could test lists the same way, but with only one number. Here are two ways that fails:
- Choose by clicks. In What success means, a list of cheap deals got almost twice the clicks. It added less than a third of the money.
- Choose by money alone. A list that pushes impulse products can raise revenue this week. It can also double the returns, or make families unsubscribe from the email after one embarrassing suggestion.
The shop needs one number to decide, and a few numbers to protect it.
The idea: an A/B test with a main measure and guardrails
Section titled “The idea: an A/B test with a main measure and guardrails”An online test, or A/B test, splits customers at random into groups. Each group sees one version for the same period. Because the split is random, the groups are alike, so the difference in results comes from the versions or from chance. With three versions, people sometimes say A/B/C test; the idea is the same.
Three words describe how to read it:
- The main measure is the one number that decides. Choose it before the test starts, so that you cannot pick the measure that happens to look best. For Larkfield’s home page, Omar chose added revenue per visit.
- A guardrail is a measure that must not get clearly worse, even if the main measure improves. Larkfield has three. The return rate is returns per order. The unsubscribe rate is customers who unsubscribed per customer emailed. Speed is the time to answer one request.
- The decision rule says in advance what each result means: release, stop, or keep the current list.
Text description of the diagram
The picture shows Larkfield’s simulated A/B test. 5,078 customers who visited in two weeks are split at random into three groups. Trending has 1,604 customers, two-stage 1,750 and ALS 1,724. For each group, the test measures the main measure, added revenue per visit with its interval, and the guardrails: returns, unsubscribes and speed. Together they lead to one decision: release, stop, or test longer.
Omar wrote his decision rule before the test:
| Result | Decision |
|---|---|
| The main measure is clearly worse than trending | Stop this list. |
| The main measure is clearly better, and no guardrail is clearly worse | Release it. |
| Anything else | Keep the simpler list, or test longer if a small gain is worth the wait. |
“Clearly” has the same meaning as in the comparison table: the 95% interval of the difference does not include 0. Speed has a fixed limit instead: Amira’s budget of 100 milliseconds (ms) per request, an assumption of the course.
A tiny example with every value visible
Section titled “A tiny example with every value visible”Here is a constructed example. The café has 200 tables in each group.
| Menu | Tables | Money per table | Plates sent back | Mean wait |
|---|---|---|---|---|
| Old | 200 | 31 dollars | 6 of 600 plates (1.0%) | 12 minutes |
| New | 200 | 34 dollars (+3, interval +1 to +5) | 18 of 620 plates (2.9%) | 13 minutes |
The main measure is clearly better: the interval of +3 dollars does not include 0. But plates sent back went from 1.0% to 2.9%. If that guardrail is clearly worse, the rule says: do not release. The owner first finds out which dishes come back.
Larkfield’s simulated A/B test
Section titled “Larkfield’s simulated A/B test”The course ran this test on its behaviour model. It is simulated: no real customer saw these lists. The 5,078 customers are every customer who visited in the test period, not only the 2,118 who ordered. Each group saw one list on the home page at every visit, and in one email a week, for two weeks. All three lists used the business rules of Module 7.
Guided practice: read the test with recs online
Section titled “Guided practice: read the test with recs online”Your project at the end of Module 7 cannot read the online test yet. You copy one command from the end-of-module-8 snapshot. The test’s data, data/online/ab_test.csv, is already in your data folder.
-
In your
larkfield-recsfolder, with the virtual environment active, copy theonlinecommand:Terminal window S=../nextia-learning-labs/C37/snapshots/end-of-m08cp $S/larkfield_recs/commands/online.py larkfield_recs/commands/Terminal window $S = "..\nextia-learning-labs\C37\snapshots\end-of-m08"Copy-Item "$S\larkfield_recs\commands\online.py" larkfield_recs\commands\ -
Open
larkfield_recs/commands/online.py. It reads the file, computes each measure per group, and resamples customers 1,000 times for each interval. -
Read the test:
Terminal window recs onlineSIMULATED A/B test on the course's behaviour model (not real customers): 5,078 customers, each shown one model's lists for two weeksmodel customers visits clicks/visit from suggestions added 95% interval returns unsubscribed (dollars per visit)als 1,724 2,723 0.061 3.31 3.18 2.25 to 4.10 7.7% 0.75%two-stage 1,750 2,695 0.114 7.06 6.82 5.45 to 8.25 8.8% 0.40%trending 1,604 2,511 0.123 6.65 6.64 5.33 to 8.01 6.3% 0.62%Difference in added revenue per visit against trending (95% interval, customers resampled):als -3.45 dollars (-5.12 to -1.87): cleartwo-stage +0.19 dollars (-1.75 to +2.05): not clear (the interval includes 0) -
Compare every list with two-stage instead:
Terminal window recs online --baseline two-stageThe last three lines change:
Difference in added revenue per visit against two-stage (95% interval, customers resampled):als -3.64 dollars (-5.40 to -2.03): cleartrending -0.19 dollars (-2.25 to +1.66): not clear (the interval includes 0) -
Run the tests with
python -m pytest -q. You see26 passed. The tests for this module’s commands come in the next lesson.
The columns, from left to right: customers and visits of the group; home page clicks per visit; revenue from suggestions per visit. (The last lesson’s table counted email clicks too, so its clicks per visit are higher: 0.174 for trending.) Then added revenue per visit with its interval, the return rate and the unsubscribe rate. If recs online stops with invalid choice: 'online', read the first common mistake below.
The main measure and the guardrails, side by side
Section titled “The main measure and the guardrails, side by side”The table gives each list’s own value. A decision needs the difference against trending, with its interval. The notebook computes the differences for the guardrails too, with the same resampling as recs online.
Text description of the chart
The figure has three panels. Each has a vertical line at zero, no difference, and one row for two-stage and one for ALS, with a 95% interval. Main measure, added revenue per visit minus trending: two-stage +0.19 dollars (−1.75 to +2.05); ALS −3.45 dollars (−5.12 to −1.87). The ALS interval is the only one entirely on one side of zero. Return rate minus trending, in percentage points: two-stage +2.5 (−4.3 to +8.8); ALS +1.4 (−6.7 to +8.8). Unsubscribe rate minus trending, in percentage points: two-stage −0.22 (−0.75 to +0.22); ALS +0.13 (−0.42 to +0.67). Every guardrail interval crosses zero.
A percentage point is the difference between two percentages: 8.8% − 6.3% = +2.5 points. Now read the panels with Omar’s rule.
ALS: stop
Section titled “ALS: stop”ALS adds 3.45 dollars less per visit than trending, from 1.87 to 5.12 dollars less. That is clearly worse, so the rule says stop. The test itself had a price. ALS’s group had 2,723 visits. So in this simulation, the two weeks cost about 3.45 × 2,723 ≈ 9,400 dollars of added revenue. An online test spends real money on every list that loses.
Two-stage: not clearly better, no clear harm
Section titled “Two-stage: not clearly better, no clear harm”Two-stage adds +0.19 dollars per visit. The interval, −1.75 to +2.05, includes 0, so it is not clearly better. Its guardrails look mixed: the highest return rate (8.8%) and the lowest unsubscribe rate (0.40%). Neither difference is clear.
Few events behind a rate
Section titled “Few events behind a rate”Why are the guardrail intervals so wide? Count the events behind them. The notebook prints them:
| List | Orders | Returns | Return rate | Customers emailed | Unsubscribed | Unsubscribe rate |
|---|---|---|---|---|---|---|
| Trending | 111 | 7 | 6.3% | 1,604 | 10 | 0.62% |
| Two-stage | 125 | 11 | 8.8% | 1,750 | 7 | 0.40% |
| ALS | 65 | 5 | 7.7% | 1,724 | 13 | 0.75% |
Two-stage’s “highest return rate” is 11 returns. Its “lowest unsubscribe rate” is 7 people. Four more returns, or three fewer unsubscribes, would reorder the table. A rate made of a handful of events is mostly noise.
What the guardrails cost, under the course’s assumptions
Section titled “What the guardrails cost, under the course’s assumptions”The course assumes 12 dollars to handle a return and 18 dollars a year for an email subscriber. Under these assumptions:
- Return handling costs 0.033 dollars per visit for trending and 0.049 for two-stage: about 2 cents of difference.
- Per 1,000 customers emailed, trending loses 6.2 subscribers (worth 112 dollars a year) and two-stage 4 (worth 72 dollars a year).
Both are small next to 6 to 7 dollars of added revenue per visit. Guardrails here protect against a large harm. In this test, none appeared.
Speed: a guardrail with a fixed limit
Section titled “Speed: a guardrail with a fixed limit”Speed comes from Module 9’s measurements on the course laptop, not from the simulation. Two-stage needs a nightly batch of 111 seconds for 40,000 customers. Looking up a saved list in the service takes 0.72 ms (median). Computing a two-stage list at request time takes 10.3 ms (median). Both are well under Amira’s 100 ms budget. Trending is one list for everybody, so it needs almost nothing. You build this in Batch or real time.
Your turn: decide which list to release
Section titled “Your turn: decide which list to release”You have everything Omar has: the main measure, three guardrails and the decision rule. Decide for each list: release, stop, or keep trending. Write one sentence of evidence for each. Then open the answer.
Show the answer
- ALS: stop. −3.45 dollars per visit against trending (−5.12 to −1.87): clearly worse on the main measure.
- Two-stage: do not release yet. +0.19 dollars per visit (−1.75 to +2.05): not clearly better. No guardrail is clearly worse. Returns: +2.5 points (−4.3 to +8.8). Unsubscribes: −0.22 points (−0.75 to +0.22). Both speeds are far below 100 ms.
- Trending: keep it. It is as good as two-stage within the noise. It is also much simpler to run: no ranker to train, no nightly batch, nothing to monitor per customer.
Could a longer test show that two-stage is better? The notebook estimates it. The interval is about ±1.90 dollars wide. An interval’s width shrinks with the square root of the number of visits. To make it as small as the difference of 0.19 dollars, the test needs about (1.90 ÷ 0.19)² ≈ 100 times more visits. That is about 270,000 visits per group. In this synthetic shop, that would take years. So Omar keeps trending, and asks Priya for a version of two-stage that is different enough to be worth a new test.
This is a rough rule, not a test design. This course does not teach test design; that belongs to a planned course on experiments.
When to run an online test, and what it costs
Section titled “When to run an online test, and what it costs”Run an online test when an offline shortlist has two or three good candidates, and when the decision matters. Examples are a new home page, a new email or a new ranker. Do not run one for every small change: it is slow, and each losing list costs money while it runs.
Choose the main measure and the guardrails before the test. Write the decision rule down. After the test, read the intervals, not only the values, and count the events behind every rate.
The costs, in Larkfield’s simulated case:
- Time. Two weeks per test, and much longer to see small differences.
- Money. The losing list pays: about 9,400 dollars for ALS in two weeks of this simulation.
- Risk. A bad list reaches real customers. Guardrails limit the damage; they do not remove it.
Common mistakes
Section titled “Common mistakes”Common mistake
recs: error: argument COMMAND: invalid choice: 'online'
Symptom: recs online prints:
usage: recs [-h] COMMAND ...recs: error: argument COMMAND: invalid choice: 'online' (choose from candidates, check, compare, evaluate, explain, info, leakage, llm, matrix, rank, reset, rules, session, similar, tune)Cause: the project finds its commands by file. larkfield_recs/commands/online.py is not there yet: your project is still at the end of Module 7.
Fix: copy the file as in step 1, from the folder larkfield-recs. Then run recs online again.
Common mistake
'Two-stage reduces unsubscribes by a third'
Symptom: Tomás reads 0.40% against 0.62% and reports that two-stage cuts unsubscribes by a third.
Cause: the two rates come from 7 and 10 people. The interval of the difference, −0.75 to +0.22 points, includes 0.
Fix: report the difference with its interval, and the counts behind it: “7 against 10 unsubscribes, not a clear difference.”
Knowledge check
Section titled “Knowledge check”Check your understanding
Why must Omar choose the main measure before the test starts, and not after he sees the results?
Show the answer
A test produces many numbers: clicks, orders, revenue from suggestions, added revenue, returns, unsubscribes. By chance, some of them will favour one list. If Omar chooses the measure after he sees the results, he can pick the one that happens to look best. Then the test only confirms what he wanted. Chosen in advance, the main measure and the decision rule make the result honest.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
Two-stage has the highest return rate (8.8%) of the three lists. Is the return guardrail broken? Use the counts and the interval.
Show the answer
No, not clearly. Two-stage had 11 returns for 125 orders; trending had 7 for 111. The difference is +2.5 percentage points, with a 95% interval from −4.3 to +8.8, which includes 0. Under the course’s assumptions, the extra return handling is about 2 cents per visit. A guardrail is broken only when it is clearly worse; here, a few more or fewer returns would change the order.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
In your project, run recs online --baseline two-stage. Which line tells you that trending is not clearly worse than two-stage, and what are its numbers?
Show the answer
The last line: trending -0.19 dollars (-2.25 to +1.66): not clear (the interval includes 0). Trending adds 0.19 dollars less per visit than two-stage, but the interval runs from 2.25 dollars less to 1.66 dollars more. It is the same comparison as before, seen from the other side. The edges differ a little from −1.75 to +2.05 because the resampling draws other customers.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Recap
- An A/B test splits customers at random, so a difference comes from the versions or from chance.
- Choose in advance one main measure (added revenue per visit), a few guardrails (return rate, unsubscribe rate, speed) and a decision rule.
- In Larkfield’s simulated test: ALS −3.45 dollars per visit against trending (−5.12 to −1.87), so stop it. Two-stage +0.19 (−1.75 to +2.05), not clearly better, with no clear harm on any guardrail.
- Guardrail rates rest on few events: 11 returns and 7 unsubscribes for two-stage. Report counts and intervals.
- A difference of 0.19 dollars would need about 100 times more visits to see. Larkfield keeps the simpler trending list.
The test lasted two weeks. But a recommender keeps running after the test, and it learns from the clicks on its own suggestions. Next, you see what happens to a home page that learns only from itself.