Skip to content
Français

Real tests at an online shop

  • Case study 2
  • 150 min
  • Optional · 1 case study needed for the certificate

Where we are

In eight modules, we analysed Larkfield's simulated tests and their traps. In this case study, we read the daily results of real online experiments at a fashion shop, check them, compare peeking with a sequential test, and decide each one.

What you will be able to do

Check and decide real online tests from their daily numbers, and choose the rules that a team should follow. You will check each test’s split, read its result at the planned end, and replay what daily peeking would have declared. Then you compare peeking with a sequential test, and count how many winners luck alone gives. Last, you run the finished check on tests that you set aside.

Case studyPortfolio piece

Licence
CC BY 4.0
Time
About 2.5 hours
Runs in
Colab, Kaggle or VS Code
Skills used
  • Plan before the result
  • Split check without the plan
  • Intervals from summary numbers
  • Peeking replayed on real tests
  • Sequential test and its tau
  • Many tests and Benjamini–Hochberg
  • Rebuilt A/A tests
  • A decision rule and its cost

Colab and Kaggle open your own copy. To use VS Code, download the notebook from GitHub. Stuck? See the solution notebook.

The notebook runs every step of this page, with the same code and the same output, plus four tasks with checks. Its setup cells install the packages and download the data (2.7 MB) from the dataset’s own page, so it needs no local setup. It runs in about 10 seconds after the install.

Before you startWhat to know and to have readyAbout 150 minutes

What you need to know

This case study uses Modules 2 to 4 most. These lessons matter most:

What you need on your computer

Nothing: the notebook installs its own packages, so it can also run in Colab or Kaggle. To run it locally, use a virtual environment with Python 3.12, pandas 3.1.1, SciPy 1.15.3 and matplotlib 3.11.2, as in the project larkfield-experiments. This case study does not change the project. The setup cell makes a new folder asos-experiments. To start again, delete that folder.

Tested on macOS with Python 3.12, in Jupyter from a fresh start. Colab, Kaggle, Windows and Linux are not tested.

TermLarkfieldA made-up Canadian online shop for home and garden products. It is not a real company. All Nextia courses use its help desk and data as the example.Learn more Meet Larkfield runs about 40 tests a year. Mateo, who keeps the test log, is writing next year’s rules for every test plan. Two questions keep coming back.

TermTomásLarkfield’s AI engineer. He often suggests a quick shortcut, and the lesson shows why it fails. asks the first one: “Can we stop a test when it wins? Waiting four weeks is slow.” TermOmarLarkfield’s product manager. He asks the business questions: what a change costs, and what it gives the shop. asks the second one: “Most of our tests show nothing. Is that normal, and how many of our winners are luck?”

Larkfield’s own tests are too few to answer. So TermPriyaLarkfield’s data scientist. She measures how well the assistant works, on test sets and not on guesses. and Mateo read the tests of a much bigger shop. The online fashion shop ASOS published the daily results of 78 real experiments, with a licence that lets anyone use them. Unlike Larkfield’s simulator, nobody knows their truth. That makes them harder, and closer to real work.

Priya, Mateo, Tomás and Omar are Larkfield’s invented team. They do not work at ASOS. The tests and their numbers are real.

You will build a checking pipeline for real tests and a decision rule that a team can write into its plans. For every test, the pipeline checks the split and gives the result at the planned end. It also replays what three stopping rules would have done. Then you compare the rules on one fair measure. You choose one, and you run it once on eight experiments that you set aside at the start.

The data is the ASOS Digital Experiments Dataset. It has 24,153 rows from 78 experiments that ran in 2019 and 2020.

Each experiment has a control group and one to four treatment groups. A comparison is one treatment group against its control: the data has 99 of them. Each row is one comparison, on one of four metrics, at one moment of the test. It holds the number of users, the mean and the variance of each group.

The numbers are cumulative: each row counts everything from the start of the test to that moment. Here is one comparison on metric 1:

one = tests[(tests.experiment_id == "036afc") & (tests.variant_id == 2) & (tests.metric_id == 1)]
print(one[["time_since_start", "count_c", "count_t", "mean_c", "mean_t"]].iloc[[0, 1, 2, -1]].to_string(index=False))
time_since_start count_c count_t mean_c mean_t
1.5 188065.0 186686.0 0.1078 0.1078
2.0 245041.0 243694.0 0.1318 0.1314
2.5 277237.0 275949.0 0.1431 0.1427
76.5 1050994.0 1050508.0 0.5868 0.5873

What this tells us. The test was read every 12 hours. The groups grew from about 188,000 users each to about 1.05 million. The mean grew too, from 0.11 to 0.59. Metric 1 is a 0/1 value for each user, and it counts anything that the user did since the start. So a user who comes back later can still change it. The data gives no names: we do not know what the metric measures, what each test changed, or when it ran.

tests = pd.read_csv("asos_digital_experiments_dataset.csv", dtype={"experiment_id": str})
print(tests.shape)
print("rows with no users yet:", int(((tests.count_c == 0) | (tests.count_t == 0)).sum()))
tests = tests[(tests.count_c > 0) & (tests.count_t > 0)]
tests = tests.sort_values(KEY + ["metric_id", "time_since_start"]).reset_index(drop=True)
print("experiments:", tests.experiment_id.nunique(), "| comparisons:", len(tests[KEY].drop_duplicates()))
print("rows per metric:", tests.metric_id.value_counts().sort_index().to_dict())
print("rows without a variance:", tests.groupby("metric_id").variance_c.apply(lambda x: int(x.isna().sum())).to_dict())
(24153, 10)
rows with no users yet: 8
experiments: 78 | comparisons: 99
rows per metric: {1: 6037, 2: 6036, 3: 6036, 4: 6036}
rows without a variance: {1: 0, 2: 261, 3: 261, 4: 256}

What this tells us. The 8 empty rows are the first look of one experiment, before any user arrived; we drop them. Metric 1 is complete. The other three metrics miss some variances, so they cannot always give an TermConfidence intervalA range around a measured value, made by a method that holds the true value 95 times in 100. If the interval of a difference does not include zero, the difference is probably not chance.Learn more Estimate uncertaintyEssential Mathematics and Statistics for AI. Priya chooses metric 1 as the metric that decides. It is the only 0/1 metric, and it has no gaps.

Priya writes the plan before she computes any result. It has the course’s form, from Write the plan first:

  • Unit: one comparison. Metric: metric 1.
  • Planned look: the last row of each comparison. The real plans are not published, so the last row stands in for the planned end. That is an assumption.
  • Decision at the planned look: 95% interval above 0, ship; below 0, do not ship; otherwise, no clear effect.
  • Split check first: a sample ratio mismatch (p < 0.001) means no decision.
  • The rules to compare: peeking every day, one planned look, and the course’s sequential test.
  • Hold-out: before any result, put 8 of the 78 experiments aside, chosen at random with a fixed seed. Choose the rules on the other 70. Run the chosen rule on the 8 once, at the end.
ids = sorted(tests.experiment_id.unique())
holdout = sorted(np.random.default_rng(2021).choice(ids, 8, replace=False).tolist())
past = tests[~tests.experiment_id.isin(holdout)]
print("set aside:", holdout)
print("past tests:", past.experiment_id.nunique(), "experiments,", len(past[KEY].drop_duplicates()), "comparisons")
set aside: ['530a76', '834947', '8b436e', '9ed9d5', 'adea31', 'b8a409', 'bac0d3', 'e4c4a1']
past tests: 70 experiments, 90 comparisons

The notebook defines a few small functions. Where the course project has the same function, they are copies. add_result() gives each row its difference, standard error, 95% interval and p-value from the summary numbers alone. split_check() is the split check of Check the test before the result, with one change that you will see in a moment.

def split_check(count_c, count_t):
"""'not checkable' (more than 0.5 points from 50%: ask for the planned split), 'mismatch' or 'ok'."""
if abs(count_t / (count_c + count_t) - 0.5) > 0.005:
return "not checkable"
return "mismatch" if srm_p(count_c, count_t) < 0.001 else "ok"

Priya applies it to the last look of every comparison:

m1 = add_result(past[past.metric_id == 1])
last = m1.groupby(KEY).tail(1).reset_index(drop=True)
first = m1.groupby(KEY).head(1).reset_index(drop=True)
last["share_first"] = first.count_t / (first.count_c + first.count_t)
last["share_t"] = last.count_t / (last.count_c + last.count_t)
last["srm_p"] = [srm_p(c, t) for c, t in zip(last.count_c, last.count_t)]
last["split"] = [split_check(c, t) for c, t in zip(last.count_c, last.count_t)]
print(last.split.value_counts().to_dict())
{'ok': 80, 'not checkable': 8, 'mismatch': 2}

The ten comparisons that did not pass, from the notebook’s table. A p-value of 0 is smaller than the computer can show.

Comparison Treatment share, first look Treatment share, last look p against 50% Split check
54a85a, variant 0 2.96% 2.99% 0 not checkable
c3d89d, variant 2 46.23% 46.16% 0 not checkable
329386, variants 1 and 2 49.28%, 49.24% 49.25%, 49.24% 0 not checkable
64dc88, variants 1 and 2 49.23%, 49.26% 49.26%, 49.24% 0 not checkable
eeefa3, variant 3 66.71% 66.64% 0 not checkable
81761c, variant 2 88.85% 88.67% 0 not checkable
26bd38, variant 1 49.99% 49.90% 6.5 × 10⁻⁸ mismatch
a68bf4, variant 1 49.94% 49.93% 4.1 × 10⁻²⁵ mismatch

What this tells us. The first version of the plan tested every comparison against an equal split. Eight comparisons failed by far. But look at their shares. Each one is the same at the first look and the last. Each is a simple ratio, such as 2 : 1 (66.7%), or 33 : 34 (49.25%) in tests with three groups. A redirect bug rarely gives the same share from the first hour. A planned unequal split does.

The plans are not published, so we cannot check these eight. Priya changes the plan and records the change: a comparison more than 0.5 points from 50% is not checkable. It gets no decision, and the team must ask for the planned split. The change uses only the shares, not any result.

Two comparisons are close to 50%, but too far for chance. 26bd38 has 49.90% treatment: about 7,500 users short in 7.6 million. With so many users, even a small gap is a clear sign that users went missing, as with the redirect bug. That leaves 80 comparisons with a good split, in 66 experiments.

For the 80, the plan’s rule gives:

ok = last[last.split == "ok"].copy()
ok["decision"] = [decision(lo, hi) for lo, hi in zip(ok.low, ok.high)]
print(ok.decision.value_counts().to_dict())
print(f"median size of a change: {ok.relative.abs().median():.2%} of the control mean")
print(f"median standard error: {(ok.se / ok.mean_c).median():.2%} of the control mean")
print(f"median length: {ok.time_since_start.median():.0f} days; median users: {(ok.count_c + ok.count_t).median():,.0f}")
{'no clear effect': 59, 'ship': 12, 'do not ship': 9}
median size of a change: 0.18% of the control mean
median standard error: 0.16% of the control mean
median length: 43 days; median users: 5,675,413

What this tells us. Three tests in four end with no clear effect. The changes are tiny: a TermMedianThe middle value when you sort the values. Half are lower and half are higher. One very slow call does not change it much, so it is a good “usual” time.Learn more Measure practical performanceLLMs: How They Work and How to Choose One of 0.18% of the control’s mean. That is the size of the noise, 0.16%, even with 5.7 million users. On a big shop, real changes move metrics by fractions of a percent. Larkfield’s simulated changes were 2% or more, so its tests were easier.

“No clear effect” is not the same as “no effect”. A small real effect can hide in a wide interval. Can the data say how many changes really did nothing? Not for one test, but for the 80 together.

Think of a quiz with true-or-false answers. Students who guess get scores spread evenly from low to high. Students who know the subject get high scores. So nearly every low score comes from a guesser. Count the low scores, and you can estimate how many guessed.

p-values work the same way. When a change does nothing, its p-value is equally likely to be anywhere from 0 to 1. When a change works, its p-value is usually small. So p-values above 0.5 come almost only from changes that did nothing. Count them and double the count: that estimates how many did nothing.

The p-values of the 80 comparisons at the planned look, in bins of 0.1. If no change did anything, each bin would hold about 8. The first bin holds 26: some changes did something.
Text description of the chart

A histogram with ten bins of p-values from 0 to 1. Counts: 26, 9, 4, 6, 5, 4, 13, 4, 1, 8. The first bin, from 0 to 0.1, is far above the others. A marker at 0.5: the five bins above it hold 30 comparisons in total.

Above 0.5 there are 4 + 13 + 4 + 1 + 8 = 30 comparisons, 37.5% of 80. Doubled, that is 75%. So about three changes in four did nothing measurable, and about one in four did something. This answers Omar’s first question: most tests showing nothing is normal. It is also normal on a shop with millions of users.

The plan’s rule decided 21 comparisons: 12 ship and 9 do not ship. How many of them are luck? Experiments as a team habit showed the reasoning on a made-up year. Here it runs on real tests.

n = len(ok)
print("p < 0.05:", int((ok.p < 0.05).sum()), "| luck alone, if no change worked:", 0.05 * n)
no_effect = min(1.0, float((ok.p > 0.5).mean()) * 2) # the share of no-effect tests, as above
print("expected false winners, with the share of no-effect tests:", round(0.05 * no_effect * n, 1))
print("Bonferroni:", int((ok.p < 0.05 / n).sum()), "| Benjamini-Hochberg:", int(benjamini_hochberg(ok.p).sum()))
p < 0.05: 21 | luck alone, if no change worked: 4.0
expected false winners, with the share of no-effect tests: 3.0
Bonferroni: 11 | Benjamini-Hochberg: 16

What this tells us. If no change worked, luck alone would give about 4 decisions of 80. There are 21, so most are real. 75% of the changes did nothing. So we expect about 0.05 × 0.75 × 80 = 3 of the 21 to be false: about one in seven. Benjamini–Hochberg keeps 16, with a false share of at most 5% on average. Bonferroni, which is stricter, keeps 11. Each test was its own decision, so each one keeps its own 5% rule. Mateo puts the corrected count in the test log, to report how many of the year’s winners to trust.

Now Tomás’s question. The data has a look every 12 or 24 hours, so we can replay what a dashboard showed each day. daily_looks() keeps one look a day. stops() applies three stopping rules to one comparison:

  • Peeking: stop at the first day with p < 0.05.
  • The sequential test (mSPRT) from Peeking: stop at the first day when its always-valid p-value is below 0.05. Its tau is a guess of the size of a real effect, as a share of the control mean.
  • The confirmed sequential test, which comes later on this page.

Predict. Of the 80 comparisons, how many will peeking stop with a “winner” or a “loser”? The planned look decided 21.

paths = all_stops(m1, ok, tau_rel=0.05)
peek = paths[paths.peek_day.notna()]
print("peeking calls:", len(peek), "of", len(paths), "| median day of the call:", peek.peek_day.median())
print("of these, no clear effect at the planned look:", int((peek.decision == "no clear effect").sum()))
peeking calls: 35 of 80 | median day of the call: 7.0
of these, no clear effect at the planned look: 14

What this tells us. Peeking would have stopped 35 tests, most in the first week. Fourteen of them end with no clear effect at the planned look. The other 21 are exactly the planned look’s 21 decisions, with the same sign. So peeking found no new real effect. It added 14 calls that the full test does not support.

Here is one of the 14, experiment b382c6, in its first four days:

time_since_start relative p
1.0 0.0463 0.1435
2.0 0.0454 0.0402
3.0 0.0407 0.0238
4.0 0.0283 0.0636
  • Usual p-value
  • Sequential p-value
  • p = 0.05
Experiment b382c6, one look a day for 61 days. The usual p-value dips below 0.05 on days 2 and 3, when the change looks like +4.5%. Then it rises; on day 60.5 it is 0.69. The sequential p-value never goes below 0.55.
Text description of the chart

A line chart over 61 days. The usual p-value starts at 0.14 on day 1, falls to 0.040 on day 2 and 0.024 on day 3, then rises above 0.05 on day 4. For the rest of the test it moves between about 0.2 and 1, and it ends at 0.69. The sequential p-value stays between 0.55 and 1. A dashed line marks 0.05.

What this tells us. On day 2, the dashboard said +4.5% with p = 0.040. A team that stopped would have shipped a “4.5% winner”. At the end, the change was −0.18%. In the first days, few users decide the numbers, and chance moves them a lot. This is the checkout button of Peeking, on a real shop.

The sequential test needs tau, its guess of how big real effects are. The course’s default is 5% of the control mean. That suits Larkfield’s simulator, but here the changes are about 0.2%. If tau is much too large, the test is slow to see small effects.

A better guess comes from past tests. The estimates spread for two reasons: the real effects differ, and noise adds its own spread. Take the spread of the estimates and remove the spread that the noise alone gives. What is left is the spread of the real effects.

rel, rel_se = ok.relative.to_numpy(), (ok.se / ok.mean_c).to_numpy()
tau_past = float(np.sqrt(np.var(rel) - np.mean(rel_se ** 2)))
print(f"spread of the estimates {np.std(rel):.2%}, of the noise {np.sqrt(np.mean(rel_se ** 2)):.2%}: tau {tau_past:.2%}")
for name, tau in [("tau 5%", 0.05), ("tau from past tests", tau_past)]:
s = all_stops(m1, ok, tau)
seq = s[s.seq_day.notna()]
print(f"{name}: {len(seq)} calls, median day {seq.seq_day.median()}, "
f"{int((seq.end_day - seq.seq_day).sum())} test-days saved")
spread of the estimates 0.70%, of the noise 0.38%: tau 0.59%
tau 5%: 15 calls, median day 5.0, 484 test-days saved
tau from past tests: 18 calls, median day 7.0, 580 test-days saved

What this tells us. The square of 0.70% minus the square of 0.38% leaves the square of 0.59%: the typical size of a real effect. With this tau, the sequential test decides 18 comparisons instead of 15, at a median of day 7 instead of day 43. Over the 18, that frees 580 test-days for other tests. A team gains this when it takes tau from its own test log.

Of the 18 sequential calls, two end with no clear effect. One of them, 162a38, stopped on day 38. The dataset’s own datasheet warns about this test and this day: the treatment mean jumps, probably because some tracking data was lost.

  • Change
  • 95% interval
Experiment 162a38 from day 30, every 12 hours. On day 38, the change jumps to +0.57% (p = 0.00003). At the next look it is back at −0.04%. Lost tracking data is the probable cause.
Text description of the chart

A line chart from day 30 to day 42, with a dot every 12 hours. The change stays between about −0.08% and 0%, inside an interval of about −0.3% to +0.25%. At day 38, one dot jumps to +0.57%, with an interval from +0.30% to +0.83%. At day 38.5 it is back at −0.04%.

The always-valid interval keeps a running intersection: once a value is out, it stays out. So one bad look is enough to decide, forever. A sequential test protects you from chance. It does not protect you from wrong data.

The fix is cheap. Confirm the stop: stop only if the look’s own always-valid p-value is below 0.05 at two looks in a row. A real effect stays; a glitch goes away. The rule is stricter, so it cannot add false winners. It costs one day. The notebook keeps the stops with tau from past tests in paths_past, and stops() already computes the confirmed stop as conf_day.

conf = paths_past[paths_past.conf_day.notna()]
agree = ((conf.conf_sign == "up") & (conf.decision == "ship")) | ((conf.conf_sign == "down") & (conf.decision == "do not ship"))
print("confirmed early stops:", len(conf), "| same answer as the planned look:", int(agree.sum()),
"| median day:", conf.conf_day.median())
confirmed early stops: 17 | same answer as the planned look: 16 | median day: 8.0

What this tells us. The glitch is gone. Sixteen of the 17 early stops agree with the planned look. The one that does not, 54a85a variant 2, is the other test that the datasheet names. Its treatment rises faster than its control over time. That may be a primacy effect, the opposite of novelty: users needed time to get used to the change. Early in the test, the change looked harmful: −1.2% on day 8. At the end, it was −0.23%, with p = 0.085. No stopping rule can see that an effect will change later. This is why a plan names a minimum length of one or two full weeks.

Common mistake

“The sequential test is always safe”

Symptom: the sequential test stops experiment 162a38 on day 38 with a “winner”. At the planned end, the change is −0.02%, p = 0.90.

Cause: one look had wrong data. The test’s promise covers chance, not a broken log. Its running intersection keeps the bad look forever.

Fix: confirm every early stop with a second look. Check the data behind it: the counts and the means of both groups at that look and the next one.

Which rule should Mateo write into the plans? Counting decisions on the real tests does not answer it. Peeking makes the most decisions, and nobody knows which of them are false. We need a measure where the truth is known.

Think of a smoke alarm. You cannot judge it only by the fires it finds. You must also count how often it rings when there is no fire. So you test it many times with no fire.

For tests, “no fire” is an A/A test: both groups get the same version, so every winner is false. We cannot rerun ASOS’s tests as A/A tests. But a test’s looks and standard errors are enough to rebuild one. With no effect, a test’s difference wanders like a random walk. Each look adds a random step. The size of the step comes from how much the standard error shrank since the last look.

So we rebuild each of the 80 comparisons 1,000 times, with its own looks and standard errors and no effect. Then we count how often each rule calls a winner or a loser. Every rule sees the same 80,000 A/A tests: one measure for all of them.

Predict. Which rules will stay under 5% false winners?

rng = np.random.default_rng(40)
rates = {}
for e, v in ok[KEY].itertuples(index=False):
looks = daily_looks(m1[(m1.experiment_id == e) & (m1.variant_id == v)])
calls = aa_versions(looks.se.to_numpy(), looks.mean_c.iloc[-1], tau_past, 1000, rng)
for rule, called in calls.items():
rates.setdefault(rule, []).append(called.mean())
aa = pd.Series({rule: np.mean(r) for rule, r in rates.items()}).round(3)
print(aa.to_string())
one planned look 0.051
peeking every day 0.175
sequential, tau 5% 0.004
sequential, tau from past tests 0.010
confirmed sequential 0.006
confirmed early stop, else planned look 0.052
  • Peeking
  • Planned look, and the combination
  • Sequential tests
Six stopping rules. Top: 80,000 rebuilt A/A tests, 1,000 per comparison, each with its own looks and standard errors and no effect. Peeking calls a false winner or loser in 17.5%, the planned look 5.1%, the sequential tests under 1%. Bottom: on the 80 real comparisons, peeking decides 35, but 14 of them end with no clear effect.
Text description of the chart

Two panels with the same six rows. False winners in rebuilt A/A tests: peeking every day 17.5%; one planned look 5.1%; sequential test with tau 5% 0.4%; sequential test with tau from past tests 1.0%; confirmed sequential test 0.6%; confirmed early stop, else the planned look, 5.2%. A line marks the promised 5%. Real comparisons decided, of 80: peeking 35; planned look 21; sequential with tau 5% 15; sequential with tau from past tests 18; confirmed sequential 17; confirmed early stop, else the planned look, 22.

What this tells us. Read the two panels together:

  • Peeking fails. It calls a winner or a loser in 17.5% of tests with no effect: three and a half times the promise. That is lower than the course’s simulated 27.6%. These tests have fewer looks on average, and their first look already holds much of the information (a median of 36%). With 40 or more looks, it reaches 19.7%.
  • The planned look keeps its promise. One look is exactly 5% in theory; 5.1% is that plus the noise of the simulation. It decides 21 comparisons, but only after a median of 43 days.
  • The sequential tests are very safe and a little weaker. They stay under 1%. They decide 15 to 18 comparisons, most of them in the first week or two.
  • The combination stops early only with the confirmed sequential test, and otherwise decides at the planned look. It decides 22 comparisons, 17 of them early, for 5.2% false winners: about 0.1 points more than the planned look alone.

The plan said: choose the rule that stays at or below 5% and decides the most. By that rule, the planned look wins, and only just. The combination decides one more comparison and saves weeks, for about 0.1 points more false winners. And Priya added it after she saw the glitch. A rule chosen after looking at the data must prove itself on new data. The hold-out is there for that.

Priya writes to Mateo in plain words, for every Larkfield test plan:

  • Never stop at the first p < 0.05. On these real tests, peeking calls a false winner in about 17.5% of tests with no effect. Fourteen of its 35 calls did not hold.
  • Plan a fixed length, and decide at the end. Name a minimum of one or two full weeks, for novelty and primacy effects.
  • Allow one early stop: the confirmed sequential test. Take tau from the team’s own past tests, not from a default. Stop only when two looks in a row agree, and check the data at those looks.
  • Check the split first, and write the planned split in the plan. Eight of these 90 comparisons could not be checked. Their planned split was not written down anywhere.
  • Count the year’s winners against luck. Report the Benjamini–Hochberg count in the test log, next to the raw count.
A flow of four steps. A test runs, and each day gives the users, mean and variance of both groups. Step 1, the split check: share near 50% and p at least 0.001? If no, a red box: no decision; for a mismatch, find the cause; for another split, ask for the plan. If yes, step 2, each day: the confirmed sequential test, always-valid p below 0.05 at two looks in a row? If yes, a green box: stop early, ship or do not ship. If not yet, until the planned day, step 3: the 95% interval; an orange box says above 0 ship, below 0 do not ship, contains 0 no clear effect. Step 4: the test log counts the winners against luck with Benjamini–Hochberg. A red dashed box: never stop at the first day with p below 0.05.
The finished check that Priya recommends. The split check comes first. The confirmed sequential test is the only early stop. Everything else waits for the planned day.

The data has no money in it, and nobody knows what metric 1 measures. So Priya counts in test-days and in false winners.

  • Worth: on the past tests, the confirmed early stops freed 530 test-days: about 31 days for each of the 17. A shop that can run only so many tests at a time gets those days back for new tests.
  • Cost of peeking: in 100 tests where about 75 changes do nothing, peeking makes about 75 × (17.5% − 5.1%) ≈ 9 more false winners than the plan. Priya assumes that a change that does nothing still costs two developer-weeks to build, keep and later remove. That is an assumption, not from the data. At that price, peeking wastes about 18 developer-weeks per 100 tests. The course’s money assumptions (26 dollars margin per order) are Larkfield’s, and they do not fit a metric whose meaning is unknown.

The rule was chosen on 70 experiments. Now it runs once on the 8 that were set aside. review() does every step for new tests:

new = tests[tests.experiment_id.isin(holdout)]
result = review(new, tau_past)
print(result.to_string(index=False))
experiment_id variant_id users treatment share change at the end p at the end verdict decided on day planned length (days)
530a76 1 112933005 0.4998 +0.07% 0.4766 no decision (mismatch) NaN 54.0
834947 1 30113549 0.4995 -0.01% 0.9658 no decision (mismatch) NaN 43.5
8b436e 1 66803872 0.4999 +0.16% 0.1405 no clear effect 45.5 45.5
9ed9d5 1 8597234 0.4998 +0.34% 0.0158 ship (early) 6.0 30.5
adea31 1 99468529 0.5000 -0.17% 0.0827 no clear effect 84.5 84.5
adea31 2 99465130 0.5000 -0.10% 0.3281 no clear effect 84.5 84.5
b8a409 1 3251862 0.4998 +0.39% 0.0431 ship 64.5 64.5
bac0d3 1 5495175 0.5000 -0.08% 0.5670 no clear effect 35.0 35.0
e4c4a1 1 5875066 0.4999 -0.84% 0.0000 do not ship (early) 9.0 36.0
  • Ship
  • Do not ship
  • No clear effect
The 7 held-out comparisons with a good split, at their planned end. Two intervals are above 0 (ship), one is below 0 (do not ship), and four contain 0.
Text description of the chart

An interval plot with seven rows, as a percentage of the control mean. 8b436e variant 1: +0.16%, from −0.05% to +0.38%. 9ed9d5 variant 1: +0.34%, from +0.06% to +0.61%. adea31 variant 1: −0.17%, from −0.37% to +0.02%. adea31 variant 2: −0.10%, from −0.29% to +0.10%. b8a409 variant 1: +0.39%, from +0.01% to +0.78%. bac0d3 variant 1: −0.08%, from −0.35% to +0.19%. e4c4a1 variant 1: −0.84%, from −1.17% to −0.50%. A vertical line marks no effect.

What this tells us. The rule worked on new tests:

  • The split check caught two mismatches. 530a76 has 113 million users and a treatment share of 49.98%. The gap is tiny, but with so many users it is not chance (p = 0.0006). The team must find the missing users before it reads the result.
  • The two early stops agree with the end. 9ed9d5 shipped on day 6 instead of day 30.5. e4c4a1 stopped on day 9 instead of day 36. Both planned-end intervals agree. Together, the two stops freed 51.5 test-days.
  • b8a409 waited for the planned end and shipped there, with p = 0.043. The sequential test was not sure enough to stop it early: a real effect near the edge needs the full test.
  • Peeking would have been wrong twice. It would have stopped bac0d3 on day 1 and adea31 variant 1 on day 10.5. Both end with no clear effect.
  • There is no truth to check against. Unlike Larkfield’s simulator, nobody knows these true effects. The rebuilt A/A tests check the rules, not any single result. They assume that a test’s difference moves like a random walk.
  • The data breaks the assumptions a little. Metric 1 counts each user’s whole history, so a new look is not a fresh sample. Some standard errors even grow between looks. The sequential test’s promise holds only roughly here, as for revenue per customer in Peeking.
  • What we could not do. There are no rows for single users, so no CUPED, no segments and no bootstrap. We do not know the metrics, the planned lengths or the planned splits. The “planned look” is the last row, and the team may itself have stopped some tests early.
  • What a fair evaluation would need. First, each test’s plan, with its length, split and metric, written before the start. Second, per-user data, for CUPED and segments. Third, real A/A tests on the same platform, to check the rebuilt ones.
  • Winners exaggerate. The 21 decided comparisons are the ones that luck helped over the line. Their estimates are probably a little too large, as in Experiments as a team habit.
  • Draw no conclusions about ASOS. The datasheet says that these results do not represent ASOS’s business or its test programme. They are 78 tests, chosen and anonymised for research.
  • The data has no people in it: keep it that way. Each row sums at least a hundred thousand users. Do not try to link the anonymous experiments to real changes on the shop’s website.
  • Credit the source. CC BY 4.0 requires credit. Use the attribution text in the box above, with a link to the licence, wherever you show the data or its results.
  • Rules are for the team, not for blame. A test log that counts false winners is there to improve the rules. Do not use it to judge a person whose test showed nothing. Most tests show nothing.

This case study shows an employer that you can work with real test data, find its problems, and turn the analysis into rules for a team. Make it yours with one change, such as metric 4 or weekly looks. Then make it public and easy to read in five minutes.

What to put in the repository:

  • README.md, the page that people read first (template below).
  • The notebook, run from top to bottom, with its outputs.
  • PLAN.md: the plan from “The plan before the result”, and the recorded change to the split check, with its reason.
  • rules.csv: each rule’s false-winner rate in the rebuilt A/A tests, and its decisions on the real tests.
  • holdout.csv: the output of review() on the 8 held-out experiments.
  • The data source and licence, with the attribution text above. CC BY 4.0 allows a copy with credit, but a link and the checksum are enough.

A README template:

# Real tests at an online shop: when may a test stop?
## Problem
A team wants to stop tests early when they win. Which stopping rule keeps
false winners at 5%, and how many of a year's winners are luck?
## Data and licence
ASOS Digital Experiments Dataset (Liu, Cardoso, Couturier and McCoy, 2021),
78 experiments, 99 comparisons, daily cumulative counts, means and variances,
https://osf.io/64jsb/, CC BY 4.0. Decided on metric 1; 8 empty rows dropped.
## Approach
Plan written first; 8 experiments held out. Split check (8 not checkable,
2 mismatches). 95% interval at the last look. Peeking, the mSPRT with tau
from past tests, and a confirmed mSPRT, replayed day by day. Rules compared
on 80,000 A/A tests rebuilt from each test's own looks and standard errors.
## Results
Peeking: 17.5% false winners, 35 calls (14 not supported at the end).
Planned look: 5.1%, 21 decisions. Confirmed early stop, else planned look:
5.2%, 22 decisions, 17 early. About 75% of changes did nothing; about 3 of
21 decisions are expected to be luck.
## Limits
No truth, no per-user data, no plans; cumulative metric; the rebuilt A/A
tests assume a random walk.

How to describe it in an interview, in two sentences: “I replayed 78 real online experiments day by day: stopping at the first significant result made three and a half times the promised false winners. I recommended a fixed length with one allowed early stop, a confirmed sequential test, which matched the full tests on held-out data and saved weeks.”

Go further:

  • Decide on metric 4 (a skewed amount), and compare its share of no-effect tests with metric 1. Predict first.
  • Try weekly looks for the peeking rule, and compare its rebuilt A/A rate with daily looks.
  • Make the planned split an input of split_check(), and test eeefa3 variant 3 against 2 : 1.

Check your understanding

Peeking stopped 35 of the 80 real comparisons, and the planned look decided only 21. Tomás says: 'Peeking finds more effects.' What do you answer?

Show the answer

Peeking did not find new real effects. Its 35 calls include all 21 decisions of the planned look, with the same sign, plus 14 calls that end with no clear effect. More calls are not more findings. On rebuilt A/A tests, where nothing works, peeking calls a winner or a loser 17.5% of the time, against 5.1% for the planned look. So its extra calls are what you expect from a rule that makes false winners. If the team needs early stops, the confirmed sequential test gives them with under 1% false winners.

Check your understanding

Eight comparisons had treatment shares such as 2.99%, 66.64% and 49.25%, far from 50%. Why did Priya give them no decision, instead of calling them sample ratio mismatches?

Show the answer

A sample ratio check compares the counts with the planned split, and these plans are not published. Each share is the same at the first look and the last, at a simple ratio such as 2 : 1 or 33 : 34. A bug rarely does that; a planned unequal split does. Against 50%, they would all fail, but that test asks the wrong question. Without the plan, the split cannot be checked. So the honest answer is “no decision: ask for the planned split”. The lesson for Larkfield is to write the split in every plan.

Check your understanding

The sequential test stopped experiment 162a38 on day 38, but the full test shows no effect. The sequential test promises at most 5% false winners. Did it break its promise?

Show the answer

No. Its promise is about chance: with correct data and no effect, it calls a false winner at most 5% of the time, and under 1% here. On day 38, the data was wrong. The treatment mean jumped and fell back at the next look, probably because tracking data was lost. A test cannot protect you from a broken log. The always-valid interval keeps a running intersection, so one bad look decides forever. The fix is to confirm an early stop at a second look and to check the data behind it.

Recap

  • You wrote the plan first and set aside 8 experiments. The split check found 2 mismatches, and 8 splits that cannot be checked without the plan.
  • Real effects were tiny (a median of 0.18%). About 75% of the changes probably did nothing, and about 3 of the 21 decisions are expected to be luck.
  • Peeking stopped 35 tests; 14 of those calls did not hold. On rebuilt A/A tests, it made 17.5% false winners.
  • A sequential test with tau from past tests, confirmed at two looks, stopped most real effects early. On the held-out tests, it agreed with the full tests and freed 51.5 test-days.

Next, the last case study: a change that nobody randomised. We read a minimum-wage rise with difference-in-differences, on public employment data.

Next: A minimum wage without a test