Skip to content
Français

Randomise

  • Lesson 5
  • 35 min
  • Module 2: Design an A/B test

Where we are

The simulator showed us that a naive comparison can miss the truth by a lot. A randomised test came much closer. In this lesson, we see why a coin flip removes confounders, and how the choice of what we randomise, a customer or a session, changes the answer.

Plays from YouTube. Nothing loads from YouTube until you press play. Privacy

What you will be able to do

Let a coin decide who gets a change, so that the two groups are alike before the change starts. Then a difference after the change can only come from the change or from chance. Choose what the coin is flipped for, the customer or the session, because the wrong unit breaks the test.

NotebookDo this lesson in a notebookColab · Kaggle · GitHub

The notebook compares the users of the recommendation block with the other customers before the block existed. Then it flips a coin for the same customers, 1,000 times, and measures how alike the coin’s groups are. Its setup cells install pandas and download the practice data, so it needs no local setup. About 30 minutes.

Colab and Kaggle open your own copy. They need a free Google or Kaggle account. Stuck? See the solution notebook.

Before you startWhat to know and to have readyAbout 35 minutes

What you need to know

What you need on your computer

  • The project folder larkfield-experiments from Meet the simulator, with its virtual environment active. python -m pytest -q prints 11 passed.

Tested on macOS with Python 3.12, pandas 3.1.1 and the start snapshot of the project. Windows was not tested.

A baker makes a new seed bread. She wants to know if a free taste makes people buy it. For two weeks, she offers a taste to the customers she knows well. They chat with her, they come every day, and they buy a lot of bread anyway. At the end, the customers who tasted bought twice as much bread as the others.

Did the taste do that? She cannot say. She chose her best customers for the taste. They would buy more bread with or without it. The comparison mixes two things: the effect of the taste, and who got the taste.

Now she tries again. At the door, she flips a coin for each customer. Heads, a taste; tails, no taste. The coin does not know who is a regular. So regulars and new customers end up on both sides, in about the same numbers. If the tasters now buy more, the taste is the most likely reason. The only other reason is the luck of the coin, and that luck gets small when many customers come in.

That is the whole idea of this lesson. Let a coin decide who gets the change.

The problem: the users were different before

Section titled “The problem: the users were different before”

You met this problem in Module 1. TermLarkfieldA made-up Canadian online shop for home and garden products. It is not a real company. All Nextia courses use its help desk and data as the example.Learn more Meet Larkfield’s logs show that the customers who used the recommendation block spent 9.70 dollars more than the others. The simulator knows the truth: the block added only 2.73 dollars for its users. Where does the rest come from?

The logs have a column for each customer’s spending in the 8 weeks before the launch: pre_spend. The block cannot have changed it, because the block did not exist yet. Here are the two groups before the launch (simulated):

Before the launch Users of the block The other customers
Customers 13,338 26,662
Spending in the 8 weeks before (dollars) 51.77 41.28
Orders in the 8 weeks before 0.62 0.48
Days as a customer 884 782
Share of new customers 26% 32%

The users already spent 10.49 dollars more before the block existed. They were customers for longer, and fewer of them were new. They are the bakery’s regulars. The customers chose to use the block, and the keen customers chose it more often. This is self-selection: the people choose their own group, so the groups differ in ways that also change the result.

Two panels. Left, customers chose: a box 'keen, active customer' has a dashed red arrow to 'uses the block' and a grey arrow to 'spends more'; 'uses the block' has a blue arrow to 'spends more'. Right, a coin chose: a 'coin flip' has a blue arrow to 'sees the block', which has a blue arrow to 'spends more'. The keen customer still has an arrow to 'spends more', and its arrow to 'sees the block' is crossed out.
When customers choose, being keen leads both to the block and to spending: the gap mixes the two paths. When a coin chooses, the arrow from the customer to the block is gone.
Text description of the diagram

The diagram has two panels. In the left panel, the customers chose. A box “keen, active customer (spent more before)” has two arrows. A dashed red arrow goes to “uses the block”, and a grey arrow goes to “spends more”. A blue arrow goes from “uses the block” to “spends more”. So there are two paths to “spends more”: the block itself, and the kind of customer who uses it. The note says that the gap of +9.70 dollars mixes them. In the right panel, a coin chose. A “coin flip” has a blue arrow to “sees the block”, and “sees the block” has a blue arrow to “spends more”. The keen customer still spends more, but its arrow to “sees the block” is crossed out: the coin ignores the customer. Keen customers are in both groups, so only the block can make a gap.

You drew this kind of graph in Draw the causes. Being keen is a confounder: it causes both “uses the block” and “spends more”. A confounder opens a second path between the change and the result, so the naive gap is wrong.

Randomisation means that chance alone decides which group each unit goes to. A computer’s random numbers play the coin. A test that does this is a randomised controlled trial, often called an A/B test. Group A, the control group, keeps today’s version. Group B, the treatment group, gets the change.

Why does a coin remove confounders? Because a confounder works only through the choice. Keen customers ended up with the block because they chose it. A coin cannot be influenced by keenness, or by anything else about the customer. So every kind of customer has the same chance to be in A or in B. On average, the two groups are alike in every column, before the change starts.

“Every column” includes the columns that you did not measure. The logs scenario hides one cause on purpose: whether the customer has a garden. It is not in the data, so no method that uses the data can correct for it. A coin balances it anyway, because the coin does not need to see it. No other method in this course can promise that. Modules 5 and 6 show methods for when you cannot randomise, and each one needs an assumption that a coin does not need.

Here are 8 customers and their spending before the change. In the first column, three keen customers chose the block. In the second, a coin decided (a constructed example):

Customer Spending before (dollars) Chose the block The coin
C1 120 yes B
C2 90 yes A
C3 80 yes A
C4 60 no B
C5 30 no A
C6 20 no B
C7 10 no B
C8 0 no A

Compare the groups before anything happened:

  • Chose the block: (120 + 90 + 80) ÷ 3 = 96.67 dollars. The others: (60 + 30 + 20 + 10 + 0) ÷ 5 = 24.00. A gap of 72.67 dollars before the change.
  • The coin: B is (120 + 60 + 20 + 10) ÷ 4 = 52.50 dollars. A is (90 + 80 + 30 + 0) ÷ 4 = 50.00. A gap of 2.50 dollars.

The coin’s gap is not zero. With 8 customers, luck still matters: a different flip could put C1 and C2 together. But the coin’s gap has no direction. It favours A as often as B, and it shrinks as the groups grow. The choice’s gap has a direction, and it stays however many customers you add.

How close is “close”? Flip the coin 1,000 times

Section titled “How close is “close”? Flip the coin 1,000 times”

Now take the 40,000 customers in Larkfield’s logs. Flip a coin for each of them, and compare the spending before the launch of the two coin groups. Then do it again, 1,000 times, with a new coin each time. The notebook does exactly this. The chart shows the 1,000 gaps, and the users’ gap:

1,000 coin flips for the same 40,000 customers. 95% of the gaps fall between −2.52 and +2.76 dollars, and the largest is 4.85. The users' gap, +10.49, is far outside.
Text description of the chart

A histogram of 1,000 gaps in spending before the launch, one for each coin flip. The gaps form a hill centred on 0 dollars. 95% of them are between −2.52 and +2.76 dollars, and none is larger than 4.85 dollars either way. A line on the right marks the gap between the users of the block and the other customers: +10.49 dollars, more than twice the largest coin gap.

The coin’s gaps are small, and they fall on both sides of zero. The users’ gap is far outside all of them. It is not bad luck. It is self-selection.

A second way to see it: the standardised difference. It is the gap between the group means, divided by the typical spread of the column, so you can compare columns with different units. Values near 0 mean “alike”. The notebook computes it for four columns:

  • Users of the block against the rest
  • Coin flip (seed 1)
The users differ from the rest in all four columns. The coin's groups are within 0.015 of zero in all four.
Text description of the chart

A chart with eight dots on an axis from −0.2 to +0.2, with a line at 0 labelled “alike”. For each of four columns, one dot shows the users of the block against the other customers, and one shows two coin groups. Users: spending before +0.076, orders before +0.104, days as a customer +0.143, new customer −0.146. Coin: +0.001, −0.003, 0.000 and −0.014. The coin’s dots sit on the zero line; the users’ dots are away from it.

A common rule of thumb calls a standardised difference below 0.1 “balanced”. By that rule, the users’ spending before (+0.076) looks fine. The histogram shows that it is not: the gap is far larger than any coin makes. Use the rule as a quick screen, not as proof.

Rafael’s reminder-email test in the simulator works like the bakery’s second try. TermAmiraLarkfield’s developer. She writes the code of the help-desk tools in the courses.’s assignment code flips a coin for each customer. The line is in larkfield_experiments/ab.py:

variant = (rng.random(n) < share_b).astype(int) # the coin flip

rng.random(n) gives one random number between 0 and 1 for each of the n customers. A number below share_b (0.5) puts the customer in B. So each customer has a 50% chance to be in B, whatever they did before.

You ran this test in Meet the simulator. The test’s estimate was +1.47 points, with a 95% TermConfidence intervalA range around a measured value, made by a method that holds the true value 95 times in 100. If the interval of a difference does not include zero, the difference is probably not chance.Learn more Estimate uncertaintyEssential Mathematics and Statistics for AI of +1.00 to +1.94. The true effect, +1.19 points, is inside the interval. The users-against-non-users answer on the logs was 3.6 times the truth. In the guided practice below, you check why the test works: its groups were alike before the email.

The coin is flipped for something. That something is the unit of randomisation. The unit is the thing that gets A or B, and that stays in its group for the whole test. It is a design choice, and it matters as much as the coin itself.

Unit Example Good when Breaks when
Customer (or user, account) the free-delivery threshold, the reminder email the change is something a person notices and remembers customers share something that the change uses up, such as stock
Session (one visit) the colour of a button the change has no memory: what you saw last time does not matter the change teaches the customer something, or the result is counted per customer
Order a payment page each order is a separate decision the same customer gets both versions and compares them
Region or store (a cluster) a delivery promise for a whole province the change works for everyone in the region together there are only a few regions, so the groups cannot be alike

The rule: choose a unit that is at least as large as the place where the effect spreads. If what a customer sees in one visit changes what they do in the next visit, then the customer is the smallest safe unit.

Two rows. Top, a coin flipped once for each customer: Ana's three visits on Monday, Wednesday and Saturday are all B; Ana is in B for the whole test. Bottom, a coin flipped for each session: Monday B, Wednesday A, Saturday B, with a dashed red arrow 'carries over' from Monday to Wednesday; 50.3% of customers saw both versions, and the interval missed the truth.
One customer, three visits. Flipped per customer, Ana stays in B. Flipped per session, she sees both versions, and what she saw in a B visit changes what she does in an A visit.
Text description of the diagram

The diagram has two rows. In the top row, the coin is flipped once for each customer. Ana’s three visits, on Monday, Wednesday and Saturday, are all in B. Ana is in B for the whole test, everything she buys counts for B, and group A stays a clean “no block” group. Each customer sees one version. In the bottom row, the coin is flipped for each session. Ana’s Monday visit is B, her Wednesday visit is A and her Saturday visit is B. A dashed red arrow labelled “carries over” goes from Monday to Wednesday: what she saw in the B visit changes what she buys in the A visit, so A’s numbers are no longer “no block”. In the simulator, 50.3% of customers saw both versions, and the test’s interval missed the truth in 96 of 100 seeds.

Suppose Amira had shown the recommendation block by session: a new coin flip at every visit. That is simple to build and gives many more units. The simulator has this scenario too; you run it in Module 4, in When units affect each other. Here is what it finds (simulated, a practice problem we made on purpose):

  • 50.3% of the customers saw both versions during the test.
  • The block’s effect lasts after the visit. Part of it shows up in the customer’s later A sessions, so it is counted for A.
  • The test’s estimate was +0.3134 dollars per session. The block’s true effect is +0.7264. The interval, +0.0982 to +0.5287, does not contain the truth.
Randomised by session, the test finds less than half of the block's true effect, and its 95% interval misses the truth. Over 100 seeds, the interval contains the truth in 4%.
Text description of the chart

Two rows on an axis from 0 to 0.9 dollars per session. The first row is the estimate of the test randomised by session: +0.3134 dollars, with a 95% interval from +0.0982 to +0.5287. The second row is the true effect from the simulator: +0.7264 dollars. The true effect is to the right of the whole interval.

This is not bad luck. Over 100 seeds, the interval contained the truth in only 4% of the runs, and the estimate was 0.447 dollars per session too low on average. A sound test’s 95% interval contains the truth in about 95% of runs. The session design is wrong every time, by the same amount, in the same direction.

One more rule follows from the unit: analyse at the unit you randomised. The coin decided per customer, so each customer is one independent observation. The visits of one customer are not independent: a keen customer has many visits and many orders. That is why xp analyse first adds up each customer’s visits, and then compares customers. The mathematics course explains why dependent rows make an interval too narrow: Tickets that are not independent.

Use randomisation whenever you can control who gets the change: a page, a price, an email, a prompt. It is the only design that removes confounders that you did not measure.

You cannot randomise in some cases. The change happened already (the block went live for everyone). It must apply to a whole region by law. Or holding it back would be unfair or unsafe. Module 5 and Module 6 show what you can do then, and how far those methods land from the truth.

What it costs:

  • Half your customers do not get the change during the test. If B is better, they miss it for a while. If B is worse, the other half pays.
  • You need enough units. A coin needs numbers to make the groups alike: 8 customers are not enough, 40,000 are. Sample size and power computes how many.
  • You need engineering: a stable assignment that keeps each customer in the same group on every visit and device, and a log of who got what. Amira owns that code.
  1. In larkfield-experiments, with the TermVirtual environmentA folder with its own Python and its own packages, for one project. Packages that you install there do not change other projects.Learn more Isolate dependenciesPython for Practical AI Engineering active, run the logs scenario and its naive answer:

    Terminal window
    sim run --scenario logs
    xp naive
    xp check
    Scenario logs (seed 40): 28 days of logs after the recommendation block's launch: who used it and what they spent (no test).
    Wrote runs/logs-s40: customers (40,000 rows). Synthetic data. 0.3 s
    Customers who used the block: 13,338, mean revenue 39.19 dollars
    Customers who did not: 26,662, mean revenue 29.48 dollars
    Difference: +9.70 dollars (+32.9%).
    Saved to runs/logs-s40/analysis.json. Compare with the true effect: xp check
    Run logs-s40: the last analysis was `xp naive`.
    Naive answer: users spent +9.70 dollars more (+32.9%).
    True effect of the block on its users (from the simulator): +2.73 dollars per customer.
    The naive answer is 3.6 times the true effect: the users were different customers before they used the block.

    The time in the second line can be different on your computer.

  2. Run the email test, its analysis and the check:

    Terminal window
    sim run --scenario email
    xp analyse
    xp check

    The last lines are:

    difference +0.0147 (+16.4%), 95% interval +0.0100 to +0.0194, p = 1.14e-09 (Welch t-test)
    Saved to runs/email-s40/analysis.json. Compare with the true effect: xp check
    Run email-s40: the last analysis was `xp analyse`.
    True effect (from the simulator): +0.0119 share of customers with an order.
    Your estimate: +0.0147, 95% interval +0.0100 to +0.0194: the interval contains the true effect.
  3. Make a file balance.py in the project folder, with this code. It is a helper for this lesson, not part of the project:

    """Compare the two groups of an email test before the email. Usage: python balance.py email-s40"""
    import sys
    import pandas as pd
    df = pd.read_csv(f"runs/{sys.argv[1]}/customers.csv.gz")
    before = ["spend_12m", "orders_12m", "tenure_days", "new_customer", "email_open_rate"]
    print(df.groupby("variant")[before].mean().round(3).T)

    The five columns describe each customer before the email: spending and orders in the last 12 months, days as a customer, new or not, and the share of past emails they opened.

  4. Run it:

    Terminal window
    python balance.py email-s40
    variant A B
    spend_12m 313.894 313.285
    orders_12m 3.683 3.678
    tenure_days 1027.661 1031.862
    new_customer 0.252 0.252
    email_open_rate 0.301 0.302
  5. Compare the two columns. A and B spent the same before the email, to within 61 cents of 314 dollars. They are equally new and open as many emails. So the gap in orders after the email (+1.47 points) can only come from the email, or from chance. The interval measures the chance.

Each seed of the simulator is a new set of customers and a new coin. Predict first: with seed 41, will the groups still be alike before the email? Will the interval still contain the truth? Then run the email test with --seed 41, analyse and check it, and run balance.py on the new run (its folder is email-s41).

Show the answer
Terminal window
sim run --scenario email --seed 41
xp analyse
xp check
python balance.py email-s41

The last lines of xp analyse and xp check:

difference +0.0152 (+17.0%), 95% interval +0.0105 to +0.0199, p = 3.22e-10 (Welch t-test)
Saved to runs/email-s41/analysis.json. Compare with the true effect: xp check
Run email-s41: the last analysis was `xp analyse`.
True effect (from the simulator): +0.0145 share of customers with an order.
Your estimate: +0.0152, 95% interval +0.0105 to +0.0199: the interval contains the true effect.

And the groups before the email:

variant A B
spend_12m 312.466 313.736
orders_12m 3.677 3.669
tenure_days 1030.488 1029.975
new_customer 0.247 0.252
email_open_rate 0.301 0.301

The groups are alike again, with different small gaps: a new coin makes new luck. The true effect is different too (+1.45 points), because seed 41 has other customers. The interval contains it. Over 200 seeds, the email test’s interval contained the truth in 97% of runs.

Common mistake

The first two weeks get A, the last two weeks get B

Symptom: TermTomásLarkfield’s AI engineer. He often suggests a quick shortcut, and the lesson shows why it fails. proposes a simpler design for the free-delivery test: “Keep 75 dollars for the first two weeks, then switch to 50 dollars for two weeks. No coin, no assignment code.” B wins by a lot.

Cause: time is a confounder. The two halves of the test are different weeks, and weeks differ: the season moves, and holidays and sales come and go. Suppose the switch to 50 dollars came on 6 October 2025. Then Larkfield’s autumn sale (9 to 19 October) and Thanksgiving (13 October) would fall in B’s weeks only. B would win with no help from the change. The threshold and the calendar move together, so the test cannot separate them.

Fix: flip the coin for each customer, and run A and B at the same time, for the same full weeks. Then the sale and the holiday fall on both groups alike.

Consequence in AI work: “before and after” designs come back in many forms. Examples are the old model in May and the new model in June, or the old prompt last week and the new prompt this week. Each one confounds the change with the calendar. Run the old and the new version side by side, on a random split of the users.

Check your understanding

Omar says: 'The users of the block spent 32.9% more, and the groups are big, so the result is reliable.' Explain why the size of the groups does not help, and what would.

Show the answer

The customers chose to use the block, and keen customers chose it more often. The users already spent 51.77 dollars before the launch, against 41.28 for the others. Self-selection gives a gap with a direction, and a bigger sample does not remove it: it only measures the wrong gap more precisely. The naive gap, +9.70 dollars, is 3.6 times the true effect on the users (+2.73). A coin flip would help. It ignores the customer, so the groups are alike before the change in every column, even the hidden ones such as owning a garden.

Check your understanding

Predict: you flip a coin for 8 customers, and for 40,000 customers. In which case can the two groups differ a lot before the change, and why is that not the same problem as self-selection?

Show the answer

With 8 customers, the coin can easily put the big spenders together: in the lesson’s example the gap was 2.50 dollars, and another flip could give much more. With 40,000 customers, 95% of 1,000 coin flips gave gaps between −2.52 and +2.76 dollars. The coin’s gap is chance: it has no direction, and it shrinks as the groups grow. The test’s interval already accounts for it. Self-selection’s gap has a direction, does not shrink with size, and no interval accounts for it.

Check your understanding

Amira wants to randomise the recommendation block by session, because it gives three times more units. What do you tell her? Which unit do you propose, and why?

Show the answer

The block’s effect carries over: what a customer sees in a B session also changes what they buy in their later A sessions. So the same customer sees both versions, and part of the block’s effect is counted for A. In the simulator, 50.3% of customers saw both, and the estimate was +0.31 dollars per session against a true +0.73. Its interval contained the truth in only 4% of 100 seeds. Propose the customer as the unit: each customer stays in one group for the whole test, and the analysis compares customers.

Recap

  • When people choose their own group, the groups differ before the change. Larkfield’s users of the block spent 51.77 dollars before the launch, against 41.28: self-selection, a confounder.
  • Randomisation lets a coin decide. The coin ignores the customer, so the groups are alike before the change in every column, also the ones that you did not measure.
  • A coin still leaves a gap from chance. With 40,000 customers, 95% of 1,000 flips gave gaps between −2.52 and +2.76 dollars. The users’ gap was +10.49.
  • The unit of randomisation must be at least as large as the place where the effect spreads. Randomised by session, half the customers saw both versions, and the interval contained the truth in 4% of seeds.
  • Analyse at the unit that you randomised: one row per customer.

The groups are now fair. Next, you decide what to measure: one primary metric for Hana’s free-delivery test, and the guardrails that must not get worse.

Next: Choose the metrics