When units affect each other
- Lesson 16
- 35 min
- Module 4: Traps in real tests
Where we are
So far, one customer's version did not change another customer's result. In a shop with shared stock, it can. In this lesson, we see how shared stock and randomisation by session bias a test, and how cluster randomisation and switchback tests help.
Lesson 16 of 30 · Experiments and Causal Impact: Did Our Change Work?When units affect each other
1. Why "it went up" is not an answer
2. Design an A/B test
3. Analyse the test
4. Traps in real tests
5. When you cannot randomise: natural experiments
6. Adjust for confounders in observational data
7. Uplift: who should get the treatment?
8. Experiments for AI features and in production
What you will be able to do
Find tests where one customer’s version changes another customer’s result. Shared stock, shared budgets and customers who see both versions make a test wrong in a way that no check of the split can find. Randomise bigger units: whole regions, or whole days.
NotebookDo this lesson in a notebook
In the notebook, you simulate a shop with one best seller and short stock. You compare a test by customer with the true effect of a launch, then run a switchback test and check its coverage over 100 seeds. Its setup cells install the packages, so it needs no local setup. About 25 minutes.
Colab and Kaggle open your own copy. They need a free Google or Kaggle account. Stuck? See the solution notebook.
Before you startWhat to know and to have ready
What you need to know
- The two worlds that the simulator compares, from Meet the simulator.
- The unit of randomisation, from Randomise.
- The sample ratio check, from Check the test before the result.
What you need on your computer (only for the project steps)
- The project
larkfield-experimentsat the snapshotend-of-m04, from Peeking, with.venvactive.
Tested on macOS with Python 3.12 and the end-of-m04 snapshot of the project. Windows was not tested.
Desserts at the school canteen
Section titled “Desserts at the school canteen”A school canteen makes 100 desserts a day. To test a new idea, half of the pupils get a card: “Go to the dessert counter first.” At the end of the month, the card pupils ate many more desserts than the others. The test says: the card works.
But the canteen still made 100 desserts a day, and they were all eaten, before and during the test. The card pupils took desserts that the other pupils would have eaten. If every pupil got a card, nobody would eat more. The test measured a race between the two groups, not the effect of the card.
The problem: a badge that wins a race
Section titled “The problem: a badge that wins a race”TermLarkfieldA made-up Canadian online shop for home and garden products. It is not a real company. All Nextia courses use its help desk and data as the example.Learn more Meet Larkfield tested a “popular now” badge on its 40 best sellers. A coin gave each customer version A (no badge) or B (the badge). The stock of each best seller is short: each day, the shop has about as many as group A would buy. The simulator has this test as a scenario. Its shared stock is a practice problem we made on purpose.
-
Run the badge test.
Terminal window sim run --scenario stockScenario stock (seed 40): A 'popular now' badge on best sellers with short stock, tested by customer (planted interference).Wrote runs/stock-s40: customers (83,052 rows), visits (268,761 rows). Synthetic data. 4.5 s -
Check the split first, as in Module 3.
Terminal window xp srm stockRun stock-s40: A 41,488, B 41,564 customers; B is 50.05% (planned 50%).chi-square 0.1, p = 0.792: no mismatch. -
Analyse it, and compare it with the truth.
Terminal window xp analysexp checkRun stock-s40 (synthetic): A 'popular now' badge on best sellers with short stock, tested by customer (planted interference).Metric: revenue. A: 41,488 customers, mean 22.2288. B: 41,564 customers, mean 25.0801.difference +2.8513 (+12.8%), 95% interval +1.8972 to +3.8053, p = 4.71e-09 (Welch t-test)Saved to runs/stock-s40/analysis.json. Compare with the true effect: xp checkRun stock-s40: the last analysis was `xp analyse`.True effect (from the simulator): +0.6869 dollars per customer, badge for everyone against nobody.Your estimate: +2.8513, 95% interval +1.8972 to +3.8053: the interval does NOT contain the true effect.
The split is perfect. The test says +2.85 dollars per customer (+12.8%), with a tiny p-value. The truth is +0.69: the badge for everyone against the badge for nobody. The test is four times too large. Over 100 seeds, its TermConfidence intervalA range around a measured value, made by a method that holds the true value 95 times in 100. If the interval of a difference does not include zero, the difference is probably not chance.Learn more Estimate uncertaintyEssential Mathematics and Statistics for AI never contained the truth (coverage 0%), and it was 2.60 dollars too high on average.
Where the extra 2 dollars come from
Section titled “Where the extra 2 dollars come from”The simulator can run each customer in three worlds: nobody has the badge, everyone has it, and the test. The table shows revenue per customer, in dollars.
| World | Group A | Group B |
|---|---|---|
| Nobody has the badge | about 23.21 | about 23.21 |
| Everyone has the badge | about 23.90 | about 23.90 |
| The test | 22.23 | 25.08 |
In the test, B is about 1.87 dollars above the no-badge world, and A is about 0.98 below it. A lost money that it would have had without the test: B customers bought the best sellers first, and A customers found them sold out. The test’s +2.85 counts B’s gain and A’s loss together. A launch has no A group to take from, so it gives only +0.69.
The idea: interference
Section titled “The idea: interference”A test compares two groups and assumes that each customer’s result depends only on their own version. Interference is when that is false: one unit’s version changes another unit’s result. Statisticians call the assumption SUTVA (the stable unit treatment value assumption). You only need its meaning: no unit affects another.
Interference makes a test wrong in a way that no check finds. The split was 50.05%. The groups were alike before the test. The interval was narrow. Everything looked healthy, and the answer was four times too large.
Look for interference wherever the groups share something limited:
- Stock: a product that both groups buy, as here.
- Budgets: in an advertising auction, B’s better bids spend a shared daily budget, and A’s ads run out early.
- Marketplaces: in a ride or delivery app, B’s riders take the drivers that A’s riders would have had.
- People who talk: friends who share a discount code, or a review that both groups read.
The second form: customers who see both
Section titled “The second form: customers who see both”Interference can also come from the unit of randomisation itself. TermAmiraLarkfield’s developer. She writes the code of the help-desk tools in the courses. once randomised the recommendation block by session (one visit) instead of by customer. Each visit got its own coin flip. The block helps in the visit where it shows, and it also helps later: a customer who saw it comes back to the products it showed. That later help reaches A visits too.
-
Run the block, randomised by session.
Terminal window sim run --scenario sessionScenario session (seed 40): The recommendation block randomised by session: customers see both versions (planted).Wrote runs/session-s40: visits (357,952 rows), customers (110,804 rows). Synthetic data. 12.4 s -
Analyse it, and compare it with the truth.
Terminal window xp analysexp checkRun session-s40 (synthetic): The recommendation block randomised by session: customers see both versions (planted).Metric: revenue. A: 179,188 sessions, mean 8.1237. B: 178,764 sessions, mean 8.4372.difference +0.3134 (+3.9%), 95% interval +0.0982 to +0.5287, p = 0.00432 (Welch t-test)Saved to runs/session-s40/analysis.json. Compare with the true effect: xp checkRun session-s40: the last analysis was `xp analyse`.True effect (from the simulator): +0.7264 dollars per session, if everyone got the block.Your estimate: +0.3134, 95% interval +0.0982 to +0.5287: the interval does NOT contain the true effect.
Here 50.3% of the customers saw both versions. The test says +0.31 dollars per session; a launch gives +0.73. This time the test is too small: A visits got part of B’s effect, so the gap between them shrank. Over 100 seeds, coverage was 4%. You chose the unit in Randomise: this is why the customer, not the visit, is the usual unit.
The fix: randomise bigger units
Section titled “The fix: randomise bigger units”If units affect each other, put the units that share something in the same group. Then A and B share nothing, and the comparison is fair again.
Cluster randomisation flips the coin for a whole group of customers who share something: a region, a store, a warehouse. All of Quebec gets B, all of Ontario gets A. Customers in different regions do not share stock, so they cannot take from each other.
A switchback test flips the coin for a time period: each day, everyone gets the same version. A and B never share a day’s stock. The project’s switchback test switches by day, with a coin in each pair of days, for 28 days.
-
Run the badge as a switchback test.
Terminal window sim run --scenario switchbackScenario switchback (seed 40): The 'popular now' badge with short stock, switched by day for everyone (a switchback test).Wrote runs/switchback-s40: days (28 rows), visits (268,761 rows). Synthetic data. 4.9 s -
Analyse it. The unit is now a day, so the metric is revenue per visit on each day.
Terminal window xp analysexp checkRun switchback-s40 (synthetic): The 'popular now' badge with short stock, switched by day for everyone (a switchback test).Metric: revenue_per_visit. A: 14 days, mean 7.2323. B: 14 days, mean 7.3049.difference +0.0726 (+1.0%), 95% interval -0.1382 to +0.2834, p = 0.485 (Welch t-test)Saved to runs/switchback-s40/analysis.json. Compare with the true effect: xp checkRun switchback-s40: the last analysis was `xp analyse`.True effect (from the simulator): +0.2123 dollars per visit, badge for everyone against nobody (approximate).Your estimate: +0.0726, 95% interval -0.1382 to +0.2834: the interval contains the true effect.The truth per visit is approximate: the simulator knows it per customer, and converts it with the number of visits.
Text description of the chart
Three panels. Shared stock, by customer, in dollars per customer: the test gives +2.85 (+1.90 to +3.81). The true effect, +0.69, is far to the left of the interval. Shared stock, switched by day, in dollars per visit: the test +0.07 (−0.14 to +0.28) against a true effect of +0.21, inside the interval. Recommendation block, by session, in dollars per session: the test +0.31 (+0.10 to +0.53) against a true effect of +0.73, to the right of the interval.
The switchback estimate is +0.07 dollars per visit, and the truth is +0.21. The estimate is off. But the interval (−0.14 to +0.28) contains the truth. Over 100 seeds, it contained the truth 96% of the time. It is honest, and it is wide, because 28 days are only 28 units. Here it cannot even say that the badge helps (p = 0.49).
What bigger units cost
Section titled “What bigger units cost”| Design | Fixes | Costs |
|---|---|---|
| By customer | Nothing, if customers share stock | Precise and wrong |
| By session | Nothing; makes things worse | Customers see both versions |
| By cluster (region, store) | Sharing between clusters | Few clusters, so wide intervals; clusters differ from each other |
| Switchback (by day or hour) | Sharing within a period | Few periods; carry-over from one period to the next |
Carry-over means that one period’s version still acts in the next one. A customer who saw the badge on Monday may still buy that product on Tuesday, an A day. Teams leave a gap between periods, or switch by longer periods. Both cost time.
When the units are few, plan for it. Run longer, use more clusters, or accept a wider interval. A wide honest interval is better than a narrow wrong one.
Search results: a special case
Section titled “Search results: a special case”In search and recommendations, a list mixes many items, and a test can show both rankers in one list. This is interleaving: each search shows a mix of A’s and B’s results, and clicks decide which ranker wins. It needs far fewer searches than an A/B test. See Interleaving in Search and Ranking.
Guided practice: a shop with one best seller
Section titled “Guided practice: a shop with one best seller”- Open the notebook from the box Do this lesson in a notebook at the top of this page.
- Run the setup cells and the helper cell.
- In section 1, predict: will the badge group get more than half of the 100 units? Then run one day.
- In section 2, run the test by customer and the two worlds. The test’s difference is several times the true effect of a launch.
- In section 3, run a switchback test. Its estimate is close to the truth, with a wide interval.
- In section 4, compute
coverage: the share of 100 seeds whose switchback interval contains the truth. - Run the check cell. It prints
Check passed.
Show the answer to step 6
runs = [switchback(s) for s in range(100)]coverage = round(float(np.mean([low <= truth <= high for est, low, high, truth in runs])), 2)print(coverage)The switchback interval contains the truth in about 95 of 100 seeds.
Your turn: would a check have warned you?
Section titled “Your turn: would a check have warned you?”Mateo asks: “We check every test for a sample ratio mismatch. Would that have caught the shared stock?” Predict first, then look again at step 2 of the badge test.
Show the answer
No. xp srm stock gave B 50.05%, p = 0.792: no mismatch. The coin worked, and no customer was lost. Interference does not change who is in each group. It changes what happens to them. No check of the split, the logs or an A/A test can find it. You find it by thinking about what the groups share before the test, and by choosing the unit in the plan.
Common mistake
“The badge adds 12.8%”
Symptom: the badge test reports +2.85 dollars per customer (+12.8%), p = 0.000000005. The team launches the badge for everyone. Revenue per customer rises by about 0.69 dollars (+3.0%), a quarter of the promise.
Cause: interference through shared stock. B customers bought best sellers that A customers would have bought. The test counted B’s gain and A’s loss. A launch has no A group to take from.
Fix: before the test, ask: “Do A and B share anything limited?” If they do, randomise bigger units: a switchback test by day, or clusters such as regions. Accept a wider interval. Write the unit and the reason in the plan.
Check your understanding
Which of these tests risks interference? (a) A new email subject line, tested by customer. (b) A lower delivery price for B customers, when the shop has only 20 delivery slots a day. (c) A new colour for the search button, tested by customer. Say why for each.
Show the answer
(a) Low risk: one customer’s email does not change another customer’s result, unless they share codes. (b) High risk: B customers book the limited slots first, and A customers find none, so the test shows a larger difference than a launch would give. Use a switchback test or clusters. (c) Low risk: one customer’s button does not change another’s search.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
The switchback test of the badge gave +0.07 dollars per visit, with an interval from −0.14 to +0.28 and p = 0.49. Omar says: “So the badge does nothing.” What do you answer?
Show the answer
The interval contains zero, but it also contains the true effect (+0.21 dollars per visit), and values up to +0.28. The test is honest but imprecise, because it has only 28 units: the days. It cannot tell “nothing” from “a small gain”. To decide, run the switchback test longer, or switch by shorter periods if carry-over allows, to get more units.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
The recommendation block was randomised by session, and 50.3% of the customers saw both versions. Why is the estimate too small, not too large?
Show the answer
The block helps in later visits too: a customer who saw it once comes back to its products. When a customer has both A and B visits, their A visits also get some of the block’s help. So the A group’s revenue rises, and the gap between B and A shrinks. The test said +0.31 dollars per session; a launch gives +0.73. Randomising by customer keeps each customer’s visits in one group.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Recap
- Interference is when one unit’s version changes another unit’s result. The test then measures a race between the groups, not the effect of a launch.
- The badge test with shared stock (a practice problem we made on purpose) said +2.85 dollars per customer; the launch gives +0.69. Coverage over 100 seeds: 0%. The split was perfect, so no check found it.
- Randomising by session let 50.3% of customers see both versions: the estimate was too small (+0.31 against +0.73; coverage 4%).
- Cluster randomisation (regions, stores) and switchback tests (by day) keep A and B apart. The switchback interval contained the truth in 96% of 100 seeds. But 28 days give a wide interval, and periods can carry over.
- In search lists, interleaving shows both rankers in one list.
This ends the traps in running a test. Sometimes you cannot run a test at all. Next, you measure the effect of free delivery in Quebec without a coin flip.