Skip to content
Français

When units affect each other

  • Lesson 16
  • 35 min
  • Module 4: Traps in real tests

Where we are

So far, one customer's version did not change another customer's result. In a shop with shared stock, it can. In this lesson, we see how shared stock and randomisation by session bias a test, and how cluster randomisation and switchback tests help.

What you will be able to do

Find tests where one customer’s version changes another customer’s result. Shared stock, shared budgets and customers who see both versions make a test wrong in a way that no check of the split can find. Randomise bigger units: whole regions, or whole days.

NotebookDo this lesson in a notebookColab · Kaggle · GitHub

In the notebook, you simulate a shop with one best seller and short stock. You compare a test by customer with the true effect of a launch, then run a switchback test and check its coverage over 100 seeds. Its setup cells install the packages, so it needs no local setup. About 25 minutes.

Colab and Kaggle open your own copy. They need a free Google or Kaggle account. Stuck? See the solution notebook.

Before you startWhat to know and to have readyAbout 35 minutes

What you need to know

What you need on your computer (only for the project steps)

  • The project larkfield-experiments at the snapshot end-of-m04, from Peeking, with .venv active.

Tested on macOS with Python 3.12 and the end-of-m04 snapshot of the project. Windows was not tested.

A school canteen makes 100 desserts a day. To test a new idea, half of the pupils get a card: “Go to the dessert counter first.” At the end of the month, the card pupils ate many more desserts than the others. The test says: the card works.

But the canteen still made 100 desserts a day, and they were all eaten, before and during the test. The card pupils took desserts that the other pupils would have eaten. If every pupil got a card, nobody would eat more. The test measured a race between the two groups, not the effect of the card.

TermLarkfieldA made-up Canadian online shop for home and garden products. It is not a real company. All Nextia courses use its help desk and data as the example.Learn more Meet Larkfield tested a “popular now” badge on its 40 best sellers. A coin gave each customer version A (no badge) or B (the badge). The stock of each best seller is short: each day, the shop has about as many as group A would buy. The simulator has this test as a scenario. Its shared stock is a practice problem we made on purpose.

  1. Run the badge test.

    Terminal window
    sim run --scenario stock
    Scenario stock (seed 40): A 'popular now' badge on best sellers with short stock, tested by customer (planted interference).
    Wrote runs/stock-s40: customers (83,052 rows), visits (268,761 rows). Synthetic data. 4.5 s
  2. Check the split first, as in Module 3.

    Terminal window
    xp srm stock
    Run stock-s40: A 41,488, B 41,564 customers; B is 50.05% (planned 50%).
    chi-square 0.1, p = 0.792: no mismatch.
  3. Analyse it, and compare it with the truth.

    Terminal window
    xp analyse
    xp check
    Run stock-s40 (synthetic): A 'popular now' badge on best sellers with short stock, tested by customer (planted interference).
    Metric: revenue. A: 41,488 customers, mean 22.2288. B: 41,564 customers, mean 25.0801.
    difference +2.8513 (+12.8%), 95% interval +1.8972 to +3.8053, p = 4.71e-09 (Welch t-test)
    Saved to runs/stock-s40/analysis.json. Compare with the true effect: xp check
    Run stock-s40: the last analysis was `xp analyse`.
    True effect (from the simulator): +0.6869 dollars per customer, badge for everyone against nobody.
    Your estimate: +2.8513, 95% interval +1.8972 to +3.8053: the interval does NOT contain the true effect.

The split is perfect. The test says +2.85 dollars per customer (+12.8%), with a tiny p-value. The truth is +0.69: the badge for everyone against the badge for nobody. The test is four times too large. Over 100 seeds, its TermConfidence intervalA range around a measured value, made by a method that holds the true value 95 times in 100. If the interval of a difference does not include zero, the difference is probably not chance.Learn more Estimate uncertaintyEssential Mathematics and Statistics for AI never contained the truth (coverage 0%), and it was 2.60 dollars too high on average.

The simulator can run each customer in three worlds: nobody has the badge, everyone has it, and the test. The table shows revenue per customer, in dollars.

World Group A Group B
Nobody has the badge about 23.21 about 23.21
Everyone has the badge about 23.90 about 23.90
The test 22.23 25.08

In the test, B is about 1.87 dollars above the no-badge world, and A is about 0.98 below it. A lost money that it would have had without the test: B customers bought the best sellers first, and A customers found them sold out. The test’s +2.85 counts B’s gain and A’s loss together. A launch has no A group to take from, so it gives only +0.69.

Group B, which sees the badge popular now and buys best sellers sooner, and group A, with no badge, both take from one shelf: today's stock of each best seller. B's arrow says buys first. A's dashed red arrow says later: sold out. From the shelf, two results: the test compares B with A, +2.85 dollars per customer, in a red box; badge for everyone against nobody, +0.69 dollars per customer, in a green box. A note says: B's gain is partly A's loss: the test counts it twice.
Shared stock. Both groups buy from one shelf. B takes units that A would have bought, so the difference between the groups is much larger than the effect of a launch.

A test compares two groups and assumes that each customer’s result depends only on their own version. Interference is when that is false: one unit’s version changes another unit’s result. Statisticians call the assumption SUTVA (the stable unit treatment value assumption). You only need its meaning: no unit affects another.

Interference makes a test wrong in a way that no check finds. The split was 50.05%. The groups were alike before the test. The interval was narrow. Everything looked healthy, and the answer was four times too large.

Look for interference wherever the groups share something limited:

  • Stock: a product that both groups buy, as here.
  • Budgets: in an advertising auction, B’s better bids spend a shared daily budget, and A’s ads run out early.
  • Marketplaces: in a ride or delivery app, B’s riders take the drivers that A’s riders would have had.
  • People who talk: friends who share a discount code, or a review that both groups read.

Interference can also come from the unit of randomisation itself. TermAmiraLarkfield’s developer. She writes the code of the help-desk tools in the courses. once randomised the recommendation block by session (one visit) instead of by customer. Each visit got its own coin flip. The block helps in the visit where it shows, and it also helps later: a customer who saw it comes back to the products it showed. That later help reaches A visits too.

  1. Run the block, randomised by session.

    Terminal window
    sim run --scenario session
    Scenario session (seed 40): The recommendation block randomised by session: customers see both versions (planted).
    Wrote runs/session-s40: visits (357,952 rows), customers (110,804 rows). Synthetic data. 12.4 s
  2. Analyse it, and compare it with the truth.

    Terminal window
    xp analyse
    xp check
    Run session-s40 (synthetic): The recommendation block randomised by session: customers see both versions (planted).
    Metric: revenue. A: 179,188 sessions, mean 8.1237. B: 178,764 sessions, mean 8.4372.
    difference +0.3134 (+3.9%), 95% interval +0.0982 to +0.5287, p = 0.00432 (Welch t-test)
    Saved to runs/session-s40/analysis.json. Compare with the true effect: xp check
    Run session-s40: the last analysis was `xp analyse`.
    True effect (from the simulator): +0.7264 dollars per session, if everyone got the block.
    Your estimate: +0.3134, 95% interval +0.0982 to +0.5287: the interval does NOT contain the true effect.

Here 50.3% of the customers saw both versions. The test says +0.31 dollars per session; a launch gives +0.73. This time the test is too small: A visits got part of B’s effect, so the gap between them shrank. Over 100 seeds, coverage was 4%. You chose the unit in Randomise: this is why the customer, not the visit, is the usual unit.

If units affect each other, put the units that share something in the same group. Then A and B share nothing, and the comparison is fair again.

Four ways to split, each with a row of squares marked A or B. By customer: customer 1 is B on every day and customer 2 is A on every day; a red box says A and B share the stock, B takes A's orders. By session: customer 1 sees A, B, B, A and customer 2 sees B, A, A, B; a red box says each customer sees both, B's effect leaks into A. By region, a cluster: Quebec is B on every day and Ontario is A; an orange box says no sharing between regions, but few units, wide interval. By day, a switchback: everyone gets B, A, A, B on four days; an orange box says no sharing within a day, but 28 units, wide interval.
Four ways to split a test. Splitting by customer or by session lets A and B affect each other here. Splitting by region or by day keeps them apart, but leaves few units.

Cluster randomisation flips the coin for a whole group of customers who share something: a region, a store, a warehouse. All of Quebec gets B, all of Ontario gets A. Customers in different regions do not share stock, so they cannot take from each other.

A switchback test flips the coin for a time period: each day, everyone gets the same version. A and B never share a day’s stock. The project’s switchback test switches by day, with a coin in each pair of days, for 28 days.

  1. Run the badge as a switchback test.

    Terminal window
    sim run --scenario switchback
    Scenario switchback (seed 40): The 'popular now' badge with short stock, switched by day for everyone (a switchback test).
    Wrote runs/switchback-s40: days (28 rows), visits (268,761 rows). Synthetic data. 4.9 s
  2. Analyse it. The unit is now a day, so the metric is revenue per visit on each day.

    Terminal window
    xp analyse
    xp check
    Run switchback-s40 (synthetic): The 'popular now' badge with short stock, switched by day for everyone (a switchback test).
    Metric: revenue_per_visit. A: 14 days, mean 7.2323. B: 14 days, mean 7.3049.
    difference +0.0726 (+1.0%), 95% interval -0.1382 to +0.2834, p = 0.485 (Welch t-test)
    Saved to runs/switchback-s40/analysis.json. Compare with the true effect: xp check
    Run switchback-s40: the last analysis was `xp analyse`.
    True effect (from the simulator): +0.2123 dollars per visit, badge for everyone against nobody (approximate).
    Your estimate: +0.0726, 95% interval -0.1382 to +0.2834: the interval contains the true effect.

    The truth per visit is approximate: the simulator knows it per customer, and converts it with the number of visits.

The three planted tests against the true effect of a launch. The test by customer and the test by session both miss the truth, in opposite directions. The switchback interval is wide, and it contains the truth.
Text description of the chart

Three panels. Shared stock, by customer, in dollars per customer: the test gives +2.85 (+1.90 to +3.81). The true effect, +0.69, is far to the left of the interval. Shared stock, switched by day, in dollars per visit: the test +0.07 (−0.14 to +0.28) against a true effect of +0.21, inside the interval. Recommendation block, by session, in dollars per session: the test +0.31 (+0.10 to +0.53) against a true effect of +0.73, to the right of the interval.

The switchback estimate is +0.07 dollars per visit, and the truth is +0.21. The estimate is off. But the interval (−0.14 to +0.28) contains the truth. Over 100 seeds, it contained the truth 96% of the time. It is honest, and it is wide, because 28 days are only 28 units. Here it cannot even say that the badge helps (p = 0.49).

Design Fixes Costs
By customer Nothing, if customers share stock Precise and wrong
By session Nothing; makes things worse Customers see both versions
By cluster (region, store) Sharing between clusters Few clusters, so wide intervals; clusters differ from each other
Switchback (by day or hour) Sharing within a period Few periods; carry-over from one period to the next

Carry-over means that one period’s version still acts in the next one. A customer who saw the badge on Monday may still buy that product on Tuesday, an A day. Teams leave a gap between periods, or switch by longer periods. Both cost time.

When the units are few, plan for it. Run longer, use more clusters, or accept a wider interval. A wide honest interval is better than a narrow wrong one.

In search and recommendations, a list mixes many items, and a test can show both rankers in one list. This is interleaving: each search shows a mix of A’s and B’s results, and clicks decide which ranker wins. It needs far fewer searches than an A/B test. See Interleaving in Search and Ranking.

Guided practice: a shop with one best seller

Section titled “Guided practice: a shop with one best seller”
  1. Open the notebook from the box Do this lesson in a notebook at the top of this page.
  2. Run the setup cells and the helper cell.
  3. In section 1, predict: will the badge group get more than half of the 100 units? Then run one day.
  4. In section 2, run the test by customer and the two worlds. The test’s difference is several times the true effect of a launch.
  5. In section 3, run a switchback test. Its estimate is close to the truth, with a wide interval.
  6. In section 4, compute coverage: the share of 100 seeds whose switchback interval contains the truth.
  7. Run the check cell. It prints Check passed.
Show the answer to step 6
runs = [switchback(s) for s in range(100)]
coverage = round(float(np.mean([low <= truth <= high for est, low, high, truth in runs])), 2)
print(coverage)

The switchback interval contains the truth in about 95 of 100 seeds.

Mateo asks: “We check every test for a sample ratio mismatch. Would that have caught the shared stock?” Predict first, then look again at step 2 of the badge test.

Show the answer

No. xp srm stock gave B 50.05%, p = 0.792: no mismatch. The coin worked, and no customer was lost. Interference does not change who is in each group. It changes what happens to them. No check of the split, the logs or an A/A test can find it. You find it by thinking about what the groups share before the test, and by choosing the unit in the plan.

Common mistake

“The badge adds 12.8%”

Symptom: the badge test reports +2.85 dollars per customer (+12.8%), p = 0.000000005. The team launches the badge for everyone. Revenue per customer rises by about 0.69 dollars (+3.0%), a quarter of the promise.

Cause: interference through shared stock. B customers bought best sellers that A customers would have bought. The test counted B’s gain and A’s loss. A launch has no A group to take from.

Fix: before the test, ask: “Do A and B share anything limited?” If they do, randomise bigger units: a switchback test by day, or clusters such as regions. Accept a wider interval. Write the unit and the reason in the plan.

Check your understanding

Which of these tests risks interference? (a) A new email subject line, tested by customer. (b) A lower delivery price for B customers, when the shop has only 20 delivery slots a day. (c) A new colour for the search button, tested by customer. Say why for each.

Show the answer

(a) Low risk: one customer’s email does not change another customer’s result, unless they share codes. (b) High risk: B customers book the limited slots first, and A customers find none, so the test shows a larger difference than a launch would give. Use a switchback test or clusters. (c) Low risk: one customer’s button does not change another’s search.

Check your understanding

The switchback test of the badge gave +0.07 dollars per visit, with an interval from −0.14 to +0.28 and p = 0.49. Omar says: “So the badge does nothing.” What do you answer?

Show the answer

The interval contains zero, but it also contains the true effect (+0.21 dollars per visit), and values up to +0.28. The test is honest but imprecise, because it has only 28 units: the days. It cannot tell “nothing” from “a small gain”. To decide, run the switchback test longer, or switch by shorter periods if carry-over allows, to get more units.

Check your understanding

The recommendation block was randomised by session, and 50.3% of the customers saw both versions. Why is the estimate too small, not too large?

Show the answer

The block helps in later visits too: a customer who saw it once comes back to its products. When a customer has both A and B visits, their A visits also get some of the block’s help. So the A group’s revenue rises, and the gap between B and A shrinks. The test said +0.31 dollars per session; a launch gives +0.73. Randomising by customer keeps each customer’s visits in one group.

Recap

  • Interference is when one unit’s version changes another unit’s result. The test then measures a race between the groups, not the effect of a launch.
  • The badge test with shared stock (a practice problem we made on purpose) said +2.85 dollars per customer; the launch gives +0.69. Coverage over 100 seeds: 0%. The split was perfect, so no check found it.
  • Randomising by session let 50.3% of customers see both versions: the estimate was too small (+0.31 against +0.73; coverage 4%).
  • Cluster randomisation (regions, stores) and switchback tests (by day) keep A and B apart. The switchback interval contained the truth in 96% of 100 seeds. But 28 days give a wide interval, and periods can carry over.
  • In search lists, interleaving shows both rankers in one list.

This ends the traps in running a test. Sometimes you cannot run a test at all. Next, you measure the effect of free delivery in Quebec without a coin flip.

Next: Difference-in-differences