Skip to content
Français

Get more precise with CUPED

  • Lesson 11
  • 35 min
  • Module 3: Analyse the test

Where we are

Our checks pass and our interval is honest. It is also wide: spending is noisy, and a real effect can hide in the noise. In this lesson, we use each customer's spending before the test to remove part of that noise with CUPED, and we see how many customers this saves.

Plays from YouTube. Nothing loads from YouTube until you press play. Privacy

What you will be able to do

Make a test more precise without more customers. Each customer’s spending before the test predicts part of their spending during it, and removing that part makes the TermConfidence intervalA range around a measured value, made by a method that holds the true value 95 times in 100. If the interval of a difference does not include zero, the difference is probably not chance.Learn more Estimate uncertaintyEssential Mathematics and Statistics for AI narrower.

NotebookDo this lesson in a notebookColab · Kaggle · GitHub

In the notebook, you compute CUPED by hand on the free-delivery data: theta, the adjusted spending, the narrower interval and the variance it removes. Then you try a covariate from the wrong time. Its setup cells install the packages and download the practice data, so it needs no local setup. About 25 minutes.

Colab and Kaggle open your own copy. They need a free Google or Kaggle account. Stuck? See the solution notebook.

Before you startWhat to know and to have readyAbout 35 minutes

What you need to know

What you need on your computer (only for the project steps)

  • The project larkfield-experiments at the snapshot end-of-m03, with .venv active, and the runs delivery, srm and aa from the last two lessons. If one is missing, make it again with sim run --scenario NAME.

Tested on macOS with Python 3.12 and the end-of-m03 snapshot of the project. Windows was not tested.

A running club tests new shoes. A coin gives half of the runners the new shoes. A month later, everyone runs 5 kilometres, and the club compares the mean times. The times go from 18 to 40 minutes, because some runners are much faster than others. The new shoes may save 20 seconds. That is hard to see between times that differ by 20 minutes.

The coach has a better idea. Every runner ran the same 5 kilometres last month, before the shoes. So she compares each runner with their own past time. A fast runner who ran 19:00 last month and 18:40 now has improved by 20 seconds. The big differences between runners disappear, and the small effect of the shoes becomes easier to see.

The shoes did not change last month’s times. So last month is fair to use: it is the same in both groups, apart from chance.

The problem: customers differ more than the effect

Section titled “The problem: customers differ more than the effect”

TermLarkfieldA made-up Canadian online shop for home and garden products. It is not a real company. All Nextia courses use its help desk and data as the example.Learn more Meet Larkfield’s free-delivery test has the same problem. The interval for revenue per customer goes from +0.16 to +1.90 dollars: 1.74 dollars wide. The true effect is 0.70 dollars, and over 200 seeds the test finds it only 41.5% of the time.

The noise comes from the customers themselves. Some customers spend hundreds of dollars a month, and most spend nothing. The project has each customer’s spending in the 8 weeks before the test: pre_spend. The chart shows how much it predicts.

Free-delivery test, both groups together: revenue during the test by spending in the 8 weeks before it. 'Nothing' is the 69% of customers who spent nothing before. The others are cut into four groups of equal size. The highest spenders before the test spend the most during it.
Text description of the chart

Five bars. Customers who spent nothing before the test: 14.34 dollars during the test. Low spenders before (2 to 53 dollars): 19.45. Medium (53 to 95 dollars): 30.10. High (95 to 178 dollars): 44.21. Highest (178 dollars and more): 107.95.

Spending before the test predicts spending during it. The correlation measures how closely two values move together: 0 means not at all, and 1 means exactly on a straight line. Here it is 0.44. This part of the noise has nothing to do with free delivery, so we can remove it.

We adjust each customer’s spending during the test (y) by their spending before it (x):

adjusted = y − θ × (x − mean of x)

θ (theta) is how many dollars of spending during the test go with each dollar of spending before it. It is the slope of a straight line through the customers: θ = covariance(y, x) / variance(x). The covariance is like the correlation, but in the units of the data: dollars times dollars. We compute one θ from both groups together, so the adjustment is the same rule for A and for B.

A customer who spent more than average before the test gets a lower adjusted value. A customer who spent less gets a higher one. What is left is the part of their spending that their past does not explain. The change, if it works, is in that part.

Here are six customers, made up so that you can see every value. The mean of x is 60 dollars, and θ = 0.559.

Customer Before the test (x) During the test (y) x − 60 0.559 × (x − 60) Adjusted
1 0 10 −60 −33.5 43.5
2 20 0 −40 −22.4 22.4
3 40 50 −20 −11.2 61.2
4 60 30 0 0 30.0
5 100 70 40 22.4 47.6
6 140 80 80 44.7 35.3

Customer 1 spent nothing before and 10 during the test, so CUPED adds 33.5 dollars. Customer 6 spent 140 before, so CUPED takes away 44.7. The mean of y and the mean of the adjusted values are both 40 dollars: CUPED does not move the average. But the variance falls from 1,040 to 191: 82% less.

The six customers of the tiny example, before and after CUPED (a constructed example). The mean stays at 40 dollars, and the spread becomes much smaller.
Text description of the chart

Two rows of six dots. Spending in the test: 0, 10, 30, 50, 70 and 80 dollars, spread over 80 dollars. After CUPED, rounded: 22, 30, 35, 44, 48 and 61 dollars, spread over 39 dollars. Both rows have a mean of 40.

The coin chose the group after the 8 weeks of pre_spend. So free delivery could not change pre_spend, and the coin made it the same in both groups, apart from chance. Subtracting it removes noise, not the effect. If one group had a few more big spenders by chance, CUPED corrects for that too.

The share of the variance that CUPED removes is the squared correlation of x and y. In the tiny example, the correlation is 0.90, so 0.90² = 82%. At Larkfield, the correlation is 0.44, so 0.44² = 19%.

This method is CUPED: Controlled-experiment Using Pre-Experiment Data. Alex Deng, Ya Xu, Ron Kohavi and Toby Walker published it at Microsoft in 2013. It is a form of regression adjustment with one covariate. A covariate is a value that you measure for each unit, here each customer, besides the outcome.

The project’s CUPED is ten lines in larkfield_experiments/cuped.py:

def cuped_adjust(y, x):
y, x = np.asarray(y, float), np.asarray(x, float)
theta = np.cov(y, x, ddof=1)[0, 1] / x.var(ddof=1)
return y - theta * (x - x.mean()), float(theta)
def variance_reduction(y, x) -> float:
"""The share of the variance that CUPED removes: the squared correlation of outcome and covariate."""
return float(np.corrcoef(y, x)[0, 1] ** 2)

xp analyse --cuped adjusts the revenue of every customer with pre_spend, then runs the same Welch’s t-test on the adjusted values.

  1. Analyse the free-delivery test with CUPED.

    Terminal window
    xp analyse delivery --cuped
    Run delivery-s40 (synthetic): Free delivery from 50 dollars instead of 75: an A/B test by customer, 28 days.
    Metric: revenue. A: 55,430 customers, mean 25.0420. B: 55,329 customers, mean 26.0854.
    difference +1.0434 (+4.2%), 95% interval +0.2631 to +1.8237, p = 0.00877 (Welch t-test, CUPED)
    CUPED removed 19.3% of the variance (theta 0.271).
    Saved to runs/delivery-s40/analysis.json. Compare with the true effect: xp check
  2. Compare it with the truth. xp check checks the run that you analysed last, here delivery.

    Terminal window
    xp check
    Run delivery-s40: the last analysis was `xp analyse`.
    True effect (from the simulator): +0.6979 dollars per customer.
    Your estimate: +1.0434, 95% interval +0.2631 to +1.8237: the interval contains the true effect.

θ is 0.271: each dollar spent before the test goes with 27 cents spent during it. CUPED removed 19.3% of the variance. The interval is now +0.26 to +1.82 dollars: 1.56 wide instead of 1.74. It still contains the truth, +0.70.

The free-delivery test without and with CUPED, on the default seed. The estimate hardly moves, and the interval is about 10% narrower. Both contain the true effect.
Text description of the chart

Two rows on an axis from −0.5 to +2.5 dollars. Welch, no adjustment: +1.03, interval +0.16 to +1.90. Welch with CUPED: +1.04, interval +0.26 to +1.82. Dashed lines mark 0 (no effect) and the true effect, +0.70.

The estimate is still too high. CUPED removes noise, but this seed is still a lucky one. One run cannot show what a method does. Many runs can.

The chart shows the width of the interval on 20 seeds, with and without CUPED.

The width of the free-delivery test's interval on 20 seeds of the simulator, without and with CUPED. On every seed, CUPED's interval is narrower.
Text description of the chart

Two rows of 20 dots. Welch: widths from 1.71 to 1.79 dollars. CUPED: widths from 1.53 to 1.59 dollars. Every CUPED width is smaller than every Welch width.

Over 200 seeds, the course measured the following:

  • CUPED removed 20.3% of the variance on average.
  • The interval was 0.89 times as wide: 1.56 dollars instead of 1.75.
  • The interval still contained the truth: 94% of the runs, against 95.5% without CUPED.
  • The test found the gain in 48.5% of the runs, against 41.5%.

The number of customers that a test needs grows with the variance. So 20% less variance means about 20% fewer customers for the same precision. Or, with the same customers, a smaller effect becomes visible. The free-delivery test could find 1.23 dollars with 80% power. With CUPED, it can find about 1.10 dollars.

That is useful, but it is not magic. The true effect is 0.70 dollars, and the test is still too small for it. It finds the effect in only about half of the seeds. CUPED makes a test better; it does not replace a test of the right size.

Why only a fifth? Look at the chart of spending bands again: 69% of the customers spent nothing in the 8 weeks before the test. For them, pre_spend tells CUPED almost nothing. Shops whose customers buy more regularly often report larger gains, because their past spending predicts more.

Use CUPED when you have a value from before the test that predicts the metric. Past spending, past visits and past orders are good choices. Write the covariate in the test plan, so that you do not try ten covariates and keep the best interval.

Do not use it in these cases:

  • The covariate was measured during the test. The change can move it, and then CUPED removes part of the effect. The mistake below shows how much.

  • The metric is a ratio per order, such as returns per order. The project’s CUPED works on a value per customer and refuses a ratio:

    Terminal window
    xp analyse delivery --cuped --metric return_rate
    CUPED works on a metric per customer, such as revenue. 'return_rate' is a ratio (per order): run it without --cuped.
  • The test is broken. CUPED does not repair a sample ratio mismatch. On the test with the redirect bug, xp analyse srm --cuped gives +2.01 dollars (+1.22 to +2.80), and the truth is still +0.87. Run the checks first.

What it costs: one column of data from before the test, and a few lines of code. The cost is low, so many large test platforms use CUPED, or a regression with several covariates, by default.

  1. Open the notebook from the box Do this lesson in a notebook at the top of this page.
  2. Run the setup cells and the helper cell. The data cell ends with Ready: 2 files in data/.
  3. In section 1, run the tiny example. You see theta 0.559 and the variance fall from 1040.0 to 190.6.
  4. In section 2, predict: will θ for Larkfield be above or below 0.5? Then run the cell. You see theta 0.271 and correlation 0.439.
  5. In section 3, run the adjusted t-test. You see the interval +0.263 to +1.824, the same as xp analyse --cuped.
  6. In section 4, compare the two widths. You see width 1.737 without CUPED and 1.561 with it.
  7. In section 5, compute share_removed: the share of the variance that CUPED removes, from the correlation.
  8. Run the check cell. It prints Check passed.
Show the answer to step 7
share_removed = round(np.corrcoef(d.revenue, d.pre_spend)[0, 1] ** 2, 3)
print(share_removed)

The answer is 0.193, the same as the project’s CUPED removed 19.3% of the variance. The variance of the adjusted revenue is 1 − 0.193 = 80.7% of the variance of the revenue.

Change one thing: the test. Run CUPED on the A/A test from the last lesson. Without CUPED, it gave −0.84 dollars (−1.72 to +0.04), with p = 0.06. Predict first: will the estimate move? Toward zero, or away from it?

Show the answer
Terminal window
xp analyse aa --cuped
xp check aa
difference -0.3143 (-1.2%), 95% interval -1.0994 to +0.4708, p = 0.433 (Welch t-test, CUPED)
CUPED removed 20.4% of the variance (theta 0.275).
...
True effect (from the simulator): +0.0000 dollars per customer.
Your estimate: -0.3143, 95% interval -1.0994 to +0.4708: the interval contains the true effect.

The estimate moved from −0.84 to −0.31 dollars, toward the true zero, and the interval became narrower. By chance, group A had more big spenders before the test than group B. Without CUPED, that looked like a loss for B. CUPED corrected for it, because it compares each customer with their own past.

Common mistake

A covariate from the wrong time

Symptom: TermTomásLarkfield’s AI engineer. He often suggests a quick shortcut, and the lesson shows why it fails. wants a bigger gain in precision. He uses each customer’s number of orders during the test as the covariate, because it predicts revenue much better. The notebook shows: “CUPED removed 61.6% of the variance.” The estimate becomes −0.42 dollars (−0.96 to +0.12). The true effect is +0.70.

Cause: free delivery changed the number of orders: B’s customers ordered more. So the covariate carries part of the effect. CUPED subtracts it, and the effect disappears. Even the sign changes.

Fix: use only covariates measured before the coin flip. A covariate from before the test cannot carry the effect. Write it in the plan before the test starts. If a covariate removes far more variance than expected, ask first when it was measured.

Check your understanding

Explain CUPED to Hana in three sentences, without a formula. Say why it does not change the average effect.

Show the answer

Customers differ a lot in what they spend, and these differences hide the small effect of free delivery. CUPED compares each customer with their own spending before the test, so the differences between customers mostly disappear and the interval becomes narrower. Free delivery could not change the spending before the test, and the coin made it the same in both groups, apart from chance. So the adjustment removes noise, not the effect.

Check your understanding

Predict: a shop's covariate has a correlation of 0.7 with its metric. Roughly what share of the variance does CUPED remove? About how many customers does a test need, compared with no CUPED?

Show the answer

CUPED removes the squared correlation: 0.7² = 0.49, about half of the variance. The customers needed grow with the variance, so the test needs about half as many customers for the same precision. Larkfield’s correlation is only 0.44, so it removes about a fifth and saves about a fifth of the customers.

Check your understanding

Tomás says: 'CUPED gave a narrower interval on the test with the redirect bug, so now we can trust it.' What do you answer?

Show the answer

A narrower interval is not a correct one. CUPED reduces noise from the differences between customers. It cannot bring back the B customers that the redirect lost. On that test, CUPED gives +2.01 dollars (+1.22 to +2.80), and the truth is +0.87: still far outside the interval. The sample ratio check comes first. With a mismatch, no adjustment makes the result usable.

Recap

  • Customers differ much more than the effect of a change, and that noise makes intervals wide. Spending before the test predicts part of the spending during it.
  • CUPED subtracts that predictable part: adjusted = y − θ × (x − mean of x). θ = cov(y, x) / var(x), from both groups together. The mean stays the same; the variance falls by the squared correlation.
  • It is fair because the covariate comes from before the coin flip. The change cannot move it, and the coin made it the same in both groups, apart from chance.
  • On the free-delivery test, CUPED removed 19.3% of the variance (θ = 0.271). The interval became +0.26 to +1.82 instead of +0.16 to +1.90. Over 200 seeds: 20.3% of the variance, about 20% fewer customers, and power from 41.5% to 48.5%.
  • Use only covariates from before the test, on metrics per customer, and only on a healthy test. A covariate from during the test removes the effect itself.

You can now get a precise average effect. But TermOmarLarkfield’s product manager. He asks the business questions: what a change costs, and what it gives the shop. asks a new question: does free delivery work better on mobile, or for new customers? Next, you look at segments, and you see why many segments find false winners.

Next: Segments and different effects