Get more precise with CUPED
- Lesson 11
- 35 min
- Module 3: Analyse the test
Where we are
Our checks pass and our interval is honest. It is also wide: spending is noisy, and a real effect can hide in the noise. In this lesson, we use each customer's spending before the test to remove part of that noise with CUPED, and we see how many customers this saves.
Lesson 11 of 30 · Experiments and Causal Impact: Did Our Change Work?Get more precise with CUPED
1. Why "it went up" is not an answer
2. Design an A/B test
3. Analyse the test
4. Traps in real tests
5. When you cannot randomise: natural experiments
6. Adjust for confounders in observational data
7. Uplift: who should get the treatment?
8. Experiments for AI features and in production
What you will be able to do
Make a test more precise without more customers. Each customer’s spending before the test predicts part of their spending during it, and removing that part makes the TermConfidence intervalA range around a measured value, made by a method that holds the true value 95 times in 100. If the interval of a difference does not include zero, the difference is probably not chance.Learn more Estimate uncertaintyEssential Mathematics and Statistics for AI narrower.
NotebookDo this lesson in a notebook
In the notebook, you compute CUPED by hand on the free-delivery data: theta, the adjusted spending, the narrower interval and the variance it removes. Then you try a covariate from the wrong time. Its setup cells install the packages and download the practice data, so it needs no local setup. About 25 minutes.
Colab and Kaggle open your own copy. They need a free Google or Kaggle account. Stuck? See the solution notebook.
Before you startWhat to know and to have ready
What you need to know
- Welch’s t-test, the interval and the free-delivery result, from The result and its uncertainty.
- The checks before the result, from Check the test before the result.
- The minimum detectable effect and the customers a test needs, from Sample size and power.
- Variance and standard deviation, from Describe a dataset in Essential Mathematics and Statistics for AI.
What you need on your computer (only for the project steps)
- The project
larkfield-experimentsat the snapshotend-of-m03, with.venvactive, and the runsdelivery,srmandaafrom the last two lessons. If one is missing, make it again withsim run --scenario NAME.
Tested on macOS with Python 3.12 and the end-of-m03 snapshot of the project. Windows was not tested.
New shoes at a running club
Section titled “New shoes at a running club”A running club tests new shoes. A coin gives half of the runners the new shoes. A month later, everyone runs 5 kilometres, and the club compares the mean times. The times go from 18 to 40 minutes, because some runners are much faster than others. The new shoes may save 20 seconds. That is hard to see between times that differ by 20 minutes.
The coach has a better idea. Every runner ran the same 5 kilometres last month, before the shoes. So she compares each runner with their own past time. A fast runner who ran 19:00 last month and 18:40 now has improved by 20 seconds. The big differences between runners disappear, and the small effect of the shoes becomes easier to see.
The shoes did not change last month’s times. So last month is fair to use: it is the same in both groups, apart from chance.
The problem: customers differ more than the effect
Section titled “The problem: customers differ more than the effect”TermLarkfieldA made-up Canadian online shop for home and garden products. It is not a real company. All Nextia courses use its help desk and data as the example.Learn more Meet Larkfield’s free-delivery test has the same problem. The interval for revenue per customer goes from +0.16 to +1.90 dollars: 1.74 dollars wide. The true effect is 0.70 dollars, and over 200 seeds the test finds it only 41.5% of the time.
The noise comes from the customers themselves. Some customers spend hundreds of dollars a month, and most spend nothing. The project has each customer’s spending in the 8 weeks before the test: pre_spend. The chart shows how much it predicts.
Text description of the chart
Five bars. Customers who spent nothing before the test: 14.34 dollars during the test. Low spenders before (2 to 53 dollars): 19.45. Medium (53 to 95 dollars): 30.10. High (95 to 178 dollars): 44.21. Highest (178 dollars and more): 107.95.
Spending before the test predicts spending during it. The correlation measures how closely two values move together: 0 means not at all, and 1 means exactly on a straight line. Here it is 0.44. This part of the noise has nothing to do with free delivery, so we can remove it.
The idea: subtract what you could predict
Section titled “The idea: subtract what you could predict”We adjust each customer’s spending during the test (y) by their spending before it (x):
adjusted = y − θ × (x − mean of x)
θ (theta) is how many dollars of spending during the test go with each dollar of spending before it. It is the slope of a straight line through the customers: θ = covariance(y, x) / variance(x). The covariance is like the correlation, but in the units of the data: dollars times dollars. We compute one θ from both groups together, so the adjustment is the same rule for A and for B.
A customer who spent more than average before the test gets a lower adjusted value. A customer who spent less gets a higher one. What is left is the part of their spending that their past does not explain. The change, if it works, is in that part.
A tiny example
Section titled “A tiny example”Here are six customers, made up so that you can see every value. The mean of x is 60 dollars, and θ = 0.559.
| Customer | Before the test (x) | During the test (y) | x − 60 | 0.559 × (x − 60) | Adjusted |
|---|---|---|---|---|---|
| 1 | 0 | 10 | −60 | −33.5 | 43.5 |
| 2 | 20 | 0 | −40 | −22.4 | 22.4 |
| 3 | 40 | 50 | −20 | −11.2 | 61.2 |
| 4 | 60 | 30 | 0 | 0 | 30.0 |
| 5 | 100 | 70 | 40 | 22.4 | 47.6 |
| 6 | 140 | 80 | 80 | 44.7 | 35.3 |
Customer 1 spent nothing before and 10 during the test, so CUPED adds 33.5 dollars. Customer 6 spent 140 before, so CUPED takes away 44.7. The mean of y and the mean of the adjusted values are both 40 dollars: CUPED does not move the average. But the variance falls from 1,040 to 191: 82% less.
Text description of the chart
Two rows of six dots. Spending in the test: 0, 10, 30, 50, 70 and 80 dollars, spread over 80 dollars. After CUPED, rounded: 22, 30, 35, 44, 48 and 61 dollars, spread over 39 dollars. Both rows have a mean of 40.
Why it is fair
Section titled “Why it is fair”The coin chose the group after the 8 weeks of pre_spend. So free delivery could not change pre_spend, and the coin made it the same in both groups, apart from chance. Subtracting it removes noise, not the effect. If one group had a few more big spenders by chance, CUPED corrects for that too.
The share of the variance that CUPED removes is the squared correlation of x and y. In the tiny example, the correlation is 0.90, so 0.90² = 82%. At Larkfield, the correlation is 0.44, so 0.44² = 19%.
The name
Section titled “The name”This method is CUPED: Controlled-experiment Using Pre-Experiment Data. Alex Deng, Ya Xu, Ron Kohavi and Toby Walker published it at Microsoft in 2013. It is a form of regression adjustment with one covariate. A covariate is a value that you measure for each unit, here each customer, besides the outcome.
CUPED on the free-delivery test
Section titled “CUPED on the free-delivery test”The project’s CUPED is ten lines in larkfield_experiments/cuped.py:
def cuped_adjust(y, x): y, x = np.asarray(y, float), np.asarray(x, float) theta = np.cov(y, x, ddof=1)[0, 1] / x.var(ddof=1) return y - theta * (x - x.mean()), float(theta)
def variance_reduction(y, x) -> float: """The share of the variance that CUPED removes: the squared correlation of outcome and covariate.""" return float(np.corrcoef(y, x)[0, 1] ** 2)xp analyse --cuped adjusts the revenue of every customer with pre_spend, then runs the same Welch’s t-test on the adjusted values.
-
Analyse the free-delivery test with CUPED.
Terminal window xp analyse delivery --cupedRun delivery-s40 (synthetic): Free delivery from 50 dollars instead of 75: an A/B test by customer, 28 days.Metric: revenue. A: 55,430 customers, mean 25.0420. B: 55,329 customers, mean 26.0854.difference +1.0434 (+4.2%), 95% interval +0.2631 to +1.8237, p = 0.00877 (Welch t-test, CUPED)CUPED removed 19.3% of the variance (theta 0.271).Saved to runs/delivery-s40/analysis.json. Compare with the true effect: xp check -
Compare it with the truth.
xp checkchecks the run that you analysed last, heredelivery.Terminal window xp checkRun delivery-s40: the last analysis was `xp analyse`.True effect (from the simulator): +0.6979 dollars per customer.Your estimate: +1.0434, 95% interval +0.2631 to +1.8237: the interval contains the true effect.
θ is 0.271: each dollar spent before the test goes with 27 cents spent during it. CUPED removed 19.3% of the variance. The interval is now +0.26 to +1.82 dollars: 1.56 wide instead of 1.74. It still contains the truth, +0.70.
Text description of the chart
Two rows on an axis from −0.5 to +2.5 dollars. Welch, no adjustment: +1.03, interval +0.16 to +1.90. Welch with CUPED: +1.04, interval +0.26 to +1.82. Dashed lines mark 0 (no effect) and the true effect, +0.70.
The estimate is still too high. CUPED removes noise, but this seed is still a lucky one. One run cannot show what a method does. Many runs can.
Over many seeds
Section titled “Over many seeds”The chart shows the width of the interval on 20 seeds, with and without CUPED.
Text description of the chart
Two rows of 20 dots. Welch: widths from 1.71 to 1.79 dollars. CUPED: widths from 1.53 to 1.59 dollars. Every CUPED width is smaller than every Welch width.
Over 200 seeds, the course measured the following:
- CUPED removed 20.3% of the variance on average.
- The interval was 0.89 times as wide: 1.56 dollars instead of 1.75.
- The interval still contained the truth: 94% of the runs, against 95.5% without CUPED.
- The test found the gain in 48.5% of the runs, against 41.5%.
What CUPED saves
Section titled “What CUPED saves”The number of customers that a test needs grows with the variance. So 20% less variance means about 20% fewer customers for the same precision. Or, with the same customers, a smaller effect becomes visible. The free-delivery test could find 1.23 dollars with 80% power. With CUPED, it can find about 1.10 dollars.
That is useful, but it is not magic. The true effect is 0.70 dollars, and the test is still too small for it. It finds the effect in only about half of the seeds. CUPED makes a test better; it does not replace a test of the right size.
Why only a fifth? Look at the chart of spending bands again: 69% of the customers spent nothing in the 8 weeks before the test. For them, pre_spend tells CUPED almost nothing. Shops whose customers buy more regularly often report larger gains, because their past spending predicts more.
When to use CUPED, and when not
Section titled “When to use CUPED, and when not”Use CUPED when you have a value from before the test that predicts the metric. Past spending, past visits and past orders are good choices. Write the covariate in the test plan, so that you do not try ten covariates and keep the best interval.
Do not use it in these cases:
-
The covariate was measured during the test. The change can move it, and then CUPED removes part of the effect. The mistake below shows how much.
-
The metric is a ratio per order, such as returns per order. The project’s CUPED works on a value per customer and refuses a ratio:
Terminal window xp analyse delivery --cuped --metric return_rateCUPED works on a metric per customer, such as revenue. 'return_rate' is a ratio (per order): run it without --cuped. -
The test is broken. CUPED does not repair a sample ratio mismatch. On the test with the redirect bug,
xp analyse srm --cupedgives +2.01 dollars (+1.22 to +2.80), and the truth is still +0.87. Run the checks first.
What it costs: one column of data from before the test, and a few lines of code. The cost is low, so many large test platforms use CUPED, or a regression with several covariates, by default.
Guided practice: CUPED by hand
Section titled “Guided practice: CUPED by hand”- Open the notebook from the box Do this lesson in a notebook at the top of this page.
- Run the setup cells and the helper cell. The data cell ends with
Ready: 2 files in data/. - In section 1, run the tiny example. You see
theta 0.559and the variance fall from1040.0to190.6. - In section 2, predict: will θ for Larkfield be above or below 0.5? Then run the cell. You see
theta 0.271andcorrelation 0.439. - In section 3, run the adjusted t-test. You see the interval
+0.263 to +1.824, the same asxp analyse --cuped. - In section 4, compare the two widths. You see
width 1.737without CUPED and1.561with it. - In section 5, compute
share_removed: the share of the variance that CUPED removes, from the correlation. - Run the check cell. It prints
Check passed.
Show the answer to step 7
share_removed = round(np.corrcoef(d.revenue, d.pre_spend)[0, 1] ** 2, 3)print(share_removed)The answer is 0.193, the same as the project’s CUPED removed 19.3% of the variance. The variance of the adjusted revenue is 1 − 0.193 = 80.7% of the variance of the revenue.
Your turn: CUPED on the A/A test
Section titled “Your turn: CUPED on the A/A test”Change one thing: the test. Run CUPED on the A/A test from the last lesson. Without CUPED, it gave −0.84 dollars (−1.72 to +0.04), with p = 0.06. Predict first: will the estimate move? Toward zero, or away from it?
Show the answer
xp analyse aa --cupedxp check aa difference -0.3143 (-1.2%), 95% interval -1.0994 to +0.4708, p = 0.433 (Welch t-test, CUPED) CUPED removed 20.4% of the variance (theta 0.275)....True effect (from the simulator): +0.0000 dollars per customer.Your estimate: -0.3143, 95% interval -1.0994 to +0.4708: the interval contains the true effect.The estimate moved from −0.84 to −0.31 dollars, toward the true zero, and the interval became narrower. By chance, group A had more big spenders before the test than group B. Without CUPED, that looked like a loss for B. CUPED corrected for it, because it compares each customer with their own past.
Common mistake
A covariate from the wrong time
Symptom: TermTomásLarkfield’s AI engineer. He often suggests a quick shortcut, and the lesson shows why it fails. wants a bigger gain in precision. He uses each customer’s number of orders during the test as the covariate, because it predicts revenue much better. The notebook shows: “CUPED removed 61.6% of the variance.” The estimate becomes −0.42 dollars (−0.96 to +0.12). The true effect is +0.70.
Cause: free delivery changed the number of orders: B’s customers ordered more. So the covariate carries part of the effect. CUPED subtracts it, and the effect disappears. Even the sign changes.
Fix: use only covariates measured before the coin flip. A covariate from before the test cannot carry the effect. Write it in the plan before the test starts. If a covariate removes far more variance than expected, ask first when it was measured.
Check your understanding
Explain CUPED to Hana in three sentences, without a formula. Say why it does not change the average effect.
Show the answer
Customers differ a lot in what they spend, and these differences hide the small effect of free delivery. CUPED compares each customer with their own spending before the test, so the differences between customers mostly disappear and the interval becomes narrower. Free delivery could not change the spending before the test, and the coin made it the same in both groups, apart from chance. So the adjustment removes noise, not the effect.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
Predict: a shop's covariate has a correlation of 0.7 with its metric. Roughly what share of the variance does CUPED remove? About how many customers does a test need, compared with no CUPED?
Show the answer
CUPED removes the squared correlation: 0.7² = 0.49, about half of the variance. The customers needed grow with the variance, so the test needs about half as many customers for the same precision. Larkfield’s correlation is only 0.44, so it removes about a fifth and saves about a fifth of the customers.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
Tomás says: 'CUPED gave a narrower interval on the test with the redirect bug, so now we can trust it.' What do you answer?
Show the answer
A narrower interval is not a correct one. CUPED reduces noise from the differences between customers. It cannot bring back the B customers that the redirect lost. On that test, CUPED gives +2.01 dollars (+1.22 to +2.80), and the truth is +0.87: still far outside the interval. The sample ratio check comes first. With a mismatch, no adjustment makes the result usable.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Recap
- Customers differ much more than the effect of a change, and that noise makes intervals wide. Spending before the test predicts part of the spending during it.
- CUPED subtracts that predictable part: adjusted = y − θ × (x − mean of x). θ = cov(y, x) / var(x), from both groups together. The mean stays the same; the variance falls by the squared correlation.
- It is fair because the covariate comes from before the coin flip. The change cannot move it, and the coin made it the same in both groups, apart from chance.
- On the free-delivery test, CUPED removed 19.3% of the variance (θ = 0.271). The interval became +0.26 to +1.82 instead of +0.16 to +1.90. Over 200 seeds: 20.3% of the variance, about 20% fewer customers, and power from 41.5% to 48.5%.
- Use only covariates from before the test, on metrics per customer, and only on a healthy test. A covariate from during the test removes the effect itself.
You can now get a precise average effect. But TermOmarLarkfield’s product manager. He asks the business questions: what a change costs, and what it gives the shop. asks a new question: does free delivery work better on mobile, or for new customers? Next, you look at segments, and you see why many segments find false winners.