Sample size and power
- Lesson 7
- 40 min
- Module 2: Design an A/B test
Where we are
We have a unit and a primary metric. The next question is how many customers we need, and for how long. In this lesson, we compute the sample size and the power of a test with statsmodels, check the answer by simulation, and see what to do when the test is too long.
Lesson 7 of 30 · Experiments and Causal Impact: Did Our Change Work?Sample size and power
1. Why "it went up" is not an answer
2. Design an A/B test
3. Analyse the test
4. Traps in real tests
5. When you cannot randomise: natural experiments
6. Adjust for confounders in observational data
7. Uplift: who should get the treatment?
8. Experiments for AI features and in production
What you will be able to do
Compute how many customers a test needs to find an effect of the size that matters, and how many days that takes. Check the number by simulating many fake tests. A test that is too small often misses a real effect, and when it does find one, it reports it too large.
NotebookDo this lesson in a notebook
The notebook computes the reminder email’s sample size with statsmodels and runs 200 fake tests to check it. Then it draws the power curves, turns customers into days, and finds the smallest effect that the free-delivery test can see. Its setup cells install pandas and statsmodels and download the practice data, so it needs no local setup. About 40 minutes.
Colab and Kaggle open your own copy. They need a free Google or Kaggle account. Stuck? See the solution notebook.
Before you startWhat to know and to have ready
What you need to know
- The primary metric, revenue per customer, and its guardrail, from Choose the metrics.
- A p-value and “p < 0.05”, from the mathematics course: The names for what you just did. The standard error, the typical size of the chance error, from Estimate uncertainty.
What you need on your computer
- The project folder
larkfield-experiments, with its virtual environment active, and the rundelivery-s40from Choose the metrics. In the guided practice, you copy this lesson’s files from the snapshotend-of-m02in../nextia-learning-labs/C40/snapshots/.
Tested on macOS with Python 3.12, statsmodels 0.15.0 and the end-of-m02 snapshot. Windows was not tested.
How many mornings does the baker need?
Section titled “How many mornings does the baker need?”The baker from Randomise now flips a coin at the door. She runs her free-taste test for one morning: 30 customers, 15 tasters and 15 others. The tasters buy a little more bread. Is that the taste? She cannot tell. On some mornings, one family buys six loaves for a party. One such family in either group moves the result more than a small taste effect could.
How many mornings would she need? It depends on three things. How large is the effect that she cares about? A taste that doubles sales shows up in a day; a taste that adds 3% needs weeks. How much do the sales jump around from customer to customer? And how sure does she want to be that she will see the effect, if it is there? This lesson turns those three questions into a number of customers, and the number of customers into days.
The problem: a real effect, and a test that misses it
Section titled “The problem: a real effect, and a test that misses it”Hana’s free-delivery test has a real effect. The simulator knows it: about +0.70 dollars per customer, +2.8% (simulated). The test ran for 28 days with about 55,000 customers per group. That sounds large. But run the same test with four different seeds, that is, four sets of customers and four coins:
Text description of the chart
Four rows on an axis from −1.2 to +2.6 dollars, with a line at 0 labelled “no effect”. Seed 40: +1.0266, TermConfidence intervalA range around a measured value, made by a method that holds the true value 95 times in 100. If the interval of a difference does not include zero, the difference is probably not chance.Learn more Estimate uncertaintyEssential Mathematics and Statistics for AI +0.1583 to +1.895, p = 0.0205. Seed 41: -0.0492, interval -0.9353 to +0.8369, p = 0.913. Seed 42: +0.3646, interval -0.4894 to +1.2186, p = 0.403. Seed 43: +1.4322, interval +0.5404 to +2.3241, p = 0.00165. The true effects are between +0.70 and +0.77 dollars. Seeds 40 and 43 have intervals above 0; seeds 41 and 42 have intervals that cross 0.
Seed 40, the default, found the effect: p = 0.0205. Seed 41 estimated −0.05 dollars, p = 0.913: “no effect at all”. Every interval contains its seed’s truth, so no test is wrong. But two of the four could not tell a real gain of 70 cents from nothing.
Over 200 seeds, the free-delivery test finds the effect in only 41.5% of the runs. The default seed is one of the lucky ones. A team that ran this test once would miss a real effect more often than it found it.
There is a second, quieter problem. Look at the two tests that found the effect: +1.03 and +1.43 dollars, against true effects of +0.70 and +0.77. When a small test finds an effect, it is usually because chance pushed the estimate up. So a “win” from a small test is often reported too large.
The idea: decide the size before the test
Section titled “The idea: decide the size before the test”Four numbers set the size of a test:
- The minimum detectable effect (MDE). The smallest effect that you care about finding. It is a business question, not a statistics question: “below this, the change is not worth it”. For the reminder email, Rafael says 1 percentage point: from 9% of customers ordering to 10%.
- The power. The chance that the test finds an effect of exactly the MDE, if it is there. The usual choice is 80%: one real effect in five is missed.
- The significance level, alpha. The p-value below which you say “found”. The usual choice is 0.05: in a test of no effect, 5% of runs “find” one by chance.
- The noise of the metric. For a share such as 9%, it follows from the share itself. For a mean such as revenue, it is the standard deviation: how far a typical customer is from the average.
From these four numbers, a formula gives the sample size: the customers per group. Then the duration is the sample size divided by the customers who enter the test each day, rounded up to full weeks. People shop differently on a Sunday and on a Tuesday, so every day of the week must be in the test equally often.
A tiny example: the email, by hand
Section titled “A tiny example: the email, by hand”A rule of thumb gives a first answer. For a share, the customers per group are about:
n ≈ 16 × p × (1 − p) ÷ MDE²
With every value for the email:
| Step | Value |
|---|---|
| p, the share who order without the email | 0.09 |
| p × (1 − p) = 0.09 × 0.91 | 0.0819 |
| MDE² = 0.01 × 0.01 | 0.0001 |
| 0.0819 ÷ 0.0001 | 819 |
| × 16 | 13,104 customers per group |
Where does 16 come from? 1.96 is the cut for alpha 0.05 and 0.84 the cut for 80% power, in standard errors. (1.96 + 0.84)² × 2 groups = 2.80² × 2 = 15.7, about 16. The mathematics course shows where such cuts come from: Confidence interval: a range with a known success rate.
The formula shows three things at once. The size grows with the noise, p × (1 − p). It falls with the square of the effect: half the effect needs four times the customers. And it does not depend on how many customers TermLarkfieldA made-up Canadian online shop for home and garden products. It is not a real company. All Nextia courses use its help desk and data as the example.Learn more Meet Larkfield has in total.
statsmodels does the exact version. It uses both shares, 9% and 10%, and gives 13,488 customers per group, close to the rule of thumb. The project’s power.py calls it:
from statsmodels.stats.power import NormalIndPowerfrom statsmodels.stats.proportion import proportion_effectsize
es = abs(proportion_effectsize(0.09 + 0.01, 0.09)) # 9% -> 10% as one standard effect sizen = NormalIndPower().solve_power(effect_size=es, power=0.8, alpha=0.05, ratio=1.0)Check the formula: 200 fake tests
Section titled “Check the formula: 200 fake tests”A formula rests on assumptions. A simulation shows what it means. Make 200 fake tests of 13,488 customers per group. In each one, A’s customers order with a chance of 9% and B’s with 10%, so the true effect is exactly 1 point. Run Welch’s t-test on each one, as xp analyse does:
Text description of the chart
A histogram of the differences that 200 fake tests found, on an axis from 0.2 to 2.2 percentage points. The true effect is 1 point in every test, marked by a line. The estimates form a hill around 1 point, from +0.29 to +2.07. A second line at about +0.7 points marks where p falls below 0.05. 79.5% of the tests are to the right of that line: they found the effect. The rest are to the left: chance pulled their difference down, and they say “no clear effect”.
The true effect is the same in all 200 tests, but each one sees a different difference. A test “finds” the effect when its difference is larger than about 0.7 points, about two standard errors. 79.5% of the fake tests did: the simulation agrees with the formula’s 80%.
Power curves
Section titled “Power curves”Power is not “yes or no”. A smaller test still finds the effect sometimes, and a larger test finds it more often. The chart shows power against the customers per group, for three effects. The dots are simulations, 200 fake tests each:
- Effect of 0.5 point
- Effect of 1 point
- Effect of 1.5 points
- 1 point, 200 fake tests each
Text description of the chart
A line chart of power, from 0 to 100%, against customers per group, from 0 to 40,000. Three curves rise from left to right and flatten near 100%. The curve for 1.5 points rises fastest and passes 80% at 6,131 customers per group. The curve for 1 point passes 80% at 13,488. The curve for half a point rises slowly and is only at about 68.5% at 40,000; it needs 52,700 per group for 80%. Six dots, from simulations of 200 fake tests each, lie close to the 1-point curve: 32.5% at 3,372, 53% at 6,744, 72% at 10,116, 79.5% at 13,488, 90.5% at 20,232, 97.5% at 26,976.
Read the chart from left to right. Power rises fast at first, then slowly. Going from 80% to 90% power costs about a third more customers. Halving the MDE costs four times as many.
Larkfield: the email plan with xp power
Section titled “Larkfield: the email plan with xp power”The project’s xp power command does the formula, the duration and the simulation in one step. About 4,000 customers a day reach “30 days without an order”, so they can enter the test:
xp power --baseline 0.09 --mde 0.01 --per-day 4000 --simulate 200To find an effect of +0.01 on a proportion (baseline 0.09) with 80% power at alpha 0.05: 13,488 customers per group, 26,976 in total. with 6,744 per group, power = 0.51 with 10,116 per group, power = 0.68 with 13,488 per group, power = 0.80 with 20,232 per group, power = 0.93 with 26,976 per group, power = 0.98 At 4,000 new customers a day: 7 days; run full weeks: 1 week (7 days). Simulated: 200 fake tests with 13,488 customers per group and a true effect of +0.01: 80% found it (p < 0.05).So the email test needs 26,976 customers: 7 days at 4,000 a day, which is exactly one full week. The simulation agrees with the formula. What about the real effect? The simulator knows that the email’s true effect is +1.19 points, larger than the MDE. Over 200 simulated email tests of this size, made by the simulator itself, 94% found it (the formula says 91%).
The free-delivery test was too small
Section titled “The free-delivery test was too small”Revenue per customer is a much noisier metric. 78% of the customers spend nothing, and a few spend hundreds of dollars. In group A, the mean is 25.05 dollars and the standard deviation is 73.22 dollars: about three times the mean. Turn the question around: with 55,430 customers per group, what is the smallest effect that the test finds with 80% power?
xp power --from-run delivery --mde 1.23 --per-day 3956From delivery, group A: revenue mean 25.0504, standard deviation 73.2223.To find an effect of +1.23 on a mean with standard deviation 73.222 (baseline 25.0504) with 80% power at alpha 0.05: 55,632 customers per group, 111,264 in total. with 27,816 per group, power = 0.51 with 41,724 per group, power = 0.68 with 55,632 per group, power = 0.80 with 83,448 per group, power = 0.93 with 111,264 per group, power = 0.98 At 3,956 new customers a day: 29 days; run full weeks: 5 weeks (35 days).About 1.23 dollars, or 4.9%. The test’s real size, 55,430 per group, is just below 55,632. The true effect, +0.70 dollars, is much smaller than 1.23. So the test was under-powered: too small to find the effect reliably. By the formula, its power for +0.70 is about 35.5%. Over [object Object][object Object][object Object][object Object] seeds of the simulator, it found the effect in 41.5%. The two numbers differ a little because the true effect changes from seed to seed (+0.74 dollars on average). Both say the same thing: far below 80%.
Text description of the chart
A column chart with four columns, on an axis from 0 to 100%. Planned power: 80%. The formula’s power for the true effect of +0.70 dollars: 35.5%. The share of 200 simulated seeds in which the test found the effect: 41.5%. The same with CUPED, a method of Module 3 that removes some noise: 48.5%. All three real values are far below the planned 80%.
What to do when the test would take a year
Section titled “What to do when the test would take a year”How large would the free-delivery test need to be to find small effects? Larkfield gets about 3,956 new customers a day:
Text description of the chart
A column chart of the days that the free-delivery test needs, for four effect sizes. 1% of revenue per customer (0.25 dollars): 1,341,205 customers per group, 679 days. 2%: 335,302 per group, 170 days. 3%: 149,024 per group, 76 days. 5%: 53,650 per group, 28 days, the length of the real test.
A 3% effect, close to the truth, needs 149,024 customers per group: 76 days, 11 full weeks. A 1% effect needs almost two years. That is too long: the season changes, the shop changes, and Hana cannot wait. You have six options, and each one has a cost:
- Ask what effect really matters. The MDE is a business question. If a gain below about 1.25 dollars (5%) per customer does not pay for the extra delivery cost, then 4 weeks is enough for Hana’s decision. Missing a smaller effect costs little, because it would not pay. The next lesson writes this into the plan.
- Reduce the noise. CUPED uses each customer’s spending before the test to remove part of the noise. Over 200 seeds, it removed about a fifth of the variance: about 20% fewer customers for the same precision. You use it in Module 3, Get more precise with CUPED.
- Use more of the traffic. Put every eligible customer in the test, and split 50/50: an uneven split, such as 90/10, needs more customers in total.
- Choose a less noisy metric, with care. The share who ordered is less noisy than revenue. But it must still be tied to the decision. Choose it before the test, never because it gave the smallest number.
- Accept less, knowingly. 70% power or a larger alpha shortens the test. Write the choice into the plan, with its cost: more missed effects or more false wins.
- Do not run the test. A test that cannot find the effect that matters wastes time, and a lucky “win” from it will be too large. Decide with other evidence, or come back with the methods of Module 5 when the change has rolled out.
Guided practice: add xp power to the project
Section titled “Guided practice: add xp power to the project”This lesson adds two files to the project, power.py and the command xp power, and their tests.
-
In
larkfield-experiments, with the TermVirtual environmentA folder with its own Python and its own packages, for one project. Packages that you install there do not change other projects.Learn more Isolate dependenciesPython for Practical AI Engineering active, copy the files from the snapshotend-of-m02:Terminal window S=../nextia-learning-labs/C40/snapshots/end-of-m02for f in larkfield_experiments/power.py larkfield_experiments/commands/power.py tests/test_design.py; docp "$S/$f" "$f"doneTerminal window $S = "..\nextia-learning-labs\C40\snapshots\end-of-m02"foreach ($f in 'larkfield_experiments\power.py', 'larkfield_experiments\commands\power.py', 'tests\test_design.py') { Copy-Item "$S\$f" $f }If you downloaded the ZIP file in Meet the simulator, the labs folder is
nextia-learning-labs-main: use that name inS.xpfinds its commands by the files inlarkfield_experiments/commands/, so you do not need to install again. -
Run the tests. One test in
test_design.pyneedsxp plan, which comes in the next lesson, so leave it out for now:Terminal window python -m pytest -q -k "not plan"13 passed, 1 deselected in 1.26sThe time can be different on your computer.
-
Read
larkfield_experiments/power.py.sample_sizeusesNormalIndPowerfor a share andTTestIndPowerfor a mean.simulated_powerruns a test function many times and counts the p-values below alpha. -
Size the email test, with the simulation:
Terminal window xp power --baseline 0.09 --mde 0.01 --per-day 4000 --simulate 200You see
13,488 customers per group,7 daysand80% found it, as above. -
Find the free-delivery test’s smallest detectable effect. It reads the standard deviation from the run
delivery-s40:Terminal window xp power --from-run delivery --mde 1.23 --per-day 3956You see
55,632 customers per groupand29 days. In a real plan, you take the standard deviation from the weeks before the test. The course uses group A as a stand-in: it has the same kind of customers. -
Ask for a 3% effect: 3% of 25.05 dollars is 0.7515 dollars.
Terminal window xp power --from-run delivery --mde 0.7515 --per-day 3956The second and last lines:
149,029 customers per group, 298,058 in total.At 3,956 new customers a day: 76 days; run full weeks: 11 weeks (77 days).The chart above used the exact 3% (0.75151 dollars) and shows 149,024: the same answer, rounded differently.
Your turn: a smaller list
Section titled “Your turn: a smaller list”Rafael finds that only about 1,000 customers a day reach “30 days without an order” outside the holiday season. He keeps the MDE of 1 point. Predict first: how many full weeks does the email test need now, and why not 7 × 4 = 28 days exactly? Then run xp power with --per-day 1000.
Show the answer
xp power --baseline 0.09 --mde 0.01 --per-day 1000The last line:
At 1,000 new customers a day: 27 days; run full weeks: 4 weeks (28 days).The sample size does not change: 13,488 per group, 26,976 in total, because it depends on the MDE, the noise, the power and alpha, not on the traffic. At 1,000 a day, 26,976 customers take 27 days. Rounded up to full weeks, that is 4 weeks, 28 days. The extra day keeps every day of the week in the test four times.
Common mistake
Two days of email, and 'no effect'
Symptom: TermTomásLarkfield’s AI engineer. He often suggests a quick shortcut, and the lesson shows why it fails. runs the reminder email for two days, without a power calculation: about 4,000 customers per group. The difference is not significant. He writes: “The email does not work. Let’s stop sending it.”
Cause: the test was too small. With 4,000 per group, the power to find a 1-point effect is about 33%. Even for the email’s true effect, +1.19 points, it is about 44%. More than half of such tests say “no clear effect” for an email that works. “Not significant” in an under-powered test means “we cannot tell”, not “no effect”.
Fix: compute the sample size before the test, from the MDE that matters, and run for the full number of weeks. If a test has ended already, report its interval, not only the p-value. A wide interval that includes both 0 and the MDE says plainly: the test cannot tell.
Consequence in AI work: a model or a prompt test that is too small kills good changes (“no effect”) and ships lucky ones with exaggerated numbers. Size every online test of a model, a prompt or a ranking change before it starts, as you would size an offline TermTest setA fixed set of examples with known correct answers. You run each version of a prompt or model on the same set, so the results compare fairly.Learn more Build a representative test setLLMs: How They Work and How to Choose One.
Knowledge check
Section titled “Knowledge check”Check your understanding
Completion check: a test of the order rate has a baseline of 9%, an MDE of 1 percentage point, 80% power and alpha 0.05. 2,000 eligible customers arrive each day. How many days does it need, and why?
Show the answer
The sample size is 13,488 customers per group, 26,976 in total (statsmodels; the rule of thumb 16 × 0.09 × 0.91 ÷ 0.01² gives about 13,100). At 2,000 a day, that is 26,976 ÷ 2,000 = 13.5, so 14 days. 14 days is exactly 2 full weeks, so the test runs 14 days. Full weeks matter because customers shop differently on different days of the week. The size comes from the MDE, the noise, the power and alpha; the traffic only turns it into days.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
Predict: Rafael wants to find an effect of half a point instead of 1 point. Roughly how many customers per group does he need, and why?
Show the answer
About four times as many: the size grows with 1 ÷ MDE², and (1 ÷ 0.5)² = 4. statsmodels gives 52,700 per group, against 13,488 for 1 point. At 4,000 customers a day, the test grows from 1 week to 27 days (4 full weeks). A smaller effect is harder to see through the noise.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
The free-delivery test found +1.03 dollars with p = 0.0205. Hana says: 'It worked, and it adds 1.03 dollars a customer.' What do you tell her about the test's power and about the number?
Show the answer
The test was under-powered for the true effect. With about 55,000 customers per group, it finds an effect of 1.23 dollars with 80% power, but the true effect is about 0.70. Over 200 simulated seeds, it found the effect in only 41.5% of runs; seed 41 of the same test estimated −0.05. The default seed was lucky. When a small test finds an effect, chance usually pushed the estimate up, so 1.03 is probably too high: the truth is 0.70. Report the interval, +0.16 to +1.90, not only the point estimate.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Recap
- The sample size comes from four numbers: the minimum detectable effect (a business choice), the power (usually 80%), alpha (usually 0.05) and the metric’s noise. Half the effect needs four times the customers.
- The email plan: 9% baseline, 1 point, 80% power: 13,488 customers per group. 200 fake tests found the effect in 80%. At 4,000 customers a day, it takes 7 days: one full week.
- Duration = customers ÷ customers per day, rounded up to full weeks.
- The free-delivery test could find 1.23 dollars (4.9%) with 80% power; the truth is 0.70. Over 200 seeds, it found the effect in only 41.5% (48.5% with CUPED). The default seed is a lucky one, and its +1.03 is too large.
- When a test would take a year: ask which effect really matters, remove noise (CUPED), use all the traffic with a 50/50 split, or do not run the test.
You can now size a test. Next, you write every choice of this module into a one-page plan before the test starts, with the decision for each possible result.
Next: Write the plan first