Skip to content
Français

Sample size and power

  • Lesson 7
  • 40 min
  • Module 2: Design an A/B test

Where we are

We have a unit and a primary metric. The next question is how many customers we need, and for how long. In this lesson, we compute the sample size and the power of a test with statsmodels, check the answer by simulation, and see what to do when the test is too long.

Plays from YouTube. Nothing loads from YouTube until you press play. Privacy

What you will be able to do

Compute how many customers a test needs to find an effect of the size that matters, and how many days that takes. Check the number by simulating many fake tests. A test that is too small often misses a real effect, and when it does find one, it reports it too large.

NotebookDo this lesson in a notebookColab · Kaggle · GitHub

The notebook computes the reminder email’s sample size with statsmodels and runs 200 fake tests to check it. Then it draws the power curves, turns customers into days, and finds the smallest effect that the free-delivery test can see. Its setup cells install pandas and statsmodels and download the practice data, so it needs no local setup. About 40 minutes.

Colab and Kaggle open your own copy. They need a free Google or Kaggle account. Stuck? See the solution notebook.

Before you startWhat to know and to have readyAbout 40 minutes

What you need to know

What you need on your computer

  • The project folder larkfield-experiments, with its virtual environment active, and the run delivery-s40 from Choose the metrics. In the guided practice, you copy this lesson’s files from the snapshot end-of-m02 in ../nextia-learning-labs/C40/snapshots/.

Tested on macOS with Python 3.12, statsmodels 0.15.0 and the end-of-m02 snapshot. Windows was not tested.

The baker from Randomise now flips a coin at the door. She runs her free-taste test for one morning: 30 customers, 15 tasters and 15 others. The tasters buy a little more bread. Is that the taste? She cannot tell. On some mornings, one family buys six loaves for a party. One such family in either group moves the result more than a small taste effect could.

How many mornings would she need? It depends on three things. How large is the effect that she cares about? A taste that doubles sales shows up in a day; a taste that adds 3% needs weeks. How much do the sales jump around from customer to customer? And how sure does she want to be that she will see the effect, if it is there? This lesson turns those three questions into a number of customers, and the number of customers into days.

The problem: a real effect, and a test that misses it

Section titled “The problem: a real effect, and a test that misses it”

Hana’s free-delivery test has a real effect. The simulator knows it: about +0.70 dollars per customer, +2.8% (simulated). The test ran for 28 days with about 55,000 customers per group. That sounds large. But run the same test with four different seeds, that is, four sets of customers and four coins:

The same free-delivery test with four seeds. The true effect is about +0.70 to +0.77 dollars each time. Two tests find it (blue); two say 'no clear effect' (grey), and one of them estimates −0.05.
Text description of the chart

Four rows on an axis from −1.2 to +2.6 dollars, with a line at 0 labelled “no effect”. Seed 40: +1.0266, TermConfidence intervalA range around a measured value, made by a method that holds the true value 95 times in 100. If the interval of a difference does not include zero, the difference is probably not chance.Learn more Estimate uncertaintyEssential Mathematics and Statistics for AI +0.1583 to +1.895, p = 0.0205. Seed 41: -0.0492, interval -0.9353 to +0.8369, p = 0.913. Seed 42: +0.3646, interval -0.4894 to +1.2186, p = 0.403. Seed 43: +1.4322, interval +0.5404 to +2.3241, p = 0.00165. The true effects are between +0.70 and +0.77 dollars. Seeds 40 and 43 have intervals above 0; seeds 41 and 42 have intervals that cross 0.

Seed 40, the default, found the effect: p = 0.0205. Seed 41 estimated −0.05 dollars, p = 0.913: “no effect at all”. Every interval contains its seed’s truth, so no test is wrong. But two of the four could not tell a real gain of 70 cents from nothing.

Over 200 seeds, the free-delivery test finds the effect in only 41.5% of the runs. The default seed is one of the lucky ones. A team that ran this test once would miss a real effect more often than it found it.

There is a second, quieter problem. Look at the two tests that found the effect: +1.03 and +1.43 dollars, against true effects of +0.70 and +0.77. When a small test finds an effect, it is usually because chance pushed the estimate up. So a “win” from a small test is often reported too large.

Four numbers set the size of a test:

  1. The minimum detectable effect (MDE). The smallest effect that you care about finding. It is a business question, not a statistics question: “below this, the change is not worth it”. For the reminder email, Rafael says 1 percentage point: from 9% of customers ordering to 10%.
  2. The power. The chance that the test finds an effect of exactly the MDE, if it is there. The usual choice is 80%: one real effect in five is missed.
  3. The significance level, alpha. The p-value below which you say “found”. The usual choice is 0.05: in a test of no effect, 5% of runs “find” one by chance.
  4. The noise of the metric. For a share such as 9%, it follows from the share itself. For a mean such as revenue, it is the standard deviation: how far a typical customer is from the average.

From these four numbers, a formula gives the sample size: the customers per group. Then the duration is the sample size divided by the customers who enter the test each day, rounded up to full weeks. People shop differently on a Sunday and on a Tuesday, so every day of the week must be in the test equally often.

A rule of thumb gives a first answer. For a share, the customers per group are about:

n ≈ 16 × p × (1 − p) ÷ MDE²

With every value for the email:

Step Value
p, the share who order without the email 0.09
p × (1 − p) = 0.09 × 0.91 0.0819
MDE² = 0.01 × 0.01 0.0001
0.0819 ÷ 0.0001 819
× 16 13,104 customers per group

Where does 16 come from? 1.96 is the cut for alpha 0.05 and 0.84 the cut for 80% power, in standard errors. (1.96 + 0.84)² × 2 groups = 2.80² × 2 = 15.7, about 16. The mathematics course shows where such cuts come from: Confidence interval: a range with a known success rate.

The formula shows three things at once. The size grows with the noise, p × (1 − p). It falls with the square of the effect: half the effect needs four times the customers. And it does not depend on how many customers TermLarkfieldA made-up Canadian online shop for home and garden products. It is not a real company. All Nextia courses use its help desk and data as the example.Learn more Meet Larkfield has in total.

statsmodels does the exact version. It uses both shares, 9% and 10%, and gives 13,488 customers per group, close to the rule of thumb. The project’s power.py calls it:

from statsmodels.stats.power import NormalIndPower
from statsmodels.stats.proportion import proportion_effectsize
es = abs(proportion_effectsize(0.09 + 0.01, 0.09)) # 9% -> 10% as one standard effect size
n = NormalIndPower().solve_power(effect_size=es, power=0.8, alpha=0.05, ratio=1.0)

A formula rests on assumptions. A simulation shows what it means. Make 200 fake tests of 13,488 customers per group. In each one, A’s customers order with a chance of 9% and B’s with 10%, so the true effect is exactly 1 point. Run Welch’s t-test on each one, as xp analyse does:

200 fake tests with a true effect of exactly 1 point. Chance spreads their estimates from about +0.3 to +2.1 points. The 79.5% to the right of the dashed line have p < 0.05: that share is the power.
Text description of the chart

A histogram of the differences that 200 fake tests found, on an axis from 0.2 to 2.2 percentage points. The true effect is 1 point in every test, marked by a line. The estimates form a hill around 1 point, from +0.29 to +2.07. A second line at about +0.7 points marks where p falls below 0.05. 79.5% of the tests are to the right of that line: they found the effect. The rest are to the left: chance pulled their difference down, and they say “no clear effect”.

The true effect is the same in all 200 tests, but each one sees a different difference. A test “finds” the effect when its difference is larger than about 0.7 points, about two standard errors. 79.5% of the fake tests did: the simulation agrees with the formula’s 80%.

Power is not “yes or no”. A smaller test still finds the effect sometimes, and a larger test finds it more often. The chart shows power against the customers per group, for three effects. The dots are simulations, 200 fake tests each:

  • Effect of 0.5 point
  • Effect of 1 point
  • Effect of 1.5 points
  • 1 point, 200 fake tests each
Power against customers per group, for the email (9% order without it). 80% power needs 6,131 per group for 1.5 points, 13,488 for 1 point, and 52,700 for half a point.
Text description of the chart

A line chart of power, from 0 to 100%, against customers per group, from 0 to 40,000. Three curves rise from left to right and flatten near 100%. The curve for 1.5 points rises fastest and passes 80% at 6,131 customers per group. The curve for 1 point passes 80% at 13,488. The curve for half a point rises slowly and is only at about 68.5% at 40,000; it needs 52,700 per group for 80%. Six dots, from simulations of 200 fake tests each, lie close to the 1-point curve: 32.5% at 3,372, 53% at 6,744, 72% at 10,116, 79.5% at 13,488, 90.5% at 20,232, 97.5% at 26,976.

Read the chart from left to right. Power rises fast at first, then slowly. Going from 80% to 90% power costs about a third more customers. Halving the MDE costs four times as many.

The project’s xp power command does the formula, the duration and the simulation in one step. About 4,000 customers a day reach “30 days without an order”, so they can enter the test:

Terminal window
xp power --baseline 0.09 --mde 0.01 --per-day 4000 --simulate 200
To find an effect of +0.01 on a proportion (baseline 0.09) with 80% power at alpha 0.05:
13,488 customers per group, 26,976 in total.
with 6,744 per group, power = 0.51
with 10,116 per group, power = 0.68
with 13,488 per group, power = 0.80
with 20,232 per group, power = 0.93
with 26,976 per group, power = 0.98
At 4,000 new customers a day: 7 days; run full weeks: 1 week (7 days).
Simulated: 200 fake tests with 13,488 customers per group and a true effect of +0.01: 80% found it (p < 0.05).

So the email test needs 26,976 customers: 7 days at 4,000 a day, which is exactly one full week. The simulation agrees with the formula. What about the real effect? The simulator knows that the email’s true effect is +1.19 points, larger than the MDE. Over 200 simulated email tests of this size, made by the simulator itself, 94% found it (the formula says 91%).

Revenue per customer is a much noisier metric. 78% of the customers spend nothing, and a few spend hundreds of dollars. In group A, the mean is 25.05 dollars and the standard deviation is 73.22 dollars: about three times the mean. Turn the question around: with 55,430 customers per group, what is the smallest effect that the test finds with 80% power?

Terminal window
xp power --from-run delivery --mde 1.23 --per-day 3956
From delivery, group A: revenue mean 25.0504, standard deviation 73.2223.
To find an effect of +1.23 on a mean with standard deviation 73.222 (baseline 25.0504) with 80% power at alpha 0.05:
55,632 customers per group, 111,264 in total.
with 27,816 per group, power = 0.51
with 41,724 per group, power = 0.68
with 55,632 per group, power = 0.80
with 83,448 per group, power = 0.93
with 111,264 per group, power = 0.98
At 3,956 new customers a day: 29 days; run full weeks: 5 weeks (35 days).

About 1.23 dollars, or 4.9%. The test’s real size, 55,430 per group, is just below 55,632. The true effect, +0.70 dollars, is much smaller than 1.23. So the test was under-powered: too small to find the effect reliably. By the formula, its power for +0.70 is about 35.5%. Over [object Object][object Object][object Object][object Object] seeds of the simulator, it found the effect in 41.5%. The two numbers differ a little because the true effect changes from seed to seed (+0.74 dollars on average). Both say the same thing: far below 80%.

The power that a test plan aims for (80%), and the free-delivery test's real power for its true effect: 35.5% by the formula, 41.5% over 200 simulated seeds, 48.5% with CUPED (a way to remove noise, taught in Module 3).
Text description of the chart

A column chart with four columns, on an axis from 0 to 100%. Planned power: 80%. The formula’s power for the true effect of +0.70 dollars: 35.5%. The share of 200 simulated seeds in which the test found the effect: 41.5%. The same with CUPED, a method of Module 3 that removes some noise: 48.5%. All three real values are far below the planned 80%.

What to do when the test would take a year

Section titled “What to do when the test would take a year”

How large would the free-delivery test need to be to find small effects? Larkfield gets about 3,956 new customers a day:

Days that the free-delivery test needs for 80% power, by the size of the effect, at 3,956 customers a day. A 1% effect needs 679 days: almost two years.
Text description of the chart

A column chart of the days that the free-delivery test needs, for four effect sizes. 1% of revenue per customer (0.25 dollars): 1,341,205 customers per group, 679 days. 2%: 335,302 per group, 170 days. 3%: 149,024 per group, 76 days. 5%: 53,650 per group, 28 days, the length of the real test.

A 3% effect, close to the truth, needs 149,024 customers per group: 76 days, 11 full weeks. A 1% effect needs almost two years. That is too long: the season changes, the shop changes, and Hana cannot wait. You have six options, and each one has a cost:

  1. Ask what effect really matters. The MDE is a business question. If a gain below about 1.25 dollars (5%) per customer does not pay for the extra delivery cost, then 4 weeks is enough for Hana’s decision. Missing a smaller effect costs little, because it would not pay. The next lesson writes this into the plan.
  2. Reduce the noise. CUPED uses each customer’s spending before the test to remove part of the noise. Over 200 seeds, it removed about a fifth of the variance: about 20% fewer customers for the same precision. You use it in Module 3, Get more precise with CUPED.
  3. Use more of the traffic. Put every eligible customer in the test, and split 50/50: an uneven split, such as 90/10, needs more customers in total.
  4. Choose a less noisy metric, with care. The share who ordered is less noisy than revenue. But it must still be tied to the decision. Choose it before the test, never because it gave the smallest number.
  5. Accept less, knowingly. 70% power or a larger alpha shortens the test. Write the choice into the plan, with its cost: more missed effects or more false wins.
  6. Do not run the test. A test that cannot find the effect that matters wastes time, and a lucky “win” from it will be too large. Decide with other evidence, or come back with the methods of Module 5 when the change has rolled out.

Guided practice: add xp power to the project

Section titled “Guided practice: add xp power to the project”

This lesson adds two files to the project, power.py and the command xp power, and their tests.

  1. In larkfield-experiments, with the TermVirtual environmentA folder with its own Python and its own packages, for one project. Packages that you install there do not change other projects.Learn more Isolate dependenciesPython for Practical AI Engineering active, copy the files from the snapshot end-of-m02:

    Terminal window
    S=../nextia-learning-labs/C40/snapshots/end-of-m02
    for f in larkfield_experiments/power.py larkfield_experiments/commands/power.py tests/test_design.py; do
    cp "$S/$f" "$f"
    done

    If you downloaded the ZIP file in Meet the simulator, the labs folder is nextia-learning-labs-main: use that name in S. xp finds its commands by the files in larkfield_experiments/commands/, so you do not need to install again.

  2. Run the tests. One test in test_design.py needs xp plan, which comes in the next lesson, so leave it out for now:

    Terminal window
    python -m pytest -q -k "not plan"
    13 passed, 1 deselected in 1.26s

    The time can be different on your computer.

  3. Read larkfield_experiments/power.py. sample_size uses NormalIndPower for a share and TTestIndPower for a mean. simulated_power runs a test function many times and counts the p-values below alpha.

  4. Size the email test, with the simulation:

    Terminal window
    xp power --baseline 0.09 --mde 0.01 --per-day 4000 --simulate 200

    You see 13,488 customers per group, 7 days and 80% found it, as above.

  5. Find the free-delivery test’s smallest detectable effect. It reads the standard deviation from the run delivery-s40:

    Terminal window
    xp power --from-run delivery --mde 1.23 --per-day 3956

    You see 55,632 customers per group and 29 days. In a real plan, you take the standard deviation from the weeks before the test. The course uses group A as a stand-in: it has the same kind of customers.

  6. Ask for a 3% effect: 3% of 25.05 dollars is 0.7515 dollars.

    Terminal window
    xp power --from-run delivery --mde 0.7515 --per-day 3956

    The second and last lines:

    149,029 customers per group, 298,058 in total.
    At 3,956 new customers a day: 76 days; run full weeks: 11 weeks (77 days).

    The chart above used the exact 3% (0.75151 dollars) and shows 149,024: the same answer, rounded differently.

Rafael finds that only about 1,000 customers a day reach “30 days without an order” outside the holiday season. He keeps the MDE of 1 point. Predict first: how many full weeks does the email test need now, and why not 7 × 4 = 28 days exactly? Then run xp power with --per-day 1000.

Show the answer
Terminal window
xp power --baseline 0.09 --mde 0.01 --per-day 1000

The last line:

At 1,000 new customers a day: 27 days; run full weeks: 4 weeks (28 days).

The sample size does not change: 13,488 per group, 26,976 in total, because it depends on the MDE, the noise, the power and alpha, not on the traffic. At 1,000 a day, 26,976 customers take 27 days. Rounded up to full weeks, that is 4 weeks, 28 days. The extra day keeps every day of the week in the test four times.

Common mistake

Two days of email, and 'no effect'

Symptom: TermTomásLarkfield’s AI engineer. He often suggests a quick shortcut, and the lesson shows why it fails. runs the reminder email for two days, without a power calculation: about 4,000 customers per group. The difference is not significant. He writes: “The email does not work. Let’s stop sending it.”

Cause: the test was too small. With 4,000 per group, the power to find a 1-point effect is about 33%. Even for the email’s true effect, +1.19 points, it is about 44%. More than half of such tests say “no clear effect” for an email that works. “Not significant” in an under-powered test means “we cannot tell”, not “no effect”.

Fix: compute the sample size before the test, from the MDE that matters, and run for the full number of weeks. If a test has ended already, report its interval, not only the p-value. A wide interval that includes both 0 and the MDE says plainly: the test cannot tell.

Consequence in AI work: a model or a prompt test that is too small kills good changes (“no effect”) and ships lucky ones with exaggerated numbers. Size every online test of a model, a prompt or a ranking change before it starts, as you would size an offline TermTest setA fixed set of examples with known correct answers. You run each version of a prompt or model on the same set, so the results compare fairly.Learn more Build a representative test setLLMs: How They Work and How to Choose One.

Check your understanding

Completion check: a test of the order rate has a baseline of 9%, an MDE of 1 percentage point, 80% power and alpha 0.05. 2,000 eligible customers arrive each day. How many days does it need, and why?

Show the answer

The sample size is 13,488 customers per group, 26,976 in total (statsmodels; the rule of thumb 16 × 0.09 × 0.91 ÷ 0.01² gives about 13,100). At 2,000 a day, that is 26,976 ÷ 2,000 = 13.5, so 14 days. 14 days is exactly 2 full weeks, so the test runs 14 days. Full weeks matter because customers shop differently on different days of the week. The size comes from the MDE, the noise, the power and alpha; the traffic only turns it into days.

Check your understanding

Predict: Rafael wants to find an effect of half a point instead of 1 point. Roughly how many customers per group does he need, and why?

Show the answer

About four times as many: the size grows with 1 ÷ MDE², and (1 ÷ 0.5)² = 4. statsmodels gives 52,700 per group, against 13,488 for 1 point. At 4,000 customers a day, the test grows from 1 week to 27 days (4 full weeks). A smaller effect is harder to see through the noise.

Check your understanding

The free-delivery test found +1.03 dollars with p = 0.0205. Hana says: 'It worked, and it adds 1.03 dollars a customer.' What do you tell her about the test's power and about the number?

Show the answer

The test was under-powered for the true effect. With about 55,000 customers per group, it finds an effect of 1.23 dollars with 80% power, but the true effect is about 0.70. Over 200 simulated seeds, it found the effect in only 41.5% of runs; seed 41 of the same test estimated −0.05. The default seed was lucky. When a small test finds an effect, chance usually pushed the estimate up, so 1.03 is probably too high: the truth is 0.70. Report the interval, +0.16 to +1.90, not only the point estimate.

Recap

  • The sample size comes from four numbers: the minimum detectable effect (a business choice), the power (usually 80%), alpha (usually 0.05) and the metric’s noise. Half the effect needs four times the customers.
  • The email plan: 9% baseline, 1 point, 80% power: 13,488 customers per group. 200 fake tests found the effect in 80%. At 4,000 customers a day, it takes 7 days: one full week.
  • Duration = customers ÷ customers per day, rounded up to full weeks.
  • The free-delivery test could find 1.23 dollars (4.9%) with 80% power; the truth is 0.70. Over 200 seeds, it found the effect in only 41.5% (48.5% with CUPED). The default seed is a lucky one, and its +1.03 is too large.
  • When a test would take a year: ask which effect really matters, remove noise (CUPED), use all the traffic with a 50/50 split, or do not run the test.

You can now size a test. Next, you write every choice of this module into a one-page plan before the test starts, with the decision for each possible result.

Next: Write the plan first