The result and its uncertainty
- Lesson 9
- 40 min
- Module 3: Analyse the test
Where we are
Our test plan is written and the test has run. Now we read the result. In this lesson, we compute the difference between the groups, its confidence interval and its p-value, with a t-test and with the bootstrap, and we report the effect and its range.
Lesson 9 of 30 · Experiments and Causal Impact: Did Our Change Work?The result and its uncertainty
1. Why "it went up" is not an answer
2. Design an A/B test
3. Analyse the test
4. Traps in real tests
5. When you cannot randomise: natural experiments
6. Adjust for confounders in observational data
7. Uplift: who should get the treatment?
8. Experiments for AI features and in production
What you will be able to do
Report a test’s result as an effect with its range, not as one number or a p-value. One number hides how much of the difference could be luck, and a decision made on luck is a wrong decision that nobody sees.
NotebookDo this lesson in a notebook
The notebook computes the free-delivery result step by step: the means, Welch’s t-test, the bootstrap and the TermConfidence intervalA range around a measured value, made by a method that holds the true value 95 times in 100. If the interval of a difference does not include zero, the difference is probably not chance.Learn more Estimate uncertaintyEssential Mathematics and Statistics for AI for a share of customers. It checks each answer against the simulator’s true effect. Its setup cells install the packages and download the practice data, so it needs no local setup. About 30 minutes.
Colab and Kaggle open your own copy. They need a free Google or Kaggle account. Stuck? See the solution notebook.
Before you startWhat to know and to have ready
What you need to know
- Hana’s free-delivery test, its primary metric (revenue per customer) and its guardrail (returns per order), from Choose the metrics.
- Why this test is small for its effect, from Sample size and power.
- The test plan, from Write the plan first.
- The standard error, confidence intervals and the bootstrap, from Estimate uncertainty in Essential Mathematics and Statistics for AI. P-values are in Compare alternatives in the same course.
What you need on your computer (only for the project steps; the notebook needs none of it)
- The folder
larkfield-experimentswith its.venv, from Meet the simulator, and the labs foldernextia-learning-labsbeside it. Ifsim listfails, activate.venvagain.
Tested on macOS with Python 3.12 and the end-of-m03 snapshot of the project. Windows was not tested.
A new shop window
Section titled “A new shop window”Go back to the gift-shop owner and her new window display. In the next two weeks, 46 customers come in each day. Before the change, the average was 40. Did the display bring 6 more customers a day?
She is right to doubt it. Her daily count was never fixed at 40. Some days it was 30, other days 55: rain, a holiday, a bus that stopped nearby. So the real question is not “6 more?” but “6 more, give or take how much?” If the give-or-take is 2 customers, the display worked. If it is 15, she knows almost nothing yet.
Every A/B test has the same question. The difference between the groups is one number. Luck alone also makes a difference between two groups, even when the change does nothing. Before you trust the difference, you must know how big that luck can be.
One number hides how sure we are
Section titled “One number hides how sure we are”Here is a tiny test, made up so that you can see every value. Six customers saw version A of a shop, and six saw version B. This is what each one spent during the test, in dollars:
| Group | Customer 1 | 2 | 3 | 4 | 5 | 6 | Mean |
|---|---|---|---|---|---|---|---|
| A | 0 | 0 | 0 | 40 | 0 | 80 | 20 |
| B | 0 | 60 | 0 | 0 | 50 | 70 | 30 |
B’s mean is 10 dollars higher: +50%. TermOmarLarkfield’s product manager. He asks the business questions: what a change costs, and what it gives the shop., the product manager, would write “B raised revenue by 50%”. But look at the rows. Most customers spent nothing, and one big order more or less would change everything. If customer 6 in A had spent 140 instead of 80, the two means would be equal.
The idea: a range around the difference
Section titled “The idea: a range around the difference”We want a range of effects that agrees with the data. The width of that range depends on two things. The first is how much customers differ from each other: the standard deviation. The second is how many customers each group has.
The standard error
Section titled “The standard error”The standard error (SE) is the typical size of the chance error in the difference. For a difference of two means, it is:
SE = √(sdA² / nA + sdB² / nB)
In the tiny test, both groups have a standard deviation of 33.47 dollars and 6 customers. So SE = √(33.47² / 6 + 33.47² / 6) = 19.32 dollars.
The 95% confidence interval
Section titled “The 95% confidence interval”The 95% confidence interval is the difference plus or minus about two standard errors (1.96, to be exact):
10 − 1.96 × 19.32 = −27.9 and 10 + 1.96 × 19.32 = +47.9 dollars.
So B could be 28 dollars worse or 48 dollars better. The interval contains zero, the line of “no effect”. The tiny test cannot tell a good change from a bad one.
Now imagine the same pattern with 100 times more customers: 600 in each group, with the same means and the same spread. The standard error falls by √100 = 10, to 1.93 dollars. The interval becomes +6.2 to +13.8 dollars. The difference is the same, but now zero is far outside the range.
Text description of the chart
Two rows. The first row, 6 customers in each group, shows the difference +10.0 with an interval from −27.9 to +47.9 dollars. It crosses the dashed no-effect line at 0. The second row, 600 customers in each group, shows +10.0 with an interval from +6.2 to +13.8 dollars, entirely to the right of 0.
The names: t-test and p-value
Section titled “The names: t-test and p-value”The interval comes from a t-test. With very few customers, the test uses a number a little larger than 1.96. For 6 customers per group, the exact interval is −33.1 to +53.1 dollars. With thousands of customers, the number is 1.96. Welch’s t-test is the version that allows the two groups to have different spreads. It is the safe default, and the course project uses it.
The test also gives a p-value. It answers one question: if the change did nothing, how often would luck alone make a difference at least this large? For the tiny test, p = 0.62: luck does this very often. A small p-value, such as 0.01, says that “no effect at all” is hard to believe. It does not say how large the effect is, and it does not say that the change is worth it. The interval says both.
A 95% interval is not a promise about one test. It is a method that contains the true effect in about 95 of 100 tests. Our simulator can check this promise, because it knows the truth.
Hana’s test: the free-delivery result
Section titled “Hana’s test: the free-delivery result”Hana, the merchandising lead, tested free delivery from 50 dollars instead of 75. For 28 days, a coin flip gave each customer version A (today’s threshold) or version B. The simulator made the data, and it knows the true effect. You run the analysis with the course project.
First, update the project to the snapshot end-of-m03. It adds every tool of this module: the bootstrap, the sample ratio check, CUPED and segments. You copy its files over your project folder. Your own runs and plans stay.
-
Open a terminal in the folder that holds
larkfield-experimentsandnextia-learning-labs, for example~/projects. If you cloned the labs repository with Git, update it first.Terminal window git -C nextia-learning-labs pull -
Copy the snapshot over your project.
Terminal window cp -R nextia-learning-labs/C40/snapshots/end-of-m03/. larkfield-experiments/cd larkfield-experimentssource .venv/bin/activateTerminal window Copy-Item -Recurse -Force nextia-learning-labs\C40\snapshots\end-of-m03\* larkfield-experimentscd larkfield-experiments.venv\Scripts\Activate.ps1The packages are the same as in Module 2, so you install nothing new.
-
Run the tests.
Terminal window python -m pytest -q............... [100%]17 passed in 2.72sSeventeen tests pass. Your time will differ.
-
Run the free-delivery test.
Terminal window sim run --scenario deliveryScenario delivery (seed 40): Free delivery from 50 dollars instead of 75: an A/B test by customer, 28 days.Wrote runs/delivery-s40: customers (110,759 rows), visits (358,302 rows). Synthetic data. 21.7 s -
Analyse it.
Terminal window xp analyseRun delivery-s40 (synthetic): Free delivery from 50 dollars instead of 75: an A/B test by customer, 28 days.Metric: revenue. A: 55,430 customers, mean 25.0504. B: 55,329 customers, mean 26.0770.difference +1.0266 (+4.1%), 95% interval +0.1583 to +1.8950, p = 0.0205 (Welch t-test)Saved to runs/delivery-s40/analysis.json. Compare with the true effect: xp check
Read the result
Section titled “Read the result”Each group has about 55,000 customers. A spent 25.05 dollars per customer in the 28 days, and B spent 26.08. The difference is +1.03 dollars, or +4.1%. The 95% interval goes from +0.16 to +1.90 dollars, and p = 0.0205.
The interval does not cross zero, so the test found a gain. But look at its width. The gain could be 16 cents or 1.90 dollars per customer. Over a year of customers, these two numbers lead to very different decisions. Free delivery costs TermLarkfieldA made-up Canadian online shop for home and garden products. It is not a real company. All Nextia courses use its help desk and data as the example.Learn more Meet Larkfield money for every extra order, so Hana needs the whole range.
Now ask the simulator.
xp checkRun delivery-s40: the last analysis was `xp analyse`.True effect (from the simulator): +0.6979 dollars per customer.Your estimate: +1.0266, 95% interval +0.1583 to +1.8950: the interval contains the true effect.The true effect is +0.70 dollars per customer (+2.8%). The interval contains it, as it should. But the estimate, +1.03, is almost 50% higher than the truth. This run was lucky.
One lucky run
Section titled “One lucky run”Each seed of the simulator is a new month of customers with the same true change. The chart shows the test on 20 other seeds.
Text description of the chart
Twenty rows, one per seed. Each row shows the estimated difference with its 95% interval, on an axis from −1.5 to +3 dollars. A dashed line marks 0 (no effect) and another marks the true effect, about +0.74. Eighteen intervals contain their seed’s true effect. Only 8 stay entirely above 0; the other 12 cross the no-effect line, so on those seeds the test finds nothing.
Eighteen of the 20 intervals contain their seed’s truth, and only 8 stay above zero. Over 200 seeds, the course measured that 95.5% of the intervals contain the truth: the promise of a 95% interval holds. But only 41.5% of the intervals stay above zero. On most seeds, Hana’s test would say “no clear effect”, although the change works. You saw why in Sample size and power. The test can find 1.23 dollars with 80% power, and the truth is 0.70. Our default seed is one of the lucky ones. Keep this in mind for the rest of the module.
The bootstrap: the same interval without a formula
Section titled “The bootstrap: the same interval without a formula”The formula for the standard error assumes that the mean behaves well. Revenue is a strange metric: 78% of customers spend nothing, and a few spend hundreds of dollars. Is the formula still right? The bootstrap checks it with a computer instead of a formula.
Think of the test as a bag of customer cards. You draw a card, write it down, and put it back. You do this until you have as many cards as the group had. This new sample is a little different from the real group: some customers appear twice, some not at all. Its difference B − A is a little different too. Do this 1,000 times, and the spread of the 1,000 differences shows how much luck moves the result.
xp analyse --bootstrap 1000Run delivery-s40 (synthetic): Free delivery from 50 dollars instead of 75: an A/B test by customer, 28 days.Metric: revenue. A: 55,430 customers, mean 25.0504. B: 55,329 customers, mean 26.0770. difference +1.0266 (+4.1%), 95% interval +0.1927 to +1.8592, p = 0.014 (bootstrap (1000 resamples))Saved to runs/delivery-s40/analysis.json. Compare with the true effect: xp checkThe bootstrap interval, +0.19 to +1.86, is almost the same as Welch’s, +0.16 to +1.90. Here is the spread of the notebook’s 1,000 bootstrap differences:
Text description of the chart
A histogram of 1,000 values from about −0.4 to +2.4 dollars, shaped like a bell centred near +1.0. Markers show 0 (no effect), the 2.5% point at +0.19 and the 97.5% point at +1.86.
With 110,000 customers, the formula and the bootstrap agree. Use the bootstrap when no simple formula exists, for example for a TermMedianThe middle value when you sort the values. Half are lower and half are higher. One very slow call does not change it much, so it is a good “usual” time.Learn more Measure practical performanceLLMs: How They Work and How to Choose One, a percentile or a metric made of several steps. It costs computer time: 1,000 rounds over all customers take a few seconds here, and much longer on a large shop’s data. Over 50 seeds, the bootstrap interval contained the truth 98% of the time.
Shares and ratios
Section titled “Shares and ratios”Revenue per customer is a mean. Two other kinds of metric need a small change.
A share, such as the share of customers who ordered, is a mean of 0 and 1 values. The same test works. Its standard error has a short form: √(pA(1 − pA)/nA + pB(1 − pB)/nB). This is the z-test for two proportions. With thousands of customers, it gives the same answer as Welch’s t-test.
xp analyse --metric convertedRun delivery-s40 (synthetic): Free delivery from 50 dollars instead of 75: an A/B test by customer, 28 days.Metric: converted. A: 55,430 customers, mean 0.2146. B: 55,329 customers, mean 0.2261. difference +0.0115 (+5.4%), 95% interval +0.0066 to +0.0164, p = 3.84e-06 (Welch t-test)Saved to runs/delivery-s40/analysis.json. Compare with the true effect: xp checkThe share who ordered rose from 21.46% to 22.61%: +1.15 percentage points, the plain difference of two percentages. In relative terms, it is +5.4%. Always say which one you mean. xp check gives the truth: +0.75 points, inside the interval.
A ratio, such as returns per order, needs the delta method, so that the customer stays the unit. You met it in Choose the metrics. This is the plan’s guardrail:
xp analyse --metric return_rateRun delivery-s40 (synthetic): Free delivery from 50 dollars instead of 75: an A/B test by customer, 28 days.Metric: return_rate. A: 11,893 customers, mean 0.0831. B: 12,508 customers, mean 0.0960. difference +0.0129 (+15.5%), 95% interval +0.0068 to +0.0190, p = 3.7e-05 (ratio, delta method)Saved to runs/delivery-s40/analysis.json. Compare with the true effect: xp checkReturns rose from 8.31% to 9.60% of orders: +1.29 points, from +0.68 to +1.90. The truth is +1.17 points. The guardrail is clearly worse.
Three results on one chart
Section titled “Three results on one chart”Put the results side by side, each with its interval and the no-effect line. This is how a result should reach the people who decide.
Text description of the chart
Three panels, each with its own axis. The reminder email: +1.47 percentage points, interval +1.00 to +1.94; true effect +1.19. Free delivery, revenue per customer: +1.03 dollars, interval +0.16 to +1.90; true effect +0.70. Free delivery, returns per order: +1.29 percentage points, interval +0.68 to +1.90; true effect +1.17. All three intervals are to the right of the no-effect line, and each contains its true effect.
TermPriyaLarkfield’s data scientist. She measures how well the assistant works, on test sets and not on guesses. writes the free-delivery result for Hana like this:
Free delivery from 50 dollars raised revenue per customer by 1.03 dollars (+4.1%) in our test. The real effect is probably between 0.16 and 1.90 dollars. Returns rose too, from 8.3% to 9.6% of orders (+0.7 to +1.9 points). Before we decide, we must check the test itself, and compare the gain with the cost of the extra deliveries and returns.
Each sentence has the effect, its unit and its range, and no p-value without an effect.
Guided practice: the result in the notebook
Section titled “Guided practice: the result in the notebook”- Open the notebook from the box Do this lesson in a notebook at the top of this page.
- Run the setup cells and the helper cell. The data cell ends with
Ready: 2 files in data/. - In section 1, predict: will the mean or the median show the change? Then run the cell. Both medians are 0, and
share with no order: 0.78. - In section 2, run Welch’s t-test. You see
difference +1.027 dollars (+4.1%), SE 0.443, 95% interval +0.158 to +1.895, p = 0.0205. - In section 3, run the bootstrap. You see
bootstrap 95% interval +0.193 to +1.859. - In section 4, compare with the truth. You see
inside Welch's interval: True. - In section 5, compute
conv_low: the lower end of the interval for the share who ordered. - Run the check cell. It prints
Check passed.
Show the answer to step 7
pa = (d.loc[d.variant == "A", "orders"] > 0).mean()pb = (d.loc[d.variant == "B", "orders"] > 0).mean()na, nb = len(a), len(b)se_c = np.sqrt(pa * (1 - pa) / na + pb * (1 - pb) / nb)conv_low = round(pb - pa - 1.96 * se_c, 4)print(pa, pb, conv_low)The lower end is 0.0066: +0.66 percentage points, the same as xp analyse --metric converted. The formula for two proportions and Welch’s t-test agree.
Your turn: orders per customer
Section titled “Your turn: orders per customer”Change one thing: the metric. Free delivery could bring more orders, smaller orders, or both. Analyse orders per customer in the project, with --metric orders. Before you run it, predict: will the interval be above zero? Then check your interval against the truth.
Show the answer
xp analyse --metric ordersxp check difference +0.0171 (+5.8%), 95% interval +0.0090 to +0.0252, p = 3.19e-05 (Welch t-test)...True effect (from the simulator): +0.0130 orders per customer.Your estimate: +0.0171, 95% interval +0.0090 to +0.0252: the interval contains the true effect.B’s customers made 0.0171 more orders each: +5.8%, from +0.0090 to +0.0252. The truth, +0.0130, is inside. Orders rose by more than revenue (+4.1%), because the orders are a little smaller. This fits the change: customers now order smaller baskets to get free delivery. It is a diagnostic metric: it explains the primary metric, but it does not decide.
Common mistake
“p < 0.05, so it works”
Symptom: TermTomásLarkfield’s AI engineer. He often suggests a quick shortcut, and the lesson shows why it fails. reads p = 0.0205 and writes to Omar: “Free delivery raises revenue by 4.1%. It is significant, so we ship.” The simulator says that the true gain is +2.8%, and the returns guardrail is clearly worse.
Cause: the p-value only says that “no effect at all” is hard to believe. It does not say how big the effect is. On a small test, the estimate that passes p < 0.05 is often a lucky high one. On this seed, +4.1% against a true +2.8%.
Fix: report the effect with its interval and its unit: “+1.03 dollars per customer, probably between +0.16 and +1.90”. Report the guardrails in the same way. Let the people who decide compare the whole range with the cost. A p-value can go in a footnote.
Check your understanding
Two tests both give p = 0.03. Test 1: +0.20 dollars per customer, 95% interval +0.02 to +0.38. Test 2: +3.00 dollars, interval +0.30 to +5.70. Are they equally good news? What does each interval tell Hana?
Show the answer
No. The same p-value only says that both effects are probably not zero. Test 1 is precise and small: the gain is at most 38 cents per customer. If free delivery costs more than that, it loses money. Test 2 may be large, but it is very uncertain: anything from 30 cents to 5.70 dollars. Hana should decide on test 1 now, and probably run test 2 for longer, or with more customers, before she counts on its gain.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
Predict and explain: Hana runs the free-delivery test again with 4 times as many customers in each group. The customers spend in the same way. What happens to the width of the interval? And to the true effect?
Show the answer
The interval becomes about half as wide, because the standard error falls with the square root of the customers: √4 = 2. The interval of 0.16 to 1.90 dollars (1.74 wide) would be about 0.87 wide. The true effect does not change: it is about +0.70 dollars, because the change is the same. More customers make the measurement sharper. They do not make the effect bigger.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
Over 200 seeds of the free-delivery test, 95.5% of the intervals contain the true effect, but only 41.5% are above zero. Explain both numbers in your own words. Which one tells you that the test is too small?
Show the answer
The first number is the coverage: the interval method is honest, because it contains the truth about 95% of the time, as it promises. The second number is the power: how often the test finds a gain that is really there. It is low because the true effect, 0.70 dollars, is smaller than the 1.23 dollars that this test can find with 80% power. The second number shows that the test is too small. An honest interval can still be too wide to give an answer.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Recap
- A difference between two groups is one number, and luck alone also makes differences. The 95% confidence interval shows the range of effects that agree with the data. The standard error sets its width, and it falls with the square root of the number of customers.
- Welch’s t-test gives the interval and the p-value for a difference of means. A share uses the same test (the z-test for two proportions). A ratio per order uses the delta method, so that the customer stays the unit.
- The bootstrap gives an interval without a formula, from 1,000 samples drawn with replacement. On the free-delivery test, it agrees with Welch: +0.19 to +1.86 against +0.16 to +1.90 dollars.
- The free-delivery test found +1.03 dollars (+4.1%), from +0.16 to +1.90; the simulator’s truth is +0.70. The intervals contain the truth in 95.5% of 200 seeds, but find the gain in only 41.5%: our seed is a lucky one. Returns per order rose by 1.29 points.
- Report the effect, its unit and its range, for the primary metric and the guardrails. A p-value alone hides how big the effect is.
You can now read a result and its range. But a result is only as good as the test that made it. Next, you check the test itself before you trust its result.