Check the test before the result
- Lesson 10
- 35 min
- Module 3: Analyse the test
Where we are
We can compute a result and its interval. But a result is only as good as the test behind it. In this lesson, we check the test before we read the result: a sample ratio mismatch, the exposure logs, and an A/A test that checks the system itself.
Lesson 10 of 30 · Experiments and Causal Impact: Did Our Change Work?Check the test before the result
1. Why "it went up" is not an answer
2. Design an A/B test
3. Analyse the test
4. Traps in real tests
5. When you cannot randomise: natural experiments
6. Adjust for confounders in observational data
7. Uplift: who should get the treatment?
8. Experiments for AI features and in production
What you will be able to do
Check that a test worked before you read its result. A broken split or a lost group of customers gives a confident, precise and wrong answer, and the result itself never shows it.
NotebookDo this lesson in a notebook
In the notebook, you plant a redirect bug in the free-delivery data yourself. Then you find it with a chi-square test, see what it does to the result, and run 1,000 A/A tests. Its setup cells install the packages and download the practice data, so it needs no local setup. About 25 minutes.
Colab and Kaggle open your own copy. They need a free Google or Kaggle account. Stuck? See the solution notebook.
Before you startWhat to know and to have ready
What you need to know
- The result and its interval, and the free-delivery test, from The result and its uncertainty.
- Why a coin flip makes the groups alike, from Randomise.
- Hypothesis tests and p-values, from Compare alternatives in Essential Mathematics and Statistics for AI.
What you need on your computer (only for the project steps)
- The project
larkfield-experimentsat the snapshotend-of-m03, from The result and its uncertainty, with.venvactive.
Tested on macOS with Python 3.12 and the end-of-m03 snapshot of the project. Windows was not tested.
A class that lost some students
Section titled “A class that lost some students”A school tests a new homework app. A teacher flips a coin for each of 60 students: 30 get the app, 30 keep the paper homework. At the end of the term, she compares the marks. But only 24 students in the app group took the final test, against 30 in the paper group.
Who are the six missing students? The app needed a phone at home. Students without one stopped using it, fell behind and missed the test. So the app group that took the test is not the group that the coin made. Its average looks better, but part of that is only who is left.
Before she compares the marks, the teacher must count the students. If the groups are not the size that the coin made, the comparison is broken. No statistics can repair it afterwards.
The problem: a precise answer that is wrong
Section titled “The problem: a precise answer that is wrong”TermLarkfieldA made-up Canadian online shop for home and garden products. It is not a real company. All Nextia courses use its help desk and data as the example.Learn more Meet Larkfield had the same problem. TermAmiraLarkfield’s developer. She writes the code of the help-desk tools in the courses., the developer, served version B of the free-delivery test through a redirect: a second page load to the new page. The simulator has this test as a scenario. Its bug is a practice problem we made on purpose.
-
Run the test with the redirect bug.
Terminal window sim run --scenario srmScenario srm (seed 40): The free-delivery test with a redirect bug that loses some B customers (planted): a 52/48 split.Wrote runs/srm-s40: customers (106,651 rows), visits (353,064 rows). Synthetic data. 11.6 s -
Analyse it as in the last lesson.
Terminal window xp analyseRun srm-s40 (synthetic): The free-delivery test with a redirect bug that loses some B customers (planted): a 52/48 split.Metric: revenue. A: 55,354 customers, mean 24.6616. B: 51,297 customers, mean 27.5628.difference +2.9012 (+11.8%), 95% interval +2.0226 to +3.7798, p = 9.71e-11 (Welch t-test)Saved to runs/srm-s40/analysis.json. Compare with the true effect: xp check -
Compare it with the truth.
Terminal window xp checkRun srm-s40: the last analysis was `xp analyse`.True effect (from the simulator): +0.8671 dollars per customer.Your estimate: +2.9012, 95% interval +2.0226 to +3.7798: the interval does NOT contain the true effect.
The result looks wonderful: +11.8%, a narrow TermConfidence intervalA range around a measured value, made by a method that holds the true value 95 times in 100. If the interval of a difference does not include zero, the difference is probably not chance.Learn more Estimate uncertaintyEssential Mathematics and Statistics for AI, p = 0.0000000001. It is also wrong. The true effect is +0.87 dollars, and the estimate is more than three times larger. The interval is far from the truth.
Text description of the chart
Four rows on an axis from −0.5 to +4.5 dollars, with a dashed no-effect line at 0. Clean test, estimate: +1.03, interval +0.16 to +1.90. Clean test, true effect: +0.70, inside that interval. Redirect bug, estimate: +2.90, interval +2.02 to +3.78. Redirect bug, true effect: +0.87, far to the left of that interval.
This is not bad luck. Over 200 seeds of this scenario, the interval contained the truth in only 3% of the runs. On average, the estimate was 1.67 dollars too high. A bigger test does not help. It makes the wrong answer more precise.
Nothing in the result warns you. You must check the test itself first.
The idea: count the customers first
Section titled “The idea: count the customers first”The plan said 50/50: the coin gives each customer an equal chance of A or B. With 106,651 customers, we expect about 53,326 in each group. The log has 55,354 in A and 51,297 in B. B has 48.10% of the customers instead of 50%.
A difference from the planned split is a sample ratio mismatch (SRM). The “sample ratio” is the ratio of the group sizes. A coin never gives exactly 50/50, so we need a test: is the gap larger than a coin makes by chance?
The chi-square test
Section titled “The chi-square test”The chi-square test compares the counts we see with the counts we expect. For each group, it computes (seen − expected)² / expected, and adds them. A large total means that the counts are far from the plan.
Here is a split of 52/48 at three sizes. Each test planned 50/50.
| Customers | A | B | Expected in each | Chi-square | p-value |
|---|---|---|---|---|---|
| 1,000 | 520 | 480 | 500 | 0.8 + 0.8 = 1.6 | 0.21 |
| 10,000 | 5,200 | 4,800 | 5,000 | 8 + 8 = 16 | 0.00006 |
| 100,000 | 52,000 | 48,000 | 50,000 | 80 + 80 = 160 | 1 × 10⁻³⁶ |
For 1,000 customers: (520 − 500)² / 500 = 400 / 500 = 0.8 for A, and the same for B. A coin makes a gap like this one test in five (p = 0.21). With 100,000 customers, the same 52/48 is almost impossible by chance. A large test must have a split that is very close to its plan.
The p-value says how often a fair coin makes a split at least this far from the plan. Most teams flag an SRM when p < 0.001. The limit is strict on purpose: you check the split of every test, and you do not want false alarms.
Check Larkfield’s splits
Section titled “Check Larkfield’s splits”Run the check on the clean test first, then on the broken one.
-
Check the clean free-delivery test.
Terminal window xp srm deliveryRun delivery-s40: A 55,430, B 55,329 customers; B is 49.95% (planned 50%).chi-square 0.1, p = 0.762: no mismatch. -
Check the test with the redirect bug.
Terminal window xp srm srmRun srm-s40: A 55,354, B 51,297 customers; B is 48.10% (planned 50%).chi-square 154.3, p = 1.96e-35: SAMPLE RATIO MISMATCH: do not use this result; find the cause first. -
Look for where the split breaks. Check each device.
Terminal window xp srm srm --by deviceRun srm-s40: A 55,354, B 51,297 customers; B is 48.10% (planned 50%).chi-square 154.3, p = 1.96e-35: SAMPLE RATIO MISMATCH: do not use this result; find the cause first.device = desktop: A 23,265, B 23,299 (B 50.04%), p = 0.875device = mobile: A 32,089, B 27,998 (B 46.60%), p = 1.57e-62
The first argument of xp srm names the run. Without it, xp uses the last run that sim run made.
Text description of the chart
Three bars on an axis that starts at 40%. All customers: 48.10%. Desktop: 50.04%. Mobile: 46.60%. The planned share is 50%.
Desktop is on plan. Mobile is not: B has only 46.60% of the mobile customers. Amira looks at the redirect. On a slow mobile connection, the second page load sometimes fails, and the customer leaves before any page appears.
Why the result is wrong, not only smaller
Section titled “Why the result is wrong, not only smaller”The lost customers are not a random part of B. They are mobile customers on slow connections, who visit less and buy less. The B group that the log sees has lost its weakest buyers, so its average is too high. A, which had no redirect, kept all of them.
So the two groups are no longer alike. The coin made them alike, and the bug undid that. The difference now mixes the effect of free delivery with the effect of who was lost. This is why the rule is: with an SRM, do not use the result. Find the cause, fix it, and run the test again.
The simulator tells you what it planted:
xp checkRun srm-s40: the last analysis was `xp srm`.Your check: a mismatch (p = 1.96e-35).The simulator: A redirect bug loses B customers on slow mobile connections. They visit and buy less than average, so the B group that the log sees is better than the real B group.Assignment and exposure logs
Section titled “Assignment and exposure logs”Where could Amira see the loss? A test usually writes two logs.
- The assignment log records the coin flip: customer 123 got B, at 10:02. It is written before the page loads.
- The exposure log records that the customer really saw the version: the page with the 50-dollar threshold appeared, at 10:02.
In a healthy test, almost every assigned customer is exposed, in both groups. Compare the two logs group by group. Here, the assignment log is 50/50 and the exposure log is 52/48. The missing B customers are between the two logs: assigned, never exposed. The loss tells Amira exactly where to look.
The analysis uses the exposure log, because only exposed customers could react to the change. That is right only while both groups lose customers in the same way. An SRM shows that they do not.
Two more checks belong here. A metric that the change cannot move, such as spending before the test (pre_spend), should be the same in both groups. A gap means that the groups differ in who they are. And the test should run for the days in the plan, with no gaps in the logs.
A/A tests: check the system
Section titled “A/A tests: check the system”An SRM check finds a broken split in one test. But what if the system itself is broken? For example, the analysis code could make intervals that are too narrow. Each test would then find “effects” too often.
Think of a new kitchen scale. To check it, you weigh the same bag of flour twice. If it shows two different weights, the scale is broken, not the flour.
An A/A test does the same with the test system. Both groups get the same shop, today’s version, so the true effect is zero. A healthy system finds an effect with p < 0.05 in about 5% of A/A tests: that is what the 0.05 means. If it finds one in 20% of A/A tests, something in the system is wrong.
-
Run an A/A test.
Terminal window sim run --scenario aaScenario aa (seed 41): An A/A test: both groups get today's shop, 28 days. The true effect is zero.Wrote runs/aa-s41: customers (111,078 rows), visits (361,090 rows). Synthetic data. 10.0 s -
Check its split.
Terminal window xp srmRun aa-s41: A 55,497, B 55,581 customers; B is 50.04% (planned 50%).chi-square 0.1, p = 0.801: no mismatch. -
Analyse it.
Terminal window xp analyseRun aa-s41 (synthetic): An A/A test: both groups get today's shop, 28 days. The true effect is zero.Metric: revenue. A: 55,497 customers, mean 25.8076. B: 55,581 customers, mean 24.9696.difference -0.8380 (-3.2%), 95% interval -1.7179 to +0.0418, p = 0.0619 (Welch t-test)Saved to runs/aa-s41/analysis.json. Compare with the true effect: xp check
The two groups saw the same shop, and B spent 84 cents less per customer: −3.2%, with p = 0.06. This is luck, and it came close to a “significant” result. Remember this number. Two identical groups of 55,000 customers can differ by 3% in a month.
One A/A test says little. Many A/A tests check the system. The chart shows the p-values of 100 A/A tests on 100 seeds.
Text description of the chart
Twenty bars of width 0.05 from 0 to 1. The bars have roughly the same height, about 5 tests each, with random ups and downs. A marker shows p = 0.05; the first bar, below it, has about the same height as the others.
The p-values are spread evenly: in a healthy system, every p-value is equally likely when nothing changes. Over 200 A/A runs, the course measured 5.5% false positives, close to the 5% we expect. The system is healthy.
Run an A/A test when the test system is new or changes: new assignment code, a new logging library, a new analysis. It costs traffic and time, and it finds nothing about a bug that only affects one version, such as the redirect. That is the job of the SRM check.
The checks before the result
Section titled “The checks before the result”Mateo, who runs Larkfield’s test log, adds this list to every test review. All of it comes before anyone reads the effect.
- The split: chi-square test of the group sizes, overall and by device. With p < 0.001, stop.
- The logs: the assigned and exposed customers per group. A loss in one group only is a bug.
- A sanity metric: spending before the test, the same in both groups.
- The dates: the planned days, with no gaps.
- The system: a recent A/A test with about 5% false positives.
Guided practice: plant and find a bug
Section titled “Guided practice: plant and find a bug”- Open the notebook from the box Do this lesson in a notebook at the top of this page.
- Run the setup cells and the helper cell. The data cell ends with
Ready: 2 files in data/. - In section 1, run the split check of the clean test. You see
chi-square 0.09, p = 0.762. - In section 2, plant the redirect bug: the cell removes some B customers on mobile who visited only once. Predict: will B’s mean revenue go up or down? Then run it.
- Run the split check on the broken log. You see a mismatch, and only mobile is off plan.
- In section 3, compare the estimate before and after the bug with the true effect.
- In section 4, run 1,000 A/A tests made from group A alone. You see a false-positive share close to 5%.
- In section 5, compute
chi_mobile: the chi-square statistic for the mobile customers of the broken log. - Run the check cell. It prints
Check passed.
Show the answer to step 8
seen = broken[broken.device == "mobile"].variant.value_counts()expected = seen.sum() / 2chi_mobile = round(((seen["A"] - expected) ** 2 + (seen["B"] - expected) ** 2) / expected, 1)print(seen["A"], seen["B"], chi_mobile)The notebook prints the counts and the statistic, and stats.chisquare gives the same value. A chi-square above about 10.8 means p < 0.001 for two groups.
Your turn: is it new customers?
Section titled “Your turn: is it new customers?”Mateo asks: “Is the loss in new customers or returning ones?” Check the broken test by new_customer (1 for new, 0 for returning). Predict first: will one kind be on plan, as desktop was?
Show the answer
xp srm srm --by new_customerRun srm-s40: A 55,354, B 51,297 customers; B is 48.10% (planned 50%). chi-square 154.3, p = 1.96e-35: SAMPLE RATIO MISMATCH: do not use this result; find the cause first. new_customer = 0: A 38,657, B 35,947 (B 48.18%), p = 3.35e-23 new_customer = 1: A 16,697, B 15,350 (B 47.90%), p = 5.29e-14Both kinds are off plan, by about the same amount. The bug does not care if a customer is new: it depends on the device and the connection. A check that breaks in every segment of one column, but only in one segment of another, points to the cause. Here, device is the column to follow.
Common mistake
“It is only 2 points off”
Symptom: TermTomásLarkfield’s AI engineer. He often suggests a quick shortcut, and the lesson shows why it fails. sees B is 48.10% and says: “That is close to 50%. Let’s use the result, and maybe cut the effect a little to be safe.” The report shows +2.90 dollars; the truth is +0.87.
Cause: the size of the mismatch does not show the size of the error. 1.9 points of the split hide a group of customers who were lost for a reason. They were the weakest mobile buyers, so the error in revenue is much larger than 2%. No simple correction removes it, because you do not know who was lost.
Fix: with p < 0.001, do not use the result. Find the cause in the logs, fix it, and run the test again. Record the broken test in the test log with its cause, so that the next test does not repeat it.
Check your understanding
A test planned 50/50 has 201,000 customers in A and 199,000 in B. Is this a sample ratio mismatch? Compute the chi-square statistic and say what you do next.
Show the answer
The expected count is 200,000 in each group. Chi-square = (201,000 − 200,000)² / 200,000 + (199,000 − 200,000)² / 200,000 = 5 + 5 = 10. For two groups, that gives p ≈ 0.0016. This is above the limit of 0.001, so it is not flagged as an SRM by the usual rule. But it is close. Check the split by device and by browser, and compare the assignment and exposure logs. If one segment is far off plan, treat it as a mismatch.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
Completion check. In the test with the redirect bug, B has 48.10% of the customers and the estimate is +2.90 dollars per customer. Say why this result must not be used, in three or four sentences, as you would to Omar.
Show the answer
The coin gave each customer a 50% chance of B, but the log has only 48.10% in B. With 106,651 customers, a fair coin almost never does that (p = 2e-35). So some B customers are missing, and the check by device shows who: mobile customers, lost by a redirect that fails on slow connections. They buy less than average, so the B group that we see is better than the real B group. The +2.90 dollars is mostly who was lost, not free delivery. The true effect is +0.87. We fix the redirect and run the test again before we decide.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
Over 100 A/A tests, a new analysis tool finds p < 0.05 in 19 of them. What does this tell you? What would a healthy tool show?
Show the answer
The groups in an A/A test see the same shop, so the true effect is zero. A healthy tool finds p < 0.05 in about 5 of 100 tests. Nineteen is far more. The tool is broken: for example, its intervals are too narrow, or it counts each visit as independent when the coin chose customers. Every “winner” that it found in real tests is now in doubt. Fix the tool and run the A/A tests again before you trust it.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Recap
- A broken test gives a confident, precise, wrong answer. The redirect bug lost B customers on slow mobile connections. The estimate was +2.90 dollars against a truth of +0.87, and the interval missed the truth in 97% of 200 seeds.
- A sample ratio mismatch (SRM) is a split that differs from the plan by more than chance. The chi-square test compares the counts with the plan. Flag an SRM at p < 0.001. Larkfield’s broken test had 48.10% in B, p = 2e-35, and the check by device found mobile (46.60%).
- With an SRM, do not use the result. The lost customers are not random, so the groups are no longer alike. Find the cause, fix it, and run the test again.
- Compare the assignment log (the coin flip) with the exposure log (who saw the version), group by group. A loss in one group shows where the bug is.
- An A/A test gives both groups the same version. A healthy system finds p < 0.05 in about 5% of them: the course measured 5.5% over 200 runs.
You can now trust a test before you read it. But a healthy test can still be too noisy to answer: the free-delivery interval was 1.74 dollars wide. Next, you use what customers spent before the test to remove part of that noise.