Experiments as a team habit
- Lesson 30
- 30 min
- Module 8: Experiments for AI features and in production
Where we are
One good report is not enough: a team runs many tests, and the traps come back. In this lesson, Mateo sets up a test log, pre-registration and a review before launch, and we look at the common traps of teams that test.
Lesson 30 of 30 · Experiments and Causal Impact: Did Our Change Work?Experiments as a team habit
1. Why "it went up" is not an answer
2. Design an A/B test
3. Analyse the test
4. Traps in real tests
5. When you cannot randomise: natural experiments
6. Adjust for confounders in observational data
7. Uplift: who should get the treatment?
8. Experiments for AI features and in production
What you will be able to do
Make good tests a habit of the whole team, not of one careful analyst. Keep a test log of every test. Write the plan before the test, and review it before launch. Know the traps that appear only when a team runs many tests: shipping only “winners”, and testing until something wins.
NotebookDo this lesson in a notebook
In the notebook, you simulate a year of tests for a team: 1,000 ideas, some of which work. You count the false wins, measure how much the winners exaggerate, and see what “testing until something wins” does. It simulates the tests, so it needs no practice data. About 25 minutes.
Colab and Kaggle open your own copy. They need a free Google or Kaggle account. Stuck? See the solution notebook.
Before you startWhat to know and to have ready
What you need to know
- The test plan, from Write the plan first.
- Power and the minimum detectable effect, from Sample size and power.
- Peeking and many metrics, from Peeking and Many metrics and many tests.
- The result report, from Report a result people trust.
What you need on your computer (only for the project steps)
- The project
larkfield-experimentsat the snapshotend-of-m08, from Test a model or a prompt online, with.venvactive. - Your plans
plans/free-delivery.mdandplans/reminder-email.md, from Write the plan first and its Your turn.xp log addchecks that each plan file exists.
Tested on macOS with Python 3.12 and the end-of-m08 snapshot. Windows was not tested.
A gardener’s notebook
Section titled “A gardener’s notebook”A gardener tries something new every year: a new fertiliser, a new place for the tomatoes, a new way to water. In a good year, she tells everyone about the new idea. In a bad year, she forgets it.
After ten years, she remembers ten “great ideas”. But many good years came from the sun and the rain, not from her ideas. And she has no record of the ideas that did nothing. So she tries some of them again.
Her neighbour keeps a notebook. Before each season, he writes what he will try and how he will judge it. At the end, he writes the result, good or bad. After ten years, he knows which ideas work. He also knows how often a “great year” was only luck.
A team that runs tests needs the neighbour’s notebook. Medicine learned this the hard way. Many countries now ask that a clinical trial is registered before it starts, because unpublished failed trials made treatments look better than they were.
The problem: a team that ships only “winners”
Section titled “The problem: a team that ships only “winners””Each test in this course had a 5% chance of a false winner, or less. That sounds safe. But a team does not run one test. It runs hundreds, and it ships the ones that win. Two things then go wrong.
Some winners are false
Section titled “Some winners are false”Here is a year of tests at a made-up company. The numbers are an example, not a measurement. The team tests 1,000 ideas. 200 of them really work. The tests have 80% power. A test of an idea that does nothing shows a false win, above zero, in 2.5% of cases: half of the 5%.
Follow the numbers:
- real wins: 200 × 80% = 160;
- false wins: 800 × 2.5% = 20;
- wins in total: 160 + 20 = 180, and 20 / 180 = about 1 in 9 are false.
If only 100 ideas of 1,000 worked, the false wins would be 900 × 2.5% = 22.5, against 80 real ones: more than 1 in 5. Most ideas do not work. This is why xp log stats prints: “Most changes do not win”.
The winners exaggerate
Section titled “The winners exaggerate”The second problem is quieter. Even when a winner is real, its estimate is usually too large. This is called the winner’s curse.
Look at Hana’s free-delivery test. The course ran it 200 times with the simulator. The true effect is about +0.75 dollars per customer on average over these runs. The test is under-powered: an estimate wins only when it is above about +0.88.
Text description of the chart
A histogram of 200 estimates from about −0.55 to +2.32 dollars, shaped like a bell centred near +0.75. Markers: the truth on average, +0.75, and the mean of the 83 winning estimates, +1.19. The smallest winning estimate is about +0.88.
A test can only win when its luck pushes the estimate up. So the winners, as a group, carry good luck. Over the 200 runs, the 83 winners estimated +1.19 dollars on average. Their true effect was +0.75. A team that adds up its winners promises more than the changes deliver.
Testing until something wins
Section titled “Testing until something wins”TermTomásLarkfield’s AI engineer. He often suggests a quick shortcut, and the lesson shows why it fails. tried five layouts of the product page, one test after another. None won. He tried five more. The tenth won, and he shipped it.
If none of the ten layouts did anything, what was his chance of a false win? Each test has a 2.5% chance. The chance that none of k tests wins is 0.975 to the power k:
- 1 test: 1 − 0.975 = 2.5%;
- 5 tests: 1 − 0.9755 = 11.9%;
- 10 tests: 1 − 0.97510 = 22.4%;
- 20 tests: 1 − 0.97520 = 39.7%.
Text description of the chart
A column chart. One test with fixed rules: 2.5% chance of a false win above zero. Up to 10 tests until one wins: 22.4%. Stopping at the first daily win in 28 days: 27.6% (10,000 simulated A/A tests). Reporting the best of 20 metrics: 56% (300 simulated tests).
Each habit looks harmless alone. Together, they turn a test of nothing into a likely “winner”. The single test does not lie. The team’s habits do.
The idea: habits that keep the team honest
Section titled “The idea: habits that keep the team honest”Pre-registration: the plan, with a date
Section titled “Pre-registration: the plan, with a date”Pre-registration means: write the plan, and store it where it gets a date, before the test starts. You wrote plans with xp plan in Write the plan first. Keep them in version control (git), so everybody can see when each line was written. A change after the start needs a dated note and a reason.
A review before launch
Section titled “A review before launch”Before a test starts, a second person reads the plan. Mateo, TermLarkfieldA made-up Canadian online shop for home and garden products. It is not a real company. All Nextia courses use its help desk and data as the example.Learn more Meet Larkfield’s growth analyst, uses five questions:
- Does
xp plan checksay the plan is ready? - Is the sample size big enough for the smallest effect that matters? Check with
xp power. - Is the unit right? Can customers in A and B affect each other?
- Does the split work? Run an A/A test or check the first day’s sample ratio.
- Are the guardrails in the plan, with a rule for each?
The review takes 20 minutes. It catches the redirect bug before 28 days are lost, not after.
A test log
Section titled “A test log”A test log is one line per finished test: what was tested, the plan, the effect with its range, and the decision. Every test goes in: winners, losers, “no clear effect” and “do not use”. The log shows the team’s habits. It also shows how often ideas really work, which helps to plan the next ones.
Larkfield: Mateo’s test log
Section titled “Larkfield: Mateo’s test log”xp log keeps the log in a file test-log.csv in the project folder. xp log add takes the last analysis of a run.
-
Check that the log is empty.
Terminal window xp log statsThe test log is empty. Add a test with: xp log add RUN --decision ... -
Analyse the free-delivery test, and log it with Hana’s decision from the report.
Terminal window sim run --scenario deliveryxp analysexp log add delivery --decision discuss --plan plans/free-delivery.md...difference +1.0266 (+4.1%), 95% interval +0.1583 to +1.8950, p = 0.0205 (Welch t-test)Saved to runs/delivery-s40/analysis.json. Compare with the true effect: xp checkLogged delivery-s40: discuss.We show only the last lines of the output.
-
Log three more tests: the reminder email, the test with the redirect bug, and Tomás’s checkout button from Peeking. The team shipped the button on day 3, without a plan.
Terminal window sim run --scenario emailxp analysexp log add email --decision ship --plan plans/reminder-email.mdsim run --scenario srmxp analysexp log add srm --decision "do not use" --plan plans/free-delivery.mdsim run --scenario early-stopxp analysexp log add early-stop --decision shipThe last command prints a warning:
Logged early-stop-s1197: ship. Warning: no test plan. -
List the log, and show its statistics.
Terminal window xp log listxp log statsYYYY-MM-DD delivery-s40 discuss revenue: +1.0266 (+0.1583 to +1.8950) plan: plans/free-delivery.mdYYYY-MM-DD email-s40 ship ordered: +0.0147 (+0.0100 to +0.0194) plan: plans/reminder-email.mdYYYY-MM-DD srm-s40 do not use revenue: +2.9012 (+2.0226 to +3.7798) plan: plans/free-delivery.mdYYYY-MM-DD early-stop-s1197 ship revenue: +0.8510 (-0.7827 to +2.4847) plan: NONE4 tests: 2 shipped (50%), 1 without a written plan.Most changes do not win: a team that ships most of its tests may be shipping noise.Your log shows today’s date in place of
YYYY-MM-DD.
What the log shows
Section titled “What the log shows”Read the last line of the list. The checkout button was shipped, with no plan, and its TermConfidence intervalA range around a measured value, made by a method that holds the true value 95 times in 100. If the interval of a difference does not include zero, the difference is probably not chance.Learn more Estimate uncertaintyEssential Mathematics and Statistics for AI over the full 28 days goes from −0.78 to +2.48 dollars: it crosses zero. The simulator knows that the button does nothing. One look at the log finds the test that should not have shipped.
The log has limits. xp log add checks that the plan file exists, but not that the team wrote it first: that is the review’s job. It logs the last effect that you analysed for the run, for example with CUPED. If your last command was xp srm, it logs the primary metric without CUPED, and says so. So run your primary analysis last, just before xp log add.
Two shipped of four is a small sample. In a real log of a hundred tests, a team that ships 50% or more should ask whether it ships noise.
When to set up these habits, and what they cost
Section titled “When to set up these habits, and what they cost”Start the log with the first test. It costs one line per test, and it gets more useful every month.
Add the review before launch when more than one person runs tests, or when a test costs more than a week of traffic.
Pre-registration costs almost nothing once the plans live in git. The hard part is not the tool. It is the discipline to log the losers and to keep a plan that you would like to change.
Do not turn the habits into paperwork. A one-page plan, a 20-minute review and one line in a log are enough for most tests.
Guided practice: a year of tests in a notebook
Section titled “Guided practice: a year of tests in a notebook”- Open the notebook from the box Do this lesson in a notebook at the top of this page.
- In section 1, simulate 1,000 tests: 200 ideas with a true effect equal to the minimum detectable effect, 800 with no effect.
- In section 2, count the wins, and the false wins among them. You see 158 real wins and 16 false wins: 9.2% of the wins are false, close to the 1 in 9 of the tree.
- In section 3, compare the mean estimate of the real winners with their true effect. You see 1.10 against 1.00: the winners exaggerate.
- In section 4, simulate “testing until something wins” with up to 10 tests of nothing. Put the share of teams that find a false win in
false_win_share. Run the check cell. It printsCheck passed.
Show the answer to step 5
wins = (rng.normal(0, SE, size=(TEAMS, 10)) > 1.96 * SE).any(axis=1)false_win_share = round(float(wins.mean()), 3)The value is 0.224, the same as the formula 1 − 0.97510 = 0.224.
Your turn: review a plan before launch
Section titled “Your turn: review a plan before launch”Rafael sends Mateo this plan for a new subject line test. Use the five review questions. Find at least three problems before you read the answer.
# Test plan: subject-line
- **Hypothesis:** the new subject line is better.- **Unit:** the email (each email sent gets A or B at random).- **Primary metric:** we will look at opens, clicks and orders, and choose the clearest.- **Guardrail metrics:** TODO- **Minimum detectable effect:** 0.2 points of orders- **Sample size:** 5,000 emails per group- **Duration:** until it wins, at most 4 weeks- **Stopping rule:** we check every morning- **Decision if B is better:** ship B- **Decision if B is worse:** keep A- **Decision if no clear effect:** try another subject lineShow the answer
- The primary metric is not chosen: “the clearest” of three is the many-metrics trap. Choose one, for example orders within 14 days.
- The guardrail is missing.
xp plan checkwould say “Not ready”. Unsubscribes are the obvious guardrail for an email. - The unit is the email. A customer who gets several emails could see both lines. Use the customer.
- The size is far too small. At a 9% base rate, finding 0.2 points needs 324,616 customers per group:
xp power --baseline 0.09 --mde 0.002says so. Either test a bigger effect or say that this question cannot be answered. - “Until it wins” and “every morning” are peeking. Fix the duration, or use the sequential test of Peeking.
- “Try another subject line” after no clear effect is testing until something wins. Log the result, and count all the tries.
Common mistake
“Our tests won 40 times this year, so sales should be up 40 × 2%”
Symptom: a team adds up the effects of all its winning tests and promises a large rise in sales. The rise never comes.
Cause: some winners were false, because most ideas do nothing. The real winners were estimated too high, because a test wins only when its luck pushes the estimate up. Some tests were run until they won, or checked every day. The losers were never written down, so nobody could see how often ideas fail.
Fix: log every test, with its plan and its range. Expect real effects to be smaller than the winning estimates. For big decisions, run the test again. Keep a holdout: a small group of customers who get none of the year’s changes. At the end of the year, compare it with everyone else.
Check your understanding
A team tests 1,000 ideas a year. 100 really work, the tests have 80% power, and an idea that does nothing wins by chance in 2.5% of tests. How many wins are in the log, and what share of them is false?
Show the answer
Real wins: 100 × 80% = 80. False wins: 900 × 2.5% = 22.5. Wins in total: about 102.5. Of these, 22.5 / 102.5 = about 22% are false: more than 1 in 5. Fewer good ideas means a larger share of false wins.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
In 200 simulated runs of the free-delivery test, the 83 winning runs estimated +1.19 dollars on average, but their true effect was about +0.75. Explain why, in two sentences.
Show the answer
The test is under-powered, so a run wins only when its random error pushes the estimate above about +0.88 dollars. The winners are therefore the runs with good luck, and their average is too high: this is the winner’s curse.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
Name three habits that protect a team that runs many tests, and the trap that each one stops.
Show the answer
A plan written and dated before the test (pre-registration) stops choosing the metric or the stopping day after seeing the result. A review before launch stops a broken design, such as a wrong unit or a too-small sample, from wasting a test. A test log of every test, with losers, stops shipping only “winners” and testing until something wins, because every try is counted.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Recap
- Most ideas do not work. In a made-up year of 1,000 tests with 200 good ideas, about 1 win in 9 is false. With 100 good ideas, more than 1 in 5 is false.
- Winners exaggerate: in 200 simulated free-delivery runs, the winners estimated +1.19 dollars, against a truth of about +0.75. This is the winner’s curse.
- Testing until something wins: up to 10 tries of nothing give a false win in 22.4% of cases.
- The habits: a dated plan (pre-registration), a review before launch, a report, and a test log of every test.
xp logkeeps it, andxp log statsshows how much the team ships and how many tests had no plan.
This is the end of the teaching modules. Next, the case studies: real data from a mailing, an online shop and a minimum wage.