Skip to content
Français

Experiments as a team habit

  • Lesson 30
  • 30 min
  • Module 8: Experiments for AI features and in production

Where we are

One good report is not enough: a team runs many tests, and the traps come back. In this lesson, Mateo sets up a test log, pre-registration and a review before launch, and we look at the common traps of teams that test.

What you will be able to do

Make good tests a habit of the whole team, not of one careful analyst. Keep a test log of every test. Write the plan before the test, and review it before launch. Know the traps that appear only when a team runs many tests: shipping only “winners”, and testing until something wins.

NotebookDo this lesson in a notebookColab · Kaggle · GitHub

In the notebook, you simulate a year of tests for a team: 1,000 ideas, some of which work. You count the false wins, measure how much the winners exaggerate, and see what “testing until something wins” does. It simulates the tests, so it needs no practice data. About 25 minutes.

Colab and Kaggle open your own copy. They need a free Google or Kaggle account. Stuck? See the solution notebook.

Before you startWhat to know and to have readyAbout 30 minutes

What you need to know

What you need on your computer (only for the project steps)

  • The project larkfield-experiments at the snapshot end-of-m08, from Test a model or a prompt online, with .venv active.
  • Your plans plans/free-delivery.md and plans/reminder-email.md, from Write the plan first and its Your turn. xp log add checks that each plan file exists.

Tested on macOS with Python 3.12 and the end-of-m08 snapshot. Windows was not tested.

A gardener tries something new every year: a new fertiliser, a new place for the tomatoes, a new way to water. In a good year, she tells everyone about the new idea. In a bad year, she forgets it.

After ten years, she remembers ten “great ideas”. But many good years came from the sun and the rain, not from her ideas. And she has no record of the ideas that did nothing. So she tries some of them again.

Her neighbour keeps a notebook. Before each season, he writes what he will try and how he will judge it. At the end, he writes the result, good or bad. After ten years, he knows which ideas work. He also knows how often a “great year” was only luck.

A team that runs tests needs the neighbour’s notebook. Medicine learned this the hard way. Many countries now ask that a clinical trial is registered before it starts, because unpublished failed trials made treatments look better than they were.

The problem: a team that ships only “winners”

Section titled “The problem: a team that ships only “winners””

Each test in this course had a 5% chance of a false winner, or less. That sounds safe. But a team does not run one test. It runs hundreds, and it ships the ones that win. Two things then go wrong.

Here is a year of tests at a made-up company. The numbers are an example, not a measurement. The team tests 1,000 ideas. 200 of them really work. The tests have 80% power. A test of an idea that does nothing shows a false win, above zero, in 2.5% of cases: half of the 5%.

A tree. 1,000 ideas tested split into 200 that really work and 800 that do nothing. The 200 give 160 wins found (power 80%) and 40 missed that look like no effect. The 800 give 20 false wins (2.5% of 800, by chance) and 780 no wins. The 160 wins and the 20 false wins lead to a box: 180 wins in the log; 20 of them, about 1 in 9, are false.
A made-up year of 1,000 tests. Even with good tests, about 1 win in 9 is false, because most ideas do nothing. With fewer ideas that work, or less power, the share of false wins grows.

Follow the numbers:

  • real wins: 200 × 80% = 160;
  • false wins: 800 × 2.5% = 20;
  • wins in total: 160 + 20 = 180, and 20 / 180 = about 1 in 9 are false.

If only 100 ideas of 1,000 worked, the false wins would be 900 × 2.5% = 22.5, against 80 real ones: more than 1 in 5. Most ideas do not work. This is why xp log stats prints: “Most changes do not win”.

The second problem is quieter. Even when a winner is real, its estimate is usually too large. This is called the winner’s curse.

Look at Hana’s free-delivery test. The course ran it 200 times with the simulator. The true effect is about +0.75 dollars per customer on average over these runs. The test is under-powered: an estimate wins only when it is above about +0.88.

The estimates of 200 simulated runs of the free-delivery test. They spread around the truth, about +0.75. Only the 83 runs above about +0.88 win, and their mean is +1.19: more than half again the true effect.
Text description of the chart

A histogram of 200 estimates from about −0.55 to +2.32 dollars, shaped like a bell centred near +0.75. Markers: the truth on average, +0.75, and the mean of the 83 winning estimates, +1.19. The smallest winning estimate is about +0.88.

A test can only win when its luck pushes the estimate up. So the winners, as a group, carry good luck. Over the 200 runs, the 83 winners estimated +1.19 dollars on average. Their true effect was +0.75. A team that adds up its winners promises more than the changes deliver.

TermTomásLarkfield’s AI engineer. He often suggests a quick shortcut, and the lesson shows why it fails. tried five layouts of the product page, one test after another. None won. He tried five more. The tenth won, and he shipped it.

If none of the ten layouts did anything, what was his chance of a false win? Each test has a 2.5% chance. The chance that none of k tests wins is 0.975 to the power k:

  • 1 test: 1 − 0.975 = 2.5%;
  • 5 tests: 1 − 0.9755 = 11.9%;
  • 10 tests: 1 − 0.97510 = 22.4%;
  • 20 tests: 1 − 0.97520 = 39.7%.
The chance of a false win when nothing works. Up to 10 tests until one wins is computed (each test 2.5% for a win above zero). Daily peeking and 20 metrics are the course's measurements from Module 4, at p < 0.05.
Text description of the chart

A column chart. One test with fixed rules: 2.5% chance of a false win above zero. Up to 10 tests until one wins: 22.4%. Stopping at the first daily win in 28 days: 27.6% (10,000 simulated A/A tests). Reporting the best of 20 metrics: 56% (300 simulated tests).

Each habit looks harmless alone. Together, they turn a test of nothing into a likely “winner”. The single test does not lie. The team’s habits do.

The idea: habits that keep the team honest

Section titled “The idea: habits that keep the team honest”
Five steps in a row. 1, Plan, written before the test: metrics, size, stopping rule; stops choosing the metric after the result. 2, Review: before launch, a second person reads the plan; stops a broken design that nobody noticed until the end. 3, Run: no early look at the result; stops testing until something wins. 4, Report: decision, range, checks, limits; stops telling only the good half of the result. 5, Log: one line per test, including the losers; stops forgetting the losers.
The life of a test, and the trap that each step stops. The plan and the log are the two ends: the plan is written before the result, and the log keeps every result, also the losers.

Pre-registration means: write the plan, and store it where it gets a date, before the test starts. You wrote plans with xp plan in Write the plan first. Keep them in version control (git), so everybody can see when each line was written. A change after the start needs a dated note and a reason.

Before a test starts, a second person reads the plan. Mateo, TermLarkfieldA made-up Canadian online shop for home and garden products. It is not a real company. All Nextia courses use its help desk and data as the example.Learn more Meet Larkfield’s growth analyst, uses five questions:

  1. Does xp plan check say the plan is ready?
  2. Is the sample size big enough for the smallest effect that matters? Check with xp power.
  3. Is the unit right? Can customers in A and B affect each other?
  4. Does the split work? Run an A/A test or check the first day’s sample ratio.
  5. Are the guardrails in the plan, with a rule for each?

The review takes 20 minutes. It catches the redirect bug before 28 days are lost, not after.

A test log is one line per finished test: what was tested, the plan, the effect with its range, and the decision. Every test goes in: winners, losers, “no clear effect” and “do not use”. The log shows the team’s habits. It also shows how often ideas really work, which helps to plan the next ones.

xp log keeps the log in a file test-log.csv in the project folder. xp log add takes the last analysis of a run.

  1. Check that the log is empty.

    Terminal window
    xp log stats
    The test log is empty. Add a test with: xp log add RUN --decision ...
  2. Analyse the free-delivery test, and log it with Hana’s decision from the report.

    Terminal window
    sim run --scenario delivery
    xp analyse
    xp log add delivery --decision discuss --plan plans/free-delivery.md
    ...
    difference +1.0266 (+4.1%), 95% interval +0.1583 to +1.8950, p = 0.0205 (Welch t-test)
    Saved to runs/delivery-s40/analysis.json. Compare with the true effect: xp check
    Logged delivery-s40: discuss.

    We show only the last lines of the output.

  3. Log three more tests: the reminder email, the test with the redirect bug, and Tomás’s checkout button from Peeking. The team shipped the button on day 3, without a plan.

    Terminal window
    sim run --scenario email
    xp analyse
    xp log add email --decision ship --plan plans/reminder-email.md
    sim run --scenario srm
    xp analyse
    xp log add srm --decision "do not use" --plan plans/free-delivery.md
    sim run --scenario early-stop
    xp analyse
    xp log add early-stop --decision ship

    The last command prints a warning:

    Logged early-stop-s1197: ship. Warning: no test plan.
  4. List the log, and show its statistics.

    Terminal window
    xp log list
    xp log stats
    YYYY-MM-DD delivery-s40 discuss revenue: +1.0266 (+0.1583 to +1.8950) plan: plans/free-delivery.md
    YYYY-MM-DD email-s40 ship ordered: +0.0147 (+0.0100 to +0.0194) plan: plans/reminder-email.md
    YYYY-MM-DD srm-s40 do not use revenue: +2.9012 (+2.0226 to +3.7798) plan: plans/free-delivery.md
    YYYY-MM-DD early-stop-s1197 ship revenue: +0.8510 (-0.7827 to +2.4847) plan: NONE
    4 tests: 2 shipped (50%), 1 without a written plan.
    Most changes do not win: a team that ships most of its tests may be shipping noise.

    Your log shows today’s date in place of YYYY-MM-DD.

Read the last line of the list. The checkout button was shipped, with no plan, and its TermConfidence intervalA range around a measured value, made by a method that holds the true value 95 times in 100. If the interval of a difference does not include zero, the difference is probably not chance.Learn more Estimate uncertaintyEssential Mathematics and Statistics for AI over the full 28 days goes from −0.78 to +2.48 dollars: it crosses zero. The simulator knows that the button does nothing. One look at the log finds the test that should not have shipped.

The log has limits. xp log add checks that the plan file exists, but not that the team wrote it first: that is the review’s job. It logs the last effect that you analysed for the run, for example with CUPED. If your last command was xp srm, it logs the primary metric without CUPED, and says so. So run your primary analysis last, just before xp log add.

Two shipped of four is a small sample. In a real log of a hundred tests, a team that ships 50% or more should ask whether it ships noise.

When to set up these habits, and what they cost

Section titled “When to set up these habits, and what they cost”

Start the log with the first test. It costs one line per test, and it gets more useful every month.

Add the review before launch when more than one person runs tests, or when a test costs more than a week of traffic.

Pre-registration costs almost nothing once the plans live in git. The hard part is not the tool. It is the discipline to log the losers and to keep a plan that you would like to change.

Do not turn the habits into paperwork. A one-page plan, a 20-minute review and one line in a log are enough for most tests.

Guided practice: a year of tests in a notebook

Section titled “Guided practice: a year of tests in a notebook”
  1. Open the notebook from the box Do this lesson in a notebook at the top of this page.
  2. In section 1, simulate 1,000 tests: 200 ideas with a true effect equal to the minimum detectable effect, 800 with no effect.
  3. In section 2, count the wins, and the false wins among them. You see 158 real wins and 16 false wins: 9.2% of the wins are false, close to the 1 in 9 of the tree.
  4. In section 3, compare the mean estimate of the real winners with their true effect. You see 1.10 against 1.00: the winners exaggerate.
  5. In section 4, simulate “testing until something wins” with up to 10 tests of nothing. Put the share of teams that find a false win in false_win_share. Run the check cell. It prints Check passed.
Show the answer to step 5
wins = (rng.normal(0, SE, size=(TEAMS, 10)) > 1.96 * SE).any(axis=1)
false_win_share = round(float(wins.mean()), 3)

The value is 0.224, the same as the formula 1 − 0.97510 = 0.224.

Rafael sends Mateo this plan for a new subject line test. Use the five review questions. Find at least three problems before you read the answer.

# Test plan: subject-line
- **Hypothesis:** the new subject line is better.
- **Unit:** the email (each email sent gets A or B at random).
- **Primary metric:** we will look at opens, clicks and orders, and choose the clearest.
- **Guardrail metrics:** TODO
- **Minimum detectable effect:** 0.2 points of orders
- **Sample size:** 5,000 emails per group
- **Duration:** until it wins, at most 4 weeks
- **Stopping rule:** we check every morning
- **Decision if B is better:** ship B
- **Decision if B is worse:** keep A
- **Decision if no clear effect:** try another subject line
Show the answer
  • The primary metric is not chosen: “the clearest” of three is the many-metrics trap. Choose one, for example orders within 14 days.
  • The guardrail is missing. xp plan check would say “Not ready”. Unsubscribes are the obvious guardrail for an email.
  • The unit is the email. A customer who gets several emails could see both lines. Use the customer.
  • The size is far too small. At a 9% base rate, finding 0.2 points needs 324,616 customers per group: xp power --baseline 0.09 --mde 0.002 says so. Either test a bigger effect or say that this question cannot be answered.
  • “Until it wins” and “every morning” are peeking. Fix the duration, or use the sequential test of Peeking.
  • “Try another subject line” after no clear effect is testing until something wins. Log the result, and count all the tries.

Common mistake

“Our tests won 40 times this year, so sales should be up 40 × 2%”

Symptom: a team adds up the effects of all its winning tests and promises a large rise in sales. The rise never comes.

Cause: some winners were false, because most ideas do nothing. The real winners were estimated too high, because a test wins only when its luck pushes the estimate up. Some tests were run until they won, or checked every day. The losers were never written down, so nobody could see how often ideas fail.

Fix: log every test, with its plan and its range. Expect real effects to be smaller than the winning estimates. For big decisions, run the test again. Keep a holdout: a small group of customers who get none of the year’s changes. At the end of the year, compare it with everyone else.

Check your understanding

A team tests 1,000 ideas a year. 100 really work, the tests have 80% power, and an idea that does nothing wins by chance in 2.5% of tests. How many wins are in the log, and what share of them is false?

Show the answer

Real wins: 100 × 80% = 80. False wins: 900 × 2.5% = 22.5. Wins in total: about 102.5. Of these, 22.5 / 102.5 = about 22% are false: more than 1 in 5. Fewer good ideas means a larger share of false wins.

Check your understanding

In 200 simulated runs of the free-delivery test, the 83 winning runs estimated +1.19 dollars on average, but their true effect was about +0.75. Explain why, in two sentences.

Show the answer

The test is under-powered, so a run wins only when its random error pushes the estimate above about +0.88 dollars. The winners are therefore the runs with good luck, and their average is too high: this is the winner’s curse.

Check your understanding

Name three habits that protect a team that runs many tests, and the trap that each one stops.

Show the answer

A plan written and dated before the test (pre-registration) stops choosing the metric or the stopping day after seeing the result. A review before launch stops a broken design, such as a wrong unit or a too-small sample, from wasting a test. A test log of every test, with losers, stops shipping only “winners” and testing until something wins, because every try is counted.

Recap

  • Most ideas do not work. In a made-up year of 1,000 tests with 200 good ideas, about 1 win in 9 is false. With 100 good ideas, more than 1 in 5 is false.
  • Winners exaggerate: in 200 simulated free-delivery runs, the winners estimated +1.19 dollars, against a truth of about +0.75. This is the winner’s curse.
  • Testing until something wins: up to 10 tries of nothing give a false win in 22.4% of cases.
  • The habits: a dated plan (pre-registration), a review before launch, a report, and a test log of every test. xp log keeps it, and xp log stats shows how much the team ships and how many tests had no plan.

This is the end of the teaching modules. Next, the case studies: real data from a mailing, an online shop and a minimum wage.

Next: Did the mailing work, and for whom?