Skip to content
Français

Write the plan first

  • Lesson 8
  • 30 min
  • Module 2: Design an A/B test

Where we are

We now know the unit, the metrics and the sample size. These decisions are easy to change after we see the data, and that is a trap. In this lesson, we write them in a one-page test plan before the test starts, with the decision we will take for each result.

What you will be able to do

Write the test’s choices on one page before the test starts. The page fixes the hypothesis, the unit, the metrics, the size and the duration. It also fixes the stopping rule and the decision for each result. Choices made after a look at the data can make almost any change look like a winner.

Before you startWhat to know and to have readyAbout 30 minutes

What you need to know

  • The unit of randomisation, from Randomise.
  • The primary metric and the guardrails of the free-delivery test, from Choose the metrics.
  • The minimum detectable effect, the sample size and the duration, and xp power, from Sample size and power.

What you need on your computer

  • The project folder larkfield-experiments with the files of Sample size and power, and its virtual environment active. You copy this lesson’s command from the snapshot end-of-m02 in ../nextia-learning-labs/C40/snapshots/.

Tested on macOS with Python 3.12 and the end-of-m02 snapshot. Windows was not tested.

Before a football match, everybody knows the rules. The match lasts 90 minutes. A goal counts when the whole ball crosses the line. The team with more goals wins. Nobody may change these rules during the match.

Now imagine a team that may change them while it plays. It is one goal ahead after 60 minutes, so it ends the match. Behind? It plays on. It loses on goals, so it says that corners count too. Such a team wins almost every match, and its wins mean nothing. The rules must be fixed before the first kick, by people who do not know the score yet.

A test needs the same. Its rules are the unit, the metrics, the size, the duration, when it may stop, and what each result will mean for the decision. The test plan writes them down before the first customer enters the test.

Every test has many small choices. Which metric decides? When does the test end? Which customers count? It is easy to make them after you see the numbers, often without noticing. Each such choice gives chance one more way to produce a “winner”. Module 4 measures two of them in the simulator, with tests of a change that has no effect at all:

Tests of a change with no effect (simulated). With the rules fixed, 5.2% find a false winner. Stopping at the first day with p < 0.05 gives 27.6%. Reporting any of 20 metrics that wins gives 56%.
Text description of the chart

A column chart of the share of tests of no effect that declare a winner. With the rules fixed (one look at day 28, one primary metric): 5.2% of 10,000 simulated tests. Stopping at the first day with p below 0.05, looking every day for 28 days: 27.6%. Reporting a winner when any one of 20 metrics wins: 56% of 300 simulated tests.

A fixed test finds a false winner about as often as alpha promises: 5.2%. A team that looks every day and stops at the first “win” finds one in 27.6% of tests where nothing changed. A team that reports whichever of 20 metrics won finds one in 56%. Combine a few of these freedoms, and a test of nothing finds a “winner” most of the time. Module 4 explains both traps in detail: Peeking and Many metrics and many tests.

There is a quieter version of the same problem. After the result, people argue about what it means. The free-delivery test raised revenue and also raised returns. Is that a success? If nobody said so before the test, the answer depends on who argues best.

A test plan is one page that fixes the test’s choices before the test starts. Writing it first is called pre-registration: you register what you will do before you see any data. Scientists do it for medical trials for the same reason.

TermLarkfieldA made-up Canadian online shop for home and garden products. It is not a real company. All Nextia courses use its help desk and data as the example.Learn more Meet Larkfield’s plan has eleven fields:

Field What it says Why it must be fixed in advance
Hypothesis what the change does, and why it says what the test is for
Unit what the coin is flipped for a different unit is a different test
Primary metric the one number that decides, exactly defined choosing it afterwards picks luck
Guardrail metrics what must not get clearly worse they decide when to stop a launch
Minimum detectable effect the smallest effect that matters it sets the size
Sample size customers per group, and how it was computed a test that ends when it “wins” is not this size
Duration days, in full weeks stopping on a good day picks luck
Stopping rule when the test may end early looking every day finds false winners
Decision if B is better what you will do the result is not argued about afterwards
Decision if B is worse what you will do the same
Decision if no clear effect what you will do the most common result needs a decision too

The plan’s life is a straight line. You write it, the test runs, and nobody looks at the result until the end:

A row of six boxes joined by arrows: write the plan, before day 1; test starts, Monday 8 Sep; test runs, no look at the result, health checks only; test ends, day 28, 4 full weeks; check the test, sample ratio, full weeks; read the result, apply the decision written in the plan. An arrow from 'write the plan' goes up to a dashed box: fixed when the test starts. A red line below: changed after a look at the data? Then the test can show whatever you hoped.
The plan is written before day 1. During the test, only health checks; after it, first check the test, then apply the decision that the plan already contains.
Text description of the diagram

The diagram is a row of six boxes joined by arrows (a column on a phone). First, write the plan, before day 1. Second, the test starts, on Monday 8 September. Third, the test runs: nobody looks at the result, only health checks such as errors. Fourth, the test ends on day 28, after 4 full weeks. Fifth, check the test: the sample ratio and the full weeks. Sixth, read the result, and apply the decision written in the plan. An arrow goes from the first box to a dashed box that lists what is fixed when the test starts: the hypothesis, the unit, the primary metric, the guardrails, the minimum detectable effect, the sample size, the duration, the stopping rule, and a decision for each result. A red line under the row warns: changed after a look at the data? Then the test can show whatever you hoped.

“No look at the result” does not mean “no look at all”. During the test, someone checks that the site works and that customers enter both groups in the planned ratio. If B breaks checkout, you stop at once. You just do not look at revenue, and you do not stop because B is ahead.

The decision fields are the heart of the plan. Write each one in terms of the 95% TermConfidence intervalA range around a measured value, made by a method that holds the true value 95 times in 100. If the interval of a difference does not include zero, the difference is probably not chance.Learn more Estimate uncertaintyEssential Mathematics and Statistics for AI of the primary metric, so that nobody has to interpret the result afterwards. Four typical cases, as constructed examples:

Four constructed results and the decision for each. Above 0: B is better. Crossing 0: no clear effect. Below 0: B is worse. A guardrail whose interval is above 0 (worse) turns 'ship' into 'discuss'.
Text description of the chart

Four constructed results on an axis from −2.5 to +2.5 dollars, with a line at 0. “B better, guardrails fine”: +1.10, interval +0.30 to +1.90, entirely above 0: ship B. “B better, a guardrail worse”: the same interval, but a guardrail’s interval is entirely on the worse side: discuss before shipping. “No clear effect”: +0.20, interval −0.70 to +1.10, which crosses 0: keep A, and write down the range that the test rules out. “B worse”: −1.20, interval −2.00 to −0.40, entirely below 0: keep A.

The result The decision in Larkfield’s plans
The primary metric’s interval is above 0, and no guardrail is clearly worse Ship B
The primary metric’s interval is above 0, but a guardrail’s interval shows it is clearly worse Discuss before shipping, with the people named in the plan
The primary metric’s interval is below 0 Keep A
The interval crosses 0 No clear effect: keep A. Write down the range that the test rules out

“No clear effect” is the most common result of real tests. It still teaches something: the test rules out effects outside its interval. If the interval is −0.70 to +1.10 dollars, a gain of 2 dollars per customer is very unlikely.

Larkfield: Hana’s plan for free delivery

Section titled “Larkfield: Hana’s plan for free delivery”

TermPriyaLarkfield’s data scientist. She measures how well the assistant works, on test sets and not on guesses. and Hana write the plan for the free-delivery test before Monday 8 September. Hana gives one number that the statistics cannot give: a gain below about 1.25 dollars per customer does not pay for the extra delivery cost. (This break-even is an assumption of the course, not a measured cost.) So the MDE is 1.25 dollars. xp power --from-run delivery --mde 1.25 --per-day 3956 gives 53,866 customers per group. That is 28 days, exactly 4 full weeks.

Here is the plan, written with xp plan new free-delivery and filled in:

# Test plan: free-delivery
Written on YYYY-MM-DD, before the test starts. Change it only with a dated note that says why.
- **Hypothesis:** Free delivery from 50 dollars instead of 75 raises revenue per customer, because more visits end with an order.
- **Unit:** the customer. A coin flip on the first visit puts each customer in A (75 dollars) or B (50 dollars), 50/50, for the whole test.
- **Primary metric:** revenue per customer in the test (every customer in the test, also those with no order).
- **Guardrail metrics:** return rate (returns per order) must not be clearly worse. Watched, not decided on: orders per customer, order value.
- **Minimum detectable effect:** +1.25 dollars per customer (+5%). Hana: a smaller gain does not pay for the extra delivery cost.
- **Sample size:** 53,866 customers per group (xp power, standard deviation 73.22 dollars, 80% power, alpha 0.05).
- **Duration:** 28 days (4 full weeks), from Monday 8 September 2025; about 3,956 new customers a day.
- **Stopping rule:** no early stop and no look at the result before day 28. Stop early only for a broken site or a sample ratio mismatch.
- **Decision if B is better:** the interval of revenue is above 0 and the return rate is not clearly worse: ship B. If the return rate is clearly worse: Hana and Priya discuss the cost before shipping.
- **Decision if B is worse:** the interval of revenue is below 0: keep A.
- **Decision if no clear effect:** keep A. Write down the range that the test rules out.
## Checks before reading the result
- Sample ratio: the share of customers in B is 50% (chi-square test, p > 0.001).
- The test ran for the planned number of full weeks.

Two lines of the plan name a check that you learn in Module 3. A sample ratio mismatch means that the groups’ sizes are far from the planned 50/50. It is a sign of a broken split, so the result cannot be trusted. Check the test before the result shows how to test for it.

The test ran for 28 days (simulated). Then Priya reads the result, and the plan tells her what it means:

  • Primary metric: revenue per customer +1.03 dollars, interval +0.16 to +1.90: above 0. B is better.
  • Guardrail: return rate +1.29 points, interval +0.68 to +1.90: above 0, so clearly worse.
  • The plan’s row: “B is better, but the return rate is clearly worse”: Hana and Priya discuss the cost before shipping.

Nobody can now say “orders went up 5.8%, ship it”, or “returns went up 15.5%, kill it”. Both numbers are in the plan’s place for them: a diagnostic, and a guardrail with a rule. In Module 8, xp report delivery writes a result report from the same rules, and its first line is Decision: Discuss before shipping.

The plan also shows a limit of this test. Its MDE, 1.25 dollars, is larger than the true effect, about 0.70 dollars. The test found a significant result partly because seed 40 is a lucky one. Over 200 seeds, it finds the true effect in only 41.5% of runs. By Hana’s own break-even, that is acceptable: an effect of 0.70 dollars would not pay for the delivery. But the estimate of +1.03 is too large, and the plan’s discussion should use the interval, not the point.

Write one for every test that a decision depends on. The page takes an hour. The arguments that it prevents take longer.

Keep it short. One page, the eleven fields, numbers instead of adjectives. A plan that nobody reads before the test starts is not a plan.

Change it only with a dated note. Sometimes a plan must change: a bug stops the test for a day, or a metric turns out to be logged wrongly. Add a dated note that says what changed and why, before you look at the result. Keep the plans in version control (git), so that the history shows when each line was written. Mateo keeps Larkfield’s test log in Module 8, Experiments as a team habit.

What it costs: an hour of thinking before the test, and the discipline not to change your mind when the numbers come in. It does not cost customers or days.

Guided practice: write and check the free-delivery plan

Section titled “Guided practice: write and check the free-delivery plan”
  1. In larkfield-experiments, with the TermVirtual environmentA folder with its own Python and its own packages, for one project. Packages that you install there do not change other projects.Learn more Isolate dependenciesPython for Practical AI Engineering active, copy the command and the project’s new README from the snapshot end-of-m02:

    Terminal window
    S=../nextia-learning-labs/C40/snapshots/end-of-m02
    cp "$S/larkfield_experiments/commands/plan.py" larkfield_experiments/commands/plan.py
    cp "$S/README.md" README.md

    Your project is now the same as the snapshot end-of-m02.

  2. Run all the tests:

    Terminal window
    python -m pytest -q
    14 passed in 1.29s
  3. Write a new plan:

    Terminal window
    xp plan new free-delivery
    Wrote plans/free-delivery.md. Fill in every TODO before the test starts, then: xp plan check plans/free-delivery.md

    Open plans/free-delivery.md. It has the eleven fields, each with TODO, and the checks before reading the result. Its second line shows today’s date in place of YYYY-MM-DD.

  4. Check the empty plan:

    Terminal window
    xp plan check plans/free-delivery.md
    Not ready: 11 of 11 fields are missing or TODO: Hypothesis, Unit, Primary metric, Guardrail metrics, Minimum detectable effect, Sample size, Duration, Stopping rule, Decision if B is better, Decision if B is worse, Decision if no clear effect.

    The command ends with an error code, so a script or a review step can stop on it.

  5. Replace each TODO with Hana’s values, from the plan above. Save the file.

  6. Check it again:

    Terminal window
    xp plan check plans/free-delivery.md
    Ready: all 11 fields are filled in.

    xp plan check checks that every field has an answer. It cannot check that the answers are good: that is the job of the review before the test.

  7. Try to write the plan a second time:

    Terminal window
    xp plan new free-delivery
    plans/free-delivery.md exists. A plan is written once: add a dated note instead of a new plan.

Write Rafael’s plan for the reminder-email test, with xp plan new reminder-email. Use what you computed in this module: the unit, the primary metric, the guardrail from Choose the metrics, and the size and duration from Sample size and power. Before you write the sample size, check it by simulation with xp power --baseline 0.09 --mde 0.01 --per-day 4000 --simulate 200. Then run xp plan check. Think about one detail first: the outcome is “ordered in the 14 days after the email”. When can you read the result?

Show the answer

The simulation’s last lines:

At 4,000 new customers a day: 7 days; run full weeks: 1 week (7 days).
Simulated: 200 fake tests with 13,488 customers per group and a true effect of +0.01: 80% found it (p < 0.05).

One possible plan, after xp plan new reminder-email:

- **Hypothesis:** a reminder email to customers with no order for 30 days makes more of them order in the next 14 days.
- **Unit:** the customer; a coin flip, 50/50: A no email, B the email.
- **Primary metric:** the share of customers who order in the 14 days after the email (ordered).
- **Guardrail metrics:** unsubscribes (unsubscribed) must not be clearly worse.
- **Minimum detectable effect:** +1 percentage point (9% to 10%).
- **Sample size:** 13,488 customers per group (xp power; 200 fake tests: 80% found it).
- **Duration:** 1 full week of sending (7 days at 4,000 customers a day), then 14 days to count the orders.
- **Stopping rule:** no early stop; no look at the orders before the 14 days are over.
- **Decision if B is better:** if unsubscribes are not clearly worse, send the email to everyone; if they are, Rafael decides who gets it.
- **Decision if B is worse:** stop the email.
- **Decision if no clear effect:** stop the email; write down the range that the test rules out.
Terminal window
xp plan check plans/reminder-email.md
Ready: all 11 fields are filled in.

The detail: the last customers get the email on day 7, and their 14 days end on day 21. You can read the result only then. In the simulator, the email test’s unsubscribes were clearly worse (0.24% against 0.84%), so the plan’s “Rafael decides” row applies. Module 7 shows how to decide who gets the email.

Common mistake

The plan changed after the result

Symptom: after the free-delivery test, TermTomásLarkfield’s AI engineer. He often suggests a quick shortcut, and the lesson shows why it fails. opens plans/free-delivery.md and changes the primary metric to “orders per customer”. The plan now says “B is better”, with p = 0.00003. xp plan check prints Ready.

Cause: the plan was changed after a look at the data, so it is no longer a plan. It is a description of the result that Tomás liked. xp plan check checks that the fields are filled, not when they were written.

Fix: commit the plan to version control before the test starts, and have someone review it. Any later change is a dated note under the plan, written before anyone reads the result, and it says why. A reviewer can then compare the plan at the start with the plan at the end.

Consequence in AI work: an evaluation whose metric or TermTest setA fixed set of examples with known correct answers. You run each version of a prompt or model on the same set, so the results compare fairly.Learn more Build a representative test setLLMs: How They Work and How to Choose One changes after the first scores are in can show any model as the best. Freeze the plan of an online test of a model or a prompt as you freeze a test set: before the first look.

Check your understanding

Name five fields of a test plan, and for two of them, explain what goes wrong if they are chosen after the test.

Show the answer

Any five of: hypothesis, unit, primary metric, guardrail metrics, minimum detectable effect, sample size, duration, stopping rule, and the decisions if B is better, worse or not clearly different. Examples of what goes wrong: a primary metric chosen afterwards picks the luckiest metric (with 20 metrics and no effect, one “wins” in 56% of simulated tests). A duration chosen afterwards, stopping at the first good day, finds a false winner in 27.6% of tests of no effect, against 5.2% with a fixed length. Decisions chosen afterwards turn the result into an argument.

Check your understanding

The free-delivery test gives revenue +1.03 dollars (interval +0.16 to +1.90) and return rate +1.29 points (interval +0.68 to +1.90). Which row of Hana's plan applies, and why is it useful that it was written before?

Show the answer

“B is better, but a guardrail is clearly worse”: the revenue interval is above 0, and the return rate’s interval is entirely on the worse side. The plan says that Hana and Priya discuss the cost before shipping. Because it was written before, nobody can choose the convenient number afterwards: neither “orders +5.8%, ship it” nor “returns +15.5%, kill it”. The discussion starts from agreed rules and from the intervals.

Check your understanding

Predict: a test's primary metric has the interval −0.70 to +1.10 dollars. Omar says: 'So the change does nothing.' What does the plan say, and what does the result really tell you?

Show the answer

The interval crosses 0, so the plan’s row is “no clear effect: keep A”. But it does not show that the change does nothing. The test cannot tell a small gain from a small loss. It does rule out effects outside the interval: a gain of 2 dollars per customer, or a loss of 1 dollar, is very unlikely. If the minimum detectable effect was chosen well, the test also says that an effect worth acting on is unlikely. Write down the range in the result.

Recap

  • Choices made after a look at the data turn chance into winners. In simulated tests of no effect: 5.2% false winners with fixed rules, 27.6% when stopping at the first daily win, 56% when any of 20 metrics may win.
  • A test plan fixes eleven fields before the test starts: hypothesis, unit, primary metric, guardrails, MDE, sample size, duration, stopping rule, and a decision for each result. Writing it first is pre-registration.
  • Write decisions as intervals: above 0 ship (if the guardrails are fine), above 0 with a worse guardrail discuss, below 0 keep A, crossing 0 no clear effect.
  • Hana’s plan met the result: revenue up, return rate clearly worse, so “discuss before shipping”. Nobody could pick the convenient number.
  • xp plan new writes the template; xp plan check checks that every field is filled. Change a plan only with a dated note, before the result.

You can now design a test from the coin to the decision. Next, in Module 3, you analyse the free-delivery test properly: the difference, its interval, and why the interval matters more than the p-value.

Next: The result and its uncertainty