Skip to content
Français

Report a result people trust

  • Lesson 29
  • 30 min
  • Module 8: Experiments for AI features and in production

Where we are

We can now run and analyse many kinds of tests. A result helps only if people trust it and act on it. In this lesson, we write a one-page result report with the decision, the effect and its range, the checks we ran and the limits.

What you will be able to do

Write the result of a test on one page that a busy person can act on. Put the decision first. Then give the effect with its range, the checks you ran, and the limits. Learn to say “no clear effect” so that it is still useful.

Before you startWhat to know and to have readyAbout 30 minutes

What you need to know

What you need on your computer (only for the project steps)

  • The project larkfield-experiments at the snapshot end-of-m08, from Test a model or a prompt online, with .venv active.
  • Hana’s filled-in plan plans/free-delivery.md, from Write the plan first. The snapshots do not contain your plans. If you do not have it, run xp plan new free-delivery and fill in its fields from that lesson.

Tested on macOS with Python 3.12 and the end-of-m08 snapshot. Windows was not tested.

You leave your car at a garage. In the evening, the mechanic gives you a note. A good note looks like this:

  • What to do: “Change the front brakes this month.”
  • How sure: “The pads have 2 to 3 mm left. New pads have 10.”
  • What I checked: “Brakes, tyres and lights.”
  • What I did not check: “The engine. That needs a longer test.”

You can act on this note in one minute. You know how urgent it is. You know what was checked. And you know what was not checked, so nobody can surprise you later.

A bad note says “Brakes: 72% worn, significant”. It sounds exact, but you do not know what to do.

A test result needs the same kind of note. This lesson calls it a result report.

After Hana’s free-delivery test, TermOmarLarkfield’s product manager. He asks the business questions: what a change costs, and what it gives the shop. puts one line on his slide: “Free delivery from 50 dollars: +4.1% revenue (p = 0.02)”. The board approves it in five minutes.

The line is true, and it is not enough. It leaves out three things that change the decision:

  • The range. The gain could be 0.16 dollars per customer or 1.90. Hana needs 1.25 to pay for the extra deliveries.
  • The guardrail. The return rate rose by 1.29 points, and that is clear.
  • The luck. This test is under-powered. With the same true effect, most runs of it would say “no clear effect”.

The simulator can run Hana’s test again, with new random customers. The true effect stays about the same: about +0.70 dollars per customer. Here is what the report would say for three seeds.

The free-delivery test run with three seeds (simulated). The true effect is about +0.70 dollars in each. Seed 40, the course's default, is a lucky run: its interval stays above zero. Seeds 41 and 42 give 'no clear effect'.
Text description of the chart

A panel with a line at 0 and a line at the true effect, about +0.70 dollars. Seed 40, Discuss before shipping: +1.03 (+0.16 to +1.90). Seed 41, No clear effect: keep A: −0.05 (−0.94 to +0.84). Seed 42, No clear effect: keep A: +0.36 (−0.49 to +1.22). Each TermConfidence intervalA range around a measured value, made by a method that holds the true value 95 times in 100. If the interval of a difference does not include zero, the difference is probably not chance.Learn more Estimate uncertaintyEssential Mathematics and Statistics for AI contains its seed’s truth (+0.70, +0.76, +0.74).

The course also ran the report’s decision rule on 200 seeds of the same test.

The decision of the result report over 200 simulated runs of the same free-delivery test. The change truly works (about +0.75 dollars on average), but the test says 'no clear effect' in 117 runs.
Text description of the chart

A column chart. Ship B: 4 runs. Discuss before shipping: 79 runs. No clear effect, keep A: 117 runs. Together, 83 runs (41.5%) find a clear gain in revenue.

A report with only “+4.1%” hides all of this. A report people trust shows it.

A one-page report for free delivery from 50 dollars instead of 75, in four blocks, each with a note. Decision: Discuss before shipping; B is better on the primary metric, but a guardrail is clearly worse. Note: what do we do? The first line says it; a busy reader stops here. The effect: revenue per customer +1.03 dollars (+4.1%), 95% interval +0.16 to +1.90; return rate +1.29 points, +0.68 to +1.90. Note: how big is it and how sure are we? The range, not only the number; the guardrail is on the same page. Checks: split A 55,430, B 55,329, p = 0.762; 28 days as planned; the metrics were in the plan before the test. Note: can we trust the test? Limits: the effect is for these customers and these weeks; this test is under-powered, the default seed is lucky. Note: what does it not tell us? Say it before someone asks.
The four parts of a one-page result report, with the free-delivery test. Each part answers one question that the reader will ask.

A result report has four parts, in this order:

  1. The decision, in one line. A busy reader reads only this. Use the words of the plan: ship B, keep A, discuss, do not use.
  2. The effect, with its range: the primary metric and each guardrail, with the 95% interval. Give the units: dollars per customer, points.
  3. The checks: the sample ratio, the duration, and that the metrics came from the plan.
  4. The limits: what the test does not tell. Who was in it, which weeks, how much power, which costs are assumptions.

The decision is not a new opinion. The plan of Write the plan first wrote it down before the test. The rules are:

A decision tree. Is the split as planned (sample ratio test)? No: do not use this result, find the cause, test again. Yes: where is the interval of the primary metric? Below 0: keep A. Crosses 0: no clear effect, keep A, say which effects it rules out. Above 0: is a guardrail clearly worse? No: ship B. Yes: discuss before shipping.
The decision rules of the result report: the same rules as the test plan. The sample ratio check comes first, because a broken split makes every other number wrong.

“No clear effect” is the most common result, and the most often written badly. “The test failed” or “it has no effect” are both wrong. The interval tells you more.

Here are two tests with the same estimate. The numbers are made up for this example. Both intervals cross zero.

Two tests with 'no clear effect' and the same estimate, −0.05 dollars. The numbers are made up. Test X is precise: it rules out any gain that would pay, so the answer is 'no'. Test Y is too small to say anything: the gain could be above the break-even, or a large loss.
Text description of the chart

A panel with a line at 0 and a line at the break-even, +1.25 dollars. Test X: −0.05 (−0.30 to +0.20), a narrow interval far below the break-even. Test Y: −0.05 (−2.00 to +1.90), a wide interval that reaches past the break-even.

  • Test X: “No clear effect. The change is between −0.30 and +0.20 dollars per customer. A gain of 1.25 dollars, which we need to pay for it, is ruled out. Keep A, and stop testing this idea.”
  • Test Y: “No clear effect. The change is between −2.00 and +1.90 dollars per customer: the test was too small to tell. Keep A for now. If the idea still matters, test it with about four times more customers.”

So a good “no clear effect” says three things: the range, what the range rules out, and what to do next.

xp report writes the first draft of a report from a run. It uses the primary metric and the guardrails of the scenario, the sample ratio test, and the rules above. With --plan, it also compares the run with the test plan that you wrote in Write the plan first.

  1. Simulate the free-delivery test again, then write its report with its plan.

    Terminal window
    sim run --scenario delivery
    xp report delivery --plan plans/free-delivery.md
    Scenario delivery (seed 40): Free delivery from 50 dollars instead of 75: an A/B test by customer, 28 days.
    Wrote runs/delivery-s40: customers (110,759 rows), visits (358,302 rows). Synthetic data. 2.9 s
    Decision: Discuss before shipping. Wrote reports/delivery-s40.md.
  2. Open reports/delivery-s40.md.

    # Result: Free delivery from 50 dollars instead of 75: an A/B test by customer, 28 days
    **Decision: Discuss before shipping.** B is better on the primary metric, but a guardrail is clearly worse.
    ## The effect
    - revenue (primary): A 25.0504, B 26.0770; difference +1.0266 (+4.1%), 95% interval +0.1583 to +1.8950 (Welch t-test).
    - return_rate (guardrail): difference +0.0129, 95% interval +0.0068 to +0.0190.
    ## Checks
    - Split: A 55,430, B 55,329 customers; sample ratio test p = 0.762.
    - Duration: 28 days, 2025-09-08 to 2025-10-05, as in the plan.
    - Primary metric: revenue, as in the plan.
    - Guardrail: return_rate, as in the plan.
    - Plan: plans/free-delivery.md. The tool compares names and days only: check that the plan was written before the test.
    ## Limits
    - The effect is for the customers and the weeks of the test. A season or a new customer mix can change it.
    - Synthetic data from the course simulator.
  3. Write the report of the test with the redirect bug.

    Terminal window
    sim run --scenario srm
    xp report srm
    Scenario srm (seed 40): The free-delivery test with a redirect bug that loses some B customers (planted): a 52/48 split.
    Wrote runs/srm-s40: customers (106,651 rows), visits (353,064 rows). Synthetic data. 2.8 s
    Decision: Do not use this result. Wrote reports/srm-s40.md.

    Its first lines say why: Sample ratio mismatch (p = 2e-35): find the cause, then test again. The revenue gain of +2.90 dollars is still in the report, but nobody should read past the first line.

The tool’s report is a good start. Read it like a reviewer, and you find three problems.

The plan checks are shallow. The tool compares the names of the metrics and the number of days with the plan. It cannot know when someone wrote the plan, or whether someone changed it after the result. Without --plan, the report says that it did not compare at all. TermPriyaLarkfield’s data scientist. She measures how well the assistant works, on test sets and not on guesses. opens plans/free-delivery.md and checks its date before she keeps the lines.

The limits are general. They are true for any test. The real limit of this test is its power, and it is missing.

The words are for analysts. “return_rate +0.0129” means “returns rose from 8.3% to 9.6% of orders”. Hana should not have to translate.

Here is Priya’s final report. The numbers are the tool’s. The words are hers.

# Result: free delivery from 50 dollars instead of 75
**Decision: Discuss before shipping.** Revenue rose, but returns rose too. Hana and Priya meet before any change.
## The effect (28 days, 110,759 customers)
- Revenue per customer: +1.03 dollars (+4.1%), probably between +0.16 and +1.90. We need +1.25 to pay for the extra deliveries: the test cannot say whether we reach it.
- Returns: from 8.3% to 9.6% of orders, +1.29 points (+0.68 to +1.90). Clearly worse.
## Checks
- Split: A 55,430, B 55,329 customers; sample ratio test p = 0.76: healthy.
- 28 days, 8 September to 5 October 2025, as in the plan of 1 September. Metrics and guardrail from the plan; I checked.
## Limits
- The test is under-powered: it finds a gain of 1.23 dollars with 80% power. Our +1.03 is smaller, so a test this size finds such a gain in fewer than half of runs. This result is partly luck. Expect the true gain to be smaller than +1.03.
- The cost of returns per order is an estimate from finance.
- Only September and early October customers. The holiday season can differ.

The date of the plan, the finance estimate and the meeting are Priya’s own additions for this example. The simulator knows the truth, about +0.70 dollars, so here we know that the limit about power was right. A real shop never knows it.

Write one for every test that a decision depends on, also when the result is “no clear effect” or “do not use”. A report of a test that did not win teaches the team as much as a winner.

Do not write a report:

  • before the planned end of the test, except to stop it for a broken split or a broken site;
  • with new metrics that were not in the plan. If you found something interesting, add it under a heading “Not planned: explore later”, never in the decision.

What it costs: the tool writes the draft in a second. The review of the checks and the honest limits takes Priya about an hour. The next lesson keeps all reports in one test log.

Guided practice: build the report rules in a notebook

Section titled “Guided practice: build the report rules in a notebook”
NotebookDo this lesson in a notebookColab · Kaggle · GitHub

In the notebook, you compute the free-delivery test’s effect and guardrail. Then you write the decision rules as one small function, and turn the numbers into a short report. Its setup cells download the data. About 25 minutes.

Colab and Kaggle open your own copy. They need a free Google or Kaggle account. Stuck? See the solution notebook.

  1. Open the notebook from the box above, and run the setup cells.
  2. In section 1, compute revenue per customer and the return rate, with their intervals. You see revenue +1.0266 (+0.1583 to +1.8950).
  3. In section 2, write decide(primary, guardrail) with the rules of the decision tree. Run it on the free-delivery numbers. You see Discuss before shipping.
  4. In section 3, run decide on the two made-up tests X and Y. Both give No clear effect: keep A.
  5. In section 4, write the “no clear effect” sentence for test Y in sentence_y, with its range. Run the check cell. It prints Check passed.
Show the answer to step 5
sentence_y = ("No clear effect. The change is between -2.00 and +1.90 dollars per customer: "
"the test was too small to tell. Keep A for now.")

The check looks for the two ends of the range in your sentence. The words are yours.

Run the free-delivery test with seed 41, and write its report. Then write the report’s first two lines for Hana, as Priya would. Use the range, what it rules out (Hana’s break-even is +1.25 dollars) and what to do next.

Terminal window
sim run --scenario delivery --seed 41
xp report delivery-s41
Show the answer
Scenario delivery (seed 41): Free delivery from 50 dollars instead of 75: an A/B test by customer, 28 days.
Wrote runs/delivery-s41: customers (111,078 rows), visits (361,090 rows). Synthetic data. 3.4 s
Decision: No clear effect: keep A. Wrote reports/delivery-s41.md.

The report says: The effect is between -0.935 and +0.837: the test cannot tell a small gain from a small loss. It does rule out effects outside this range.

One good version for Hana:

Decision: keep the threshold at 75 dollars. The change was between −0.94 and +0.84 dollars per customer. The test cannot tell a small gain from a small loss, but a gain of 1.25 dollars, which we need to pay for free delivery, is unlikely. Returns rose by 0.67 points (+0.07 to +1.27).

“Rules out” is a little strong: a 95% interval misses the truth in about 1 run of 20. “Unlikely” is more honest.

The simulator knows that the change truly works here: about +0.76 dollars per customer. So “no clear effect” was correct, and “no effect” would have been wrong.

Independent task: the prompt test’s report

Section titled “Independent task: the prompt test’s report”

Write the one-page report for the support prompt test of Test a model or a prompt online. Start from the draft of xp report prompt: it counts customers, not tickets, as the test did. Then finish the report by hand, from the output of xp prompt and xp prompt --offline. Use the four parts: decision, effect with range, checks, limits.

Show one possible report
# Result: the support assistant's new prompt (B) against today's (A)
**Decision: Ship prompt B.** It resolves more tickets and costs less, and no guardrail is worse.
## The effect (14 days, 21,357 tickets from 9,000 customers)
- Resolved by the assistant: 70.1% → 79.8% of tickets, +9.8 points (+8.6 to +10.9).
- Handed to a person: 27.0% → 17.2%, −9.9 points (−11.0 to −8.8).
- Cost per ticket: 1.76 → 1.12 dollars, −0.64 (−0.71 to −0.57), at 6.50 dollars per hand-off (an assumption).
- Complaints (guardrail): −0.33 points (−0.77 to +0.10): no clear change.
- Unsafe answers (guardrail, offline): 0 of 120 for both prompts; both handed over all 20 questions that need a person.
## Checks
- Unit: the customer, so all tickets of a customer got the same prompt. Ratio metrics with the delta method.
- Offline evaluation before the test: correct 90.8% (A) and 99.2% (B).
- Split: A 4,426, B 4,574 customers; sample ratio test p = 0.12 (from xp report prompt): healthy.
## Limits
- Zero unsafe answers in 120 means the true rate could be up to about 2.5%. The unsafe check misses some wordings. Keep grading a weekly sample of real answers.
- The cost saving depends on the price of a hand-off.
- One recorded answer per question: the model's answer can vary, and model versions change.

The draft of xp report prompt gives the decision line (Ship B), the primary metric (resolved, +0.0976) and the guardrail (complaint). It also checks the split: 4,426 against 4,574 customers is 50.8% in B, with p = 0.119. The offline grades, the cost and the limits are yours.

Common mistake

“+4.1%, p = 0.02, ship it”

Symptom: a slide shows one number and a p-value. The board ships the change. Six months later, the gain is much smaller, and returns cost more than expected.

Cause: the slide left out the range, the guardrail and the luck. The estimate of a test that wins by a small margin is usually too large, and a guardrail on another slide is forgotten.

Fix: write a one-page report, with the decision from the plan first. Show the primary metric and every guardrail with its interval. List the checks you really ran. Write the limits of this test, not general ones. For “no clear effect”, say the range, what it rules out, and what to do next.

Check your understanding

Put the four parts of a one-page result report in order, and say in a few words what question each part answers.

Show the answer
  1. The decision: what do we do? 2. The effect with its range: how big is it, and how sure are we? It includes every guardrail. 3. The checks: can we trust the test? (sample ratio, duration, metrics from the plan). 4. The limits: what does the test not tell us? (who and when, power, assumptions).

Check your understanding

A test of a new checkout page gives −0.10 dollars per customer, 95% interval −0.25 to +0.05. The page needs +0.50 dollars to pay for itself. Write the decision line and one sentence about the effect.

Show the answer

Decision: no clear effect, keep the current page. The change is between −0.25 and +0.05 dollars per customer, so the gain of 0.50 that the page needs is very unlikely: stop testing this idea. This is a precise “no clear effect”, like test X: it answers the question.

Check your understanding

With --plan, the report of xp report says 'Duration: 28 days, as in the plan' and 'Primary metric: revenue, as in the plan.' Why must Priya still check the plan before she sends the report?

Show the answer

The tool compares only the names of the metrics and the number of days. It cannot know when Priya wrote the plan. If someone edited the plan after the result to match it, the report would still say “as in the plan”. So Priya checks the plan’s date and its dated notes, and corrects the lines if they are wrong.

Recap

  • A result report is one page: the decision, the effect with its range (also the guardrails), the checks, the limits.
  • The decision comes from the plan’s rules: the sample ratio first, then where the interval lies, then the guardrails.
  • The same free-delivery change gives “Discuss” at seed 40. It gives “No clear effect” at seeds 41 and 42. Over 200 runs: 79 discuss, 4 ship, 117 no clear effect. The default seed is lucky.
  • Say “no clear effect” with the range, what it rules out, and what to do next.
  • xp report writes a draft. With --plan, it compares the metrics and the days with the plan. Check the plan’s date yourself, and write limits for this test.

Next, Mateo turns single reports into a habit for the whole team.

Next: Experiments as a team habit