Choose the metrics
- Lesson 6
- 30 min
- Module 2: Design an A/B test
Where we are
We can now split customers fairly with a coin flip. Next we must decide what to measure. In this lesson, we choose one primary metric for Hana's free-delivery test, add guardrail metrics that must not get worse, and handle ratio metrics such as revenue per visitor.
Lesson 6 of 30 · Experiments and Causal Impact: Did Our Change Work?Choose the metrics
1. Why "it went up" is not an answer
2. Design an A/B test
3. Analyse the test
4. Traps in real tests
5. When you cannot randomise: natural experiments
6. Adjust for confounders in observational data
7. Uplift: who should get the treatment?
8. Experiments for AI features and in production
What you will be able to do
Choose one primary metric that decides the test, before the test starts. Add guardrail metrics that must not get worse, and compute ratio metrics such as the return rate correctly. With six metrics and no rule, a test can tell any story that you want.
Before you startWhat to know and to have ready
What you need to know
- Randomisation and the unit of randomisation (the customer), from Randomise.
- A 95% interval for a difference: a range that contains the true difference in about 95 of 100 tests. The mathematics course explains it in Confidence interval: a range with a known success rate.
What you need on your computer
- The project folder
larkfield-experiments, with its virtual environment active, as in Randomise. This lesson adds no code: it usesxp analyse --metric.
Tested on macOS with Python 3.12 and the start snapshot of the project. Windows was not tested.
The new window display
Section titled “The new window display”The gift-shop owner from Module 1 has had her new window display for a month. She asks: did it work? Her assistant brings six numbers. More people came in. The share of visitors who bought something went up. But the average sale was a little smaller. Returns went up. Sales per visitor went up a bit. And visits per customer did not move.
So did it work? It depends on which number she reads. If she wanted more sales, perhaps yes. If returns cost her a lot, perhaps no. Suppose she chooses the number after she sees all six. Then she can tell almost any story, and she will usually tell the one that she hoped for.
A careful owner decides first. “The display is for sales. I will judge it on sales per visitor. And I will not keep it if returns get clearly worse.” One number decides. Another number can stop her. The rest help her understand why.
The problem: six metrics, six stories
Section titled “The problem: six metrics, six stories”Hana’s free-delivery test is the same situation. Group A keeps free delivery from 75 dollars; group B gets free delivery from 50 dollars. The simulator ran it for 28 days with 110,759 customers (simulated). xp analyse --metric can measure many things on the same test. Here are six, each as a change against group A, with the simulator’s true effect where it knows one:
- The test's estimate and interval
- True effect (from the simulator)
Text description of the chart
A chart of relative changes from −6% to +24%, with a line at 0 labelled “no effect”. Revenue per customer: +4.1%, TermConfidence intervalA range around a measured value, made by a method that holds the true value 95 times in 100. If the interval of a difference does not include zero, the difference is probably not chance.Learn more Estimate uncertaintyEssential Mathematics and Statistics for AI +0.63% to +7.56%, true +2.79%. Share who ordered: +5.36%, interval +3.09% to +7.64%, true +3.49%. Orders per customer: +5.83%, interval +3.08% to +8.57%, true +4.43%. Order value: -1.63%, interval -3.72% to +0.45%, no true value in the simulator. Visits per customer: +0.37%, interval -0.69% to +1.42%, no true value. Return rate: +15.53%, interval +8.15% to +22.91%, true +14.12%.
Each metric tells a different story. TermOmarLarkfield’s product manager. He asks the business questions: what a change costs, and what it gives the shop. can say “orders went up 5.8%”. Hana’s finance colleague can say “returns went up 15.5%”. Someone else can say “order value fell”. All of them read the same test. Without a rule, the loudest person picks the number.
There is a second problem. Every metric has its own chance to show a difference by luck. Module 4 measures it: in a test with 20 metrics and no true effect, at least one metric “won” in 56% of simulated runs (Many metrics and many tests). Choosing the best metric after the test picks up that luck.
The idea: one metric decides, others guard
Section titled “The idea: one metric decides, others guard”Before the test, sort the metrics into four kinds:
Text description of the diagram
The diagram has five boxes from top to bottom, joined by arrows. At the top is the decision: lower free delivery from 75 to 50 dollars for everyone? Below it is the primary metric, one, chosen before the test: revenue per customer. It decides. Next are the guardrail metrics, which must not get clearly worse: the return rate, support contacts and page speed. They can stop a launch. Next are the diagnostic metrics, which explain why: the share who ordered, orders per customer, order value and visits. At the bottom is the sanity check, a metric that the change cannot move: spending before the test. A gap there means that the split is broken.
- The primary metric is the one number that decides the test. There is exactly one, and you choose it before the test starts. Some teams call it the “decision metric” or the “overall evaluation criterion”.
- Guardrail metrics are numbers that must not get clearly worse. They do not decide that B is better. They can stop a launch. Typical guardrails are returns, support contacts, page speed, errors and unsubscribes.
- Diagnostic metrics explain why the primary metric moved. They never decide on their own.
- A sanity check is a metric that the change cannot move, such as spending before the test. If it shows a clear gap, the split is broken. You saw why in Randomise.
What makes a good primary metric
Section titled “What makes a good primary metric”Hana’s decision is: “Should TermLarkfieldA made-up Canadian online shop for home and garden products. It is not a real company. All Nextia courses use its help desk and data as the example.Learn more Meet Larkfield pay for delivery on more orders?” The change costs money on every order between 50 and 75 dollars. It pays only if customers spend more. So the primary metric is revenue per customer. It is all that the group’s customers spent during the test, divided by the number of customers in the group.
A good primary metric has five properties:
- It is tied to the decision. Revenue pays for delivery. The share of customers who ordered does not: a customer who now buys one small item adds a delivery cost and little revenue.
- It is measured per unit of randomisation. The coin flipped per customer, so the metric is per customer.
- It counts everyone in the group, also the customers who bought nothing. In this test, 78% of the customers have no order. They count as 0 dollars.
- It can move during the test. “Customers who are still active next year” is a better goal, but a 4-week test cannot see it.
- It is hard to game. A metric such as “clicks on the delivery banner” goes up when the banner is bigger, whether or not anyone buys more.
A trap: order value only counts the buyers
Section titled “A trap: order value only counts the buyers”Look at order value in the chart: B’s orders are 1.6% smaller. It is tempting to read this as “free delivery makes people spend less per order”. But order value is computed only on the customers who ordered: 11,893 in A and 12,508 in B. B has 615 more buyers than A: the change made more customers buy. Those extra buyers may differ from the others, for example with smaller baskets.
So A’s buyers and B’s buyers may no longer be the same kind of people. The coin made the groups alike; keeping only the customers who “bought something” after the change can undo that. Order value then mixes two things: a change in what each customer buys, and a change in who buys. The test cannot separate them. Order value is fine as a diagnostic: it helps explain the result. It must never be the primary metric. The same is true for “revenue per buyer”. Revenue per customer has no filter, so the coin’s balance holds.
Ratio metrics
Section titled “Ratio metrics”Some metrics are not “per customer”. The return rate is returns per order. Revenue per visit is revenue per visit. The unit in the bottom of the fraction is not the unit of randomisation. These are ratio metrics, and they need care in two places.
A tiny example: four customers and their returns
Section titled “A tiny example: four customers and their returns”Here are four customers of group A with their orders and returns (a constructed example):
| Customer | Orders | Returns | Their own return rate |
|---|---|---|---|
| K1 | 4 | 1 | 1 ÷ 4 = 25% |
| K2 | 1 | 1 | 1 ÷ 1 = 100% |
| K3 | 0 | 0 | no orders |
| K4 | 3 | 0 | 0 ÷ 3 = 0% |
| All | 8 | 2 |
There are two ways to compute “the return rate of group A”:
- Ratio of the sums: 2 returns ÷ 8 orders = 25%. Each order counts once.
- Mean of the customers’ rates: (25% + 100% + 0%) ÷ 3 = 41.7%. Each customer with an order counts once, so K2’s single order weighs as much as K1’s four.
They answer different questions. Larkfield pays for each returned order, so Hana wants “returns per order”: the ratio of the sums. Write the definition into the test plan, so that nobody changes it later.
The second point is the interval. It is tempting to treat the 8 orders as 8 independent rows. They are not: K1’s four orders come from one person, who has one habit about returns. The coin flipped per customer, so the customers are the independent units. xp analyse therefore keeps one row per customer: their orders and their returns. It computes the ratio of the sums, and its interval with the delta method. This is a formula for the uncertainty of a ratio, when both the top and the bottom of the fraction vary from customer to customer. You do not need the formula: the output says (ratio, delta method). The course’s measurements show that it works: over 200 seeds, its 95% interval for the return rate contained the truth in 94% of runs.
“Revenue per visitor”, which many shops use, is the easy case. The visitor is the customer, the unit of randomisation, so it is a plain mean per customer. That is Hana’s primary metric. “Revenue per visit” is a ratio metric, like the return rate.
Larkfield: Hana’s metrics
Section titled “Larkfield: Hana’s metrics”TermPriyaLarkfield’s data scientist. She measures how well the assistant works, on test sets and not on guesses. and Hana fix the metrics of the free-delivery test before it starts:
| Kind | Metric | In the project |
|---|---|---|
| Primary | Revenue per customer, during the 28 days | revenue (the scenario’s default) |
| Guardrail | Return rate: returns per order | return_rate |
| Guardrail | Support contacts per customer; page speed | not in the simulator: a real shop would add them |
| Diagnostic | Share who ordered; orders per customer; order value; visits per customer | converted, orders, order_value, visits |
| Sanity check | Spending in the 8 weeks before the test | pre_spend |
Now read the chart again, with the rule. The primary metric went up: +1.03 dollars per customer (+4.1%), with an interval from +0.16 to +1.90. That interval contains the true effect, +0.70. But this seed was lucky: over 200 seeds of the simulator, a test of this size finds the revenue effect in only 41.5% of runs. Sample size and power explains why. The guardrail is clearly worse: the return rate rose from 8.31% to 9.60% of orders, +1.29 points (true +1.17). The diagnostics explain the story: more customers ordered, and their orders were a little smaller.
So the test does not end with “B wins”. It ends with a question for Hana: is +1.03 dollars of revenue worth 1.29 more returns per 100 orders? That trade-off is a business decision. The plan in Write the plan first says in advance who makes it, and how.
When to add a metric, and what it costs
Section titled “When to add a metric, and what it costs”Add a guardrail for every way the change could hurt: money (returns, refunds), people (support contacts, unsubscribes, complaints) and the product (page speed, errors). A guardrail is cheap insurance.
Do not add primary metrics. One test, one primary metric. If two goals really matter, combine them into one number before the test, for example revenue minus the cost of returns.
What it costs: every extra metric is one more chance for a false winner, and one more number to argue about. Each guardrail that can stop a launch also needs its own limit. Here, “clearly worse” means that its whole interval is on the bad side of 0. Ratio metrics need the right analysis, as above.
Guided practice: the metrics of the free-delivery test
Section titled “Guided practice: the metrics of the free-delivery test”-
In
larkfield-experiments, with the TermVirtual environmentA folder with its own Python and its own packages, for one project. Packages that you install there do not change other projects.Learn more Isolate dependenciesPython for Practical AI Engineering active, run the free-delivery test:Terminal window sim run --scenario deliveryScenario delivery (seed 40): Free delivery from 50 dollars instead of 75: an A/B test by customer, 28 days.Wrote runs/delivery-s40: customers (110,759 rows), visits (358,302 rows). Synthetic data. 8.6 sIt takes a few seconds. The time can be different on your computer.
-
Analyse the primary metric. With no
--metric,xp analyseuses the scenario’s primary metric:Terminal window xp analyseRun delivery-s40 (synthetic): Free delivery from 50 dollars instead of 75: an A/B test by customer, 28 days.Metric: revenue. A: 55,430 customers, mean 25.0504. B: 55,329 customers, mean 26.0770.difference +1.0266 (+4.1%), 95% interval +0.1583 to +1.8950, p = 0.0205 (Welch t-test)Saved to runs/delivery-s40/analysis.json. Compare with the true effect: xp check -
Analyse the guardrail, then check it against the simulator’s truth:
Terminal window xp analyse --metric return_ratexp checkRun delivery-s40 (synthetic): Free delivery from 50 dollars instead of 75: an A/B test by customer, 28 days.Metric: return_rate. A: 11,893 customers, mean 0.0831. B: 12,508 customers, mean 0.0960.difference +0.0129 (+15.5%), 95% interval +0.0068 to +0.0190, p = 3.7e-05 (ratio, delta method)Saved to runs/delivery-s40/analysis.json. Compare with the true effect: xp checkRun delivery-s40: the last analysis was `xp analyse`.True effect (from the simulator): +0.0117 returns per order.Your estimate: +0.0129, 95% interval +0.0068 to +0.0190: the interval contains the true effect.The return rate counts only customers with an order (11,893 and 12,508), and it says
ratio, delta method. Its interval is above 0: the guardrail is clearly worse. -
Analyse one diagnostic, the order value:
Terminal window xp analyse --metric order_valueRun delivery-s40 (synthetic): Free delivery from 50 dollars instead of 75: an A/B test by customer, 28 days.Metric: order_value. A: 11,893 customers, mean 85.3280. B: 12,508 customers, mean 83.9334.difference -1.3947 (-1.6%), 95% interval -3.1745 to +0.3851, p = 0.125 (ratio, delta method)Saved to runs/delivery-s40/analysis.json. Compare with the true effect: xp check -
Run the sanity check. The change cannot move spending from before the test:
Terminal window xp analyse --metric pre_spendRun delivery-s40 (synthetic): Free delivery from 50 dollars instead of 75: an A/B test by customer, 28 days.Metric: pre_spend. A: 55,430 customers, mean 46.6598. B: 55,329 customers, mean 46.5977.difference -0.0621 (-0.1%), 95% interval -1.4699 to +1.3458, p = 0.931 (Welch t-test)Saved to runs/delivery-s40/analysis.json. Compare with the true effect: xp checkThe interval is wide and centred near 0, as it should be. The split is healthy.
Your turn: the email’s guardrail
Section titled “Your turn: the email’s guardrail”Rafael’s reminder email has the primary metric ordered: the share of customers who order in the 14 days after the email. Which guardrail would you add for an email? Predict whether it gets worse. Then run the email test again, analyse the metric unsubscribed (1 if the customer unsubscribed from Larkfield’s emails), and check it.
Show the answer
Unsubscribes are the natural guardrail: an email that annoys people costs Larkfield every future email to them.
sim run --scenario emailxp analyse --metric unsubscribedxp checkThe last lines:
Metric: unsubscribed. A: 30,208 customers, mean 0.0024. B: 29,792 customers, mean 0.0084. difference +0.0060 (+250.7%), 95% interval +0.0048 to +0.0071, p = 1.61e-23 (Welch t-test)Saved to runs/email-s40/analysis.json. Compare with the true effect: xp checkRun email-s40: the last analysis was `xp analyse`.True effect (from the simulator): +0.0064 share of customers.Your estimate: +0.0060, 95% interval +0.0048 to +0.0071: the interval contains the true effect.The guardrail is clearly worse: unsubscribes went from 0.24% to 0.84% of the customers. That is small in points and large in relative terms (+251%). Module 7 shows who unsubscribes: mostly the heavy buyers, whom the email annoys. A test that looked only at orders would miss it.
Common mistake
The metric chosen after the test
Symptom: TermTomásLarkfield’s AI engineer. He often suggests a quick shortcut, and the lesson shows why it fails. reads the free-delivery test. “Revenue is only significant at p = 0.0205, but orders per customer went up 5.8% with p = 0.00003. Let’s report orders: it is the stronger result.”
Cause: he chose the metric after he saw the results. Orders are not tied to Hana’s decision: more small orders can cost more in delivery than they bring in. And picking the strongest of six metrics picks up luck: with 20 metrics and no true effect, at least one “wins” in 56% of simulated runs.
Fix: report the primary metric that was chosen before the test, with its interval, and the guardrails. Report the diagnostics as explanations only. Write the primary metric into the test plan before the test starts.
Consequence in AI work: the same happens with model and prompt tests. A team that reports “the metric that moved most” (accuracy one week, user rating the next) will find a winner in almost every test. Its launches then fail to improve the number that the business cares about.
Knowledge check
Section titled “Knowledge check”Check your understanding
Omar wants the free-delivery test's primary metric to be the share of customers who order, 'because it moved the most clearly'. What do you answer?
Show the answer
The primary metric must be tied to the decision, and chosen before the test, not because it moved most. Hana’s decision is whether to pay for delivery on more orders. The share who ordered can rise while revenue does not cover the delivery cost, for example with many small orders. Revenue per customer, counting every customer in the group (also those with 0 dollars), is tied to that decision. The share who ordered is a useful diagnostic: it explains why revenue moved.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
In the free-delivery test, B's orders are 1.6% smaller than A's. Tomás concludes: 'free delivery makes each customer spend less per order.' What is wrong with this reading?
Show the answer
Order value is computed only on customers who ordered: 11,893 in A and 12,508 in B. The change made more customers buy. So the buyers in B may not be the same kind of customers as the buyers in A. The filter on “bought something” can break the balance that the coin made. The 1.6% then mixes a change in what each customer buys with a change in who buys, and the test cannot separate them. Order value can explain the result as a diagnostic. Revenue per customer has no filter, so it measures the effect on comparable groups.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
Four customers have 4, 1, 0 and 3 orders and 1, 1, 0 and 0 returns. Compute the return rate as Larkfield defines it, and say why the analysis keeps one row per customer instead of one row per order.
Show the answer
Larkfield’s return rate is returns per order: the ratio of the sums, 2 ÷ 8 = 25%. (The mean of the customers’ own rates would be 41.7%, which answers another question.) The orders of one customer are not independent: they share that customer’s habits. The coin flipped per customer, so customers are the independent units. One row per customer, with the delta method for the ratio, gives an honest interval; one row per order would make it too narrow.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Recap
- On the same free-delivery test, six metrics tell six stories: revenue +4.1%, orders +5.8%, order value −1.6%, return rate +15.5%. Choosing one after the test picks the story you hoped for.
- One primary metric decides, chosen before the test and tied to the decision: revenue per customer, counting customers with no order as 0.
- Guardrails must not get clearly worse: the return rate rose from 8.31% to 9.60% of orders, so the test ends with a trade-off for Hana, not with “B wins”. Diagnostics explain; a sanity check (
pre_spend) tests the split. - A metric computed only on buyers, such as order value, compares different people. Never make it primary.
- Ratio metrics (returns per order, revenue per visit): define the ratio, keep one row per customer, and use the delta method for the interval.
You know what to measure. Next, you compute how many customers the test needs to see an effect of the size that matters, and check it by simulation.
Next: Sample size and power