Many metrics and many tests
- Lesson 14
- 35 min
- Module 4: Traps in real tests
Where we are
Peeking looks at one metric many times. Another trap looks at many metrics once. In this lesson, we check 20 metrics in one test, find a winner that is not real, and correct for it with Bonferroni and Benjamini–Hochberg.
Lesson 14 of 30 · Experiments and Causal Impact: Did Our Change Work?Many metrics and many tests
1. Why "it went up" is not an answer
2. Design an A/B test
3. Analyse the test
4. Traps in real tests
5. When you cannot randomise: natural experiments
6. Adjust for confounders in observational data
7. Uplift: who should get the treatment?
8. Experiments for AI features and in production
What you will be able to do
Read a test with many metrics without finding winners that are not there. Check 20 metrics with the 5% rule, and about one of them “wins” by luck. A correction and a primary metric chosen before the test keep the false winners rare.
NotebookDo this lesson in a notebook
In the notebook, you simulate 1,000 A/A tests with 20 metrics each, count the false winners, and apply Bonferroni and Benjamini–Hochberg. Then you do Benjamini–Hochberg by hand on five p-values. Its setup cells install the packages, so it needs no local setup. About 25 minutes.
Colab and Kaggle open your own copy. They need a free Google or Kaggle account. Stuck? See the solution notebook.
Before you startWhat to know and to have ready
What you need to know
- Peeking and the meaning of the 5%, from Peeking.
- Primary, guardrail and diagnostic metrics, from Choose the metrics.
- Hypothesis tests and p-values, from Compare alternatives in Essential Mathematics and Statistics for AI.
What you need on your computer (only for the project steps)
- The project
larkfield-experimentsat the snapshotend-of-m04, from Peeking, with.venvactive.
Tested on macOS with Python 3.12 and the end-of-m04 snapshot of the project. Windows was not tested.
A new breakfast and twenty school results
Section titled “A new breakfast and twenty school results”A school gives its pupils a new breakfast for a month. Then the head teacher compares the pupils with last year’s on 20 results: maths, reading, spelling, running, attendance, and 15 more. One result is better, with p = 0.03: spelling.
The newspaper writes: “New breakfast improves spelling.” But nobody expected spelling to change. And with 20 results, one of them is likely to look better by chance alone. The head teacher did not find an effect. He found the luckiest of 20 numbers.
The problem: a layout that “raised wishlist adds”
Section titled “The problem: a layout that “raised wishlist adds””TermLarkfieldA made-up Canadian online shop for home and garden products. It is not a real company. All Nextia courses use its help desk and data as the example.Learn more Meet Larkfield tested a new product page layout. The plan named one primary metric, revenue per customer. The dashboard shows 19 more metrics: orders, visits, cart adds, wishlist adds, video plays and others. The simulator has this test as a scenario. Its false winner is a practice problem we made on purpose: the layout changes nothing.
-
Run the layout test.
Terminal window sim run --scenario many-metricsScenario many-metrics (seed 2001): A new product page layout with no real effect, checked on 20 metrics (planted: one false winner).Wrote runs/many-metrics-s2001: customers (33,357 rows), visits (107,695 rows). Synthetic data. 3.9 s -
Analyse the primary metric.
Terminal window xp analyseRun many-metrics-s2001 (synthetic): A new product page layout with no real effect, checked on 20 metrics (planted: one false winner).Metric: revenue. A: 16,451 customers, mean 25.4117. B: 16,906 customers, mean 24.6490.difference -0.7627 (-3.0%), 95% interval -2.2962 to +0.7709, p = 0.33 (Welch t-test)Saved to runs/many-metrics-s2001/analysis.json. Compare with the true effect: xp check -
Test all 20 metrics at once.
Terminal window xp metricsRun many-metrics-s2001: 20 metrics. Primary metric: revenue.metric relative p p<0.05 Bonferroni BHrevenue -3.00% 0.3297 - - -orders -0.81% 0.7448 - - -converted -0.80% 0.6996 - - -returns -8.83% 0.1948 - - -visits -0.13% 0.8942 - - -pages_viewed -0.16% 0.8994 - - -add_to_cart -1.07% 0.4960 - - -wishlist_adds +5.31% 0.0307 yes - -reviews_read -0.68% 0.6344 - - -search_uses +1.07% 0.4363 - - -newsletter_signups +11.35% 0.1746 - - -minutes_on_site -0.09% 0.9562 - - -support_contacts +0.47% 0.9115 - - -coupon_uses -0.06% 0.9874 - - -share_clicks +11.72% 0.0637 - - -photo_zooms +0.65% 0.6465 - - -size_guide_views +0.38% 0.8931 - - -video_plays +0.27% 0.8978 - - -compare_uses +0.79% 0.7636 - - -stock_alerts -3.55% 0.4553 - - -Without a correction: 1 winner(s) ['wishlist_adds']. Bonferroni: 0. Benjamini-Hochberg: 0. -
Compare with the truth.
Terminal window xp checkRun many-metrics-s2001: the last analysis was `xp metrics`.Metrics with a real effect (from the simulator): none.Your winners without a correction: wishlist_adds; false winners: wishlist_adds.After Benjamini-Hochberg: none.
The primary metric did not move: −3.0%, p = 0.33. But one metric “won”: wishlist adds, +5.3%, p = 0.031. A report could say: “The new layout raises wishlist adds by 5%.” The simulator says that no metric has a real effect.
Text description of the chart
Twenty rows on an axis from −25% to +30%, with a no-effect line at 0. Each row is a metric with its estimate and its 95% TermConfidence intervalA range around a measured value, made by a method that holds the true value 95 times in 100. If the interval of a difference does not include zero, the difference is probably not chance.Learn more Estimate uncertaintyEssential Mathematics and Statistics for AI. revenue: -3.0% (-9.0 to 3.0); orders: -0.8% (-5.7 to 4.0); converted: -0.8% (-4.9 to 3.3); returns: -8.8% (-22.2 to 4.5); visits: -0.1% (-2.1 to 1.8); pages_viewed: -0.2% (-2.6 to 2.3); add_to_cart: -1.1% (-4.1 to 2.0); wishlist_adds: 5.3% (0.5 to 10.1); reviews_read: -0.7% (-3.5 to 2.1); search_uses: 1.1% (-1.6 to 3.8); newsletter_signups: 11.3% (-5.0 to 27.7); minutes_on_site: -0.1% (-3.4 to 3.3); support_contacts: 0.5% (-7.8 to 8.7); coupon_uses: -0.1% (-7.0 to 6.8); share_clicks: 11.7% (-0.7 to 24.1); photo_zooms: 0.7% (-2.1 to 3.4); size_guide_views: 0.4% (-5.1 to 5.9); video_plays: 0.3% (-3.8 to 4.4); compare_uses: 0.8% (-4.4 to 5.9); stock_alerts: -3.5% (-12.9 to 5.8). Only the wishlist-adds interval lies fully above zero. Revenue, the primary metric, is at −3.0%, with an interval that contains zero.
Nineteen intervals contain zero, and one misses it. That is what 95% intervals do: about 1 in 20 misses the truth. With 20 metrics, you should expect about one miss.
The idea: many chances, one false winner
Section titled “The idea: many chances, one false winner”Each metric is one test. Each test with no real effect has a 5% chance to show p < 0.05. If you run 20 such tests, the chance that at least one of them crosses the line is much larger than 5%. This is the multiple testing problem. A winner that is only chance is a false discovery.
If the 20 metrics were independent, the chance would be 1 − 0.9520 = 64%, the same sum as for 20 segments in Segments and different effects. Real metrics move together: a customer who views more pages also adds more to the cart. So they give fewer independent chances. The course ran the layout test on 300 seeds:
Text description of the chart
Five bars. The formula for 20 independent metrics: 64.2%. Measured, with no correction: 56.0%. Bonferroni: 4.0%. Benjamini–Hochberg: 4.7%. The primary metric alone: 7.0%.
More than half of the runs have a false winner. The primary metric alone gives 7.0%, close to 5%: with only 300 runs, chance moves the measured rate by a point or two.
Two corrections
Section titled “Two corrections”A correction makes each metric’s test stricter, so that the whole set of tests keeps a promise. The two common corrections keep two different promises.
Bonferroni: a stricter line for each metric
Section titled “Bonferroni: a stricter line for each metric”Bonferroni divides 0.05 by the number of metrics. With 20 metrics, a metric wins only with p < 0.05 / 20 = 0.0025. It promises: the chance of any false winner, over all metrics, stays under 5%. This chance is called the family-wise error rate.
It is simple and safe. It is also strict: a real effect of medium size may not reach p < 0.0025, so you may miss it.
Benjamini–Hochberg: control the share of false winners
Section titled “Benjamini–Hochberg: control the share of false winners”Benjamini–Hochberg (BH) keeps a different promise. Among the metrics that you call winners, the share of false ones stays under 5% on average. This share is the false discovery rate (FDR).
The method has three steps:
- Sort the m p-values from small to large.
- Give the k-th smallest the limit k / m × 0.05.
- Find the largest k whose p-value is under its limit. Every metric up to that k wins.
Here is a small example with five metrics. We made up the p-values for this example.
| Rank k | p-value | Bonferroni limit (0.05 / 5) | BH limit (k / 5 × 0.05) | BH: under the limit? |
|---|---|---|---|---|
| 1 | 0.004 | 0.01: wins | 0.01 | yes |
| 2 | 0.012 | 0.01: no | 0.02 | yes |
| 3 | 0.030 | 0.01: no | 0.03 | yes |
| 4 | 0.30 | 0.01: no | 0.04 | no |
| 5 | 0.70 | 0.01: no | 0.05 | no |
The largest k under its limit is 3. So BH keeps three winners: ranks 1, 2 and 3. Bonferroni keeps only one. BH is less strict, so it finds more real effects, and it accepts that a few winners may be false.
Larkfield’s 20 metrics
Section titled “Larkfield’s 20 metrics”Apply BH to the layout test. The five smallest p-values of the 20 are:
| Rank k | Metric | p-value | BH limit (k / 20 × 0.05) |
|---|---|---|---|
| 1 | wishlist_adds | 0.0307 | 0.0025 |
| 2 | share_clicks | 0.0637 | 0.0050 |
| 3 | newsletter_signups | 0.1746 | 0.0075 |
| 4 | returns | 0.1948 | 0.0100 |
| 5 | revenue | 0.3297 | 0.0125 |
Even the smallest p-value, 0.0307, is far above its limit, 0.0025. No metric wins. This is what xp metrics printed: Bonferroni 0, Benjamini–Hochberg 0. Over 300 runs, BH left at least one false winner in 4.7% of the runs, and Bonferroni in 4.0%.
Choose the primary metric before the test
Section titled “Choose the primary metric before the test”Corrections help, but the strongest rule is older: one primary metric decides, and you choose it before the test. You wrote it in the plan in Write the plan first. One metric, one look at the end: the 5% holds.
The other metrics still have jobs:
- Diagnostic metrics explain the result: did revenue rise through more orders or bigger baskets? Read them with BH, and treat a surprise as an idea for the next test, not a result.
- Guardrail metrics protect the shop: returns, complaints, page speed. A guardrail asks “did it get worse?”, so it is not a search for winners. Name the guardrails in the plan, and do not correct them away.
What is not allowed: choose the metric after you see the data. If wishlist adds had been the plan’s primary metric, p = 0.031 would be a fair result. Chosen after the test from 20 metrics, it is the luckiest of 20 numbers.
Many tests, not only many metrics
Section titled “Many tests, not only many metrics”The same problem appears across tests. Mateo, who runs Larkfield’s test log, counts 40 tests a year. If most changes do nothing, about 5% of them still “win”. The test log keeps every test, with its plan and its result, so the team can see how many winners to expect from luck alone. Segments are another form of many tests: you saw 20 segments in Segments and different effects.
| You check | Promise you need | Use |
|---|---|---|
| One primary metric, chosen in the plan | 5% false winners | The usual test, no correction |
| A few diagnostic metrics, to explain the result | Few false winners among the winners | Benjamini–Hochberg |
| Many metrics where any false winner is costly | No false winner at all, at 5% | Bonferroni |
| Guardrails | Catch harm quickly | A separate rule for each, written in the plan |
Guided practice: count and correct the false winners
Section titled “Guided practice: count and correct the false winners”- Open the notebook from the box Do this lesson in a notebook at the top of this page.
- Run the setup cells and the helper cell.
- In section 1, predict how many of 20 metrics in one A/A test show p < 0.05. Then run the cell.
- In section 2, run 1,000 A/A tests. You see about 65% of tests with a false winner for independent metrics, and fewer for linked metrics.
- In section 3, apply Bonferroni and Benjamini–Hochberg to the same tests. Both bring the rate near 5%.
- In section 4, compute the five BH limits and the number of metrics that BH keeps.
- Run the check cells. They print
Check passed.
Show the answer to step 6
p5 = [0.004, 0.012, 0.030, 0.30, 0.70] # already sortedbh_limits = [round(k / 5 * 0.05, 3) for k in range(1, 6)]passes = [p <= limit for p, limit in zip(p5, bh_limits)]kept = max(k + 1 for k, ok in enumerate(passes) if ok)print(bh_limits, passes, kept)The limits are 0.01, 0.02, 0.03, 0.04 and 0.05. The first three p-values pass, so kept is 3.
Your turn: is the wishlist effect real?
Section titled “Your turn: is the wishlist effect real?”TermOmarLarkfield’s product manager. He asks the business questions: what a change costs, and what it gives the shop. asks: “Forget the other 19. Just check wishlist adds on their own. Is the effect real?” Predict first: will a test of this one metric say yes or no? Will the interval contain the truth? Then analyse wishlist_adds alone and check it.
Show the answer
xp analyse --metric wishlist_addsxp checkRun many-metrics-s2001 (synthetic): A new product page layout with no real effect, checked on 20 metrics (planted: one false winner).Metric: wishlist_adds. A: 16,451 customers, mean 0.2839. B: 16,906 customers, mean 0.2989. difference +0.0151 (+5.3%), 95% interval +0.0014 to +0.0287, p = 0.0307 (Welch t-test)Saved to runs/many-metrics-s2001/analysis.json. Compare with the true effect: xp checkRun many-metrics-s2001: the last analysis was `xp analyse`.True effect (from the simulator): +0.0000 every metric: no real effect.Your estimate: +0.0151, 95% interval +0.0014 to +0.0287: the interval does NOT contain the true effect.The test of one metric says p = 0.031, and its interval misses the truth. The test is not wrong: it is one of the 1 in 20 intervals that miss. The mistake is the choice. We chose this metric because it was the smallest of 20 p-values. To check the idea fairly, run a new test with wishlist adds as the primary metric.
Common mistake
“Revenue did not move, but wishlist adds did”
Symptom: the layout test ends with no change in revenue. The report says: “The new layout raises wishlist adds by 5.3% (p = 0.03). We recommend shipping it.” Months later, wishlist adds are where they were.
Cause: the report picked the metric after the test, from 20 metrics. With 20 tests at 5%, about one false winner is normal. In the course’s 300 runs, 56% had at least one. The p-value of one metric does not count the 19 other chances.
Fix: decide on the primary metric in the plan. Report every metric that you checked, with a correction: xp metrics prints Bonferroni and Benjamini–Hochberg. Treat a surprise in a diagnostic metric as an idea for a new test.
Check your understanding
A test has 10 metrics. With Bonferroni at 5%, which p-value must a metric have to win? Three metrics have p = 0.003, 0.008 and 0.04. Which of them win?
Show the answer
Bonferroni’s limit is 0.05 / 10 = 0.005. Only the metric with p = 0.003 wins. The metric with p = 0.008 is above 0.005, and so is the one with p = 0.04.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
Apply Benjamini–Hochberg at 5% to four metrics with p = 0.001, 0.020, 0.030 and 0.50. Which metrics win?
Show the answer
Sort the p-values. Give the k-th the limit k / 4 × 0.05: 0.0125, 0.025, 0.0375 and 0.05. Then compare. 0.001 ≤ 0.0125 yes; 0.020 ≤ 0.025 yes; 0.030 ≤ 0.0375 yes; 0.50 ≤ 0.05 no. The largest k under its limit is 3, so the first three metrics win. Bonferroni (0.05 / 4 = 0.0125) would keep only the first one.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
Tomás says: “We check 20 metrics and correct with Bonferroni, so we do not need a primary metric.” What is wrong with this?
Show the answer
Bonferroni keeps false winners rare. But it makes each metric’s test very strict (p < 0.0025). So the test may miss the real effect on the metric that matters. It also does not say which result decides. A primary metric chosen in the plan keeps the normal 5% rule for the metric that decides. We also plan the test’s sample size for that metric. The other metrics explain the result or guard against harm.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Recap
- With many metrics, some “win” by chance. This is the multiple testing problem, and a winner that is only chance is a false discovery.
- The layout test (a practice problem we made on purpose) had no real effect. One of 20 metrics, wishlist adds, showed +5.3% with p = 0.031. Over 300 runs, 56% had at least one false winner.
- Bonferroni divides 0.05 by the number of metrics: it controls the chance of any false winner (4.0% over 300 runs).
- Benjamini–Hochberg compares the k-th smallest p-value with k / m × 0.05. It controls the share of false winners among the winners (4.7% of runs had one). It finds more real effects than Bonferroni.
- The strongest rule: one primary metric, chosen in the plan, decides. Diagnostic metrics explain; guardrails protect.
You can now control false winners over days and over metrics. Next, a real effect that does not last: a widget that wins in its first weeks and then loses.