Skip to content
Français

Many metrics and many tests

  • Lesson 14
  • 35 min
  • Module 4: Traps in real tests

Where we are

Peeking looks at one metric many times. Another trap looks at many metrics once. In this lesson, we check 20 metrics in one test, find a winner that is not real, and correct for it with Bonferroni and Benjamini–Hochberg.

What you will be able to do

Read a test with many metrics without finding winners that are not there. Check 20 metrics with the 5% rule, and about one of them “wins” by luck. A correction and a primary metric chosen before the test keep the false winners rare.

NotebookDo this lesson in a notebookColab · Kaggle · GitHub

In the notebook, you simulate 1,000 A/A tests with 20 metrics each, count the false winners, and apply Bonferroni and Benjamini–Hochberg. Then you do Benjamini–Hochberg by hand on five p-values. Its setup cells install the packages, so it needs no local setup. About 25 minutes.

Colab and Kaggle open your own copy. They need a free Google or Kaggle account. Stuck? See the solution notebook.

Before you startWhat to know and to have readyAbout 35 minutes

What you need to know

What you need on your computer (only for the project steps)

  • The project larkfield-experiments at the snapshot end-of-m04, from Peeking, with .venv active.

Tested on macOS with Python 3.12 and the end-of-m04 snapshot of the project. Windows was not tested.

A school gives its pupils a new breakfast for a month. Then the head teacher compares the pupils with last year’s on 20 results: maths, reading, spelling, running, attendance, and 15 more. One result is better, with p = 0.03: spelling.

The newspaper writes: “New breakfast improves spelling.” But nobody expected spelling to change. And with 20 results, one of them is likely to look better by chance alone. The head teacher did not find an effect. He found the luckiest of 20 numbers.

The problem: a layout that “raised wishlist adds”

Section titled “The problem: a layout that “raised wishlist adds””

TermLarkfieldA made-up Canadian online shop for home and garden products. It is not a real company. All Nextia courses use its help desk and data as the example.Learn more Meet Larkfield tested a new product page layout. The plan named one primary metric, revenue per customer. The dashboard shows 19 more metrics: orders, visits, cart adds, wishlist adds, video plays and others. The simulator has this test as a scenario. Its false winner is a practice problem we made on purpose: the layout changes nothing.

  1. Run the layout test.

    Terminal window
    sim run --scenario many-metrics
    Scenario many-metrics (seed 2001): A new product page layout with no real effect, checked on 20 metrics (planted: one false winner).
    Wrote runs/many-metrics-s2001: customers (33,357 rows), visits (107,695 rows). Synthetic data. 3.9 s
  2. Analyse the primary metric.

    Terminal window
    xp analyse
    Run many-metrics-s2001 (synthetic): A new product page layout with no real effect, checked on 20 metrics (planted: one false winner).
    Metric: revenue. A: 16,451 customers, mean 25.4117. B: 16,906 customers, mean 24.6490.
    difference -0.7627 (-3.0%), 95% interval -2.2962 to +0.7709, p = 0.33 (Welch t-test)
    Saved to runs/many-metrics-s2001/analysis.json. Compare with the true effect: xp check
  3. Test all 20 metrics at once.

    Terminal window
    xp metrics
    Run many-metrics-s2001: 20 metrics. Primary metric: revenue.
    metric relative p p<0.05 Bonferroni BH
    revenue -3.00% 0.3297 - - -
    orders -0.81% 0.7448 - - -
    converted -0.80% 0.6996 - - -
    returns -8.83% 0.1948 - - -
    visits -0.13% 0.8942 - - -
    pages_viewed -0.16% 0.8994 - - -
    add_to_cart -1.07% 0.4960 - - -
    wishlist_adds +5.31% 0.0307 yes - -
    reviews_read -0.68% 0.6344 - - -
    search_uses +1.07% 0.4363 - - -
    newsletter_signups +11.35% 0.1746 - - -
    minutes_on_site -0.09% 0.9562 - - -
    support_contacts +0.47% 0.9115 - - -
    coupon_uses -0.06% 0.9874 - - -
    share_clicks +11.72% 0.0637 - - -
    photo_zooms +0.65% 0.6465 - - -
    size_guide_views +0.38% 0.8931 - - -
    video_plays +0.27% 0.8978 - - -
    compare_uses +0.79% 0.7636 - - -
    stock_alerts -3.55% 0.4553 - - -
    Without a correction: 1 winner(s) ['wishlist_adds']. Bonferroni: 0. Benjamini-Hochberg: 0.
  4. Compare with the truth.

    Terminal window
    xp check
    Run many-metrics-s2001: the last analysis was `xp metrics`.
    Metrics with a real effect (from the simulator): none.
    Your winners without a correction: wishlist_adds; false winners: wishlist_adds.
    After Benjamini-Hochberg: none.

The primary metric did not move: −3.0%, p = 0.33. But one metric “won”: wishlist adds, +5.3%, p = 0.031. A report could say: “The new layout raises wishlist adds by 5%.” The simulator says that no metric has a real effect.

The 20 metrics of the layout test, each with its 95% interval, relative to group A. No metric has a real effect. One interval, wishlist adds, misses zero by chance.
Text description of the chart

Twenty rows on an axis from −25% to +30%, with a no-effect line at 0. Each row is a metric with its estimate and its 95% TermConfidence intervalA range around a measured value, made by a method that holds the true value 95 times in 100. If the interval of a difference does not include zero, the difference is probably not chance.Learn more Estimate uncertaintyEssential Mathematics and Statistics for AI. revenue: -3.0% (-9.0 to 3.0); orders: -0.8% (-5.7 to 4.0); converted: -0.8% (-4.9 to 3.3); returns: -8.8% (-22.2 to 4.5); visits: -0.1% (-2.1 to 1.8); pages_viewed: -0.2% (-2.6 to 2.3); add_to_cart: -1.1% (-4.1 to 2.0); wishlist_adds: 5.3% (0.5 to 10.1); reviews_read: -0.7% (-3.5 to 2.1); search_uses: 1.1% (-1.6 to 3.8); newsletter_signups: 11.3% (-5.0 to 27.7); minutes_on_site: -0.1% (-3.4 to 3.3); support_contacts: 0.5% (-7.8 to 8.7); coupon_uses: -0.1% (-7.0 to 6.8); share_clicks: 11.7% (-0.7 to 24.1); photo_zooms: 0.7% (-2.1 to 3.4); size_guide_views: 0.4% (-5.1 to 5.9); video_plays: 0.3% (-3.8 to 4.4); compare_uses: 0.8% (-4.4 to 5.9); stock_alerts: -3.5% (-12.9 to 5.8). Only the wishlist-adds interval lies fully above zero. Revenue, the primary metric, is at −3.0%, with an interval that contains zero.

Nineteen intervals contain zero, and one misses it. That is what 95% intervals do: about 1 in 20 misses the truth. With 20 metrics, you should expect about one miss.

Each metric is one test. Each test with no real effect has a 5% chance to show p < 0.05. If you run 20 such tests, the chance that at least one of them crosses the line is much larger than 5%. This is the multiple testing problem. A winner that is only chance is a false discovery.

If the 20 metrics were independent, the chance would be 1 − 0.9520 = 64%, the same sum as for 20 segments in Segments and different effects. Real metrics move together: a customer who views more pages also adds more to the cart. So they give fewer independent chances. The course ran the layout test on 300 seeds:

The layout test on 300 seeds, with 20 metrics and no real effect. Without a correction, 56% of the runs show at least one false winner. Both corrections bring it near 5%. The primary metric alone gives 7.0%.
Text description of the chart

Five bars. The formula for 20 independent metrics: 64.2%. Measured, with no correction: 56.0%. Bonferroni: 4.0%. Benjamini–Hochberg: 4.7%. The primary metric alone: 7.0%.

More than half of the runs have a false winner. The primary metric alone gives 7.0%, close to 5%: with only 300 runs, chance moves the measured rate by a point or two.

A correction makes each metric’s test stricter, so that the whole set of tests keeps a promise. The two common corrections keep two different promises.

Bonferroni: a stricter line for each metric

Section titled “Bonferroni: a stricter line for each metric”

Bonferroni divides 0.05 by the number of metrics. With 20 metrics, a metric wins only with p < 0.05 / 20 = 0.0025. It promises: the chance of any false winner, over all metrics, stays under 5%. This chance is called the family-wise error rate.

It is simple and safe. It is also strict: a real effect of medium size may not reach p < 0.0025, so you may miss it.

Benjamini–Hochberg: control the share of false winners

Section titled “Benjamini–Hochberg: control the share of false winners”

Benjamini–Hochberg (BH) keeps a different promise. Among the metrics that you call winners, the share of false ones stays under 5% on average. This share is the false discovery rate (FDR).

The method has three steps:

  1. Sort the m p-values from small to large.
  2. Give the k-th smallest the limit k / m × 0.05.
  3. Find the largest k whose p-value is under its limit. Every metric up to that k wins.

Here is a small example with five metrics. We made up the p-values for this example.

Rank k p-value Bonferroni limit (0.05 / 5) BH limit (k / 5 × 0.05) BH: under the limit?
1 0.004 0.01: wins 0.01 yes
2 0.012 0.01: no 0.02 yes
3 0.030 0.01: no 0.03 yes
4 0.30 0.01: no 0.04 no
5 0.70 0.01: no 0.05 no

The largest k under its limit is 3. So BH keeps three winners: ranks 1, 2 and 3. Bonferroni keeps only one. BH is less strict, so it finds more real effects, and it accepts that a few winners may be false.

Apply BH to the layout test. The five smallest p-values of the 20 are:

Rank k Metric p-value BH limit (k / 20 × 0.05)
1 wishlist_adds 0.0307 0.0025
2 share_clicks 0.0637 0.0050
3 newsletter_signups 0.1746 0.0075
4 returns 0.1948 0.0100
5 revenue 0.3297 0.0125

Even the smallest p-value, 0.0307, is far above its limit, 0.0025. No metric wins. This is what xp metrics printed: Bonferroni 0, Benjamini–Hochberg 0. Over 300 runs, BH left at least one false winner in 4.7% of the runs, and Bonferroni in 4.0%.

Corrections help, but the strongest rule is older: one primary metric decides, and you choose it before the test. You wrote it in the plan in Write the plan first. One metric, one look at the end: the 5% holds.

The other metrics still have jobs:

  • Diagnostic metrics explain the result: did revenue rise through more orders or bigger baskets? Read them with BH, and treat a surprise as an idea for the next test, not a result.
  • Guardrail metrics protect the shop: returns, complaints, page speed. A guardrail asks “did it get worse?”, so it is not a search for winners. Name the guardrails in the plan, and do not correct them away.

What is not allowed: choose the metric after you see the data. If wishlist adds had been the plan’s primary metric, p = 0.031 would be a fair result. Chosen after the test from 20 metrics, it is the luckiest of 20 numbers.

The same problem appears across tests. Mateo, who runs Larkfield’s test log, counts 40 tests a year. If most changes do nothing, about 5% of them still “win”. The test log keeps every test, with its plan and its result, so the team can see how many winners to expect from luck alone. Segments are another form of many tests: you saw 20 segments in Segments and different effects.

You check Promise you need Use
One primary metric, chosen in the plan 5% false winners The usual test, no correction
A few diagnostic metrics, to explain the result Few false winners among the winners Benjamini–Hochberg
Many metrics where any false winner is costly No false winner at all, at 5% Bonferroni
Guardrails Catch harm quickly A separate rule for each, written in the plan

Guided practice: count and correct the false winners

Section titled “Guided practice: count and correct the false winners”
  1. Open the notebook from the box Do this lesson in a notebook at the top of this page.
  2. Run the setup cells and the helper cell.
  3. In section 1, predict how many of 20 metrics in one A/A test show p < 0.05. Then run the cell.
  4. In section 2, run 1,000 A/A tests. You see about 65% of tests with a false winner for independent metrics, and fewer for linked metrics.
  5. In section 3, apply Bonferroni and Benjamini–Hochberg to the same tests. Both bring the rate near 5%.
  6. In section 4, compute the five BH limits and the number of metrics that BH keeps.
  7. Run the check cells. They print Check passed.
Show the answer to step 6
p5 = [0.004, 0.012, 0.030, 0.30, 0.70] # already sorted
bh_limits = [round(k / 5 * 0.05, 3) for k in range(1, 6)]
passes = [p <= limit for p, limit in zip(p5, bh_limits)]
kept = max(k + 1 for k, ok in enumerate(passes) if ok)
print(bh_limits, passes, kept)

The limits are 0.01, 0.02, 0.03, 0.04 and 0.05. The first three p-values pass, so kept is 3.

TermOmarLarkfield’s product manager. He asks the business questions: what a change costs, and what it gives the shop. asks: “Forget the other 19. Just check wishlist adds on their own. Is the effect real?” Predict first: will a test of this one metric say yes or no? Will the interval contain the truth? Then analyse wishlist_adds alone and check it.

Show the answer
Terminal window
xp analyse --metric wishlist_adds
xp check
Run many-metrics-s2001 (synthetic): A new product page layout with no real effect, checked on 20 metrics (planted: one false winner).
Metric: wishlist_adds. A: 16,451 customers, mean 0.2839. B: 16,906 customers, mean 0.2989.
difference +0.0151 (+5.3%), 95% interval +0.0014 to +0.0287, p = 0.0307 (Welch t-test)
Saved to runs/many-metrics-s2001/analysis.json. Compare with the true effect: xp check
Run many-metrics-s2001: the last analysis was `xp analyse`.
True effect (from the simulator): +0.0000 every metric: no real effect.
Your estimate: +0.0151, 95% interval +0.0014 to +0.0287: the interval does NOT contain the true effect.

The test of one metric says p = 0.031, and its interval misses the truth. The test is not wrong: it is one of the 1 in 20 intervals that miss. The mistake is the choice. We chose this metric because it was the smallest of 20 p-values. To check the idea fairly, run a new test with wishlist adds as the primary metric.

Common mistake

“Revenue did not move, but wishlist adds did”

Symptom: the layout test ends with no change in revenue. The report says: “The new layout raises wishlist adds by 5.3% (p = 0.03). We recommend shipping it.” Months later, wishlist adds are where they were.

Cause: the report picked the metric after the test, from 20 metrics. With 20 tests at 5%, about one false winner is normal. In the course’s 300 runs, 56% had at least one. The p-value of one metric does not count the 19 other chances.

Fix: decide on the primary metric in the plan. Report every metric that you checked, with a correction: xp metrics prints Bonferroni and Benjamini–Hochberg. Treat a surprise in a diagnostic metric as an idea for a new test.

Check your understanding

A test has 10 metrics. With Bonferroni at 5%, which p-value must a metric have to win? Three metrics have p = 0.003, 0.008 and 0.04. Which of them win?

Show the answer

Bonferroni’s limit is 0.05 / 10 = 0.005. Only the metric with p = 0.003 wins. The metric with p = 0.008 is above 0.005, and so is the one with p = 0.04.

Check your understanding

Apply Benjamini–Hochberg at 5% to four metrics with p = 0.001, 0.020, 0.030 and 0.50. Which metrics win?

Show the answer

Sort the p-values. Give the k-th the limit k / 4 × 0.05: 0.0125, 0.025, 0.0375 and 0.05. Then compare. 0.001 ≤ 0.0125 yes; 0.020 ≤ 0.025 yes; 0.030 ≤ 0.0375 yes; 0.50 ≤ 0.05 no. The largest k under its limit is 3, so the first three metrics win. Bonferroni (0.05 / 4 = 0.0125) would keep only the first one.

Check your understanding

Tomás says: “We check 20 metrics and correct with Bonferroni, so we do not need a primary metric.” What is wrong with this?

Show the answer

Bonferroni keeps false winners rare. But it makes each metric’s test very strict (p < 0.0025). So the test may miss the real effect on the metric that matters. It also does not say which result decides. A primary metric chosen in the plan keeps the normal 5% rule for the metric that decides. We also plan the test’s sample size for that metric. The other metrics explain the result or guard against harm.

Recap

  • With many metrics, some “win” by chance. This is the multiple testing problem, and a winner that is only chance is a false discovery.
  • The layout test (a practice problem we made on purpose) had no real effect. One of 20 metrics, wishlist adds, showed +5.3% with p = 0.031. Over 300 runs, 56% had at least one false winner.
  • Bonferroni divides 0.05 by the number of metrics: it controls the chance of any false winner (4.0% over 300 runs).
  • Benjamini–Hochberg compares the k-th smallest p-value with k / m × 0.05. It controls the share of false winners among the winners (4.7% of runs had one). It finds more real effects than Bonferroni.
  • The strongest rule: one primary metric, chosen in the plan, decides. Diagnostic metrics explain; guardrails protect.

You can now control false winners over days and over metrics. Next, a real effect that does not last: a widget that wins in its first weeks and then loses.

Next: Novelty and learning effects