Skip to content
Français

Segments and different effects

  • Lesson 12
  • 35 min
  • Module 3: Analyse the test

Where we are

We have a precise average effect. Omar now asks whether the change works better on mobile, or for new customers. In this lesson, we look at segments, see why many segments find false winners, and plan the segments before the test.

What you will be able to do

Read the effect in segments without being fooled. With enough segments, one of them always “wins” by luck, so the segments you report must be chosen before the test.

NotebookDo this lesson in a notebookColab · Kaggle · GitHub

In the notebook, you estimate the free-delivery effect on mobile and desktop, and test whether the two segments differ. Then you count the false winners among 20 segments of 200 A/A tests. Its setup cells install the packages and download the practice data, so it needs no local setup. About 25 minutes.

Colab and Kaggle open your own copy. They need a free Google or Kaggle account. Stuck? See the solution notebook.

Before you startWhat to know and to have readyAbout 35 minutes

What you need to know

What you need on your computer (only for the project steps)

  • The project larkfield-experiments at the snapshot end-of-m03, with .venv active, and the runs delivery and aa from the earlier lessons of this module. If one is missing, make it again with sim run --scenario NAME.

Tested on macOS with Python 3.12 and the end-of-m03 snapshot of the project. Windows was not tested.

A baker tries a new bread recipe for a month. Sales rise a little, but not clearly. So she looks closer. Did children like it more? Older customers? Customers on Mondays? Customers who also buy coffee? She looks at twenty groups of customers. In one group, Tuesday mornings, sales rose a lot. “The new bread works on Tuesday mornings!”

Does it? Sales on any morning go up and down by chance. With twenty groups, one of them will go up a lot by chance alone. The baker did not find a Tuesday effect. She found the group where luck was strongest.

This is the problem with segments: parts of the customers, such as mobile users or new customers. A different effect in some segments can be real and important. But each segment has fewer customers, so more noise, and every extra segment is one more chance to be fooled.

Omar’s question: does it work on mobile?

Section titled “Omar’s question: does it work on mobile?”

TermOmarLarkfield’s product manager. He asks the business questions: what a change costs, and what it gives the shop. wants to know where free delivery works best. The project shows the effect in each segment of a column with --by.

  1. Analyse the free-delivery test by device.

    Terminal window
    xp analyse delivery --by device
    Run delivery-s40 (synthetic): Free delivery from 50 dollars instead of 75: an A/B test by customer, 28 days.
    Metric: revenue. A: 55,430 customers, mean 25.0504. B: 55,329 customers, mean 26.0770.
    difference +1.0266 (+4.1%), 95% interval +0.1583 to +1.8950, p = 0.0205 (Welch t-test)
    device = desktop: difference +0.7313 (+2.6%), 95% interval -0.7162 to +2.1787, p = 0.322 (Welch t-test)
    device = mobile: difference +1.2226 (+5.4%), 95% interval +0.1548 to +2.2904, p = 0.0248 (Welch t-test)
    Saved to runs/delivery-s40/analysis.json. Compare with the true effect: xp check
  2. Analyse it by new and returning customers. new_customer is 1 for a customer whose first order was less than 90 days ago.

    Terminal window
    xp analyse delivery --by new_customer
    Run delivery-s40 (synthetic): Free delivery from 50 dollars instead of 75: an A/B test by customer, 28 days.
    Metric: revenue. A: 55,430 customers, mean 25.0504. B: 55,329 customers, mean 26.0770.
    difference +1.0266 (+4.1%), 95% interval +0.1583 to +1.8950, p = 0.0205 (Welch t-test)
    new_customer = 0: difference +1.1829 (+4.3%), 95% interval +0.0785 to +2.2874, p = 0.0358 (Welch t-test)
    new_customer = 1: difference +0.5263 (+2.8%), 95% interval -0.7856 to +1.8382, p = 0.432 (Welch t-test)
    Saved to runs/delivery-s40/analysis.json. Compare with the true effect: xp check

TermTomásLarkfield’s AI engineer. He often suggests a quick shortcut, and the lesson shows why it fails. reads it fast: “It works on mobile, not on desktop. It works for returning customers, not for new ones. Let’s give free delivery from 50 dollars only to mobile and returning customers.”

The simulator knows the true effect in each segment, because it ran every customer in both worlds. xp check only knows the total, so the course computed the segments’ truth with its own script.

The free-delivery effect in four segments: each estimate with its 95% interval (blue: above zero, grey: crosses zero), and the simulator's true effect in that segment (orange dot). The true effects are all between +0.57 and +0.75 dollars. The estimates scatter around them by chance.
Text description of the chart

Eight rows on an axis from −1.5 to +3 dollars with a dashed no-effect line at 0. Desktop: estimate +0.73, TermConfidence intervalA range around a measured value, made by a method that holds the true value 95 times in 100. If the interval of a difference does not include zero, the difference is probably not chance.Learn more Estimate uncertaintyEssential Mathematics and Statistics for AI −0.72 to +2.18; true effect +0.75. Mobile: estimate +1.22, interval +0.15 to +2.29; true effect +0.66. Returning customers: estimate +1.18, interval +0.08 to +2.29; true effect +0.75. New customers: estimate +0.53, interval −0.79 to +1.84; true effect +0.57. Every interval contains its true effect.

Free delivery works almost the same everywhere. On desktop, the truth (+0.75) is even a little higher than on mobile (+0.66). The “winning” segments are the ones where luck pushed the estimate up. The segments that “do not work” have fewer customers, so their intervals are wider and cross zero more easily.

“Wins” and “does not win” is not a difference

Section titled ““Wins” and “does not win” is not a difference”

One segment passes p < 0.05 and the other does not. That does not show that they differ. To compare two segments, test the difference between them.

The two estimates come from different customers, so their errors add up like this: SE of the gap = √(SEmobile² + SEdesktop²).

Effect Standard error
Mobile +1.22 0.54
Desktop +0.73 0.74
Mobile − desktop +0.49 √(0.54² + 0.74²) = 0.92

The 95% interval of the gap is +0.49 ± 1.96 × 0.92: from −1.31 to +2.29 dollars. It contains zero, and it is wide. The test cannot tell whether mobile is better, worse or the same. The true gap is −0.09 dollars.

A segment needs many more customers than the total. The gap between two halves has about twice the standard error of the total effect. To find a gap as precisely as the total, a test needs about four times as many customers. Most tests are sized for the total only, so their segments are too noisy to compare.

Omar’s two columns gave four segments. A report with provinces would have ten more, and provinces times devices make twenty. What happens then?

Each segment with no true effect has a 5% chance of p < 0.05. With one segment, the chance of a false winner is 5%. With two, the chance that neither wins is 0.95 × 0.95 = 0.90, so the chance of at least one false winner is 10%. With 20 segments:

1 − 0.9520 = 1 − 0.36 = 0.64

So, with 20 segments and no effect at all, about two reports in three show at least one “winner”.

The A/A test has no effect anywhere. Cut it by province and device: 20 segments.

The A/A test (both groups saw the same shop) cut into 20 segments: 10 provinces × 2 devices. Every true effect is zero. Orange rows have p < 0.05: false winners. Here, one of the 20. Small provinces have very wide intervals.
Text description of the chart

Twenty rows, one per province and device, on an axis from −22 to +22 dollars, with a dashed line at 0, the true effect. Nineteen intervals contain 0. One row, British Columbia on desktop, is orange: its p-value is 0.026, and its interval misses 0 although nothing changed. The small provinces have intervals more than 20 dollars wide.

The course ran 100 A/A tests on 100 seeds and counted the false winners among the same 20 segments each time:

How many of 20 segments had p < 0.05, in each of 100 simulated A/A tests. In many tests, at least one segment 'wins', although nothing changed anywhere.
Text description of the chart

A bar chart. Each bar is a number of false winners in one A/A test: 0, 1, 2 and so on. Its height is the number of tests, out of 100, with that many false winners.

Over the 100 A/A tests, 65 of them had at least one false winner, and the average was 1.0 false winner per test. This is what the coin calculation predicted. A report with 20 segments almost always has a “winner” to tell a story about.

The idea: plan the segments before the test

Section titled “The idea: plan the segments before the test”

The fix is the one from Write the plan first: decide in advance. The plan names the few segments that matter for the decision, and says why. Then the report shows all of them, wins or not, with their intervals.

Two panels. Planned in advance: the plan names 2 segments and why, device and new or returning; the test runs; report both segments, wins or not, with their intervals. Green text: a false winner is rare. Found afterwards: the test runs; cut the data 20 ways, province by device; report the one segment with p below 0.05. Red text: with no effect at all, about 1 segment in 20 wins.
Planned segments against segments found afterwards. The same data, but only the planned report can be trusted.

TermLarkfieldA made-up Canadian online shop for home and garden products. It is not a real company. All Nextia courses use its help desk and data as the example.Learn more Meet Larkfield’s plan for the free-delivery test, from Write the plan first, can list them like this:

Segments (planned): device (mobile, desktop); new or returning customer.
Why: mobile baskets are smaller, so a lower threshold may matter more there; new customers may react more to delivery prices.
Decision: one threshold for everyone, unless the interval of the gap between two segments excludes zero.
Other segments: ideas for a new test only, not results.

Some changes do help some customers and harm others. Later in the course, a reminder email helps new customers and annoys heavy buyers, and a whole module finds whom to treat. Segments are worth planning when there is a reason to expect a difference:

  • The change works differently by design. For example, a new mobile page only changes mobile.
  • The cost differs. For example, free delivery costs more in a remote province.
  • A wrong answer for one group is costly. For example, a change that may confuse new customers.

A segment that you found after the test is an idea, not a result. Write it down and test it in a new test, planned for it.

The tools of this module work in each segment too. CUPED makes each segment’s interval narrower:

Terminal window
xp analyse delivery --cuped --by device
Run delivery-s40 (synthetic): Free delivery from 50 dollars instead of 75: an A/B test by customer, 28 days.
Metric: revenue. A: 55,430 customers, mean 25.0420. B: 55,329 customers, mean 26.0854.
difference +1.0434 (+4.2%), 95% interval +0.2631 to +1.8237, p = 0.00877 (Welch t-test, CUPED)
CUPED removed 19.3% of the variance (theta 0.271).
device = desktop: difference +1.1811 (+4.2%), 95% interval -0.0992 to +2.4613, p = 0.0706 (Welch t-test, CUPED)
device = mobile: difference +0.9726 (+4.3%), 95% interval -0.0010 to +1.9462, p = 0.0502 (Welch t-test, CUPED)
Saved to runs/delivery-s40/analysis.json. Compare with the true effect: xp check

With CUPED, the two devices show +1.18 and +0.97 dollars: much closer to each other, and both just miss p < 0.05. The “mobile only” story came from noise, and less noise makes it disappear. And a sample ratio mismatch can hide in one segment, as the redirect bug did on mobile, so check the split by segment too.

  1. Open the notebook from the box Do this lesson in a notebook at the top of this page.
  2. Run the setup cells and the helper cell. The data cell ends with Ready: 2 files in data/.
  3. In section 1, predict: will both devices show a clear gain? Then run the cell. Only mobile has p < 0.05.
  4. In section 2, compute the gap between mobile and desktop. You see +0.49 dollars, 95% interval -1.31 to +2.29.
  5. In section 3, predict how often 20 segments find at least one false winner. Then run the 200 A/A splits. You see a share close to 64%.
  6. In section 4, compute gap_low for returning minus new customers.
  7. Run the check cell. It prints Check passed.
Show the answer to step 6
ret, new = welch(d[d.new_customer == 0]), welch(d[d.new_customer == 1])
gap_n = ret["difference"] - new["difference"]
se_n = np.sqrt(ret["se"] ** 2 + new["se"] ** 2)
gap_low = round(gap_n - 1.96 * se_n, 2)
print(ret, new, round(gap_n, 2), gap_low)

The gap is +0.66 dollars, and its interval goes from −1.06 to +2.37: it contains zero. The test cannot say that returning customers react more. The simulator’s true gap is +0.19 dollars.

This task uses the whole module. Analyse Hana’s free-delivery test as TermPriyaLarkfield’s data scientist. She measures how well the assistant works, on test sets and not on guesses. would, and write the result for Hana in five sentences or fewer:

  1. Check the split, overall and by device.
  2. Estimate the effect on revenue per customer, with and without CUPED.
  3. Estimate the effect on the guardrail, returns per order.
  4. Report the two planned segments, device and new or returning, with their intervals.
  5. Say what you would decide, and what you would still need to know.
Show the answer
Terminal window
xp srm delivery --by device
xp analyse delivery
xp analyse delivery --cuped
xp analyse delivery --metric return_rate
xp analyse delivery --cuped --by device
xp analyse delivery --cuped --by new_customer

The split is healthy: B has 49.95% of the customers (p = 0.76), and both devices are on plan. With CUPED, free delivery from 50 dollars raised revenue per customer by +1.04 dollars, probably between +0.26 and +1.82. Without CUPED: +1.03, from +0.16 to +1.90. Returns rose from 8.3% to 9.6% of orders, +1.29 points (+0.68 to +1.90): the guardrail is clearly worse. The planned segments do not differ clearly, so one threshold for everyone is the right comparison. Before she decides, Hana must compare the gain with the cost of the extra free deliveries and returns. She must also remember that this run was lucky. Over 200 seeds, the test with CUPED finds a true gain of this size in only 48.5% of runs.

The simulator’s truth for this run: +0.70 dollars of revenue and +1.17 points of returns.

Common mistake

Shipping to the winning segment

Symptom: Omar reads the analysis by device and decides: “Free delivery from 50 dollars for mobile only.” The simulator knows what would happen. Mobile customers would gain about +0.66 dollars each, not the +1.22 of the test. Desktop customers, who would gain +0.75, would not get the offer.

Cause: the mobile segment “won” by luck. Its interval, +0.15 to +2.29, was wide, and the gap with desktop (+0.49, from −1.31 to +2.29) contained zero. The true effects were +0.66 on mobile and +0.75 on desktop. The decision followed the noise.

Fix: decide on the planned primary metric for everyone, unless the plan named the segment and the gap between segments is clear. If a segment looks different, plan a new test for it, with enough customers in that segment.

Check your understanding

A report shows the effect of a change in 30 segments. Two of them have p < 0.05, and the total effect is not significant. What do you tell the team?

Show the answer

With 30 segments and no effect at all, we expect about 1.5 false winners (30 × 0.05). At least one appears in about 79% of reports (1 − 0.95³⁰). So two winners out of 30 is what luck alone makes. They are not results. Suppose that one of them matters for a decision, and there is a reason to expect it. Then write it in the plan of a new test, with enough customers in that segment.

Check your understanding

Predict and explain: in a test, segment X gives +2.0 dollars (95% interval +0.4 to +3.6) and segment Y gives +0.5 dollars (−1.1 to +2.1). Does the change work better in X than in Y? Estimate the gap and its interval roughly.

Show the answer

Not clearly. Each interval is about ±1.6, so each standard error is about 1.6 / 1.96 ≈ 0.82. The gap is +1.5, with a standard error of √(0.82² + 0.82²) ≈ 1.16. So its interval is about +1.5 ± 2.3: from −0.8 to +3.8. It contains zero. X passes p < 0.05 and Y does not, but the data cannot say that they differ.

Check your understanding

Write the segments part of a test plan for a new product page on mobile. Which segments do you plan, and why? What do you do with a segment found after the test?

Show the answer

A good answer names few segments and a reason for each, before the test. For example, device: the change is a mobile page, so desktop should show no effect, which checks that the test works. And new or returning customers: new customers may find a new page harder. It says that both are reported with their intervals, wins or not. It also says how a decision uses them: only if the interval of the gap excludes zero. A segment found after the test is an idea for a new test, not a result.

Recap

  • A segment is a part of the customers, such as mobile or new customers. Each segment has fewer customers, so a wider interval.
  • On the free-delivery test, mobile “won” (+1.22, p = 0.025) and desktop did not (+0.73, p = 0.32). The simulator’s true effects were +0.66 and +0.75. A segment that passes p < 0.05 next to one that does not is not a difference: test the gap. Here it was +0.49, from −1.31 to +2.29.
  • With 20 segments and no effect, about 64% of reports show at least one false winner (1 − 0.95²⁰). The course’s 100 A/A tests cut into 20 segments agreed.
  • Plan the segments before the test, with a reason for each, and report them all with their intervals. A segment found afterwards is an idea for a new test.
  • The module’s checks work in segments too. CUPED narrows each segment’s interval, and a sample ratio mismatch can hide in one segment.

You can now analyse a test from the split to the segments. But every analysis of this module looked at the result once, at the end. Next, you see what happens when the team looks every day and stops at the first “win”.

Next: Peeking