Skip to content
Français

The average hides the people

  • Lesson 24
  • 30 min
  • Module 7: Uplift: who should get the treatment?

Where we are

So far, we estimated one average effect for everyone. Rafael's reminder email has a small average effect, but it hides very different people. In this lesson, we meet the four groups: persuadables, sure things, lost causes and do-not-disturbs.

Plays from YouTube. Nothing loads from YouTube until you press play. Privacy

What you will be able to do

Look under the average effect of the reminder email. Find a group that the email helps a lot, and a group that it harms. Name the four groups of customers. They explain why “email everyone” can lose orders.

NotebookDo this lesson in a notebookColab · Kaggle · GitHub

In the notebook, you compute the average effect of the email, split it for new customers and heavy buyers, and count the four groups. Its setup cells install the packages and download the email data, so it needs no local setup. About 30 minutes.

Colab and Kaggle open your own copy. They need a free Google or Kaggle account. Stuck? See the solution notebook.

Before you startWhat to know and to have readyAbout 30 minutes

What you need to know

What you need on your computer (only for the project steps)

  • The project larkfield-experiments with .venv active, at the snapshot end-of-m06 or later, from Regression adjustment. This lesson adds no new files.

Tested on macOS with Python 3.12. Windows was not tested.

A small shop sends a discount coupon to all its customers. Sales go up a little, and the owner is happy. Then she talks to four of her customers.

  • Amal came in only because of the coupon. Without it, she would not have come.
  • Bruno comes every Saturday. He used the coupon, but he would have come anyway. The shop gave him a discount for nothing.
  • Carla threw the coupon away. She never shops there, with or without a coupon.
  • David came every Saturday too. The coupon annoyed him: “They think I only come for discounts.” He stopped coming.

The average rise is real. But the coupon did four different things to four kinds of people. If the owner had known who was who, she would have sent it only to people like Amal.

This lesson is about seeing those differences. The next two lessons are about acting on them.

The problem: a good average, and customers who are hurt

Section titled “The problem: a good average, and customers who are hurt”

Rafael, the email manager at TermLarkfieldA made-up Canadian online shop for home and garden products. It is not a real company. All Nextia courses use its help desk and data as the example.Learn more Meet Larkfield, ran the reminder email test. The email went to customers who had not ordered for 30 days. Half of them got it (B), and half did not (A). You analysed this test in Meet the simulator. All the data is simulated.

The result looks simple. In the test, 10.43% of B ordered within 14 days, and 8.96% of A. The difference is +1.47 points (95% TermConfidence intervalA range around a measured value, made by a method that holds the true value 95 times in 100. If the interval of a difference does not include zero, the difference is probably not chance.Learn more Estimate uncertaintyEssential Mathematics and Statistics for AI +1.00 to +1.94). The simulator knows the true effect: +1.19 points. The interval contains it. “The email works,” Rafael says. “Next month, I will email all 50,000 customers on the list.”

But Rafael planned two segments before the test: new customers, and heavy buyers with 7 or more orders in a year. Look at the effect inside each.

The email's effect on the share who ordered, in points, with 95% intervals, in the simulated test (seed 40). The average for everyone is small and positive. It hides a large gain for new customers and a loss for heavy buyers. The truth in each segment: everyone +1.19, new customers +7.46, returning customers −0.92, heavy buyers −6.98.
Text description of the chart

A panel with a line at 0 (no effect). Everyone: +1.47 points (+1.00 to +1.94); the simulator’s truth is +1.19. New customers: +7.74 (+6.74 to +8.73); truth +7.46. Returning customers: −0.64 (−1.17 to −0.10); truth −0.92. Heavy buyers with 7 or more orders: −6.26 (−7.44 to −5.08); truth −6.98. The intervals of the new customers and of the heavy buyers are far from 0, on opposite sides.

The email raises orders among new customers by +7.74 points. It lowers orders among heavy buyers by −6.26 points. Each estimate is close to the truth that the simulator knows: +7.46 and −6.98.

So the average of +1.47 is not wrong. It is a mix of a large gain and a real loss. A plan built on the average alone would email the people it hurts.

The heavy buyers lose in a second way too. Over the whole test, the true share of customers who unsubscribe rises from 0.21% to 0.85%. Most of that rise comes from heavy buyers. They did not need a reminder, and some leave the list.

Heavy buyers order often without any reminder. In the simulator, a reminder makes some of them feel pushed, and they order less. Some leave the email list. We wrote this behaviour into the simulator: it is a practice problem we made on purpose. Real shops can see the same pattern, but its size here is ours. The lesson does not depend on the size: an average can hide people who are harmed.

For each customer, think of two possible results: what happens with the email, and what happens without it. You saw these two worlds in Causes and comparisons. Now apply them to one customer. Each customer falls into one of four groups.

The customer’s two results Name What the email does
Does not order without it, orders with it Persuadable Brings an extra order
Orders without it, orders with it Sure thing Nothing: they order anyway
Does not order without it, does not order with it Lost cause Nothing: they never order
Orders without it, does not order with it Do-not-disturb Loses an order

These names are common in marketing. Uplift is the word for the change that the email makes: the extra chance to order because of the email. A persuadable has a positive uplift. A do-not-disturb has a negative uplift. A sure thing and a lost cause have an uplift of zero.

Only two of the four groups change anything. An email costs money and attention. So the goal is to email the persuadables, and to leave the do-not-disturbs alone.

Eight customers. The numbers are made up for this example, and only a simulator could know both worlds.

Customer Orders without the email? Orders with the email? Group Effect
Ana yes yes Sure thing 0
Ben no yes Persuadable +1
Chloé yes no Do-not-disturb −1
Dev no no Lost cause 0
Elif no no Lost cause 0
Farid yes yes Sure thing 0
Grace no yes Persuadable +1
Hugo no no Lost cause 0

Add up the effects:

  • persuadables add +1 each: 2 customers, so +2;
  • do-not-disturbs add −1 each: 1 customer, so −1;
  • sure things and lost causes add nothing.

The average effect is (2 − 1) ÷ 8 = +0.125: 12.5 points. The result is positive, but the persuadables pay for the do-not-disturbs. If the shop emailed only Ben and Grace, it would get +2 orders with 2 emails instead of +1 order with 8 emails.

That sentence is the idea of the whole module.

Look at Ben again. A real test gave him the email or it did not. It never does both. So a real test cannot say “Ben is a persuadable”. If he got the email and ordered, he might be a persuadable or a sure thing. If he did not get it and ordered, he might be a sure thing or a do-not-disturb. Nothing in the log tells you which.

This is the same missing-world problem as in Causes and comparisons. The group of one person is never observed. Only groups of people have a measurable effect. So next, we will need models that predict the effect for a kind of customer. That is the next lesson.

The simulator is different. It simulates each customer in both worlds, with the same random numbers. So it can count the groups exactly.

A grid of two rows and two columns. Columns: orders with the email, does not order with the email. Rows: orders without the email, does not order without the email. Sure things, 4,592: order anyway. Do-not-disturbs, 853, in red: they would order, the email stops them. Persuadables, 1,567, in green: they order only because of the email. Lost causes, 52,988: they do not order either way. Under the grid: simulated, both worlds of 60,000 customers, the effect is 1,567 minus 853, divided by 60,000, which is plus 1.19 points.
The four groups in the simulated test of 60,000 customers. Only persuadables and do-not-disturbs change the result: (1,567 − 853) ÷ 60,000 = +1.19 points, the true effect.

Of the 60,000 customers, only 1,567 order because of the email. 853 do not order because of it. The other 57,580 are not changed by the email at all. The true effect is (1,567 − 853) ÷ 60,000 = +1.19 points: the number that xp check prints.

The email is not good or bad. It depends on who gets it. Now look at who is in the two groups that matter.

The groups inside two segments of the simulated test (P: persuadables, D: do-not-disturbs). Among new customers, 7.46% are persuadable and none are do-not-disturb. Among heavy buyers, it is the opposite: no persuadables and 6.98% do-not-disturbs.
Text description of the chart

A column chart with four columns. Among the 15,119 new customers: persuadables 7.46%, do-not-disturbs 0%. Among the 9,327 heavy buyers: persuadables 0%, do-not-disturbs 6.98%. These match the true effects of the segments: new customers +7.46 points, heavy buyers −6.98 points.

The chart explains the segment result of the test. New customers are almost all persuadable or indifferent. Heavy buyers have no persuadables, and many do-not-disturbs. So sending the email to heavy buyers can only lose orders.

Now run it yourself.

  1. Create the email data again. It is the same as in Meet the simulator.

    Terminal window
    sim run --scenario email
    Scenario email (seed 40): A reminder email to customers who have not ordered for 30 days: an A/B test by customer, 14 days.
    Wrote runs/email-s40: customers (60,000 rows), targets (50,000 rows), truth_targets (50,000 rows). Synthetic data. 0.5 s
  2. Analyse the test, with new customers as a segment.

    Terminal window
    xp analyse --by new_customer
    Run email-s40 (synthetic): A reminder email to customers who have not ordered for 30 days: an A/B test by customer, 14 days.
    Metric: ordered. A: 30,208 customers, mean 0.0896. B: 29,792 customers, mean 0.1043.
    difference +0.0147 (+16.4%), 95% interval +0.0100 to +0.0194, p = 1.14e-09 (Welch t-test)
    new_customer = 0: difference -0.0064 (-6.7%), 95% interval -0.0117 to -0.0010, p = 0.0195 (Welch t-test)
    new_customer = 1: difference +0.0774 (+106.0%), 95% interval +0.0674 to +0.0873, p = 9.98e-52 (Welch t-test)
    Saved to runs/email-s40/analysis.json. Compare with the true effect: xp check

    The line new_customer = 0 is the returning customers. Their interval is below zero: the email lowers their orders. This is the segment “returning customers” in the first chart.

  3. Compare with the truth.

    Terminal window
    xp check
    Run email-s40: the last analysis was `xp analyse`.
    True effect (from the simulator): +0.0119 share of customers with an order.
    Your estimate: +0.0147, 95% interval +0.0100 to +0.0194: the interval contains the true effect.
  4. Read the four groups from the simulator’s truth.

    Terminal window
    python -c "import json; print(json.load(open('runs/email-s40/truth.json'))['groups'])"
    {'persuadable': 1567, 'sure_thing': 4592, 'lost_cause': 52988, 'do_not_disturb': 853}

    Only the simulator’s truth file has these counts. A real test does not.

When to use it, when not, and what it costs

Section titled “When to use it, when not, and what it costs”

Think about the four groups when:

  • the treatment costs money or attention: an email, a coupon, a call, a free gift;
  • the treatment can annoy some people, or teach them to wait for a discount;
  • you have a randomised test. Without one, you cannot tell who the email changed, and you are back to the logs problems of Regression adjustment.

Do not use this idea when:

  • the change reaches everyone, such as a new checkout page, and you cannot choose who gets it;
  • your test has too few people to split into segments. A small segment has a wide interval, as you saw in Segments and different effects;
  • you choose segments after you see the result. Choose them first, or you will find a “winner” by chance. See Many metrics and many tests.

It costs one thing: a test that is large enough for the groups you care about. Before you spend a budget, you must know who you can reach, and who you can lose. The next lesson finds those people automatically.

Guided practice: split the average yourself

Section titled “Guided practice: split the average yourself”
  1. Open the notebook from the box Do this lesson in a notebook at the top of this page.
  2. Run the setup cells and the helper cell.
  3. In section 1, compute the average effect. You see everyone: +0.0147 (± 0.0047).
  4. In section 2, predict first: will the email help heavy buyers more or less than new customers? Then run the cell. You see new customers +0.0774 and heavy buyers -0.0626.
  5. In section 3, count the four groups. You see persuadables 1567 (2.6%) and do-not-disturbs 853 (1.4%).
  6. In section 4, compute the effect among customers with email_open_rate of 0.5 or more. Put it in open_effect. Run the check cell. It prints Check passed.
Show the answer to step 6
open_effect = round(float(effect(email[email.email_open_rate >= 0.5])[0]), 4)
print(open_effect)

The value is +0.0352: customers who open emails respond more than the average. The simulator’s truth for this group is +0.0416.

Your turn: find a group that the email harms

Section titled “Your turn: find a group that the email harms”

Rafael must hear about the customers who lose. Show him one group where the email lowers orders. Use the numbers from this lesson, or from your own run of the notebook. Write two or three lines with the segment, the estimate, the 95% interval, and the truth.

Show one answer

Heavy buyers (7 or more orders in 12 months): the email changes the share who ordered by −6.26 points, with a 95% interval from −7.44 to −5.08. The interval is far below zero. The simulator’s truth for this group is −6.98 points. The email does not help them, and it pushes some of them to unsubscribe.

A second correct answer: returning customers, −0.64 points (−1.17 to −0.10), from xp analyse --by new_customer; truth −0.92 points. The effect is much smaller and the interval is close to zero.

Common mistake

“The average effect is positive, so email everyone”

Symptom: the team sends the email to the full list after a good test. Orders rise less than expected, and unsubscribes rise, mostly among the best customers.

Cause: an average is a mix. It adds the persuadables and takes away the do-not-disturbs. A positive average only says that the first group is bigger than the second. It says nothing about who is who.

Fix: before the next send, look at planned segments: new and returning customers, and heavy buyers. Keep a guardrail on unsubscribes. Then use an uplift model to choose who gets the email. The next two lessons show how.

Check your understanding

A customer did not order without the email and did order with it, in the simulator's two worlds. Which of the four groups is this customer in? Would a real test be able to tell?

Show the answer

The customer is a persuadable: the email made the order happen. A real test could not tell, because it shows only one of the two worlds for each customer. If this customer got the email and ordered, they could be a persuadable or a sure thing. Only a simulator, which runs both worlds, can name the group of one person.

Check your understanding

The simulated test has 1,567 persuadables and 853 do-not-disturbs among 60,000 customers. What is the true effect in points, and why does an email to everyone lose some orders?

Show the answer

The true effect is (1,567 − 853) ÷ 60,000 = +1.19 points. The email to everyone loses orders from the do-not-disturbs: 853 customers who would have ordered without the email do not order with it. The average is positive only because persuadables outnumber them.

Check your understanding

Rafael reports: the email raised orders by 1.47 points, so it should go to every customer on the list. Using the segments in this lesson, what would you tell him?

Show the answer

The average hides very different people. The email adds about +7.74 points among new customers but removes about 6.26 points among heavy buyers. Emailing everyone gives up the orders of heavy buyers who would have ordered anyway, and it raises unsubscribes. He should choose who gets the email, not send it to everyone. He should keep a guardrail on unsubscribes. A model can help him find the persuadables.

Recap

  • An average effect can hide large and opposite effects in different people.
  • In the simulated email test, the average is +1.47 points (truth +1.19). New customers: +7.74 (truth +7.46). Heavy buyers: −6.26 (truth −6.98).
  • Each customer is one of four groups: persuadables (1,567), sure things (4,592), lost causes (52,988) and do-not-disturbs (853).
  • Uplift is the change the treatment makes for a customer. Only persuadables (positive) and do-not-disturbs (negative) have one.
  • A real test cannot see one customer’s group. Only the simulator can. So we need a model of the effect by kind of customer.

Next: train uplift models on the randomised test, and compare them with a normal “who will buy” model.

Next: Uplift models