The average hides the people
- Lesson 24
- 30 min
- Module 7: Uplift: who should get the treatment?
Where we are
So far, we estimated one average effect for everyone. Rafael's reminder email has a small average effect, but it hides very different people. In this lesson, we meet the four groups: persuadables, sure things, lost causes and do-not-disturbs.
Lesson 24 of 30 · Experiments and Causal Impact: Did Our Change Work?The average hides the people
1. Why "it went up" is not an answer
2. Design an A/B test
3. Analyse the test
4. Traps in real tests
5. When you cannot randomise: natural experiments
6. Adjust for confounders in observational data
7. Uplift: who should get the treatment?
8. Experiments for AI features and in production
What you will be able to do
Look under the average effect of the reminder email. Find a group that the email helps a lot, and a group that it harms. Name the four groups of customers. They explain why “email everyone” can lose orders.
NotebookDo this lesson in a notebook
In the notebook, you compute the average effect of the email, split it for new customers and heavy buyers, and count the four groups. Its setup cells install the packages and download the email data, so it needs no local setup. About 30 minutes.
Colab and Kaggle open your own copy. They need a free Google or Kaggle account. Stuck? See the solution notebook.
Before you startWhat to know and to have ready
What you need to know
- The two worlds of each customer, from Causes and comparisons.
- Segments planned before the test, from Segments and different effects.
- A 95% interval, from The result and its uncertainty.
What you need on your computer (only for the project steps)
- The project
larkfield-experimentswith.venvactive, at the snapshotend-of-m06or later, from Regression adjustment. This lesson adds no new files.
Tested on macOS with Python 3.12. Windows was not tested.
A coupon in a small shop
Section titled “A coupon in a small shop”A small shop sends a discount coupon to all its customers. Sales go up a little, and the owner is happy. Then she talks to four of her customers.
- Amal came in only because of the coupon. Without it, she would not have come.
- Bruno comes every Saturday. He used the coupon, but he would have come anyway. The shop gave him a discount for nothing.
- Carla threw the coupon away. She never shops there, with or without a coupon.
- David came every Saturday too. The coupon annoyed him: “They think I only come for discounts.” He stopped coming.
The average rise is real. But the coupon did four different things to four kinds of people. If the owner had known who was who, she would have sent it only to people like Amal.
This lesson is about seeing those differences. The next two lessons are about acting on them.
The problem: a good average, and customers who are hurt
Section titled “The problem: a good average, and customers who are hurt”Rafael, the email manager at TermLarkfieldA made-up Canadian online shop for home and garden products. It is not a real company. All Nextia courses use its help desk and data as the example.Learn more Meet Larkfield, ran the reminder email test. The email went to customers who had not ordered for 30 days. Half of them got it (B), and half did not (A). You analysed this test in Meet the simulator. All the data is simulated.
The result looks simple. In the test, 10.43% of B ordered within 14 days, and 8.96% of A. The difference is +1.47 points (95% TermConfidence intervalA range around a measured value, made by a method that holds the true value 95 times in 100. If the interval of a difference does not include zero, the difference is probably not chance.Learn more Estimate uncertaintyEssential Mathematics and Statistics for AI +1.00 to +1.94). The simulator knows the true effect: +1.19 points. The interval contains it. “The email works,” Rafael says. “Next month, I will email all 50,000 customers on the list.”
But Rafael planned two segments before the test: new customers, and heavy buyers with 7 or more orders in a year. Look at the effect inside each.
Text description of the chart
A panel with a line at 0 (no effect). Everyone: +1.47 points (+1.00 to +1.94); the simulator’s truth is +1.19. New customers: +7.74 (+6.74 to +8.73); truth +7.46. Returning customers: −0.64 (−1.17 to −0.10); truth −0.92. Heavy buyers with 7 or more orders: −6.26 (−7.44 to −5.08); truth −6.98. The intervals of the new customers and of the heavy buyers are far from 0, on opposite sides.
The email raises orders among new customers by +7.74 points. It lowers orders among heavy buyers by −6.26 points. Each estimate is close to the truth that the simulator knows: +7.46 and −6.98.
So the average of +1.47 is not wrong. It is a mix of a large gain and a real loss. A plan built on the average alone would email the people it hurts.
The heavy buyers lose in a second way too. Over the whole test, the true share of customers who unsubscribe rises from 0.21% to 0.85%. Most of that rise comes from heavy buyers. They did not need a reminder, and some leave the list.
Why heavy buyers order less
Section titled “Why heavy buyers order less”Heavy buyers order often without any reminder. In the simulator, a reminder makes some of them feel pushed, and they order less. Some leave the email list. We wrote this behaviour into the simulator: it is a practice problem we made on purpose. Real shops can see the same pattern, but its size here is ours. The lesson does not depend on the size: an average can hide people who are harmed.
The idea: four groups of customers
Section titled “The idea: four groups of customers”For each customer, think of two possible results: what happens with the email, and what happens without it. You saw these two worlds in Causes and comparisons. Now apply them to one customer. Each customer falls into one of four groups.
| The customer’s two results | Name | What the email does |
|---|---|---|
| Does not order without it, orders with it | Persuadable | Brings an extra order |
| Orders without it, orders with it | Sure thing | Nothing: they order anyway |
| Does not order without it, does not order with it | Lost cause | Nothing: they never order |
| Orders without it, does not order with it | Do-not-disturb | Loses an order |
These names are common in marketing. Uplift is the word for the change that the email makes: the extra chance to order because of the email. A persuadable has a positive uplift. A do-not-disturb has a negative uplift. A sure thing and a lost cause have an uplift of zero.
Only two of the four groups change anything. An email costs money and attention. So the goal is to email the persuadables, and to leave the do-not-disturbs alone.
A tiny example with every value visible
Section titled “A tiny example with every value visible”Eight customers. The numbers are made up for this example, and only a simulator could know both worlds.
| Customer | Orders without the email? | Orders with the email? | Group | Effect |
|---|---|---|---|---|
| Ana | yes | yes | Sure thing | 0 |
| Ben | no | yes | Persuadable | +1 |
| Chloé | yes | no | Do-not-disturb | −1 |
| Dev | no | no | Lost cause | 0 |
| Elif | no | no | Lost cause | 0 |
| Farid | yes | yes | Sure thing | 0 |
| Grace | no | yes | Persuadable | +1 |
| Hugo | no | no | Lost cause | 0 |
Add up the effects:
- persuadables add +1 each: 2 customers, so +2;
- do-not-disturbs add −1 each: 1 customer, so −1;
- sure things and lost causes add nothing.
The average effect is (2 − 1) ÷ 8 = +0.125: 12.5 points. The result is positive, but the persuadables pay for the do-not-disturbs. If the shop emailed only Ben and Grace, it would get +2 orders with 2 emails instead of +1 order with 8 emails.
That sentence is the idea of the whole module.
Why we cannot see the groups directly
Section titled “Why we cannot see the groups directly”Look at Ben again. A real test gave him the email or it did not. It never does both. So a real test cannot say “Ben is a persuadable”. If he got the email and ordered, he might be a persuadable or a sure thing. If he did not get it and ordered, he might be a sure thing or a do-not-disturb. Nothing in the log tells you which.
This is the same missing-world problem as in Causes and comparisons. The group of one person is never observed. Only groups of people have a measurable effect. So next, we will need models that predict the effect for a kind of customer. That is the next lesson.
The simulator is different. It simulates each customer in both worlds, with the same random numbers. So it can count the groups exactly.
Larkfield: the four groups in the test
Section titled “Larkfield: the four groups in the test”Of the 60,000 customers, only 1,567 order because of the email. 853 do not order because of it. The other 57,580 are not changed by the email at all. The true effect is (1,567 − 853) ÷ 60,000 = +1.19 points: the number that xp check prints.
The email is not good or bad. It depends on who gets it. Now look at who is in the two groups that matter.
Text description of the chart
A column chart with four columns. Among the 15,119 new customers: persuadables 7.46%, do-not-disturbs 0%. Among the 9,327 heavy buyers: persuadables 0%, do-not-disturbs 6.98%. These match the true effects of the segments: new customers +7.46 points, heavy buyers −6.98 points.
The chart explains the segment result of the test. New customers are almost all persuadable or indifferent. Heavy buyers have no persuadables, and many do-not-disturbs. So sending the email to heavy buyers can only lose orders.
Now run it yourself.
-
Create the email data again. It is the same as in Meet the simulator.
Terminal window sim run --scenario emailScenario email (seed 40): A reminder email to customers who have not ordered for 30 days: an A/B test by customer, 14 days.Wrote runs/email-s40: customers (60,000 rows), targets (50,000 rows), truth_targets (50,000 rows). Synthetic data. 0.5 s -
Analyse the test, with new customers as a segment.
Terminal window xp analyse --by new_customerRun email-s40 (synthetic): A reminder email to customers who have not ordered for 30 days: an A/B test by customer, 14 days.Metric: ordered. A: 30,208 customers, mean 0.0896. B: 29,792 customers, mean 0.1043.difference +0.0147 (+16.4%), 95% interval +0.0100 to +0.0194, p = 1.14e-09 (Welch t-test)new_customer = 0: difference -0.0064 (-6.7%), 95% interval -0.0117 to -0.0010, p = 0.0195 (Welch t-test)new_customer = 1: difference +0.0774 (+106.0%), 95% interval +0.0674 to +0.0873, p = 9.98e-52 (Welch t-test)Saved to runs/email-s40/analysis.json. Compare with the true effect: xp checkThe line
new_customer = 0is the returning customers. Their interval is below zero: the email lowers their orders. This is the segment “returning customers” in the first chart. -
Compare with the truth.
Terminal window xp checkRun email-s40: the last analysis was `xp analyse`.True effect (from the simulator): +0.0119 share of customers with an order.Your estimate: +0.0147, 95% interval +0.0100 to +0.0194: the interval contains the true effect. -
Read the four groups from the simulator’s truth.
Terminal window python -c "import json; print(json.load(open('runs/email-s40/truth.json'))['groups'])"{'persuadable': 1567, 'sure_thing': 4592, 'lost_cause': 52988, 'do_not_disturb': 853}Only the simulator’s truth file has these counts. A real test does not.
When to use it, when not, and what it costs
Section titled “When to use it, when not, and what it costs”Think about the four groups when:
- the treatment costs money or attention: an email, a coupon, a call, a free gift;
- the treatment can annoy some people, or teach them to wait for a discount;
- you have a randomised test. Without one, you cannot tell who the email changed, and you are back to the logs problems of Regression adjustment.
Do not use this idea when:
- the change reaches everyone, such as a new checkout page, and you cannot choose who gets it;
- your test has too few people to split into segments. A small segment has a wide interval, as you saw in Segments and different effects;
- you choose segments after you see the result. Choose them first, or you will find a “winner” by chance. See Many metrics and many tests.
It costs one thing: a test that is large enough for the groups you care about. Before you spend a budget, you must know who you can reach, and who you can lose. The next lesson finds those people automatically.
Guided practice: split the average yourself
Section titled “Guided practice: split the average yourself”- Open the notebook from the box Do this lesson in a notebook at the top of this page.
- Run the setup cells and the helper cell.
- In section 1, compute the average effect. You see
everyone: +0.0147 (± 0.0047). - In section 2, predict first: will the email help heavy buyers more or less than new customers? Then run the cell. You see new customers
+0.0774and heavy buyers-0.0626. - In section 3, count the four groups. You see persuadables
1567 (2.6%)and do-not-disturbs853 (1.4%). - In section 4, compute the effect among customers with
email_open_rateof 0.5 or more. Put it inopen_effect. Run the check cell. It printsCheck passed.
Show the answer to step 6
open_effect = round(float(effect(email[email.email_open_rate >= 0.5])[0]), 4)print(open_effect)The value is +0.0352: customers who open emails respond more than the average. The simulator’s truth for this group is +0.0416.
Your turn: find a group that the email harms
Section titled “Your turn: find a group that the email harms”Rafael must hear about the customers who lose. Show him one group where the email lowers orders. Use the numbers from this lesson, or from your own run of the notebook. Write two or three lines with the segment, the estimate, the 95% interval, and the truth.
Show one answer
Heavy buyers (7 or more orders in 12 months): the email changes the share who ordered by −6.26 points, with a 95% interval from −7.44 to −5.08. The interval is far below zero. The simulator’s truth for this group is −6.98 points. The email does not help them, and it pushes some of them to unsubscribe.
A second correct answer: returning customers, −0.64 points (−1.17 to −0.10), from xp analyse --by new_customer; truth −0.92 points. The effect is much smaller and the interval is close to zero.
Common mistake
“The average effect is positive, so email everyone”
Symptom: the team sends the email to the full list after a good test. Orders rise less than expected, and unsubscribes rise, mostly among the best customers.
Cause: an average is a mix. It adds the persuadables and takes away the do-not-disturbs. A positive average only says that the first group is bigger than the second. It says nothing about who is who.
Fix: before the next send, look at planned segments: new and returning customers, and heavy buyers. Keep a guardrail on unsubscribes. Then use an uplift model to choose who gets the email. The next two lessons show how.
Check your understanding
A customer did not order without the email and did order with it, in the simulator's two worlds. Which of the four groups is this customer in? Would a real test be able to tell?
Show the answer
The customer is a persuadable: the email made the order happen. A real test could not tell, because it shows only one of the two worlds for each customer. If this customer got the email and ordered, they could be a persuadable or a sure thing. Only a simulator, which runs both worlds, can name the group of one person.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
The simulated test has 1,567 persuadables and 853 do-not-disturbs among 60,000 customers. What is the true effect in points, and why does an email to everyone lose some orders?
Show the answer
The true effect is (1,567 − 853) ÷ 60,000 = +1.19 points. The email to everyone loses orders from the do-not-disturbs: 853 customers who would have ordered without the email do not order with it. The average is positive only because persuadables outnumber them.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
Rafael reports: the email raised orders by 1.47 points, so it should go to every customer on the list. Using the segments in this lesson, what would you tell him?
Show the answer
The average hides very different people. The email adds about +7.74 points among new customers but removes about 6.26 points among heavy buyers. Emailing everyone gives up the orders of heavy buyers who would have ordered anyway, and it raises unsubscribes. He should choose who gets the email, not send it to everyone. He should keep a guardrail on unsubscribes. A model can help him find the persuadables.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Recap
- An average effect can hide large and opposite effects in different people.
- In the simulated email test, the average is +1.47 points (truth +1.19). New customers: +7.74 (truth +7.46). Heavy buyers: −6.26 (truth −6.98).
- Each customer is one of four groups: persuadables (1,567), sure things (4,592), lost causes (52,988) and do-not-disturbs (853).
- Uplift is the change the treatment makes for a customer. Only persuadables (positive) and do-not-disturbs (negative) have one.
- A real test cannot see one customer’s group. Only the simulator can. So we need a model of the effect by kind of customer.
Next: train uplift models on the randomised test, and compare them with a normal “who will buy” model.
Next: Uplift models