Uplift models
- Lesson 25
- 40 min
- Module 7: Uplift: who should get the treatment?
Where we are
We know that the email helps some customers and annoys others. We cannot see the groups directly. In this lesson, we train uplift models, the T-learner and the X-learner, on the randomised email test, and compare them with a normal "who will buy" model.
Lesson 25 of 30 · Experiments and Causal Impact: Did Our Change Work?Uplift models
1. Why "it went up" is not an answer
2. Design an A/B test
3. Analyse the test
4. Traps in real tests
5. When you cannot randomise: natural experiments
6. Adjust for confounders in observational data
7. Uplift: who should get the treatment?
8. Experiments for AI features and in production
What you will be able to do
Give each customer a score for how much the email changes their chance to order. Train two uplift models on the randomised email test: the T-learner and the X-learner. Compare them with a normal “who will buy” model, and see why that model picks the wrong people.
NotebookDo this lesson in a notebook
In the notebook, you build the “who will buy” model, the T-learner and the X-learner by hand with scikit-learn. Then you look at the customers at the top of each list. Its setup cells install the packages and download the email data, so it needs no local setup. About 40 minutes.
Colab and Kaggle open your own copy. They need a free Google or Kaggle account. Stuck? See the solution notebook.
Before you startWhat to know and to have ready
What you need to know
- The four groups and the word uplift, from The average hides the people.
- What a model that predicts a chance is, and why we test it on data it did not see. For a refresher, see Train, validation and test sets and Ensembles in the machine learning course. This lesson explains what it needs.
What you need on your computer (only for the project steps)
- The project
larkfield-experimentsat the snapshotend-of-m06, from Regression adjustment, with.venvactive. This lesson updates it toend-of-m07.
Tested on macOS with Python 3.12 and the end-of-m07 snapshot of the project. Windows was not tested.
A bookshop and its best customers
Section titled “A bookshop and its best customers”A bookshop has 100 coupons for a new novel. The owner asks: “Who is most likely to buy it?” She sends the coupons to her 100 best customers. Most of them buy the novel.
Was it a success? Her best customers buy every new novel. They would have bought this one without a coupon. She gave them a discount for nothing. The customers whom a coupon would have changed were never asked.
She asked the wrong question. “Who will buy?” is not “Who will buy because of the coupon?” The second question is the one that matters for a budget.
The problem: a “who will buy” model emails the wrong people
Section titled “The problem: a “who will buy” model emails the wrong people”Rafael can send 10,000 emails next month. The list has 50,000 customers. Which 10,000 should he choose?
A data scientist on his team trains a normal model. It learns from the customers who got the email in the test (group B), and it predicts each customer’s chance to order when emailed. Then it takes the 10,000 customers with the highest chance. This is a response model: we will call it the “who will buy” model.
The simulator knows each customer’s true chances with and without the email. So it can score the list.
- The “who will buy” list brings +672 extra orders.
- A random list of 10,000 brings only +130. So the model is much better than chance.
- But the best possible 10,000 bring +943.
- And 1,193 of its customers order less because of the email: they are do-not-disturbs.
The model is not useless. It is just aimed at a different target. It finds people who will order, and many of them would order anyway. Some of them are the heavy buyers whom the email annoys.
The idea: model the change, not the chance
Section titled “The idea: model the change, not the chance”An uplift model predicts, for each customer, the change that the email makes:
uplift = (chance to order with the email) − (chance to order without the email)A high uplift is a persuadable. An uplift of zero is a sure thing or a lost cause. A negative uplift is a do-not-disturb.
No customer has both results, so no model can learn uplift directly from one row. But the random test has two groups that are alike in everything except the email. So we can learn the chance with the email from group B, and the chance without it from group A. A coin chose the groups, so the difference is fair for every kind of customer. Without a random test, the models would also learn the confounders of the logs.
There are many ways to build an uplift model. We use two: the T-learner and the X-learner.
The T-learner, on a tiny example
Section titled “The T-learner, on a tiny example”The T-learner (“T” for two) trains two models. Model A predicts the chance to order without the email. Model B predicts it with the email. The uplift is model B minus model A.
Take a tiny shop with two kinds of customers and a random test. The numbers are made up for this example.
| Kind of customer | Ordered, with the email (B) | Ordered, without it (A) | T-learner uplift (B − A) |
|---|---|---|---|
| New | 16% | 8% | +8 points |
| Heavy buyer | 28% | 34% | −6 points |
Here, each “model” is just the share who ordered in that kind. The T-learner says: email new customers (+8 points). Do not email heavy buyers (−6 points).
Now ask the “who will buy” model. It looks only at the B column. Heavy buyers have the higher chance: 28% against 16%. So it emails heavy buyers first. Those are exactly the customers it should leave alone.
The X-learner, on the same example
Section titled “The X-learner, on the same example”The T-learner has one weakness. Its two models make their own errors, and the uplift is the small difference between two big numbers. The errors do not cancel. So the uplift can be noisy.
The X-learner adds a second step. It uses the first two models to guess each customer’s own effect. Then it trains models on those guesses directly.
- A new customer in B who ordered: the guess is 1 − 0.08 = +0.92. (Model A says that a new customer orders 8% of the time without the email.)
- A new customer in B who did not order: the guess is 0 − 0.08 = −0.08.
- A new customer in A who did not order: the guess is 0.16 − 0 = +0.16. (Model B says 16% with the email.)
- A new customer in A who ordered: the guess is 0.16 − 1 = −0.84.
One guess is very wrong for one person. But the average of the guesses is right. In B, 16% of new customers ordered: 0.16 × 0.92 + 0.84 × (−0.08) = +0.08. In A, 8% ordered: 0.08 × (−0.84) + 0.92 × 0.16 = +0.08. Both give the +8 points of the T-learner.
So on this tiny shop, both learners agree. On real data with many columns, they differ. The X-learner’s second models learn the effect itself, which is often simpler than the chance to order. Its last step averages its two effect models. It gives more weight to the guesses that use the model of the larger group, because that model is more accurate. That helps most when one group is much smaller, for example when only 10% of customers got the email.
What the models are
Section titled “What the models are”Each “model” here is a gradient-boosted tree model from scikit-learn: many small decision trees, each one fixing the errors of the ones before. It reads seven columns: new or returning, tenure, orders and spending in 12 months, days since the last order, the email open rate, and mobile or desktop. You do not need to know how it works inside. For the idea, see Ensembles in the machine learning course. The T-learner and the X-learner come from EconML, an open-source library from Microsoft (MIT licence). The notebook writes both learners by hand in a few lines, and gets the same scores.
Larkfield: three lists of 10,000
Section titled “Larkfield: three lists of 10,000”First, update the project to the snapshot end-of-m07. It adds xp uplift and EconML. Your own runs and plans stay.
-
Open a terminal in the folder that holds
larkfield-experimentsandnextia-learning-labs. If you cloned the labs repository with Git, update it first.Terminal window git -C nextia-learning-labs pull -
Copy the snapshot over your project, and install the new packages.
Terminal window cp -R nextia-learning-labs/C40/snapshots/end-of-m07/. larkfield-experiments/cd larkfield-experimentssource .venv/bin/activatepython -m pip install -r requirements.txtTerminal window Copy-Item -Recurse -Force nextia-learning-labs\C40\snapshots\end-of-m07\* larkfield-experimentscd larkfield-experiments.venv\Scripts\Activate.ps1python -m pip install -r requirements.txtEconML brings a few other packages, so the install can take a few minutes.
-
Run the tests.
Terminal window python -m pytest -q............................ [100%]31 passed in 7.20sThirty-one tests pass. Your time will differ.
-
Create the email data, if you have not done so in the last lesson.
Terminal window sim run --scenario email -
Train the “who will buy” model, choose 10,000 customers, and check the list against the simulator’s truth.
Terminal window xp uplift --learner responsexp check'who will buy' model trained on 42,157 customers of the test; evaluated on the other 17,843.Extra orders in the evaluation group if we email the top share by score (uplift curve):top 10%: +147.2top 20%: +212.9top 30%: +253.2top 40%: +273.9top 50%: +280.2top 60%: +277.9top 70%: +282.4top 80%: +264.6top 90%: +227.7top 100%: +189.0Qini coefficient: +138.92 (0 = no better than random)Chose 10,000 of next month's 50,000 customers: runs/email-s40/targets_response.csv. Check them: xp checkRun email-s40: the last analysis was `xp uplift`.Your 10,000 customers: +672 extra orders expected (from the simulator's true chances).Random 10,000: +130. The best possible 10,000: +943. Emailing all 50,000: +648.1,193 of your customers order less because of the email (do-not-disturbs). -
Do the same with the T-learner and the X-learner.
Terminal window xp uplift --learner txp checkxp uplift --learner xxp checkWe show only the last lines of each
xp check:Your 10,000 customers: +795 extra orders expected (from the simulator's true chances).Random 10,000: +130. The best possible 10,000: +943. Emailing all 50,000: +648.215 of your customers order less because of the email (do-not-disturbs).Your 10,000 customers: +838 extra orders expected (from the simulator's true chances).Random 10,000: +130. The best possible 10,000: +943. Emailing all 50,000: +648.24 of your customers order less because of the email (do-not-disturbs).
xp uplift trains on 70% of the test and keeps 30% to evaluate the model. The evaluation lines (the uplift curve and the Qini coefficient) are the subject of the next lesson. Then it scores next month’s list of 50,000 customers and writes the top 10,000 to a file. xp check compares that file with the simulator’s truth. A real shop cannot run xp check. Here, it shows how good each list really is.
What the three lists bring
Section titled “What the three lists bring”The T-learner’s list brings +795 extra orders, and the X-learner’s +838. The “who will buy” list brings +672. The best possible is +943.
Look at the do-not-disturbs: 1,193 in the “who will buy” list, 215 for the T-learner, and only 24 for the X-learner.
Text description of the chart
A column chart. In each list of 10,000 customers from next month’s 50,000, the number whose true chance to order goes down with the email. Random list: 4,298. “Who will buy” model: 1,193. T-learner: 215. X-learner: 24.
A random list includes 4,298 harmed customers. That sounds like a lot. Most of them are harmed only a little, because 21,573 of the 50,000 customers have a slightly negative uplift. The uplift models leave them out almost completely.
Who is at the top of each list
Section titled “Who is at the top of each list”The table describes the three lists of 10,000, from the simulator’s view of next month’s customers.
| List of 10,000 | New customers | Heavy buyers (7+ orders) | Chance to order without the email | True uplift per customer |
|---|---|---|---|---|
| Random | 25% | 15% | 8.8% | +1.3 points |
| “Who will buy” | 57% | 2.9% | 11.4% | +6.7 points |
| T-learner | 81% | 0.35% | 8.9% | +8.0 points |
| X-learner | 88% | 0.01% | 8.5% | +8.4 points |
The “who will buy” list has the highest chance to order without the email: 11.4%. That is the bookshop’s mistake. It is full of sure things. The X-learner’s list is almost all new customers, and it has almost no heavy buyers. It is the list of the persuadables.
Does the score sort the customers well?
Section titled “Does the score sort the customers well?”Cut each list of 50,000 into ten bands of 5,000 customers, from the highest score to the lowest. Then ask the simulator for the true uplift in each band.
- X-learner
- T-learner
- "Who will buy"
Text description of the chart
A line chart with ten bands on the horizontal axis and the true uplift in points on the vertical axis. X-learner, from the highest band to the lowest: +10.03, +6.73, +4.48, +1.69, +0.54, +0.12, −0.15, −0.51, −2.96, −7.01. T-learner: +9.47, +6.42, +3.94, +2.04, +0.71, +0.06, −0.22, −0.76, −2.70, −6.00. “Who will buy” model: +8.51, +4.94, +3.42, +1.79, +0.68, +0.12, −0.59, −1.47, −1.64, −2.79.
All three lines go down, so all three sort the customers in the right order on the whole. The X-learner sorts the most sharply. Its last band, −7.01 points, is the heavy buyers: the people the email must not reach.
Over more seeds
Section titled “Over more seeds”One simulated test can be lucky. The course’s measurement repeated the whole process on 10 simulated tests, each with its own customers. On average, 10,000 emails brought +699 extra orders with the “who will buy” model, +785 with the T-learner, and +823 with the X-learner. The best possible was +936, and random +129. So the order of the three models holds. The X-learner’s lead over the T-learner is modest: about 40 orders.
When to use it, when not, and what it costs
Section titled “When to use it, when not, and what it costs”Use an uplift model when:
- you have a randomised test of the treatment, with enough customers: tens of thousands for a small effect like this one;
- the treatment has a cost or a budget, and you must choose who gets it;
- the effect differs between customers, and you have columns that describe them.
Do not use it when:
- you have only logs, with no random test. The model would learn the confounders as if they were the effect;
- the test is small. Uplift is a small difference between two noisy chances. With a few thousand customers, the scores are mostly noise;
- the effect is about the same for everyone. Then a simple rule, “email everyone” or “email nobody”, is as good.
It costs a test large enough to learn from. The model needs care, like any model. It also needs a check on data it did not see, which the next lesson covers. The T-learner is the simplest to explain. The X-learner did a little better here, and the extra work is small.
Guided practice: build the learners by hand
Section titled “Guided practice: build the learners by hand”- Open the notebook from the box Do this lesson in a notebook at the top of this page.
- Run the setup cells and the helper cell.
- In section 1, split the test. You see
train: 42,157 customers; evaluate: 17,843 customers. - In section 2, predict which customers the “who will buy” model puts at the top. Then run it. Among its top 20%, you see 58% new customers and 3% heavy buyers. The email’s effect there is +0.0597.
- In section 3, build the T-learner. Its top 20%: 81% new customers, effect +0.0771.
- In section 4, build the X-learner. Its top 20%: 89% new customers, effect +0.0781.
- In section 5, find the share of heavy buyers in the T-learner’s top 20%. Put it in
t_heavy_share. Run the check cell. It printsCheck passed.
Show the answer to step 7
top_t = hold.iloc[np.argsort(-score_t)[: len(hold) // 5]]t_heavy_share = round(float((top_t.orders_12m >= 7).mean()), 4)print(t_heavy_share)The value is 0.0039: fewer than 1 customer in 200. The T-learner learned to leave heavy buyers out.
Your turn: train on another split
Section titled “Your turn: train on another split”xp uplift splits the test at random into training and evaluation. Its --seed option changes the split. Before you run anything, predict: with --seed 1, will the X-learner’s list bring about the same extra orders?
Show the answer
xp uplift --learner x --seed 1xp checkWe show the Qini line and the check:
Qini coefficient: +233.75 (0 = no better than random) Chose 10,000 of next month's 50,000 customers: runs/email-s40/targets_x.csv. Check them: xp checkRun email-s40: the last analysis was `xp uplift`.Your 10,000 customers: +810 extra orders expected (from the simulator's true chances). Random 10,000: +130. The best possible 10,000: +943. Emailing all 50,000: +648. 43 of your customers order less because of the email (do-not-disturbs).About the same: +810 extra orders against +838, and 43 harmed customers against 24. Another training sample gives a slightly different model.
Look at the Qini coefficient too: +233.75 against +218.92. The list is a little worse, but its score is higher. The score is measured on a different evaluation group, and that group has its own luck. Keep this in mind for the next lesson. Run xp uplift --learner x again before you continue, to get back the default list.
Common mistake
“Email the customers most likely to buy”
Symptom: the target list comes from a model of who will order. The campaign report shows a high order rate in the emailed group, and the team is pleased. A later test with a held-out group shows a much smaller gain than expected, and more unsubscribes among loyal customers.
Cause: a model of the chance to order finds sure things: customers who would order anyway. Their high order rate is not caused by the email. The model also picks some heavy buyers whom the email annoys.
Fix: train an uplift model on a randomised test, so that it predicts the change the email makes. Report extra orders against a random hold-out group, not the order rate of the emailed customers.
Check your understanding
In a random test, new customers order 16% of the time with the email and 8% without it. Heavy buyers order 28% with it and 34% without it. What uplift does a T-learner give each kind, and which kind does a 'who will buy' model email first?
Show the answer
The T-learner gives new customers 16 − 8 = +8 points, and heavy buyers 28 − 34 = −6 points. So it emails new customers and leaves heavy buyers alone. The “who will buy” model looks only at the chance with the email: 28% for heavy buyers against 16% for new customers. It emails heavy buyers first, who are exactly the people the email harms.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
Why must an uplift model be trained on a randomised test, and not on the logs of past emails that the marketing team chose to send?
Show the answer
An uplift model compares the chance to order with and without the email for each kind of customer. That comparison is fair only if the two groups differ in nothing but the email. A coin makes them alike. In past campaigns, the team chose who got the email, for example the best customers. The model would then learn that difference between the groups as if it were the effect of the email: the confounder problem of observational data.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
The 'who will buy' list brings +672 extra orders and the X-learner's list +838, from 10,000 emails. A random list brings +130. Is the 'who will buy' model useless? What is wrong with it?
Show the answer
It is not useless: +672 is about five times what a random list gives. Customers who are likely to order are, on average, also a little more likely to respond. But it aims at the wrong target. It ranks by the chance to order, so it fills the list with sure things. It also includes 1,193 customers whom the email harms (the X-learner: 24). It leaves about 170 extra orders behind, compared with the X-learner.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Recap
- A “who will buy” model predicts the chance to order. An uplift model predicts the change that the email makes.
- The T-learner trains one model per group and subtracts them. The X-learner guesses each customer’s effect, then learns the effect itself.
- Both need a randomised test. A coin makes the two groups alike, so the difference is the email’s effect.
- From 10,000 emails (simulated, checked against the truth): “who will buy” +672, T-learner +795, X-learner +838, best possible +943, random +130.
- The “who will buy” list is full of sure things and includes 1,193 customers whom the email harms. The X-learner’s list has 24.
Next: evaluate the lists without the simulator, and choose how many customers to email.
Next: Evaluate and use uplift