Skip to content
Français

Uplift models

  • Lesson 25
  • 40 min
  • Module 7: Uplift: who should get the treatment?

Where we are

We know that the email helps some customers and annoys others. We cannot see the groups directly. In this lesson, we train uplift models, the T-learner and the X-learner, on the randomised email test, and compare them with a normal "who will buy" model.

Plays from YouTube. Nothing loads from YouTube until you press play. Privacy

What you will be able to do

Give each customer a score for how much the email changes their chance to order. Train two uplift models on the randomised email test: the T-learner and the X-learner. Compare them with a normal “who will buy” model, and see why that model picks the wrong people.

NotebookDo this lesson in a notebookColab · Kaggle · GitHub

In the notebook, you build the “who will buy” model, the T-learner and the X-learner by hand with scikit-learn. Then you look at the customers at the top of each list. Its setup cells install the packages and download the email data, so it needs no local setup. About 40 minutes.

Colab and Kaggle open your own copy. They need a free Google or Kaggle account. Stuck? See the solution notebook.

Before you startWhat to know and to have readyAbout 40 minutes

What you need to know

What you need on your computer (only for the project steps)

  • The project larkfield-experiments at the snapshot end-of-m06, from Regression adjustment, with .venv active. This lesson updates it to end-of-m07.

Tested on macOS with Python 3.12 and the end-of-m07 snapshot of the project. Windows was not tested.

A bookshop has 100 coupons for a new novel. The owner asks: “Who is most likely to buy it?” She sends the coupons to her 100 best customers. Most of them buy the novel.

Was it a success? Her best customers buy every new novel. They would have bought this one without a coupon. She gave them a discount for nothing. The customers whom a coupon would have changed were never asked.

She asked the wrong question. “Who will buy?” is not “Who will buy because of the coupon?” The second question is the one that matters for a budget.

The problem: a “who will buy” model emails the wrong people

Section titled “The problem: a “who will buy” model emails the wrong people”

Rafael can send 10,000 emails next month. The list has 50,000 customers. Which 10,000 should he choose?

A data scientist on his team trains a normal model. It learns from the customers who got the email in the test (group B), and it predicts each customer’s chance to order when emailed. Then it takes the 10,000 customers with the highest chance. This is a response model: we will call it the “who will buy” model.

The simulator knows each customer’s true chances with and without the email. So it can score the list.

  • The “who will buy” list brings +672 extra orders.
  • A random list of 10,000 brings only +130. So the model is much better than chance.
  • But the best possible 10,000 bring +943.
  • And 1,193 of its customers order less because of the email: they are do-not-disturbs.

The model is not useless. It is just aimed at a different target. It finds people who will order, and many of them would order anyway. Some of them are the heavy buyers whom the email annoys.

The idea: model the change, not the chance

Section titled “The idea: model the change, not the chance”

An uplift model predicts, for each customer, the change that the email makes:

uplift = (chance to order with the email) − (chance to order without the email)

A high uplift is a persuadable. An uplift of zero is a sure thing or a lost cause. A negative uplift is a do-not-disturb.

No customer has both results, so no model can learn uplift directly from one row. But the random test has two groups that are alike in everything except the email. So we can learn the chance with the email from group B, and the chance without it from group A. A coin chose the groups, so the difference is fair for every kind of customer. Without a random test, the models would also learn the confounders of the logs.

There are many ways to build an uplift model. We use two: the T-learner and the X-learner.

Top: the T-learner. Group A, no email, feeds model A, the chance to order without the email. Group B, with the email, feeds model B, the chance to order with the email. Blue arrows join them: the uplift of a customer is model B minus model A. Bottom: the X-learner in four steps. Step 1, fit model A and model B as above. Step 2, guess each customer's effect: in B, the result minus model A; in A, model B minus the result. Step 3, train two effect models on these guesses, one for B and one for A. Step 4, average them, with weights from the share in each group.
Two uplift models. The T-learner subtracts two models of the chance to order. The X-learner uses them to guess each customer's effect, and then learns the effect itself.

The T-learner (“T” for two) trains two models. Model A predicts the chance to order without the email. Model B predicts it with the email. The uplift is model B minus model A.

Take a tiny shop with two kinds of customers and a random test. The numbers are made up for this example.

Kind of customer Ordered, with the email (B) Ordered, without it (A) T-learner uplift (B − A)
New 16% 8% +8 points
Heavy buyer 28% 34% −6 points

Here, each “model” is just the share who ordered in that kind. The T-learner says: email new customers (+8 points). Do not email heavy buyers (−6 points).

Now ask the “who will buy” model. It looks only at the B column. Heavy buyers have the higher chance: 28% against 16%. So it emails heavy buyers first. Those are exactly the customers it should leave alone.

The T-learner has one weakness. Its two models make their own errors, and the uplift is the small difference between two big numbers. The errors do not cancel. So the uplift can be noisy.

The X-learner adds a second step. It uses the first two models to guess each customer’s own effect. Then it trains models on those guesses directly.

  • A new customer in B who ordered: the guess is 1 − 0.08 = +0.92. (Model A says that a new customer orders 8% of the time without the email.)
  • A new customer in B who did not order: the guess is 0 − 0.08 = −0.08.
  • A new customer in A who did not order: the guess is 0.16 − 0 = +0.16. (Model B says 16% with the email.)
  • A new customer in A who ordered: the guess is 0.16 − 1 = −0.84.

One guess is very wrong for one person. But the average of the guesses is right. In B, 16% of new customers ordered: 0.16 × 0.92 + 0.84 × (−0.08) = +0.08. In A, 8% ordered: 0.08 × (−0.84) + 0.92 × 0.16 = +0.08. Both give the +8 points of the T-learner.

So on this tiny shop, both learners agree. On real data with many columns, they differ. The X-learner’s second models learn the effect itself, which is often simpler than the chance to order. Its last step averages its two effect models. It gives more weight to the guesses that use the model of the larger group, because that model is more accurate. That helps most when one group is much smaller, for example when only 10% of customers got the email.

Each “model” here is a gradient-boosted tree model from scikit-learn: many small decision trees, each one fixing the errors of the ones before. It reads seven columns: new or returning, tenure, orders and spending in 12 months, days since the last order, the email open rate, and mobile or desktop. You do not need to know how it works inside. For the idea, see Ensembles in the machine learning course. The T-learner and the X-learner come from EconML, an open-source library from Microsoft (MIT licence). The notebook writes both learners by hand in a few lines, and gets the same scores.

First, update the project to the snapshot end-of-m07. It adds xp uplift and EconML. Your own runs and plans stay.

  1. Open a terminal in the folder that holds larkfield-experiments and nextia-learning-labs. If you cloned the labs repository with Git, update it first.

    Terminal window
    git -C nextia-learning-labs pull
  2. Copy the snapshot over your project, and install the new packages.

    Terminal window
    cp -R nextia-learning-labs/C40/snapshots/end-of-m07/. larkfield-experiments/
    cd larkfield-experiments
    source .venv/bin/activate
    python -m pip install -r requirements.txt

    EconML brings a few other packages, so the install can take a few minutes.

  3. Run the tests.

    Terminal window
    python -m pytest -q
    ............................ [100%]
    31 passed in 7.20s

    Thirty-one tests pass. Your time will differ.

  4. Create the email data, if you have not done so in the last lesson.

    Terminal window
    sim run --scenario email
  5. Train the “who will buy” model, choose 10,000 customers, and check the list against the simulator’s truth.

    Terminal window
    xp uplift --learner response
    xp check
    'who will buy' model trained on 42,157 customers of the test; evaluated on the other 17,843.
    Extra orders in the evaluation group if we email the top share by score (uplift curve):
    top 10%: +147.2
    top 20%: +212.9
    top 30%: +253.2
    top 40%: +273.9
    top 50%: +280.2
    top 60%: +277.9
    top 70%: +282.4
    top 80%: +264.6
    top 90%: +227.7
    top 100%: +189.0
    Qini coefficient: +138.92 (0 = no better than random)
    Chose 10,000 of next month's 50,000 customers: runs/email-s40/targets_response.csv. Check them: xp check
    Run email-s40: the last analysis was `xp uplift`.
    Your 10,000 customers: +672 extra orders expected (from the simulator's true chances).
    Random 10,000: +130. The best possible 10,000: +943. Emailing all 50,000: +648.
    1,193 of your customers order less because of the email (do-not-disturbs).
  6. Do the same with the T-learner and the X-learner.

    Terminal window
    xp uplift --learner t
    xp check
    xp uplift --learner x
    xp check

    We show only the last lines of each xp check:

    Your 10,000 customers: +795 extra orders expected (from the simulator's true chances).
    Random 10,000: +130. The best possible 10,000: +943. Emailing all 50,000: +648.
    215 of your customers order less because of the email (do-not-disturbs).
    Your 10,000 customers: +838 extra orders expected (from the simulator's true chances).
    Random 10,000: +130. The best possible 10,000: +943. Emailing all 50,000: +648.
    24 of your customers order less because of the email (do-not-disturbs).

xp uplift trains on 70% of the test and keeps 30% to evaluate the model. The evaluation lines (the uplift curve and the Qini coefficient) are the subject of the next lesson. Then it scores next month’s list of 50,000 customers and writes the top 10,000 to a file. xp check compares that file with the simulator’s truth. A real shop cannot run xp check. Here, it shows how good each list really is.

The T-learner’s list brings +795 extra orders, and the X-learner’s +838. The “who will buy” list brings +672. The best possible is +943.

Look at the do-not-disturbs: 1,193 in the “who will buy” list, 215 for the T-learner, and only 24 for the X-learner.

Of each list of 10,000 customers, how many order less because of the email (the simulator's true chances; simulated, seed 40). A random list has 4,298, because 43% of the whole list are harmed a little. The uplift models avoid them.
Text description of the chart

A column chart. In each list of 10,000 customers from next month’s 50,000, the number whose true chance to order goes down with the email. Random list: 4,298. “Who will buy” model: 1,193. T-learner: 215. X-learner: 24.

A random list includes 4,298 harmed customers. That sounds like a lot. Most of them are harmed only a little, because 21,573 of the 50,000 customers have a slightly negative uplift. The uplift models leave them out almost completely.

The table describes the three lists of 10,000, from the simulator’s view of next month’s customers.

List of 10,000 New customers Heavy buyers (7+ orders) Chance to order without the email True uplift per customer
Random 25% 15% 8.8% +1.3 points
“Who will buy” 57% 2.9% 11.4% +6.7 points
T-learner 81% 0.35% 8.9% +8.0 points
X-learner 88% 0.01% 8.5% +8.4 points

The “who will buy” list has the highest chance to order without the email: 11.4%. That is the bookshop’s mistake. It is full of sure things. The X-learner’s list is almost all new customers, and it has almost no heavy buyers. It is the list of the persuadables.

Cut each list of 50,000 into ten bands of 5,000 customers, from the highest score to the lowest. Then ask the simulator for the true uplift in each band.

  • X-learner
  • T-learner
  • "Who will buy"
The true uplift of next month's customers, in ten bands by each model's score (simulated, seed 40). A good uplift model starts high and ends low. The X-learner's top band gains 10.0 points, and its bottom band loses 7.0 points. The “who will buy” model sorts less sharply.
Text description of the chart

A line chart with ten bands on the horizontal axis and the true uplift in points on the vertical axis. X-learner, from the highest band to the lowest: +10.03, +6.73, +4.48, +1.69, +0.54, +0.12, −0.15, −0.51, −2.96, −7.01. T-learner: +9.47, +6.42, +3.94, +2.04, +0.71, +0.06, −0.22, −0.76, −2.70, −6.00. “Who will buy” model: +8.51, +4.94, +3.42, +1.79, +0.68, +0.12, −0.59, −1.47, −1.64, −2.79.

All three lines go down, so all three sort the customers in the right order on the whole. The X-learner sorts the most sharply. Its last band, −7.01 points, is the heavy buyers: the people the email must not reach.

One simulated test can be lucky. The course’s measurement repeated the whole process on 10 simulated tests, each with its own customers. On average, 10,000 emails brought +699 extra orders with the “who will buy” model, +785 with the T-learner, and +823 with the X-learner. The best possible was +936, and random +129. So the order of the three models holds. The X-learner’s lead over the T-learner is modest: about 40 orders.

When to use it, when not, and what it costs

Section titled “When to use it, when not, and what it costs”

Use an uplift model when:

  • you have a randomised test of the treatment, with enough customers: tens of thousands for a small effect like this one;
  • the treatment has a cost or a budget, and you must choose who gets it;
  • the effect differs between customers, and you have columns that describe them.

Do not use it when:

  • you have only logs, with no random test. The model would learn the confounders as if they were the effect;
  • the test is small. Uplift is a small difference between two noisy chances. With a few thousand customers, the scores are mostly noise;
  • the effect is about the same for everyone. Then a simple rule, “email everyone” or “email nobody”, is as good.

It costs a test large enough to learn from. The model needs care, like any model. It also needs a check on data it did not see, which the next lesson covers. The T-learner is the simplest to explain. The X-learner did a little better here, and the extra work is small.

Guided practice: build the learners by hand

Section titled “Guided practice: build the learners by hand”
  1. Open the notebook from the box Do this lesson in a notebook at the top of this page.
  2. Run the setup cells and the helper cell.
  3. In section 1, split the test. You see train: 42,157 customers; evaluate: 17,843 customers.
  4. In section 2, predict which customers the “who will buy” model puts at the top. Then run it. Among its top 20%, you see 58% new customers and 3% heavy buyers. The email’s effect there is +0.0597.
  5. In section 3, build the T-learner. Its top 20%: 81% new customers, effect +0.0771.
  6. In section 4, build the X-learner. Its top 20%: 89% new customers, effect +0.0781.
  7. In section 5, find the share of heavy buyers in the T-learner’s top 20%. Put it in t_heavy_share. Run the check cell. It prints Check passed.
Show the answer to step 7
top_t = hold.iloc[np.argsort(-score_t)[: len(hold) // 5]]
t_heavy_share = round(float((top_t.orders_12m >= 7).mean()), 4)
print(t_heavy_share)

The value is 0.0039: fewer than 1 customer in 200. The T-learner learned to leave heavy buyers out.

xp uplift splits the test at random into training and evaluation. Its --seed option changes the split. Before you run anything, predict: with --seed 1, will the X-learner’s list bring about the same extra orders?

Show the answer
Terminal window
xp uplift --learner x --seed 1
xp check

We show the Qini line and the check:

Qini coefficient: +233.75 (0 = no better than random)
Chose 10,000 of next month's 50,000 customers: runs/email-s40/targets_x.csv. Check them: xp check
Run email-s40: the last analysis was `xp uplift`.
Your 10,000 customers: +810 extra orders expected (from the simulator's true chances).
Random 10,000: +130. The best possible 10,000: +943. Emailing all 50,000: +648.
43 of your customers order less because of the email (do-not-disturbs).

About the same: +810 extra orders against +838, and 43 harmed customers against 24. Another training sample gives a slightly different model.

Look at the Qini coefficient too: +233.75 against +218.92. The list is a little worse, but its score is higher. The score is measured on a different evaluation group, and that group has its own luck. Keep this in mind for the next lesson. Run xp uplift --learner x again before you continue, to get back the default list.

Common mistake

“Email the customers most likely to buy”

Symptom: the target list comes from a model of who will order. The campaign report shows a high order rate in the emailed group, and the team is pleased. A later test with a held-out group shows a much smaller gain than expected, and more unsubscribes among loyal customers.

Cause: a model of the chance to order finds sure things: customers who would order anyway. Their high order rate is not caused by the email. The model also picks some heavy buyers whom the email annoys.

Fix: train an uplift model on a randomised test, so that it predicts the change the email makes. Report extra orders against a random hold-out group, not the order rate of the emailed customers.

Check your understanding

In a random test, new customers order 16% of the time with the email and 8% without it. Heavy buyers order 28% with it and 34% without it. What uplift does a T-learner give each kind, and which kind does a 'who will buy' model email first?

Show the answer

The T-learner gives new customers 16 − 8 = +8 points, and heavy buyers 28 − 34 = −6 points. So it emails new customers and leaves heavy buyers alone. The “who will buy” model looks only at the chance with the email: 28% for heavy buyers against 16% for new customers. It emails heavy buyers first, who are exactly the people the email harms.

Check your understanding

Why must an uplift model be trained on a randomised test, and not on the logs of past emails that the marketing team chose to send?

Show the answer

An uplift model compares the chance to order with and without the email for each kind of customer. That comparison is fair only if the two groups differ in nothing but the email. A coin makes them alike. In past campaigns, the team chose who got the email, for example the best customers. The model would then learn that difference between the groups as if it were the effect of the email: the confounder problem of observational data.

Check your understanding

The 'who will buy' list brings +672 extra orders and the X-learner's list +838, from 10,000 emails. A random list brings +130. Is the 'who will buy' model useless? What is wrong with it?

Show the answer

It is not useless: +672 is about five times what a random list gives. Customers who are likely to order are, on average, also a little more likely to respond. But it aims at the wrong target. It ranks by the chance to order, so it fills the list with sure things. It also includes 1,193 customers whom the email harms (the X-learner: 24). It leaves about 170 extra orders behind, compared with the X-learner.

Recap

  • A “who will buy” model predicts the chance to order. An uplift model predicts the change that the email makes.
  • The T-learner trains one model per group and subtracts them. The X-learner guesses each customer’s effect, then learns the effect itself.
  • Both need a randomised test. A coin makes the two groups alike, so the difference is the email’s effect.
  • From 10,000 emails (simulated, checked against the truth): “who will buy” +672, T-learner +795, X-learner +838, best possible +943, random +130.
  • The “who will buy” list is full of sure things and includes 1,193 customers whom the email harms. The X-learner’s list has 24.

Next: evaluate the lists without the simulator, and choose how many customers to email.

Next: Evaluate and use uplift