Evaluate and use uplift
- Lesson 26
- 40 min
- Module 7: Uplift: who should get the treatment?
Where we are
Our uplift models give each customer a predicted extra chance to order. Rafael can send 10,000 emails. In this lesson, we evaluate the models with uplift and Qini curves, choose whom to email, and count the extra orders that the simulator confirms.
Lesson 26 of 30 · Experiments and Causal Impact: Did Our Change Work?Evaluate and use uplift
1. Why "it went up" is not an answer
2. Design an A/B test
3. Analyse the test
4. Traps in real tests
5. When you cannot randomise: natural experiments
6. Adjust for confounders in observational data
7. Uplift: who should get the treatment?
8. Experiments for AI features and in production
What you will be able to do
Evaluate uplift models on data they did not see, with uplift and Qini curves. Choose whom Rafael should email under a budget of 10,000 emails. Then choose how many to email by profit, not by accuracy.
NotebookDo this lesson in a notebook
In the notebook, you write the uplift curve and the Qini coefficient yourself, and draw the curves of three models. Then you find the list size with the highest profit. Its setup cells install the packages and download the email data, so it needs no local setup. About 40 minutes.
Colab and Kaggle open your own copy. They need a free Google or Kaggle account. Stuck? See the solution notebook.
Before you startWhat to know and to have ready
What you need to know
- The T-learner and the X-learner, from Uplift models.
- The four groups, from The average hides the people.
What you need on your computer (only for the project steps)
- The project
larkfield-experimentsat the snapshotend-of-m07, from Uplift models, with.venvactive.
Tested on macOS with Python 3.12 and the end-of-m07 snapshot of the project. Windows was not tested.
A flyer and a box of leftovers
Section titled “A flyer and a box of leftovers”A bakery prints flyers for a new bread. It has a list of streets, sorted by a guess of where the flyer works best. The baker cannot afford every street. How does she know that her sorted list is any good?
She keeps a few houses on each street without a flyer. On the best streets, houses with a flyer buy much more. On the last streets, they buy the same. Now she sees where the flyer stops paying for the paper.
That is this lesson: hold back a random group, measure the gain along the sorted list, and stop where the gain no longer pays.
The problem: accuracy points the wrong way
Section titled “The problem: accuracy points the wrong way”In the last lesson, the simulator told us how good each list was. A real shop has no simulator. It must judge a model from its data. The first idea is to use what we use for normal models: did the chosen customers order?
Take the 17,843 customers of the test that the models did not see in training. For each model, take its top 20%: 3,568 customers. Some of them got the email (B), some did not (A).
| Top 20% by | Ordered, with the email (B) | Ordered, without it (A) | The email’s effect |
|---|---|---|---|
| “Who will buy” model | 18.38% | 12.41% | +5.97 points |
| X-learner | 16.93% | 9.13% | +7.81 points |
If you look only at B, the “who will buy” model wins: 18.38% of its customers ordered. It is the more accurate model of who orders. But look at A. Its customers order 12.41% of the time without the email. The email adds less for them. The X-learner’s customers order less, but the email changes more of them.
So, accuracy points the wrong way. We need a measure of the change, and the change is visible only when we compare B with A.
The idea: the uplift curve
Section titled “The idea: the uplift curve”An uplift curve answers one question for every list size. If we email the top part of the list, how many extra orders do we get?
- Score the evaluation group with the model, and sort it, best first.
- Take the top 10%. In it, compare the share who ordered in B with the share in A. Multiply the difference by the number of customers. That is the extra orders from emailing those customers.
- Do the same for the top 20%, 30%, and so on, up to everyone.
A random test makes this fair: inside every slice, A and B are alike except for the email.
Then draw a straight line from 0 to the last point. It shows what a random list would give: the same small effect per customer, all along. The area between the curve and that line is the Qini coefficient. A higher Qini means the model puts more of the gain at the top. A Qini of 0 means no better than random. Books and libraries define the Qini curve in slightly different ways. Compare Qini values only from the same tool.
A tiny example with every value visible
Section titled “A tiny example with every value visible”Twelve customers in an evaluation group, sorted by a model’s score. The values are made up for this example. “B” got the email; “A” did not.
| Rank | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | 11 | 12 |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Group | B | B | A | A | B | B | A | A | B | B | A | A |
| Ordered? | yes | no | no | no | yes | yes | yes | no | no | no | yes | no |
- Top 4: B ordered 1 of 2 (0.50); A ordered 0 of 2 (0). Gain = (0.50 − 0) × 4 = +2.
- Top 8: B 3 of 4 (0.75); A 1 of 4 (0.25). Gain = (0.75 − 0.25) × 8 = +4.
- All 12: B 3 of 6 (0.50); A 2 of 6 (0.33). Gain = (0.50 − 0.33) × 12 = +2.
The random line goes from 0 to +2. At one third of the list, it is at 0.67. At two thirds, it is at 1.33.
- The model
- Random list
Text description of the chart
A line chart. The model’s curve: 0 at 0%, +2 at 33%, +4 at 67%, +2 at 100%. The random line goes straight from 0 at 0% to +2 at 100%.
The last four customers lose two orders: they act like do-not-disturbs. The best list here is the top two thirds, not everyone.
The Qini coefficient is the area between the two lines. The gaps are 0, 1.33, 2.67 and 0, at shares 0, 1/3, 2/3 and 1. We add three trapezoids. The area is 1/3 × (0 + 1.33)/2 + 1/3 × (1.33 + 2.67)/2 + 1/3 × (2.67 + 0)/2 = 1.33. The project’s qini_coefficient gives the same.
Larkfield: the curves of three models
Section titled “Larkfield: the curves of three models”xp uplift prints the uplift curve of the evaluation group, and its Qini coefficient. You ran it for the three models in the last lesson.
- X-learner
- T-learner
- “Who will buy”
- Random list
Text description of the chart
A line chart with the share of the evaluation group emailed on the horizontal axis, from 0% to 100%, and the extra orders on the vertical axis. Every curve ends at +189 at 100%, where all customers are emailed. The random line goes straight from 0 to +189. The X-learner’s curve rises to +176 at 10%, +279 at 20%, +349 at 30%, +397 at 50%, then falls slowly to +189. The T-learner’s curve: +205 at 10%, +275 at 20%, +365 at 50%. The “who will buy” curve: +147 at 10%, +213 at 20%, +280 at 50%, and it stays lower.
All curves end at +189: “email everyone” does not depend on the order. The X-learner’s curve is the highest for most sizes, so its Qini is the largest: +218.92. The “who will buy” model is the lowest of the three: +138.92.
Each curve also rises and then falls. The fall at the end is the do-not-disturbs. They sit at the bottom of the good lists, and emailing them loses orders.
These curves are noisy
Section titled “These curves are noisy”The evaluation group has 17,843 customers. Each point of the curve compares two small groups, so it has its own luck. The T-learner’s curve rises from +331 at 60% to +358 at 80%, but the truth says it should fall there. Remember Your turn in the last lesson. With another split, the X-learner’s list was a little worse. But its Qini was higher: +233.75 against +218.92. Use the Qini to compare models on the same evaluation group. Do not read small differences as real.
Larkfield: choose the 10,000
Section titled “Larkfield: choose the 10,000”Rafael’s budget is 10,000 emails, from a list of 50,000. That is 20% of the list. Each model scores the 50,000 customers, and xp uplift keeps the top 10,000. The simulator then gives the true extra orders of each list.
Text description of the chart
A column chart of extra orders from next month’s list of 50,000 customers. 10,000 random customers: +130. The 10,000 chosen by the “who will buy” model: +672. By the T-learner: +795. By the X-learner: +838. The best possible 10,000: +943. Emailing all 50,000: +648.
Two results stand out. The X-learner gets 89% of the best possible (838 of 943). And emailing all 50,000 customers gets fewer extra orders than 10,000 well-chosen emails: +648 against +838. Sending more emails does not always bring more orders. It also reaches the do-not-disturbs.
Over 10 simulated tests, the averages were: “who will buy” +699, T-learner +785, X-learner +823, best +936, random +129. The order holds.
How many emails? Profit, not accuracy
Section titled “How many emails? Profit, not accuracy”The budget of 10,000 was set before anyone saw the data. Is it the right number? The simulator can draw the true extra orders for every list size.
- Best possible
- X-learner
- “Who will buy”
- Random list
Text description of the chart
A line chart with the number of customers emailed on the horizontal axis, from 0 to 50,000, and the true extra orders on the vertical axis. X-learner: +501 at 5,000, +838 at 10,000, +1,062 at 15,000, +1,147 at 20,000, +1,174 at 25,000, +1,180 at 30,000, +1,146 at 40,000, +648 at 50,000. Best possible: +943 at 10,000, a peak of about +1,313 at 27,500, +648 at 50,000. “Who will buy”: +672 at 10,000, a peak of +973 at 27,500, +648 at 50,000. Random list: a straight line from 0 to +648.
The X-learner’s list keeps adding orders until about 30,000 customers: +1,180. After that, the extra emails reach do-not-disturbs, and the total falls. But each extra email also costs something. To choose the size, we need money, not orders.
Put a price on each email
Section titled “Put a price on each email”We need three numbers. All three are assumptions of this course. Use your own shop’s numbers.
- An extra order earns 26 dollars of margin. The simulated mean order is 86.75 dollars; we assume a margin of about 30%.
- An email costs 0.02 dollars to send.
- A customer who unsubscribes costs 6 dollars: the value of the future emails we can no longer send.
Profit of a list = extra orders × 26 − emails × 0.02 − extra unsubscribes × 6.
For example, the X-learner’s 10,000 emails: 838 × 26 = 21,788 dollars, minus 10,000 × 0.02 = 200 dollars, minus a few unsubscribes. The simulator gives 21,468 dollars.
- Best possible
- X-learner
- “Who will buy”
- Random list
Text description of the chart
A line chart with the number of customers emailed on the horizontal axis and the profit in dollars on the vertical axis. X-learner: 12,877 dollars at 5,000, 21,468 at 10,000, 27,136 at 15,000, 29,164 at 20,000, 29,704 at 25,000, a peak of 29,748 at 28,500, 28,421 at 40,000 and 13,856 at 50,000. Best possible: 24,196 at 10,000, a peak of 33,269 at about 27,500. “Who will buy”: 17,104 at 10,000, a peak of 24,133 at 27,000. Random list: a straight line from 0 to 13,856.
With these prices, the most profitable list has about 28,500 customers, and it earns 29,748 dollars. The budget of 10,000 earns 21,468. Emailing everyone earns only 13,856, less than the budget list. The curve is also flat from about 20,000 to 35,000. In a flat area, a cheaper, smaller list loses little.
So Rafael can ask for about 20,000 to 30,000 emails, chosen by the X-learner, and not for “everyone”. The prices decide the size. Higher costs give a smaller best list.
A real shop has no true curve. It reads the profit from its noisier evaluation curve. The notebook does this, and finds a best share of 50% of the list: rough, but it points the same way.
Use more of the budget in the project
Section titled “Use more of the budget in the project”-
Choose 20,000 customers with the X-learner, and check them.
Terminal window xp uplift --learner x --budget 20000xp checkWe show the last lines of
xp check:Your 20,000 customers: +1147 extra orders expected (from the simulator's true chances).Random 20,000: +259. The best possible 20,000: +1256. Emailing all 50,000: +648.1,491 of your customers order less because of the email (do-not-disturbs).The best possible 20,000 customers would bring +1,256. The X-learner’s list gets 91% of that.
-
Go back to Rafael’s budget.
Terminal window xp uplift --learner xxp checkYou see
+838 extra orders expectedand24 of your customers, as in the last lesson.
Rafael’s list is the file runs/email-s40/targets_x.csv: 10,000 customer numbers. You will meet Rafael again in the course Recommendation and Personalisation: Show Each Person What Matters (coming later).
When to use it, when not, and what it costs
Section titled “When to use it, when not, and what it costs”Use uplift and Qini curves when:
- you compare uplift models, or an uplift model with a simple rule, on a held-out part of a random test;
- you must choose a list size: the curve shows where the gain stops;
- you can put a price on an extra order, on a message and on a lost customer.
Do not trust them when:
- the evaluation group is small: the curve is noisy, and small Qini differences mean nothing;
- the evaluation group was used in training: the curve will look too good;
- the world has changed since the test: a model trained in September may not fit a December sale.
It costs a held-out group that you do not train on. It also costs a small random group in every send that gets no email. That group gives up a few extra orders: the price of knowing that the model still works.
Guided practice: draw the curves yourself
Section titled “Guided practice: draw the curves yourself”- Open the notebook from the box Do this lesson in a notebook at the top of this page.
- Run the setup cells and the helper cell. Then run section 1. It trains the three models again, and prints
17843 customers to evaluate on. - In section 2, write the uplift curve. For the X-learner, the first line is
top 10%: +176.0. - In section 3, compute the Qini coefficients. You see +138.92, +196.38 and +218.92: the same as
xp uplift. - In section 4, scale the top 20% to 50,000 customers. The X-learner gives about +781 extra orders. The truth for its real list is +838: close, not exact.
- In section 5, compute the profit for each share of the X-learner’s curve. Put the best share in
best_shareand its profit inbest_profit. Run the check cells. They printCheck passed.
Show the answer to step 6
s, g = curves["X-learner"]profit = g * scale * 26 - s * 50_000 * 0.02i = int(np.argmax(profit))best_share = round(float(s[i]), 2)best_profit = round(float(profit[i]))print(best_share, best_profit)The best share is 0.5 (25,000 customers), with a profit of about 28,387 dollars. This version leaves out the unsubscribes, and the evaluation curve is noisy. It still says what the true curve says: email more than 10,000, and fewer than all.
Your turn: test a bigger budget
Section titled “Your turn: test a bigger budget”Rafael asks for 30,000 emails. Before you run anything, predict: will 30,000 emails chosen by the X-learner bring more extra orders than 10,000? More than emailing all 50,000? Then run xp uplift --learner x --budget 30000 and xp check.
Show the answer
Your 30,000 customers: +1180 extra orders expected (from the simulator's true chances). Random 30,000: +389. The best possible 30,000: +1312. Emailing all 50,000: +648. 6,232 of your customers order less because of the email (do-not-disturbs).Yes to both. 30,000 emails bring +1,180 extra orders: more than 10,000 (+838), and almost twice as many as emailing all 50,000 (+648). The list now includes 6,232 customers whose chance goes down, most of them only a little. With our assumed prices, this list earns about 29,700 dollars, close to the top of the profit curve. Run xp uplift --learner x again to get back the list of 10,000.
Common mistake
“Our model is 90% accurate, so the campaign will work”
Symptom: a team chooses its campaign model by accuracy: how well it predicts who orders. The emailed customers order often. But a random hold-out group that got no email orders almost as often. The campaign added little.
Cause: accuracy measures the chance to order, not the change the email makes. The most accurate model finds sure things, who order with or without the email.
Fix: evaluate on a held-out part of a random test with an uplift curve and its Qini coefficient. Choose the list size with a profit curve, with the costs written down. Keep a random hold-out group in every send, to check the gain again.
Check your understanding
In the top 20% of a model's list, 30% of the emailed customers ordered and 28% of the customers who got no email ordered. The slice has 1,000 customers. What are the extra orders from emailing this slice, and is this a good list?
Show the answer
The gain is (0.30 − 0.28) × 1,000 = +20 extra orders. The order rate of 30% looks high, but the email adds only 2 points. Most of these customers would order anyway: they are sure things. It is a poor list for an email, even if the model predicts orders well.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
With another training split, the X-learner's Qini rose from +218.92 to +233.75, but its list of 10,000 brought +810 extra orders instead of +838. How can both be true?
Show the answer
The Qini is measured on the evaluation group, and the new split has a different evaluation group with its own luck. The extra orders of the list are measured on next month’s 50,000 customers, by the simulator. A small change in Qini between two evaluation groups is noise. Compare models with the Qini only on the same evaluation group, and do not read small differences as real.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
Emailing all 50,000 customers brings +648 extra orders. Emailing the X-learner's top 10,000 brings +838. Explain why sending more emails gives fewer extra orders.
Show the answer
The top of the list is mostly persuadables, and the email adds orders there. Further down, the customers are lost causes, sure things and do-not-disturbs. The do-not-disturbs order less because of the email, so their part of the list takes orders away. Emailing everyone adds all their losses: the total falls to +648.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Recap
- Accuracy points the wrong way: the “who will buy” model’s customers order more (18.38%), but the email changes fewer of them (+5.97 points against +7.81).
- An uplift curve shows the extra orders for each list size, from a held-out part of a random test. The Qini coefficient is the area above the random line: X-learner +218.92, T-learner +196.38, “who will buy” +138.92. The curves are noisy.
- Rafael’s 10,000 emails by the X-learner: +838 extra orders (best possible +943). Emailing all 50,000: only +648.
- Choose the list size by profit. With our assumed prices, the best list has about 28,500 customers and earns 29,748 dollars. The 10,000 earn 21,468, and everyone 13,856.
- Keep a random hold-out group in every send, to measure the gain again.
Next, Module 8 tests AI features online: a new prompt for the support assistant.