Skip to content
Français

Did the mailing work, and for whom?

  • Case study 1
  • 150 min
  • Optional · 1 case study needed for the certificate

Where we are

In eight modules, we tested Larkfield's changes and found who benefits from an email. In this case study, we use real open data: a randomised mailing to 344,000 people. We analyse the test, build an uplift model and choose whom to mail.

What you will be able to do

Analyse a real randomised mailing from the first check to a decision you can defend. You will check the randomisation, estimate each letter’s effect with honest TermConfidence intervalA range around a measured value, made by a method that holds the true value 95 times in 100. If the interval of a difference does not include zero, the difference is probably not chance.Learn more Estimate uncertaintyEssential Mathematics and Statistics for AI, and find for whom the best letter works. Then you compare five ways to choose whom to mail under a budget, with an uplift model among them. Last, you say what the data cannot tell you.

Case studyPortfolio piece

Licence
CC0 1.0
Time
About 2.5 hours
Runs in
Colab, Kaggle or VS Code
Skills used
  • Plan before the result
  • Sample ratio and balance checks
  • Placebo outcome and A/A splits
  • Intervals clustered by household
  • Comparing several arms
  • Planned segments
  • Uplift models and uplift curves
  • Cross-fitting and a paired bootstrap
  • A decision with its cost

Colab and Kaggle open your own copy. To use VS Code, download the notebook from GitHub. Stuck? See the solution notebook.

The notebook runs every step of this page, with the same code and the same output, plus three tasks with checks. Its setup cells install the packages and download the data (20 MB) from the study’s archive, so it needs no local setup. It runs in about one minute after the install.

Before you startWhat to know and to have readyAbout 150 minutes

What you need to know

This case study uses the whole course. These lessons matter most:

What you need on your computer

Nothing: the notebook installs its own packages, so it can also run in Colab or Kaggle. To run it locally, use a virtual environment with Python 3.12, pandas 3.1.1, SciPy 1.15.3, statsmodels 0.15.0, scikit-learn 1.9.1 and matplotlib 3.11.2, as in the project larkfield-experiments. This case study does not change the project. The setup cell makes a new folder mailing-test. To start again, delete that folder.

Tested on macOS with Python 3.12, in Jupyter from a fresh start. Colab, Kaggle, Windows and Linux are not tested.

Rafael, TermLarkfieldA made-up Canadian online shop for home and garden products. It is not a real company. All Nextia courses use its help desk and data as the example.Learn more Meet Larkfield’s email manager, wants to send a printed letter to customers for the first time. A letter costs much more than an email, so he can afford one only for some households. Before he spends the money, he asks TermPriyaLarkfield’s data scientist. She measures how well the assistant works, on test sets and not on guesses. two questions: “How will we know if a letter works? And who should get it?”

Larkfield has no letter test yet. So Priya practises on a real, public one. It is large, randomised and well documented, and its data is free to share.

It is a 2006 US election study. A primary election took place in the US state of Michigan on 8 August 2006. Before it, researchers sent one of four letters to households of registered voters. Then they read from public records who voted. We read it as a mailing test: one letter, one action, and a yes-or-no result for each person. We use it to learn methods. We draw no political lessons from it, and this page gives no advice about elections.

Rafael and Priya are invented. The letters, the households and the results are real.

You will analyse the test from start to finish, as a team would before a decision. You write the plan, check the randomisation, and estimate and compare the letters’ effects. Then you find for whom the best letter works, compare five ways to choose the households, and score a new list. You end with a recommendation, its cost, and what is still unknown.

The study is Social Pressure and Voter Turnout by Gerber, Green and Larimer. Its data file has one row for each of 344,084 registered voters in 180,002 households.

Why not a shop’s data? Two well-known public uplift datasets come from shops: the Hillstrom email test and the Criteo uplift dataset. The Hillstrom data has no licence at its source, so we may not share it. Criteo’s licence (CC BY-NC-SA 4.0) does not allow commercial use. This study is in the public domain, and it has one row per person.

Every household got one of four letters, or no letter. The coin was flipped per household, so everyone at one address got the same letter.

A flow from top to bottom. 180,002 households with 344,084 registered voters go to a coin, random by household. The coin sends them to five groups: no letter, 99,999 households; Civic duty, 20,001, 'Do your civic duty and vote!'; Hawthorne, 20,002, 'You are being studied!'; Self, 20,000, your household's past votes; Neighbors, 20,000, your neighbours' past votes too, in green. All five groups go to one box: after the election, public records show whether each person voted. A note: one letter per household.
The design. One household in nine got each letter, and five in nine got none. The outcome comes from public records, for every person, in every group.

The letters, from the study’s own copies:

  • Civic duty: “Do your civic duty and vote!”
  • Hawthorne: “You are being studied!” The researchers will check public records to see who votes.
  • Self: “Who votes is public information!” It lists the household’s own past votes, and promises an updated list after the election.
  • Neighbors: “What if your neighbors knew whether you voted?” It lists names from the street with their past votes: “You and your neighbors will all know who voted and who did not.”

The last two letters use social pressure: they show who voted. We come back to that in “Responsible use”.

The notebook loads the file and adds three columns: the letter’s name, past_votes (in how many of the five earlier elections the person voted) and age.

ARMS = {0: "No letter", 1: "Hawthorne", 2: "Civic duty", 3: "Neighbors", 4: "Self"}
PAST = ["g2000", "g2002", "p2000", "p2002", "p2004"] # the five earlier elections
raw = pd.read_csv("social_pressure.tab", sep="\t")
people = raw.assign(letter=raw["treatment"].map(ARMS), past_votes=raw[PAST].sum(axis=1), age=2006 - raw["yob"])
print("rows, columns:", raw.shape, "| households:", people["hh_id"].nunique(), "| missing:", int(raw.isna().sum().sum()))
print("everyone voted in g2004:", bool((raw["g2004"] == 1).all()))
rows, columns: (344084, 16) | households: 180002 | missing: 0
everyone voted in g2004: True

What this tells us. No values are missing. Everyone in the file voted in the general election of November 2004. So the study mailed people who had voted recently. It is not the whole population, and we can only speak about people like these.

The columns that we use:

Column What it means Use
treatment 0 no letter, 1 Hawthorne, 2 Civic duty, 3 Neighbors, 4 Self the group
voted 1 if the person voted on 8 August 2006 the outcome
g2000, g2002, p2000, p2002, p2004 1 if the person voted in that earlier general (g) or primary (p) election facts from before the letters
hh_id, hh_size the household, and the number of its people in the file the unit, and a fact
yob year of birth a fact (as age)

The file also has sex. We do not use it: the decision does not need it, and a list chosen by sex would be hard to defend.

In Write the plan first, the plan came before the data. Here the data already exists, so the rule is: write the plan before you read the outcome column. Priya writes it down, and Rafael agrees:

  • The question: does a letter raise the share of people who act (here, who vote)? Which letter raises it most? For whom?
  • The decision: which letter to send, and to which households, with a budget for one household in five.
  • The metric: the share who voted. The unit of randomisation is the household, so every interval is clustered by household.
  • The comparisons: each letter against no letter: four comparisons.
  • The segments, fixed now: six groups by past_votes, 0 to 5.
  • The one measure for the lists: extra voters per 1,000 letters, if we mail the top 20% of households.

What we cannot check. In the course, the simulator knew the true effects. Real data has no truth. So we use the checks that real data allows: the counts per group, the balance, a placebo outcome and A/A splits.

The design put 5 of every 9 households in the no-letter group, and 1 of 9 in each letter group. The chi-square test from Check the test before the result compares the counts with that design.

{'No letter': 99999, 'Hawthorne': 20002, 'Civic duty': 20001, 'Neighbors': 20000, 'Self': 20000}
chi-square 0.00, p = 1.000

What this tells us. The counts match the design almost exactly. There is no sample ratio mismatch.

The facts from before the letters must be alike in every group. The standardised difference is the gap in means divided by the typical spread. Under 0.1 is the usual rule. Across 7 facts and 4 letters, the largest is:

largest standardised difference: 0.0129

What this tells us. The groups were alike before the letters. The largest gap is in the Neighbors group’s share who voted in the 2004 primary: 40.67% against 40.03%.

A placebo outcome is a result that the letter cannot change. Here, it is voting in the 2004 primary, two years before the letters. If the method finds an “effect” on it, something is wrong.

The 'effect' of each letter on voting in the 2004 primary, before any letter was sent, with 95% intervals clustered by household. All four intervals contain zero.
Text description of the chart

Four rows on an axis from −1.5 to +1.5 points, with a dashed no-effect line at 0. Civic duty −0.09 (−0.76 to +0.58). Hawthorne +0.29 (−0.38 to +0.96). Self +0.21 (−0.46 to +0.88). Neighbors +0.63 (−0.04 to +1.30).

What this tells us. All four intervals contain zero. The Neighbors line comes closest (p = 0.07). With four tests, one p-value this low is common by chance. It also matches the small balance gap above.

Think of a family dinner. If one person in a family goes out to eat, the others often go too. People who live together also tend to vote together: they go to the polling station together, or stay home together. So two people from one household are not two independent pieces of evidence.

A tiny example. Take 4 households of 2 people each. In 2 households, both voted; in 2, nobody voted. That is 8 people and 50% turnout. It looks like 8 coin flips. But the two people in a household always did the same thing, so the households are the coins: there are only 4. The uncertainty is that of 4 flips, not 8.

An interval clustered by household counts the households, not the people. You met the idea in Difference-in-differences, where the weeks of one area moved together. The notebook’s function works from the households’ totals:

def effect(g, arm, base="No letter"):
"""Letter minus no letter, in the share: the difference, a 95% interval clustered by household, and the naive one."""
parts = {}
for name in (base, arm):
h = g[g["arm"] == name]
n, y = h["n"].to_numpy(float), h["y"].to_numpy(float)
r, G = y.sum() / n.sum(), len(h)
parts[name] = (r, G / (G - 1) * np.sum((y - r * n) ** 2) / n.sum() ** 2, r * (1 - r) / (n.sum() - 1))
d = parts[arm][0] - parts[base][0]
se, se_naive = np.sqrt(parts[arm][1] + parts[base][1]), np.sqrt(parts[arm][2] + parts[base][2])
return {"diff": d, "low": d - Z * se, "high": d + Z * se, ...}

It computes one gap per household: the household’s votes minus what the group’s share predicts. A household that votes together makes one big gap, not two small ones, so the interval grows.

Real data has no truth, but it has an A/A test inside it. Take the 99,999 no-letter households. Split them at random into two fake groups: 20,000 and the rest. Nothing differs between them. Test the difference, and do it 1,000 times. An honest method calls about 5% of the splits “significant”.

1,000 random splits of the no-letter households into two fake groups. The clustered interval calls 5.8% of them significant, close to the promised 5%. The naive interval, which treats people as independent, calls 12.7%.
Text description of the chart

Two bars. Clustered by household: 5.8%. Naive, people independent: 12.7%.

What this tells us. The naive interval is too narrow. It finds a “winner” in about one split in eight: more than twice the promised rate. The clustered interval keeps its promise. From here on, we use the clustered one.

Common mistake

People counted as independent

What you see: narrow intervals and very small p-values. With 344,084 rows, almost everything looks “significant”.

Why: the coin was flipped for 180,002 households, not for 344,084 people. People in one household act alike, so the rows hold less evidence than their number says.

Fix: analyse at the unit of randomisation. Cluster the errors by household (cov_type="cluster" in statsmodels), or work from household totals as effect() does. Check the method with A/A splits of the control group.

Now, and only now, Priya reads the voted column.

Civic duty 29.66% -> 31.45%: +0.0179 clustered 95% +0.0116 to +0.0242 | naive +0.0128 to +0.0230 | width ratio 1.25
Hawthorne 29.66% -> 32.24%: +0.0257 clustered 95% +0.0194 to +0.0321 | naive +0.0206 to +0.0309 | width ratio 1.25
Self 29.66% -> 34.52%: +0.0485 clustered 95% +0.0420 to +0.0550 | naive +0.0433 to +0.0537 | width ratio 1.25
Neighbors 29.66% -> 37.79%: +0.0813 clustered 95% +0.0747 to +0.0879 | naive +0.0760 to +0.0866 | width ratio 1.25
The effect of each letter on turnout, with the 95% interval clustered by household (blue) and the naive interval (grey). The clustered intervals are 25% wider. Every interval is far from zero.
Text description of the chart

Eight rows on an axis from 0 to 10 percentage points. Civic duty +1.79: clustered 1.16 to 2.42, naive 1.28 to 2.30. Hawthorne +2.57: clustered 1.94 to 3.21, naive 2.06 to 3.09. Self +4.85: clustered 4.20 to 5.50, naive 4.33 to 5.37. Neighbors +8.13: clustered 7.47 to 8.79, naive 7.60 to 8.66.

What this tells us. Every letter worked. Without a letter, 29.66% voted. With the Neighbors letter, 37.79% did: +8.1 points (7.5 to 8.8). That is 27% more voters, from one letter. These are the same numbers as in the paper’s own table.

There are four comparisons, so Priya checks them as in Many metrics and many tests. With Bonferroni, each p-value is multiplied by 4. The largest is still far below 0.001.

The same differences come from a regression in statsmodels, with errors clustered by household. Then Priya adds the five past votes as inputs. That is regression adjustment, the idea behind Get more precise with CUPED: it takes out the differences that we could predict.

Neighbors regression +0.0813 (+0.0747 to +0.0879); with past votes +0.0807 (+0.0744 to +0.0870)

What this tells us. The plain regression gives exactly the same interval as effect(): two methods, one answer. The adjusted interval is only 4% narrower, and the estimate barely moves (+8.07). Priya reports the plain numbers, because they need no model.

Is Neighbors better than Self? As in Segments and different effects, test the difference between them, not two p-values.

Neighbors - Self: +0.0328 (95% +0.0243 to +0.0413)
Self - Hawthorne: +0.0228 (95% +0.0144 to +0.0311)
Hawthorne - Civic duty: +0.0078 (95% -0.0004 to +0.0161)

What this tells us. Neighbors beats Self by 3.3 points, and Self beats Hawthorne by 2.3. Hawthorne and Civic duty may be the same: that interval contains zero. The more a letter says “people will see what you did”, the more it works.

The plan fixed six segments: people who voted in 0, 1, … 5 of the five earlier elections.

The Neighbors letter's effect in the six planned segments, with 95% intervals clustered by household. The share of all people is in brackets. People who voted in 3 or 4 of the 5 earlier elections respond most (blue).
Text description of the chart

Six rows on an axis from 0 to 12 points, with a line at the overall effect, +8.1. 0 of 5 (6.3% of people): +2.4 (1.0 to 3.8). 1 of 5 (7.6%): +5.9 (4.2 to 7.6). 2 of 5 (23.4%): +7.2 (6.1 to 8.3). 3 of 5 (39.5%): +9.5 (8.6 to 10.5). 4 of 5 (19.1%): +9.9 (8.5 to 11.3). 5 of 5 (4.0%): +4.4 (1.7 to 7.1).

What this tells us. The effect has a hump. Without a letter, 8.3% of the “never” group voted, and the letter added only 2.4 points: these people are hard to move. In the “always” group, 64.2% voted anyway, so the letter had less room (+4.4). People who vote sometimes respond most: +9.5 and +9.9 points. In the words of The average hides the people, the “always” group has many sure things. The “never” group has many lost causes, and the middle has the most persuadables.

No segment has a negative effect. Larkfield’s email made heavy buyers order less. This letter has no group of do-not-disturbs that we can see.

We planned the segments and clustered the intervals. The gap between “0 of 5” and “3 of 5” (+7.1 points) is far larger than the noise. So this finding is solid.

Rafael’s budget pays for letters to one household in five. Which households? Priya compares five ways to rank them for the Neighbors letter. She uses the 119,999 households of the Neighbors test: 20,000 with the letter and 99,999 without.

  • Biggest households first (the baseline). One letter reaches every voter at the address, so a household of four gives more chances than a household of one.
  • Planned segments. Each person gets their segment’s effect, learnt from the training households.
  • “Who will vote”. A model of the chance to vote with the letter, like the usual “who will buy” model.
  • The T-learner and the X-learner, from Uplift models. Inputs: the five past votes, household size and age.

One letter goes to a household, so a household’s score is the sum of its people’s scores: its expected extra voters.

Cross-fitting: scores from models that never saw the household

Section titled “Cross-fitting: scores from models that never saw the household”

A model must not score the households it learnt from. In the uplift lessons, we kept 30% of the customers apart for that. Here, the differences between lists are small, so we want every household in the comparison. Cross-fitting does both.

A grid of four rounds and four folds. In round 1, fold 1 is scored and folds 2, 3 and 4 train the models. In round 2, fold 2 is scored and the others train. The same for rounds 3 and 4. A box on the right: every household gets a score from models that never saw it; then compare the lists on all 119,999 households.
Cross-fitting with 4 folds of households. Each round trains on three folds and scores the fourth. After four rounds, every household has an honest score.

The folds are made of households, so the people of one household are never split between training and scoring.

For each list, the measure: mail the top 20% of households by score. Among them, compare the share who voted with and without the letter.

size top 20%: +185.6 extra voters per 1,000 letters (95% +149.2 to +222.0); Qini 1272.7
segment top 20%: +197.9 extra voters per 1,000 letters (95% +162.3 to +233.6); Qini 1528.9
response top 20%: +223.5 extra voters per 1,000 letters (95% +188.7 to +258.2); Qini 1708.8
t top 20%: +224.7 extra voters per 1,000 letters (95% +191.0 to +258.4); Qini 1439.9
x top 20%: +231.1 extra voters per 1,000 letters (95% +197.3 to +264.9); Qini 1830.4
everyone: +154.6 per 1,000 letters (95% +141.6 to +167.6)
Extra voters per 1,000 letters if we mail the top 20% of households, for each list, with 95% intervals. Every list beats a random choice (aqua). The X-learner is highest, but its interval overlaps the others. The placebo (orange) is a model trained on a fake letter: its interval contains zero.
Text description of the chart

Seven rows on an axis from −40 to +280. Biggest households +186 (149 to 222). Planned segments +198 (162 to 234). “Who will vote” +224 (189 to 258). T-learner +225 (191 to 258). X-learner +231 (197 to 265). Random, everyone: +155 (142 to 168). Placebo, fake letter: +6 (−27 to +39).

What this tells us. Mailing at random brings about 155 extra voters per 1,000 letters. Every ranked list does better. The X-learner’s top 20% brings 231: about 50% more from the same budget. The Qini coefficients (the area between each curve and a random list) also put the X-learner first. But the intervals overlap. Are the lists really different?

The intervals above include noise that all the lists share: the same households, the same luck. A paired bootstrap removes it. Resample the households 200 times. On each resample, compute the measure for both lists, and keep their difference. Then take the middle 95% of the differences.

x - size +45.5 (95% +6.1 to +78.3)
x - segment +33.2 (95% +3.9 to +62.2)
x - response +7.7 (95% -19.8 to +35.3)
x - t +6.4 (95% -14.3 to +26.6)

What this tells us. The X-learner is clearly better than “biggest households first” and than the planned segments. It is not clearly better than the “who will vote” model or the T-learner: those intervals contain zero. The data cannot rank the three models.

Why does a “who will vote” model do so well here, when it chose badly at Larkfield? Look at the segments: the effect rises with past votes, up to 4 of 5. So likely voters are mostly persuadables, not sure things. Only the “always” group (4% of people) breaks the pattern. Larkfield’s email harmed its heavy buyers; this letter harms no group that we can see.

The curves tell the same story along the whole list:

  • X-learner
  • "Who will vote"
  • Biggest households
  • Random list
Uplift curves: the extra voters if the top part of each list gets the Neighbors letter, scored out of fold. All curves end at 18,551, the effect of mailing everyone. The X-learner and 'who will vote' curves are close; both are above a random list.
Text description of the chart

A line chart from 0% to 100% of households. At 20%: X-learner 5,546 extra voters, “who will vote” 5,362, biggest households 4,460, random list 3,710. At 50%: X-learner 11,952, “who will vote” 11,839, biggest households 11,283, random 9,276. All four end at 18,551 at 100%.

A real test cannot show an uplift model its true effects. So Priya builds a test where the truth is known to be zero. She splits the no-letter households at random: one in six goes into a “fake letter” group. She trains the X-learner on it, with the same cross-fitting. A model that finds uplift here finds noise.

placebo top 20%: +6.1 per 1,000 (95% -26.7 to +38.9)

What this tells us. The fake letter’s “top 20%” gains nothing: zero is inside the interval. So the X-learner does not make gains out of noise, and cross-fitting kept the comparison honest.

The average person in the top 20% of each list, against everyone:

Past votes (of 5) People per household Age
Everyone 2.70 1.91 49.8
Biggest households 2.34 2.89 44.3
“Who will vote” 3.21 2.50 50.8
X-learner 2.86 2.40 49.5

The X-learner picks people who vote sometimes, in households a little larger than average.

Priya writes to Rafael in his words. She keeps the election out of it and talks about the method.

  • The letter matters more than the list. The best letter added +8.1 points, and the weakest +1.8. Changing the letter multiplied the effect by 4.5. The best list adds about 50% to a random choice. Test the message first.
  • The strongest letters showed people’s own records. That works, but it is pressure, and for a shop it is not acceptable (see “Responsible use”). Larkfield should test honest versions, for example a reminder of the customer’s own saved basket.
  • To choose whom to mail, use an uplift model, and check it against a simple one. The X-learner was best. It was not clearly better than “who will vote”. Prefer the X-learner, because it targets the change. Compare the two again in the next test.
  • Keep a random hold-back in every mailing. About one household in ten on the chosen list gets no letter, chosen at random. Without it, nobody can measure the list’s effect.

The data has no costs. Priya assumes that a letter costs 0.50 dollars to print and post. This is an assumption, not from the study. (The course’s email price, 0.02 dollars, does not fit a letter.)

List Extra voters per 1,000 letters Cost of one extra voter
Random (everyone) 154.6 500 ÷ 154.6 = 3.23 dollars
Biggest households, top 20% 185.6 500 ÷ 185.6 = 2.69 dollars
X-learner, top 20% 231.1 500 ÷ 231.1 = 2.16 dollars

What this tells us. With the X-learner’s list, each extra person who acts costs 2.16 dollars instead of 3.23. For Larkfield, “an extra voter” becomes “an extra customer who orders”. Rafael can compare its cost with the course’s assumed margin of 26 dollars per order.

A list is useful only if it works on households that the test never saw. The households that got the other three letters (60,003) stand in for “next year’s list”. Priya uses only their facts from before the mailing, never their outcomes. The X-learner, trained on all 119,999 households of the Neighbors test, scores them. She takes the top 20%.

final = fit_models(two)
new = people[~people["letter"].isin(["No letter", "Neighbors"])].copy()
new["score"] = final[best](new)
plan = new.groupby("hh_id").agg(score=("score", "sum"), people=("voted", "size"))
chosen = plan.sort_values("score", ascending=False, kind="stable").head(int(round(0.2 * len(plan))))
60,003 households; chosen 12,001 letters to 30,095 people; expected extra voters 2696 (224.7 per 1,000 letters)

What this tells us. The plan is 12,001 letters to 30,095 people: about 6,000 dollars at the assumed price. The model expects about 2,700 extra voters, 225 per 1,000 letters, close to the +231 measured with cross-fitting. An expectation is not a measurement: only the hold-back group of the next mailing can confirm it.

  • “Did it work?” was easy. “For whom?” was hard. The test measured each letter to within about ±0.65 points, but it could not rank three targeting models.
  • There is no truth to check against. All the checks passed: the counts, the balance, the placebo outcome, the A/A splits and the placebo for the model. They show that the methods behave well. They do not prove an effect for any one person.
  • The winner is a little lucky. The X-learner was the best of five lists on the same data. So its number is probably a little high.
  • What a fair test of the targeting would need. The study randomised the letters, not the lists. To test a list, randomise households to “mailed by the model’s list” or “mailed at random”, with the same number of letters.
  • One election, one state, 2006. It shows the methods, and how large the effect of one letter can be. It does not predict the effect of a shop’s letter.
  • Real data is harder than the simulator. In Evaluate and use uplift, the simulator showed the X-learner’s true gain. Here we could only measure it, with a wide interval.
  • Real people are in this file. It has no names or addresses, but it has year of birth, sex, household and voting history. Do not try to identify anyone. Do not join the file with other data about people.
  • Keep sensitive facts out of targeting. We left out sex. Age is in the model, so check that a list does not leave out an age group without a reason you can defend.
  • Social pressure has a cost that the test did not measure. The Neighbors letter showed people’s voting records to their neighbours. The study measured turnout, not anger, complaints or lost trust. A shop that names or shames customers would lose them. A strong effect is not a reason to use a message.
  • Not for elections. A model that chooses which citizens to encourage to vote raises questions of fairness and law. This course does not cover them. Use the methods for products and services.
  • Credit the source. CC0 asks for nothing, but good practice does: cite the authors and the archive, as in the box above.

This case study shows an employer that you can analyse a real experiment from the checks to a decision. It also shows that you know what the data cannot tell. Make it yours with one change, such as another letter or another budget. Then make it public and easy to read in five minutes.

What to put in the repository:

  • README.md, the page that people read first (template below).
  • The notebook, run from top to bottom, with its outputs.
  • PLAN.md: the plan from “The plan before the result”, written before the analysis.
  • results.csv: each letter’s effect with both intervals, and each list’s measure with its interval.
  • assumptions.md: the letter cost and the budget.
  • The data source and licence, with the citation above. CC0 allows a copy, but a link and the checksum are enough.

A README template:

# Did the mailing work, and for whom? A real randomised mailing
## Problem
An organisation can mail one household in five. Which letter, and to whom?
## Data and licence
Gerber, Green and Larimer, "Social Pressure and Voter Turnout", Michigan 2006
(Yale ISPS Data Archive, doi:10.60600/YU/CGMWNW), CC0 1.0. 344,084 people in
180,002 households; 4 letters and no letter, randomised by household.
## Approach
Plan written before reading the outcome. Checks: sample ratio, balance,
placebo outcome, 1,000 A/A splits. Effects with intervals clustered by household.
Six planned segments by past votes. Five household lists compared with
cross-fitting (4 folds) and a paired bootstrap; a placebo test for the model.
## Results
Neighbors letter +8.1 points (7.5 to 8.8); Civic duty +1.8. Naive intervals
gave 12.7% false winners in A/A splits; clustered 5.8%. Effect by past votes:
+2.4 (never) to +9.9 (4 of 5). Top 20% by the X-learner: +231 extra voters per
1,000 letters vs +155 at random; not clearly better than a "who will vote" model.
## Limits
One election, 2006; no costs (0.50 dollars a letter assumed); the targeting was
not randomised; social pressure has costs the study did not measure.

How to describe it in an interview, in three sentences: “I analysed a real randomised mailing to 180,000 households; A/A splits showed that the intervals must be clustered by household. The best letter raised the action rate by 8 points, most among people who acted sometimes. An uplift model’s list brought about 50% more per letter than a random one, but no clear gain over a simple model. So I recommended a randomised test of the list.”

Go further:

  • Repeat the segments and the lists for the Self letter. Is the hump the same?
  • Plan the targeting test on paper: how many households would it need to see a difference of 30 extra voters per 1,000 letters? Use Sample size and power.
  • Compare with the other case studies, Real tests at an online shop and A minimum wage without a test: where was randomisation available, and what did it make easy?

Before you run it, predict: take the people who voted in 3 of the 5 earlier elections. Is the Self letter’s effect for them above or below its overall effect (+4.85 points)? In the notebook, section 5, estimate it with effect() and by_household(), and put it in self_3.

Show the answer
sub = people[people["past_votes"] == 3]
self_3 = round(float(effect(by_household(sub, "voted"), "Self")["diff"]), 4)
print(self_3)
0.0556

The Self letter adds 5.56 points in the “3 of 5” segment, above its overall +4.85. As with the Neighbors letter, people who vote sometimes respond more than average.

Check your understanding

The file has 344,084 people, but Priya computes her intervals from 180,002 households. Why? What did the A/A splits show?

Show the answer

The coin was flipped per household: everyone at one address got the same letter. People who live together also tend to vote together, so they are not independent pieces of evidence. An interval that treats every person as independent is too narrow. The A/A splits checked this on real data. The splits divided the no-letter households at random 1,000 times, so nothing differed. The naive interval called 12.7% of them “significant”, against the promised 5%. The interval clustered by household called 5.8%. So the clustered interval keeps its promise, and the naive one finds about one false winner in eight.

Check your understanding

At Larkfield, the 'who will buy' model emailed the wrong customers. Here, the 'who will vote' model is almost as good as the X-learner. Explain the difference.

Show the answer

A response model ranks people by their chance to act, not by the change that the letter makes. That goes wrong when the most likely buyers are sure things or do-not-disturbs, as with Larkfield’s heavy buyers. Here, the planned segments show that the letter’s effect rises with past votes, up to 4 of 5 elections. So people who are likely to vote are mostly people whom the letter moves. Only the “always” group (4% of people) breaks this pattern, and no segment has a negative effect. In this data, the chance and the change point the same way. That is a fact about this test, not a rule: check it with an uplift curve every time.

Check your understanding

Rafael asks: 'The X-learner won. Can we use its list and stop testing?' What does Priya answer?

Show the answer

Not yet. The X-learner had the highest measure, +231 extra voters per 1,000 letters. But the paired bootstrap could not separate it from the “who will vote” model or the T-learner. It was also chosen as the best of five on the same data, so its number is probably a little high. Most important, the study randomised the letters, not the lists. To know what the list is worth, run a new test: mail households by the model’s list or at random, with the same number of letters. And keep a random hold-back in every mailing, so that each mailing measures its own effect.

Recap

  • You wrote the plan before reading the outcome. Then you checked the test without it: the counts, the balance, a placebo outcome and 1,000 A/A splits.
  • The household was the unit, so the intervals were clustered by household. Naive intervals gave 12.7% false winners in A/A splits; clustered ones 5.8%.
  • Every letter worked. The Neighbors letter added +8.1 points (7.5 to 8.8), and the message mattered far more than the list.
  • In planned segments, people who voted sometimes responded most (+9.5 to +9.9); “never” and “always” voters least.
  • With cross-fitting, the X-learner’s top 20% brought +231 extra voters per 1,000 letters, against +155 at random. It was not clearly better than a “who will vote” model, so the next step is a randomised test of the list.

Next, the second case study: the daily results of 78 real online experiments at a fashion shop, with peeking and a sequential test.

Next: Real tests at an online shop