Did the mailing work, and for whom?
- Case study 1
- 150 min
- Optional · 1 case study needed for the certificate
Where we are
In eight modules, we tested Larkfield's changes and found who benefits from an email. In this case study, we use real open data: a randomised mailing to 344,000 people. We analyse the test, build an uplift model and choose whom to mail.
Case study 1 of 3 · Experiments and Causal Impact: Did Our Change Work?Did the mailing work, and for whom?
1. Why "it went up" is not an answer
2. Design an A/B test
3. Analyse the test
4. Traps in real tests
5. When you cannot randomise: natural experiments
6. Adjust for confounders in observational data
7. Uplift: who should get the treatment?
8. Experiments for AI features and in production
What you will be able to do
Analyse a real randomised mailing from the first check to a decision you can defend. You will check the randomisation, estimate each letter’s effect with honest TermConfidence intervalA range around a measured value, made by a method that holds the true value 95 times in 100. If the interval of a difference does not include zero, the difference is probably not chance.Learn more Estimate uncertaintyEssential Mathematics and Statistics for AI, and find for whom the best letter works. Then you compare five ways to choose whom to mail under a budget, with an uplift model among them. Last, you say what the data cannot tell you.
The notebook runs every step of this page, with the same code and the same output, plus three tasks with checks. Its setup cells install the packages and download the data (20 MB) from the study’s archive, so it needs no local setup. It runs in about one minute after the install.
Before you startWhat to know and to have ready
What you need to know
This case study uses the whole course. These lessons matter most:
- The unit of randomisation, from Randomise, and the plan written first, from Write the plan first.
- Intervals, from The result and its uncertainty. The checks before the result (sample ratio, A/A tests), from Check the test before the result.
- Planned segments, from Segments and different effects, and many comparisons, from Many metrics and many tests.
- Errors clustered by group, from Difference-in-differences.
- The four groups, the T-learner and the X-learner, and the uplift curve, from The average hides the people, Uplift models and Evaluate and use uplift.
What you need on your computer
Nothing: the notebook installs its own packages, so it can also run in Colab or Kaggle. To run it locally, use a virtual environment with Python 3.12, pandas 3.1.1, SciPy 1.15.3, statsmodels 0.15.0, scikit-learn 1.9.1 and matplotlib 3.11.2, as in the project larkfield-experiments. This case study does not change the project. The setup cell makes a new folder mailing-test. To start again, delete that folder.
Tested on macOS with Python 3.12, in Jupyter from a fresh start. Colab, Kaggle, Windows and Linux are not tested.
The situation
Section titled “The situation”Rafael, TermLarkfieldA made-up Canadian online shop for home and garden products. It is not a real company. All Nextia courses use its help desk and data as the example.Learn more Meet Larkfield’s email manager, wants to send a printed letter to customers for the first time. A letter costs much more than an email, so he can afford one only for some households. Before he spends the money, he asks TermPriyaLarkfield’s data scientist. She measures how well the assistant works, on test sets and not on guesses. two questions: “How will we know if a letter works? And who should get it?”
Larkfield has no letter test yet. So Priya practises on a real, public one. It is large, randomised and well documented, and its data is free to share.
It is a 2006 US election study. A primary election took place in the US state of Michigan on 8 August 2006. Before it, researchers sent one of four letters to households of registered voters. Then they read from public records who voted. We read it as a mailing test: one letter, one action, and a yes-or-no result for each person. We use it to learn methods. We draw no political lessons from it, and this page gives no advice about elections.
Rafael and Priya are invented. The letters, the households and the results are real.
What you will build
Section titled “What you will build”You will analyse the test from start to finish, as a team would before a decision. You write the plan, check the randomisation, and estimate and compare the letters’ effects. Then you find for whom the best letter works, compare five ways to choose the households, and score a new list. You end with a recommendation, its cost, and what is still unknown.
The data
Section titled “The data”The study is Social Pressure and Voter Turnout by Gerber, Green and Larimer. Its data file has one row for each of 344,084 registered voters in 180,002 households.
Why not a shop’s data? Two well-known public uplift datasets come from shops: the Hillstrom email test and the Criteo uplift dataset. The Hillstrom data has no licence at its source, so we may not share it. Criteo’s licence (CC BY-NC-SA 4.0) does not allow commercial use. This study is in the public domain, and it has one row per person.
The four letters
Section titled “The four letters”Every household got one of four letters, or no letter. The coin was flipped per household, so everyone at one address got the same letter.
The letters, from the study’s own copies:
- Civic duty: “Do your civic duty and vote!”
- Hawthorne: “You are being studied!” The researchers will check public records to see who votes.
- Self: “Who votes is public information!” It lists the household’s own past votes, and promises an updated list after the election.
- Neighbors: “What if your neighbors knew whether you voted?” It lists names from the street with their past votes: “You and your neighbors will all know who voted and who did not.”
The last two letters use social pressure: they show who voted. We come back to that in “Responsible use”.
Load and first checks
Section titled “Load and first checks”The notebook loads the file and adds three columns: the letter’s name, past_votes (in how many of the five earlier elections the person voted) and age.
ARMS = {0: "No letter", 1: "Hawthorne", 2: "Civic duty", 3: "Neighbors", 4: "Self"}PAST = ["g2000", "g2002", "p2000", "p2002", "p2004"] # the five earlier electionsraw = pd.read_csv("social_pressure.tab", sep="\t")people = raw.assign(letter=raw["treatment"].map(ARMS), past_votes=raw[PAST].sum(axis=1), age=2006 - raw["yob"])print("rows, columns:", raw.shape, "| households:", people["hh_id"].nunique(), "| missing:", int(raw.isna().sum().sum()))print("everyone voted in g2004:", bool((raw["g2004"] == 1).all()))rows, columns: (344084, 16) | households: 180002 | missing: 0everyone voted in g2004: TrueWhat this tells us. No values are missing. Everyone in the file voted in the general election of November 2004. So the study mailed people who had voted recently. It is not the whole population, and we can only speak about people like these.
The columns that we use:
| Column | What it means | Use |
|---|---|---|
treatment |
0 no letter, 1 Hawthorne, 2 Civic duty, 3 Neighbors, 4 Self | the group |
voted |
1 if the person voted on 8 August 2006 | the outcome |
g2000, g2002, p2000, p2002, p2004 |
1 if the person voted in that earlier general (g) or primary (p) election | facts from before the letters |
hh_id, hh_size |
the household, and the number of its people in the file | the unit, and a fact |
yob |
year of birth | a fact (as age) |
The file also has sex. We do not use it: the decision does not need it, and a list chosen by sex would be hard to defend.
The plan before the result
Section titled “The plan before the result”In Write the plan first, the plan came before the data. Here the data already exists, so the rule is: write the plan before you read the outcome column. Priya writes it down, and Rafael agrees:
- The question: does a letter raise the share of people who act (here, who vote)? Which letter raises it most? For whom?
- The decision: which letter to send, and to which households, with a budget for one household in five.
- The metric: the share who voted. The unit of randomisation is the household, so every interval is clustered by household.
- The comparisons: each letter against no letter: four comparisons.
- The segments, fixed now: six groups by
past_votes, 0 to 5. - The one measure for the lists: extra voters per 1,000 letters, if we mail the top 20% of households.
What we cannot check. In the course, the simulator knew the true effects. Real data has no truth. So we use the checks that real data allows: the counts per group, the balance, a placebo outcome and A/A splits.
Check the test before the result
Section titled “Check the test before the result”Households per group
Section titled “Households per group”The design put 5 of every 9 households in the no-letter group, and 1 of 9 in each letter group. The chi-square test from Check the test before the result compares the counts with that design.
{'No letter': 99999, 'Hawthorne': 20002, 'Civic duty': 20001, 'Neighbors': 20000, 'Self': 20000}chi-square 0.00, p = 1.000What this tells us. The counts match the design almost exactly. There is no sample ratio mismatch.
Balance before the letters
Section titled “Balance before the letters”The facts from before the letters must be alike in every group. The standardised difference is the gap in means divided by the typical spread. Under 0.1 is the usual rule. Across 7 facts and 4 letters, the largest is:
largest standardised difference: 0.0129What this tells us. The groups were alike before the letters. The largest gap is in the Neighbors group’s share who voted in the 2004 primary: 40.67% against 40.03%.
A placebo outcome
Section titled “A placebo outcome”A placebo outcome is a result that the letter cannot change. Here, it is voting in the 2004 primary, two years before the letters. If the method finds an “effect” on it, something is wrong.
Text description of the chart
Four rows on an axis from −1.5 to +1.5 points, with a dashed no-effect line at 0. Civic duty −0.09 (−0.76 to +0.58). Hawthorne +0.29 (−0.38 to +0.96). Self +0.21 (−0.46 to +0.88). Neighbors +0.63 (−0.04 to +1.30).
What this tells us. All four intervals contain zero. The Neighbors line comes closest (p = 0.07). With four tests, one p-value this low is common by chance. It also matches the small balance gap above.
Households vote together
Section titled “Households vote together”Think of a family dinner. If one person in a family goes out to eat, the others often go too. People who live together also tend to vote together: they go to the polling station together, or stay home together. So two people from one household are not two independent pieces of evidence.
A tiny example. Take 4 households of 2 people each. In 2 households, both voted; in 2, nobody voted. That is 8 people and 50% turnout. It looks like 8 coin flips. But the two people in a household always did the same thing, so the households are the coins: there are only 4. The uncertainty is that of 4 flips, not 8.
An interval clustered by household counts the households, not the people. You met the idea in Difference-in-differences, where the weeks of one area moved together. The notebook’s function works from the households’ totals:
def effect(g, arm, base="No letter"): """Letter minus no letter, in the share: the difference, a 95% interval clustered by household, and the naive one.""" parts = {} for name in (base, arm): h = g[g["arm"] == name] n, y = h["n"].to_numpy(float), h["y"].to_numpy(float) r, G = y.sum() / n.sum(), len(h) parts[name] = (r, G / (G - 1) * np.sum((y - r * n) ** 2) / n.sum() ** 2, r * (1 - r) / (n.sum() - 1)) d = parts[arm][0] - parts[base][0] se, se_naive = np.sqrt(parts[arm][1] + parts[base][1]), np.sqrt(parts[arm][2] + parts[base][2]) return {"diff": d, "low": d - Z * se, "high": d + Z * se, ...}It computes one gap per household: the household’s votes minus what the group’s share predicts. A household that votes together makes one big gap, not two small ones, so the interval grows.
A/A splits: which interval is honest?
Section titled “A/A splits: which interval is honest?”Real data has no truth, but it has an A/A test inside it. Take the 99,999 no-letter households. Split them at random into two fake groups: 20,000 and the rest. Nothing differs between them. Test the difference, and do it 1,000 times. An honest method calls about 5% of the splits “significant”.
Text description of the chart
Two bars. Clustered by household: 5.8%. Naive, people independent: 12.7%.
What this tells us. The naive interval is too narrow. It finds a “winner” in about one split in eight: more than twice the promised rate. The clustered interval keeps its promise. From here on, we use the clustered one.
Common mistake
People counted as independent
What you see: narrow intervals and very small p-values. With 344,084 rows, almost everything looks “significant”.
Why: the coin was flipped for 180,002 households, not for 344,084 people. People in one household act alike, so the rows hold less evidence than their number says.
Fix: analyse at the unit of randomisation. Cluster the errors by household (cov_type="cluster" in statsmodels), or work from household totals as effect() does. Check the method with A/A splits of the control group.
Did the mailing work?
Section titled “Did the mailing work?”Now, and only now, Priya reads the voted column.
Civic duty 29.66% -> 31.45%: +0.0179 clustered 95% +0.0116 to +0.0242 | naive +0.0128 to +0.0230 | width ratio 1.25Hawthorne 29.66% -> 32.24%: +0.0257 clustered 95% +0.0194 to +0.0321 | naive +0.0206 to +0.0309 | width ratio 1.25Self 29.66% -> 34.52%: +0.0485 clustered 95% +0.0420 to +0.0550 | naive +0.0433 to +0.0537 | width ratio 1.25Neighbors 29.66% -> 37.79%: +0.0813 clustered 95% +0.0747 to +0.0879 | naive +0.0760 to +0.0866 | width ratio 1.25Text description of the chart
Eight rows on an axis from 0 to 10 percentage points. Civic duty +1.79: clustered 1.16 to 2.42, naive 1.28 to 2.30. Hawthorne +2.57: clustered 1.94 to 3.21, naive 2.06 to 3.09. Self +4.85: clustered 4.20 to 5.50, naive 4.33 to 5.37. Neighbors +8.13: clustered 7.47 to 8.79, naive 7.60 to 8.66.
What this tells us. Every letter worked. Without a letter, 29.66% voted. With the Neighbors letter, 37.79% did: +8.1 points (7.5 to 8.8). That is 27% more voters, from one letter. These are the same numbers as in the paper’s own table.
There are four comparisons, so Priya checks them as in Many metrics and many tests. With Bonferroni, each p-value is multiplied by 4. The largest is still far below 0.001.
A second method: regression
Section titled “A second method: regression”The same differences come from a regression in statsmodels, with errors clustered by household. Then Priya adds the five past votes as inputs. That is regression adjustment, the idea behind Get more precise with CUPED: it takes out the differences that we could predict.
Neighbors regression +0.0813 (+0.0747 to +0.0879); with past votes +0.0807 (+0.0744 to +0.0870)What this tells us. The plain regression gives exactly the same interval as effect(): two methods, one answer. The adjusted interval is only 4% narrower, and the estimate barely moves (+8.07). Priya reports the plain numbers, because they need no model.
Compare the letters
Section titled “Compare the letters”Is Neighbors better than Self? As in Segments and different effects, test the difference between them, not two p-values.
Neighbors - Self: +0.0328 (95% +0.0243 to +0.0413)Self - Hawthorne: +0.0228 (95% +0.0144 to +0.0311)Hawthorne - Civic duty: +0.0078 (95% -0.0004 to +0.0161)What this tells us. Neighbors beats Self by 3.3 points, and Self beats Hawthorne by 2.3. Hawthorne and Civic duty may be the same: that interval contains zero. The more a letter says “people will see what you did”, the more it works.
For whom does the Neighbors letter work?
Section titled “For whom does the Neighbors letter work?”The plan fixed six segments: people who voted in 0, 1, … 5 of the five earlier elections.
Text description of the chart
Six rows on an axis from 0 to 12 points, with a line at the overall effect, +8.1. 0 of 5 (6.3% of people): +2.4 (1.0 to 3.8). 1 of 5 (7.6%): +5.9 (4.2 to 7.6). 2 of 5 (23.4%): +7.2 (6.1 to 8.3). 3 of 5 (39.5%): +9.5 (8.6 to 10.5). 4 of 5 (19.1%): +9.9 (8.5 to 11.3). 5 of 5 (4.0%): +4.4 (1.7 to 7.1).
What this tells us. The effect has a hump. Without a letter, 8.3% of the “never” group voted, and the letter added only 2.4 points: these people are hard to move. In the “always” group, 64.2% voted anyway, so the letter had less room (+4.4). People who vote sometimes respond most: +9.5 and +9.9 points. In the words of The average hides the people, the “always” group has many sure things. The “never” group has many lost causes, and the middle has the most persuadables.
No segment has a negative effect. Larkfield’s email made heavy buyers order less. This letter has no group of do-not-disturbs that we can see.
We planned the segments and clustered the intervals. The gap between “0 of 5” and “3 of 5” (+7.1 points) is far larger than the noise. So this finding is solid.
Whom to mail: five lists, one measure
Section titled “Whom to mail: five lists, one measure”Rafael’s budget pays for letters to one household in five. Which households? Priya compares five ways to rank them for the Neighbors letter. She uses the 119,999 households of the Neighbors test: 20,000 with the letter and 99,999 without.
- Biggest households first (the baseline). One letter reaches every voter at the address, so a household of four gives more chances than a household of one.
- Planned segments. Each person gets their segment’s effect, learnt from the training households.
- “Who will vote”. A model of the chance to vote with the letter, like the usual “who will buy” model.
- The T-learner and the X-learner, from Uplift models. Inputs: the five past votes, household size and age.
One letter goes to a household, so a household’s score is the sum of its people’s scores: its expected extra voters.
Cross-fitting: scores from models that never saw the household
Section titled “Cross-fitting: scores from models that never saw the household”A model must not score the households it learnt from. In the uplift lessons, we kept 30% of the customers apart for that. Here, the differences between lists are small, so we want every household in the comparison. Cross-fitting does both.
The folds are made of households, so the people of one household are never split between training and scoring.
The results
Section titled “The results”For each list, the measure: mail the top 20% of households by score. Among them, compare the share who voted with and without the letter.
size top 20%: +185.6 extra voters per 1,000 letters (95% +149.2 to +222.0); Qini 1272.7segment top 20%: +197.9 extra voters per 1,000 letters (95% +162.3 to +233.6); Qini 1528.9response top 20%: +223.5 extra voters per 1,000 letters (95% +188.7 to +258.2); Qini 1708.8t top 20%: +224.7 extra voters per 1,000 letters (95% +191.0 to +258.4); Qini 1439.9x top 20%: +231.1 extra voters per 1,000 letters (95% +197.3 to +264.9); Qini 1830.4everyone: +154.6 per 1,000 letters (95% +141.6 to +167.6)Text description of the chart
Seven rows on an axis from −40 to +280. Biggest households +186 (149 to 222). Planned segments +198 (162 to 234). “Who will vote” +224 (189 to 258). T-learner +225 (191 to 258). X-learner +231 (197 to 265). Random, everyone: +155 (142 to 168). Placebo, fake letter: +6 (−27 to +39).
What this tells us. Mailing at random brings about 155 extra voters per 1,000 letters. Every ranked list does better. The X-learner’s top 20% brings 231: about 50% more from the same budget. The Qini coefficients (the area between each curve and a random list) also put the X-learner first. But the intervals overlap. Are the lists really different?
A paired bootstrap
Section titled “A paired bootstrap”The intervals above include noise that all the lists share: the same households, the same luck. A paired bootstrap removes it. Resample the households 200 times. On each resample, compute the measure for both lists, and keep their difference. Then take the middle 95% of the differences.
x - size +45.5 (95% +6.1 to +78.3)x - segment +33.2 (95% +3.9 to +62.2)x - response +7.7 (95% -19.8 to +35.3)x - t +6.4 (95% -14.3 to +26.6)What this tells us. The X-learner is clearly better than “biggest households first” and than the planned segments. It is not clearly better than the “who will vote” model or the T-learner: those intervals contain zero. The data cannot rank the three models.
Why does a “who will vote” model do so well here, when it chose badly at Larkfield? Look at the segments: the effect rises with past votes, up to 4 of 5. So likely voters are mostly persuadables, not sure things. Only the “always” group (4% of people) breaks the pattern. Larkfield’s email harmed its heavy buyers; this letter harms no group that we can see.
The curves tell the same story along the whole list:
- X-learner
- "Who will vote"
- Biggest households
- Random list
Text description of the chart
A line chart from 0% to 100% of households. At 20%: X-learner 5,546 extra voters, “who will vote” 5,362, biggest households 4,460, random list 3,710. At 50%: X-learner 11,952, “who will vote” 11,839, biggest households 11,283, random 9,276. All four end at 18,551 at 100%.
A placebo for the model
Section titled “A placebo for the model”A real test cannot show an uplift model its true effects. So Priya builds a test where the truth is known to be zero. She splits the no-letter households at random: one in six goes into a “fake letter” group. She trains the X-learner on it, with the same cross-fitting. A model that finds uplift here finds noise.
placebo top 20%: +6.1 per 1,000 (95% -26.7 to +38.9)What this tells us. The fake letter’s “top 20%” gains nothing: zero is inside the interval. So the X-learner does not make gains out of noise, and cross-fitting kept the comparison honest.
Who is at the top?
Section titled “Who is at the top?”The average person in the top 20% of each list, against everyone:
| Past votes (of 5) | People per household | Age | |
|---|---|---|---|
| Everyone | 2.70 | 1.91 | 49.8 |
| Biggest households | 2.34 | 2.89 | 44.3 |
| “Who will vote” | 3.21 | 2.50 | 50.8 |
| X-learner | 2.86 | 2.40 | 49.5 |
The X-learner picks people who vote sometimes, in households a little larger than average.
The recommendation
Section titled “The recommendation”Priya writes to Rafael in his words. She keeps the election out of it and talks about the method.
- The letter matters more than the list. The best letter added +8.1 points, and the weakest +1.8. Changing the letter multiplied the effect by 4.5. The best list adds about 50% to a random choice. Test the message first.
- The strongest letters showed people’s own records. That works, but it is pressure, and for a shop it is not acceptable (see “Responsible use”). Larkfield should test honest versions, for example a reminder of the customer’s own saved basket.
- To choose whom to mail, use an uplift model, and check it against a simple one. The X-learner was best. It was not clearly better than “who will vote”. Prefer the X-learner, because it targets the change. Compare the two again in the next test.
- Keep a random hold-back in every mailing. About one household in ten on the chosen list gets no letter, chosen at random. Without it, nobody can measure the list’s effect.
What it costs
Section titled “What it costs”The data has no costs. Priya assumes that a letter costs 0.50 dollars to print and post. This is an assumption, not from the study. (The course’s email price, 0.02 dollars, does not fit a letter.)
| List | Extra voters per 1,000 letters | Cost of one extra voter |
|---|---|---|
| Random (everyone) | 154.6 | 500 ÷ 154.6 = 3.23 dollars |
| Biggest households, top 20% | 185.6 | 500 ÷ 185.6 = 2.69 dollars |
| X-learner, top 20% | 231.1 | 500 ÷ 231.1 = 2.16 dollars |
What this tells us. With the X-learner’s list, each extra person who acts costs 2.16 dollars instead of 3.23. For Larkfield, “an extra voter” becomes “an extra customer who orders”. Rafael can compare its cost with the course’s assumed margin of 26 dollars per order.
The chosen list on new input
Section titled “The chosen list on new input”A list is useful only if it works on households that the test never saw. The households that got the other three letters (60,003) stand in for “next year’s list”. Priya uses only their facts from before the mailing, never their outcomes. The X-learner, trained on all 119,999 households of the Neighbors test, scores them. She takes the top 20%.
final = fit_models(two)new = people[~people["letter"].isin(["No letter", "Neighbors"])].copy()new["score"] = final[best](new)plan = new.groupby("hh_id").agg(score=("score", "sum"), people=("voted", "size"))chosen = plan.sort_values("score", ascending=False, kind="stable").head(int(round(0.2 * len(plan))))60,003 households; chosen 12,001 letters to 30,095 people; expected extra voters 2696 (224.7 per 1,000 letters)What this tells us. The plan is 12,001 letters to 30,095 people: about 6,000 dollars at the assumed price. The model expects about 2,700 extra voters, 225 per 1,000 letters, close to the +231 measured with cross-fitting. An expectation is not a measurement: only the hold-back group of the next mailing can confirm it.
Insights and limits
Section titled “Insights and limits”- “Did it work?” was easy. “For whom?” was hard. The test measured each letter to within about ±0.65 points, but it could not rank three targeting models.
- There is no truth to check against. All the checks passed: the counts, the balance, the placebo outcome, the A/A splits and the placebo for the model. They show that the methods behave well. They do not prove an effect for any one person.
- The winner is a little lucky. The X-learner was the best of five lists on the same data. So its number is probably a little high.
- What a fair test of the targeting would need. The study randomised the letters, not the lists. To test a list, randomise households to “mailed by the model’s list” or “mailed at random”, with the same number of letters.
- One election, one state, 2006. It shows the methods, and how large the effect of one letter can be. It does not predict the effect of a shop’s letter.
- Real data is harder than the simulator. In Evaluate and use uplift, the simulator showed the X-learner’s true gain. Here we could only measure it, with a wide interval.
Responsible use
Section titled “Responsible use”- Real people are in this file. It has no names or addresses, but it has year of birth, sex, household and voting history. Do not try to identify anyone. Do not join the file with other data about people.
- Keep sensitive facts out of targeting. We left out
sex. Age is in the model, so check that a list does not leave out an age group without a reason you can defend. - Social pressure has a cost that the test did not measure. The Neighbors letter showed people’s voting records to their neighbours. The study measured turnout, not anger, complaints or lost trust. A shop that names or shames customers would lose them. A strong effect is not a reason to use a message.
- Not for elections. A model that chooses which citizens to encourage to vote raises questions of fairness and law. This course does not cover them. Use the methods for products and services.
- Credit the source. CC0 asks for nothing, but good practice does: cite the authors and the archive, as in the box above.
Make it a portfolio piece
Section titled “Make it a portfolio piece”This case study shows an employer that you can analyse a real experiment from the checks to a decision. It also shows that you know what the data cannot tell. Make it yours with one change, such as another letter or another budget. Then make it public and easy to read in five minutes.
What to put in the repository:
README.md, the page that people read first (template below).- The notebook, run from top to bottom, with its outputs.
PLAN.md: the plan from “The plan before the result”, written before the analysis.results.csv: each letter’s effect with both intervals, and each list’s measure with its interval.assumptions.md: the letter cost and the budget.- The data source and licence, with the citation above. CC0 allows a copy, but a link and the checksum are enough.
A README template:
# Did the mailing work, and for whom? A real randomised mailing
## ProblemAn organisation can mail one household in five. Which letter, and to whom?
## Data and licenceGerber, Green and Larimer, "Social Pressure and Voter Turnout", Michigan 2006(Yale ISPS Data Archive, doi:10.60600/YU/CGMWNW), CC0 1.0. 344,084 people in180,002 households; 4 letters and no letter, randomised by household.
## ApproachPlan written before reading the outcome. Checks: sample ratio, balance,placebo outcome, 1,000 A/A splits. Effects with intervals clustered by household.Six planned segments by past votes. Five household lists compared withcross-fitting (4 folds) and a paired bootstrap; a placebo test for the model.
## ResultsNeighbors letter +8.1 points (7.5 to 8.8); Civic duty +1.8. Naive intervalsgave 12.7% false winners in A/A splits; clustered 5.8%. Effect by past votes:+2.4 (never) to +9.9 (4 of 5). Top 20% by the X-learner: +231 extra voters per1,000 letters vs +155 at random; not clearly better than a "who will vote" model.
## LimitsOne election, 2006; no costs (0.50 dollars a letter assumed); the targeting wasnot randomised; social pressure has costs the study did not measure.How to describe it in an interview, in three sentences: “I analysed a real randomised mailing to 180,000 households; A/A splits showed that the intervals must be clustered by household. The best letter raised the action rate by 8 points, most among people who acted sometimes. An uplift model’s list brought about 50% more per letter than a random one, but no clear gain over a simple model. So I recommended a randomised test of the list.”
Go further:
- Repeat the segments and the lists for the Self letter. Is the hump the same?
- Plan the targeting test on paper: how many households would it need to see a difference of 30 extra voters per 1,000 letters? Use Sample size and power.
- Compare with the other case studies, Real tests at an online shop and A minimum wage without a test: where was randomisation available, and what did it make easy?
Your turn: one segment of the Self letter
Section titled “Your turn: one segment of the Self letter”Before you run it, predict: take the people who voted in 3 of the 5 earlier elections. Is the Self letter’s effect for them above or below its overall effect (+4.85 points)? In the notebook, section 5, estimate it with effect() and by_household(), and put it in self_3.
Show the answer
sub = people[people["past_votes"] == 3]self_3 = round(float(effect(by_household(sub, "voted"), "Self")["diff"]), 4)print(self_3)0.0556The Self letter adds 5.56 points in the “3 of 5” segment, above its overall +4.85. As with the Neighbors letter, people who vote sometimes respond more than average.
Knowledge check
Section titled “Knowledge check”Check your understanding
The file has 344,084 people, but Priya computes her intervals from 180,002 households. Why? What did the A/A splits show?
Show the answer
The coin was flipped per household: everyone at one address got the same letter. People who live together also tend to vote together, so they are not independent pieces of evidence. An interval that treats every person as independent is too narrow. The A/A splits checked this on real data. The splits divided the no-letter households at random 1,000 times, so nothing differed. The naive interval called 12.7% of them “significant”, against the promised 5%. The interval clustered by household called 5.8%. So the clustered interval keeps its promise, and the naive one finds about one false winner in eight.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
At Larkfield, the 'who will buy' model emailed the wrong customers. Here, the 'who will vote' model is almost as good as the X-learner. Explain the difference.
Show the answer
A response model ranks people by their chance to act, not by the change that the letter makes. That goes wrong when the most likely buyers are sure things or do-not-disturbs, as with Larkfield’s heavy buyers. Here, the planned segments show that the letter’s effect rises with past votes, up to 4 of 5 elections. So people who are likely to vote are mostly people whom the letter moves. Only the “always” group (4% of people) breaks this pattern, and no segment has a negative effect. In this data, the chance and the change point the same way. That is a fact about this test, not a rule: check it with an uplift curve every time.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
Rafael asks: 'The X-learner won. Can we use its list and stop testing?' What does Priya answer?
Show the answer
Not yet. The X-learner had the highest measure, +231 extra voters per 1,000 letters. But the paired bootstrap could not separate it from the “who will vote” model or the T-learner. It was also chosen as the best of five on the same data, so its number is probably a little high. Most important, the study randomised the letters, not the lists. To know what the list is worth, run a new test: mail households by the model’s list or at random, with the same number of letters. And keep a random hold-back in every mailing, so that each mailing measures its own effect.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Recap
- You wrote the plan before reading the outcome. Then you checked the test without it: the counts, the balance, a placebo outcome and 1,000 A/A splits.
- The household was the unit, so the intervals were clustered by household. Naive intervals gave 12.7% false winners in A/A splits; clustered ones 5.8%.
- Every letter worked. The Neighbors letter added +8.1 points (7.5 to 8.8), and the message mattered far more than the list.
- In planned segments, people who voted sometimes responded most (+9.5 to +9.9); “never” and “always” voters least.
- With cross-fitting, the X-learner’s top 20% brought +231 extra voters per 1,000 letters, against +155 at random. It was not clearly better than a “who will vote” model, so the next step is a randomised test of the list.
Next, the second case study: the daily results of 78 real online experiments at a fashion shop, with peeking and a sequential test.