Propensity scores, matching and weighting
- Lesson 22
- 40 min
- Module 6: Adjust for confounders in observational data
Where we are
Regression adjustment needs the right formula for the outcome. Another way is to model who got the treatment. In this lesson, we estimate each customer's chance of seeing the recommendation block, then match and weight customers, and check that the groups are balanced.
Lesson 22 of 30 · Experiments and Causal Impact: Did Our Change Work?Propensity scores, matching and weighting
1. Why "it went up" is not an answer
2. Design an A/B test
3. Analyse the test
4. Traps in real tests
5. When you cannot randomise: natural experiments
6. Adjust for confounders in observational data
7. Uplift: who should get the treatment?
8. Experiments for AI features and in production
What you will be able to do
Model who used the feature: give each customer a propensity score. Use it to match users with similar non-users, or to weight the customers. Check that the groups are balanced and that they overlap. Know what these checks cannot see.
NotebookDo this lesson in a notebook
In the notebook, you fit the propensity scores, measure balance, weight and match the customers, and see a mediator break the overlap. Its setup cells download the logs data and install the packages, so it needs no local setup. About 40 minutes.
Colab and Kaggle open your own copy. They need a free Google or Kaggle account. Stuck? See the solution notebook.
Before you startWhat to know and to have ready
What you need to know
- Regression adjustment, the six confounders, ATE and ATT, from Regression adjustment.
- A probability between 0 and 1. For a refresher, see Events and uncertainty in the mathematics course.
What you need on your computer (only for the project steps)
- The project
larkfield-experimentsat the snapshotend-of-m06, from Regression adjustment, with.venvactive.
Tested on macOS with Python 3.12 and the end-of-m06 snapshot of the project. Windows was not tested.
Fair pairs at the small shop
Section titled “Fair pairs at the small shop”Go back to the small shop and its new shelf. The owner now has a hundred customers. She also knows more about each one: how often they come, how much they spent last year, whether they come by car.
She cannot sort them into “keen” and “casual” any more. There are too many kinds. So she asks a simpler question about each customer: “How likely was this person to stop at the shelf?” A regular who comes every day: very likely. A tourist in a hurry: not likely.
Then she pairs each customer who stopped with a customer who did not stop, but was just as likely to stop. Two people with the same chance are alike in the ways that matter for stopping. The difference in their spending is the shelf.
The problem: too many kinds of customer
Section titled “The problem: too many kinds of customer”TermLarkfieldA made-up Canadian online shop for home and garden products. It is not a real company. All Nextia courses use its help desk and data as the example.Learn more Meet Larkfield’s logs have six confounders. Past spending alone takes thousands of values. With six columns, almost no two customers are exactly alike. You cannot make a table with one row for every kind of customer, as we did with ten customers.
Regression adjustment solved this with a formula for revenue. It assumes that each column adds a straight-line amount. If the real shape is different, the answer can be off, and the output does not warn you.
This lesson models the other side: who used the block. It is often easier to think about. Larkfield’s team knows why people click the block. They do not know exactly how spending depends on six columns.
The idea: one number for “how likely to use it”
Section titled “The idea: one number for “how likely to use it””The propensity score of a customer is the chance that this customer uses the treatment, given the confounders. “Propensity” means a natural tendency. A logistic regression estimates it: a model whose output is a probability between 0 and 1.
The useful fact: two customers with the same score are, on average, alike in all the confounders that went into the model. So you can compare users and non-users with the same score, instead of the same six columns. Six columns become one number.
A tiny example with every value visible
Section titled “A tiny example with every value visible”The same ten customers as in the last lesson. The numbers are made up for this example. Four of the five keen customers used the shelf. So a keen customer’s score is 4 / 5 = 0.8. One of the five casual customers used it. So a casual customer’s score is 0.2.
Matching. Give each user the non-user with the closest score:
| User | Score | Spent | Matched non-user | Score | Spent | Difference |
|---|---|---|---|---|---|---|
| keen 1 to 4 | 0.8 | 60 | keen 5 (reused) | 0.8 | 57 | +3 each |
| casual 6 | 0.2 | 23 | casual 7 | 0.2 | 20 | +3 |
The mean difference is +3 dollars. It is the effect on the users (ATT): each user is compared with a non-user like them. One non-user is used four times. That is allowed, and common.
Weighting. Instead of pairs, give each customer a weight: how much they count. Users count 1 / score. Non-users count 1 / (1 − score).
| Customers | Score | Weight | Total weight | Spent |
|---|---|---|---|---|
| 4 keen users | 0.8 | 1 / 0.8 = 1.25 | 5 | 60 |
| 1 casual user | 0.2 | 1 / 0.2 = 5 | 5 | 23 |
| 1 keen non-user | 0.8 | 1 / 0.2 = 5 | 5 | 57 |
| 4 casual non-users | 0.2 | 1 / 0.8 = 1.25 | 5 | 20 |
- weighted users: (5 × 60 + 5 × 23) / 10 = 41.50 dollars;
- weighted non-users: (5 × 57 + 5 × 20) / 10 = 38.50 dollars;
- difference: +3.00 dollars.
The rare customers count more. The one casual user stands for all the casual customers who might have used the shelf. After weighting, each group is half keen and half casual, like the whole shop. So weighting estimates the effect on everyone (ATE). The method is called inverse probability weighting (IPW): “inverse” because the weight is 1 divided by the chance.
Larkfield: match and weight
Section titled “Larkfield: match and weight”xp adjust runs the propensity model with the six confounders, then matching and weighting. Your project should be at end-of-m06, with the logs simulated.
sim run --scenario logsxp adjustxp checkScenario logs (seed 40): 28 days of logs after the recommendation block's launch: who used it and what they spent (no test).Wrote runs/logs-s40: customers (40,000 rows). Synthetic data. 0.3 sRun logs-s40: effect of used_block on revenue (dollars per customer, 28 days). Covariates: pre_spend, pre_orders, visits, tenure_days, new_customer, desktop naive (users − non-users) +9.70 regression +3.38 (95% interval +1.74 to +5.03; ATE) matching (nearest neighbour) +4.43 (95% interval +1.96 to +6.89; ATT) weighting (IPW) +2.87 (95% interval +0.98 to +5.16; ATE) Overlap: propensity scores 0.191–0.505 (users), 0.191–0.508 (non-users). Balance (standardised mean difference; below 0.1 is good): pre_spend +0.08 → -0.01, pre_orders +0.10 → -0.00, visits +0.09 → +0.00, tenure_days +0.14 → -0.00, new_customer -0.15 → -0.00, desktop +0.15 → +0.00Run logs-s40: the last analysis was `xp adjust`.True effect (from the simulator): ATE +2.37, ATT +2.73 dollars per customer. Naive: +9.70. regression: +3.38 against ATE +2.37: the interval contains it. matching (nearest neighbour): +4.43 against ATT +2.73: the interval contains it. weighting (IPW): +2.87 against ATE +2.37: the interval contains it.Hidden from the log: has_garden (45% of customers): users are more likely to have a garden, and gardeners spend more. It is not in the log.Matching gives +4.43 dollars (+1.96 to +6.89) against a true ATT of +2.73. Weighting gives +2.87 (+0.98 to +5.16) against a true ATE of +2.37. All three TermConfidence intervalA range around a measured value, made by a method that holds the true value 95 times in 100. If the interval of a difference does not include zero, the difference is probably not chance.Learn more Estimate uncertaintyEssential Mathematics and Statistics for AI contain their truth. All three estimates are above it.
Matching pairs all 13,338 users, with 7,994 different non-users. Weighting’s interval comes from a bootstrap: the tool repeats the estimate on 200 resamples of the customers, as in The result and its uncertainty.
Text description of the chart
Two panels. This run, seed 40, with lines at the true ATE, +2.37, and the true ATT, +2.73 dollars. Naive: +9.70. Regression, ATE: +3.38 (+1.74 to +5.03). Matching, ATT: +4.43 (+1.96 to +6.89). Weighting, ATE: +2.87 (+0.98 to +5.16). Bias over 50 seeds, with a line at no bias: regression +1.19 dollars (coverage 86%), matching +2.23 (coverage 76%), weighting +1.27 (coverage 90%).
Over 50 simulated logs, the bias stays: +1.19 dollars for regression, +2.23 for matching, +1.27 for weighting. The 95% intervals contained the truth in 86%, 76% and 90% of the runs. Matching with one partner per user is the noisiest of the three: each user’s answer rests on one other customer.
Check balance
Section titled “Check balance”The propensity score is a means, not an end. The real question is: after matching or weighting, are users and non-users alike? This is balance.
Columns have different units, so we compare them with the standardised mean difference (SMD). It is the difference of the two means, divided by the column’s typical spread (its standard deviation). An SMD of 0 means equal means. Below 0.1 is usually called balanced.
A worked example from the log: users spent 51.77 dollars before the launch, and non-users 41.28. The difference is 10.49 dollars. The standard deviation of past spending is about 137 dollars, because a few customers spend a lot. So the SMD is 10.49 / 137 = 0.08.
After matching, the matched non-users spent 51.22 dollars before the launch: almost the same as the users. The chart shows every column.
Text description of the chart
Three panels with lines at 0 and 0.1. Before adjustment: pre_spend +0.08, pre_orders +0.10, visits +0.09, tenure_days +0.14, new_customer −0.15, desktop +0.15, and the hidden garden +0.18. After matching: pre_spend 0.00, pre_orders +0.04, visits +0.06, tenure_days −0.02, new_customer +0.01, desktop −0.05, garden +0.19. After weighting: pre_spend −0.01, pre_orders 0.00, visits 0.00, tenure_days 0.00, new_customer 0.00, desktop 0.00, garden +0.18.
Weighting balances the six columns almost perfectly. Matching leaves small gaps (visits +0.06), which adds noise. But look at the orange row. The garden’s gap does not move: 0.18 before, 0.19 after matching, 0.18 after weighting. Of the users, 50.8% have a garden; of the non-users, 42.1%.
That is the limit of balance. It can only check the columns you have. A perfect balance table says nothing about a column that is missing. The garden keeps its gap, and it pushes every estimate up.
Check overlap
Section titled “Check overlap”Matching and weighting need a second condition. For every kind of user, there must be non-users of the same kind. This is overlap. If some users have a score of 0.99, almost nobody like them stayed away from the block. There is nobody to compare them with.
- Users
- Non-users
Text description of the chart
A chart of the propensity score from 0.16 to 0.54, in bands of 0.02, with the share of each group in each band. Both curves start near 0 at 0.18 and end near 0 at 0.50. Both have their highest point in the band 0.30 to 0.32: 17.9% of users and 20.1% of non-users. Users have a little more weight to the right: for example 13.6% of users and 10.5% of non-users are in the band 0.38 to 0.40.
The scores are not extreme. The six confounders explain only a little of who clicks the block. That is good for overlap: users and non-users mix well.
A mediator breaks overlap
Section titled “A mediator breaks overlap”What if someone adds pages_viewed to the propensity model? The block adds pages, so pages predict block use very well. Too well.
xp adjust --add pages_viewedRun logs-s40: effect of used_block on revenue (dollars per customer, 28 days). Covariates: pre_spend, pre_orders, visits, tenure_days, new_customer, desktop, pages_viewed naive (users − non-users) +9.70 regression +2.79 (95% interval +0.81 to +4.76; ATE) matching (nearest neighbour) -2.01 (95% interval -4.53 to +0.52; ATT) weighting (IPW) -282.02 (95% interval -313.35 to -59.44; ATE) Overlap: propensity scores 0.064–1.000 (users), 0.056–1.000 (non-users). Balance (standardised mean difference; below 0.1 is good): pre_spend +0.08 → -3.68, pre_orders +0.10 → -5.10, visits +0.09 → -3.97, tenure_days +0.14 → +0.85, new_customer -0.15 → -1.20, desktop +0.15 → +0.66, pages_viewed +0.35 → -11.35The scores now reach 1.000. Weighting gives −282.02 dollars: absurd for customers who spend about 30 dollars a month. A non-user with a score of 0.99999 gets a weight of 1 / (1 − 0.99999), about 100,000. In the notebook, the largest weight is 204,860. A handful of customers decide the whole answer. The balance line shows it too: the SMDs explode to −3.68 and −11.35, far from 0.
- Users
- Non-users
Text description of the chart
A chart of the propensity score from 0 to 1, in bands of 0.05. Non-users peak at 0.25 to 0.30 with 21.7% of the group. Users peak at 0.35 to 0.40 with 18.3%. Both groups have long thin tails to the right. From 0.65 to 1.0, each band holds under 1% of each group, but the tails reach 1.0.
There are two errors here. pages_viewed is a mediator, so it does not belong in the model at all. And any column that predicts the treatment almost perfectly breaks overlap. A better propensity model is not one that predicts better. It is one that balances the confounders.
Three methods, one blind spot
Section titled “Three methods, one blind spot”Regression gave +3.38, matching +4.43 and weighting +2.87. They roughly agree, and all three are too high. Why do they agree? Because they use the same six columns. They differ in how they use them: a formula for revenue, pairs, or weights. They do not differ in what they can see.
So agreement between the three methods is a check of the method: did the formula, the pairs and the weights work? It is not a check of the data: is a confounder missing? The garden is missing from all three, so it biases all three in the same direction. The next lesson tries to break the estimate on purpose, and asks how big a missing confounder would need to be.
When to use it, when not, and what it costs
Section titled “When to use it, when not, and what it costs”Use propensity scores when:
- you have logs with the confounders, and no test;
- you understand who takes the treatment better than how the outcome works;
- you want to show balance: a table of SMDs convinces people more than a regression formula.
Do not trust it when:
- the scores reach 0 or 1: there is no overlap, and the weights explode. Look at the score chart first;
- you added columns that the treatment changes: mediators, colliders;
- a confounder is missing: balance looks perfect, and the answer is still biased.
Matching estimates the effect on the users (ATT); weighting estimates the effect on everyone (ATE). Choose the question first. Matching with one partner is noisy; weighting is sensitive to extreme scores. Both cost little to compute. The real cost is checking: the overlap chart and the balance table, every time.
Guided practice: score, weight and match
Section titled “Guided practice: score, weight and match”- Open the notebook from the box Do this lesson in a notebook at the top of this page.
- Run the setup cells and the helper cell. The notebook reads the same logs as
sim run --scenario logs(seed 40). - In section 1, fit the propensity scores. You see
users: 0.191 to 0.505; non-users: 0.191 to 0.508. - In section 2, measure the balance before adjustment. You see
'tenure_days': 0.14and'desktop': 0.15. - In section 3, weight the customers. You see
weighting: +2.87 dollars (ATE); truth +2.37, and every SMD near 0. - In section 4, match the customers. You see
matching: +4.43 dollars (ATT). - In section 5, add
pages_viewedto the propensity model and weight again. Put the estimate inipw_mediator. Run the check cell. It printsCheck passed.
Show the answer to step 7
ps_m = propensity(CONFOUNDERS + ["pages_viewed"])w_m = np.where(users, 1 / ps_m, 1 / (1 - ps_m))ipw_mediator = round(float((w_m[users] * y[users]).sum() / w_m[users].sum() - (w_m[~users] * y[~users]).sum() / w_m[~users].sum()), 2)print(f"largest score {ps_m.max():.4f}; largest weight {w_m.max():,.0f}; weighting {ipw_mediator:+.2f}")The value is −282.02: the same as xp adjust --add pages_viewed.
Your turn: another month of logs
Section titled “Your turn: another month of logs”Mateo, the growth analyst, asks: “Is seed 40 a lucky one? Run another month.” Simulate the logs with seed 41, then run xp adjust and xp check. Before you run them, predict: which of the three methods will move the most?
Show the answer
sim run --scenario logs --seed 41xp adjustxp checkScenario logs (seed 41): 28 days of logs after the recommendation block's launch: who used it and what they spent (no test).Wrote runs/logs-s41: customers (40,000 rows). Synthetic data. 0.9 sRun logs-s41: effect of used_block on revenue (dollars per customer, 28 days). Covariates: pre_spend, pre_orders, visits, tenure_days, new_customer, desktop naive (users − non-users) +8.90 regression +2.66 (95% interval +0.98 to +4.35; ATE) matching (nearest neighbour) +6.20 (95% interval +3.94 to +8.46; ATT) weighting (IPW) +2.64 (95% interval +0.53 to +4.58; ATE) Overlap: propensity scores 0.204–0.515 (users), 0.204–0.507 (non-users). Balance (standardised mean difference; below 0.1 is good): pre_spend +0.08 → -0.00, pre_orders +0.09 → -0.01, visits +0.09 → -0.01, tenure_days +0.13 → -0.00, new_customer -0.15 → +0.00, desktop +0.15 → +0.00Run logs-s41: the last analysis was `xp adjust`.True effect (from the simulator): ATE +2.16, ATT +2.46 dollars per customer. Naive: +8.90. regression: +2.66 against ATE +2.16: the interval contains it. matching (nearest neighbour): +6.20 against ATT +2.46: the interval does NOT contain it. weighting (IPW): +2.64 against ATE +2.16: the interval contains it.Hidden from the log: has_garden (45% of customers): users are more likely to have a garden, and gardeners spend more. It is not in the log.Matching moves the most: +6.20 dollars against a true ATT of +2.46, and its interval misses the truth. Regression (+2.66) and weighting (+2.64) stay about 0.5 dollars above the true ATE of +2.16. Each user’s matched partner is one customer, so matching is noisy. It also carries the garden’s bias. So the course measured each method over 50 seeds. One run cannot show a method’s bias or its noise.
The next xp commands use the newest logs run. To go back to seed 40, run sim run --scenario logs again.
Common mistake
“Our propensity model is 99% accurate”
Symptom: a team builds a very good model of who used a feature, with every column it has. The model is almost always right. Then the weighted estimate is huge and negative, and the interval is enormous.
Cause: the model used columns that the feature itself changes, such as pages viewed. They predict the treatment almost perfectly, so the scores reach 0 and 1. Customers with a score near 1 who did not use the feature get weights in the thousands. Overlap is gone: some users have no comparable non-users.
Fix: put only the confounders into the propensity model: causes of the treatment and the outcome, measured before the treatment. Plot the scores of both groups and check their overlap. Check balance after weighting. A propensity model is good when it balances the groups, not when it predicts well.
Check your understanding
A keen customer has a propensity score of 0.75. What weight does she get if she used the shelf? And if she did not?
Show the answer
If she used it: 1 / 0.75 = 1.33. If she did not: 1 / (1 − 0.75) = 1 / 0.25 = 4. A keen non-user is rare, so she counts more: she stands for the keen customers who might have stayed away.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
After weighting, every confounder of Larkfield's log has a standardised mean difference near 0. Does that prove the estimate (+2.87 dollars) is right?
Show the answer
No. You can check balance only for the columns in the log. The garden is missing, and its gap stays at about 0.18 after weighting. Gardeners use the block more and spend more, so the estimate stays too high: +2.87 against a true +2.37. Over 50 seeds, weighting was on average 1.27 dollars too high.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
Regression (+3.38), matching (+4.43) and weighting (+2.87) roughly agree. Explain why they agree, and why they could still all be wrong.
Show the answer
They agree because they use the same six confounders: they differ only in how they use them (a formula for revenue, pairs, or weights). So their agreement checks the method, not the data. If a confounder is missing from the log, like the garden, all three miss it, and all three are biased in the same direction. Here all three are above the truth.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Recap
- A propensity score is each customer’s chance of using the treatment, given the confounders. Customers with the same score are alike in those confounders.
- Matching pairs each user with the closest non-user: the effect on the users (ATT). Weighting (IPW) counts rare customers more: the effect on everyone (ATE).
- On Larkfield’s logs: matching +4.43 (true ATT +2.73), weighting +2.87 (true ATE +2.37). Over 50 seeds, bias +2.23 and +1.27 dollars.
- Check balance (SMD below 0.1) and overlap (both groups in the same score range) every time. A mediator in the model broke overlap: scores reached 1.0 and weighting gave −282.02.
- Balance cannot see a missing column: the garden’s gap stayed at 0.18. Three methods that agree can all be wrong together.
Next, we try to break our own answer, and ask how strong a hidden confounder would need to be.
Next: Test your own answer