Skip to content
Français

Propensity scores, matching and weighting

  • Lesson 22
  • 40 min
  • Module 6: Adjust for confounders in observational data

Where we are

Regression adjustment needs the right formula for the outcome. Another way is to model who got the treatment. In this lesson, we estimate each customer's chance of seeing the recommendation block, then match and weight customers, and check that the groups are balanced.

What you will be able to do

Model who used the feature: give each customer a propensity score. Use it to match users with similar non-users, or to weight the customers. Check that the groups are balanced and that they overlap. Know what these checks cannot see.

NotebookDo this lesson in a notebookColab · Kaggle · GitHub

In the notebook, you fit the propensity scores, measure balance, weight and match the customers, and see a mediator break the overlap. Its setup cells download the logs data and install the packages, so it needs no local setup. About 40 minutes.

Colab and Kaggle open your own copy. They need a free Google or Kaggle account. Stuck? See the solution notebook.

Before you startWhat to know and to have readyAbout 40 minutes

What you need to know

What you need on your computer (only for the project steps)

  • The project larkfield-experiments at the snapshot end-of-m06, from Regression adjustment, with .venv active.

Tested on macOS with Python 3.12 and the end-of-m06 snapshot of the project. Windows was not tested.

Go back to the small shop and its new shelf. The owner now has a hundred customers. She also knows more about each one: how often they come, how much they spent last year, whether they come by car.

She cannot sort them into “keen” and “casual” any more. There are too many kinds. So she asks a simpler question about each customer: “How likely was this person to stop at the shelf?” A regular who comes every day: very likely. A tourist in a hurry: not likely.

Then she pairs each customer who stopped with a customer who did not stop, but was just as likely to stop. Two people with the same chance are alike in the ways that matter for stopping. The difference in their spending is the shelf.

TermLarkfieldA made-up Canadian online shop for home and garden products. It is not a real company. All Nextia courses use its help desk and data as the example.Learn more Meet Larkfield’s logs have six confounders. Past spending alone takes thousands of values. With six columns, almost no two customers are exactly alike. You cannot make a table with one row for every kind of customer, as we did with ten customers.

Regression adjustment solved this with a formula for revenue. It assumes that each column adds a straight-line amount. If the real shape is different, the answer can be off, and the output does not warn you.

This lesson models the other side: who used the block. It is often easier to think about. Larkfield’s team knows why people click the block. They do not know exactly how spending depends on six columns.

The idea: one number for “how likely to use it”

Section titled “The idea: one number for “how likely to use it””

The propensity score of a customer is the chance that this customer uses the treatment, given the confounders. “Propensity” means a natural tendency. A logistic regression estimates it: a model whose output is a probability between 0 and 1.

The useful fact: two customers with the same score are, on average, alike in all the confounders that went into the model. So you can compare users and non-users with the same score, instead of the same six columns. Six columns become one number.

A flow from left to right. The confounders in the log (past spending, visits, tenure, new or returning, device) go into a model of who used the block (logistic regression). It gives each customer's propensity score: the chance to use it. From the score, two roads. Match: give each user the non-user with the closest score; this estimates the effect on the users, ATT. Or weight: rare customers count more, 1 divided by the score for users, 1 divided by 1 minus the score for non-users; this estimates the effect on everyone, ATE. Both roads lead to an orange box: check balance, are the groups now alike in every column of the log, and check overlap, does every kind of user also have non-users. A note below: the score sees only the columns in the log; a hidden column, the garden, stays unbalanced.
From the confounders to a propensity score, then two ways to use it: match or weight. Both need the same two checks. And both see only the columns in the log.

The same ten customers as in the last lesson. The numbers are made up for this example. Four of the five keen customers used the shelf. So a keen customer’s score is 4 / 5 = 0.8. One of the five casual customers used it. So a casual customer’s score is 0.2.

Matching. Give each user the non-user with the closest score:

User Score Spent Matched non-user Score Spent Difference
keen 1 to 4 0.8 60 keen 5 (reused) 0.8 57 +3 each
casual 6 0.2 23 casual 7 0.2 20 +3

The mean difference is +3 dollars. It is the effect on the users (ATT): each user is compared with a non-user like them. One non-user is used four times. That is allowed, and common.

Weighting. Instead of pairs, give each customer a weight: how much they count. Users count 1 / score. Non-users count 1 / (1 − score).

Customers Score Weight Total weight Spent
4 keen users 0.8 1 / 0.8 = 1.25 5 60
1 casual user 0.2 1 / 0.2 = 5 5 23
1 keen non-user 0.8 1 / 0.2 = 5 5 57
4 casual non-users 0.2 1 / 0.8 = 1.25 5 20
  • weighted users: (5 × 60 + 5 × 23) / 10 = 41.50 dollars;
  • weighted non-users: (5 × 57 + 5 × 20) / 10 = 38.50 dollars;
  • difference: +3.00 dollars.

The rare customers count more. The one casual user stands for all the casual customers who might have used the shelf. After weighting, each group is half keen and half casual, like the whole shop. So weighting estimates the effect on everyone (ATE). The method is called inverse probability weighting (IPW): “inverse” because the weight is 1 divided by the chance.

xp adjust runs the propensity model with the six confounders, then matching and weighting. Your project should be at end-of-m06, with the logs simulated.

Terminal window
sim run --scenario logs
xp adjust
xp check
Scenario logs (seed 40): 28 days of logs after the recommendation block's launch: who used it and what they spent (no test).
Wrote runs/logs-s40: customers (40,000 rows). Synthetic data. 0.3 s
Run logs-s40: effect of used_block on revenue (dollars per customer, 28 days).
Covariates: pre_spend, pre_orders, visits, tenure_days, new_customer, desktop
naive (users − non-users) +9.70
regression +3.38 (95% interval +1.74 to +5.03; ATE)
matching (nearest neighbour) +4.43 (95% interval +1.96 to +6.89; ATT)
weighting (IPW) +2.87 (95% interval +0.98 to +5.16; ATE)
Overlap: propensity scores 0.191–0.505 (users), 0.191–0.508 (non-users).
Balance (standardised mean difference; below 0.1 is good): pre_spend +0.08 → -0.01, pre_orders +0.10 → -0.00, visits +0.09 → +0.00, tenure_days +0.14 → -0.00, new_customer -0.15 → -0.00, desktop +0.15 → +0.00
Run logs-s40: the last analysis was `xp adjust`.
True effect (from the simulator): ATE +2.37, ATT +2.73 dollars per customer. Naive: +9.70.
regression: +3.38 against ATE +2.37: the interval contains it.
matching (nearest neighbour): +4.43 against ATT +2.73: the interval contains it.
weighting (IPW): +2.87 against ATE +2.37: the interval contains it.
Hidden from the log: has_garden (45% of customers): users are more likely to have a garden, and gardeners spend more. It is not in the log.

Matching gives +4.43 dollars (+1.96 to +6.89) against a true ATT of +2.73. Weighting gives +2.87 (+0.98 to +5.16) against a true ATE of +2.37. All three TermConfidence intervalA range around a measured value, made by a method that holds the true value 95 times in 100. If the interval of a difference does not include zero, the difference is probably not chance.Learn more Estimate uncertaintyEssential Mathematics and Statistics for AI contain their truth. All three estimates are above it.

Matching pairs all 13,338 users, with 7,994 different non-users. Weighting’s interval comes from a bootstrap: the tool repeats the estimate on 200 resamples of the customers, as in The result and its uncertainty.

Three adjusted estimates of the block's effect on the same simulated logs, against the truth. Above: this run, with dashed lines at the true ATE (left) and ATT (right). Matching estimates the effect on the users (ATT), so compare it with the ATT line. Below: the average error over 50 simulated logs. Every method stays too high, because the garden is missing. Matching is the noisiest.
Text description of the chart

Two panels. This run, seed 40, with lines at the true ATE, +2.37, and the true ATT, +2.73 dollars. Naive: +9.70. Regression, ATE: +3.38 (+1.74 to +5.03). Matching, ATT: +4.43 (+1.96 to +6.89). Weighting, ATE: +2.87 (+0.98 to +5.16). Bias over 50 seeds, with a line at no bias: regression +1.19 dollars (coverage 86%), matching +2.23 (coverage 76%), weighting +1.27 (coverage 90%).

Over 50 simulated logs, the bias stays: +1.19 dollars for regression, +2.23 for matching, +1.27 for weighting. The 95% intervals contained the truth in 86%, 76% and 90% of the runs. Matching with one partner per user is the noisiest of the three: each user’s answer rests on one other customer.

The propensity score is a means, not an end. The real question is: after matching or weighting, are users and non-users alike? This is balance.

Columns have different units, so we compare them with the standardised mean difference (SMD). It is the difference of the two means, divided by the column’s typical spread (its standard deviation). An SMD of 0 means equal means. Below 0.1 is usually called balanced.

A worked example from the log: users spent 51.77 dollars before the launch, and non-users 41.28. The difference is 10.49 dollars. The standard deviation of past spending is about 137 dollars, because a few customers spend a lot. So the SMD is 10.49 / 137 = 0.08.

After matching, the matched non-users spent 51.22 dollars before the launch: almost the same as the users. The chart shows every column.

Balance of the confounders before adjustment, after matching and after weighting, on the simulated logs. Matching and weighting bring all six columns of the log close to 0. The garden is not in the log, so its gap stays at about 0.18. Only the simulator can draw that last row.
Text description of the chart

Three panels with lines at 0 and 0.1. Before adjustment: pre_spend +0.08, pre_orders +0.10, visits +0.09, tenure_days +0.14, new_customer −0.15, desktop +0.15, and the hidden garden +0.18. After matching: pre_spend 0.00, pre_orders +0.04, visits +0.06, tenure_days −0.02, new_customer +0.01, desktop −0.05, garden +0.19. After weighting: pre_spend −0.01, pre_orders 0.00, visits 0.00, tenure_days 0.00, new_customer 0.00, desktop 0.00, garden +0.18.

Weighting balances the six columns almost perfectly. Matching leaves small gaps (visits +0.06), which adds noise. But look at the orange row. The garden’s gap does not move: 0.18 before, 0.19 after matching, 0.18 after weighting. Of the users, 50.8% have a garden; of the non-users, 42.1%.

That is the limit of balance. It can only check the columns you have. A perfect balance table says nothing about a column that is missing. The garden keeps its gap, and it pushes every estimate up.

Matching and weighting need a second condition. For every kind of user, there must be non-users of the same kind. This is overlap. If some users have a score of 0.99, almost nobody like them stayed away from the block. There is nobody to compare them with.

  • Users
  • Non-users
The propensity scores of users and non-users, with the six confounders. Both groups cover the same range, from 0.19 to 0.51. Users sit a little to the right: they were more likely to use the block. Every user has non-users with a similar score. Synthetic data, seed 40.
Text description of the chart

A chart of the propensity score from 0.16 to 0.54, in bands of 0.02, with the share of each group in each band. Both curves start near 0 at 0.18 and end near 0 at 0.50. Both have their highest point in the band 0.30 to 0.32: 17.9% of users and 20.1% of non-users. Users have a little more weight to the right: for example 13.6% of users and 10.5% of non-users are in the band 0.38 to 0.40.

The scores are not extreme. The six confounders explain only a little of who clicks the block. That is good for overlap: users and non-users mix well.

What if someone adds pages_viewed to the propensity model? The block adds pages, so pages predict block use very well. Too well.

Terminal window
xp adjust --add pages_viewed
Run logs-s40: effect of used_block on revenue (dollars per customer, 28 days).
Covariates: pre_spend, pre_orders, visits, tenure_days, new_customer, desktop, pages_viewed
naive (users − non-users) +9.70
regression +2.79 (95% interval +0.81 to +4.76; ATE)
matching (nearest neighbour) -2.01 (95% interval -4.53 to +0.52; ATT)
weighting (IPW) -282.02 (95% interval -313.35 to -59.44; ATE)
Overlap: propensity scores 0.064–1.000 (users), 0.056–1.000 (non-users).
Balance (standardised mean difference; below 0.1 is good): pre_spend +0.08 → -3.68, pre_orders +0.10 → -5.10, visits +0.09 → -3.97, tenure_days +0.14 → +0.85, new_customer -0.15 → -1.20, desktop +0.15 → +0.66, pages_viewed +0.35 → -11.35

The scores now reach 1.000. Weighting gives −282.02 dollars: absurd for customers who spend about 30 dollars a month. A non-user with a score of 0.99999 gets a weight of 1 / (1 − 0.99999), about 100,000. In the notebook, the largest weight is 204,860. A handful of customers decide the whole answer. The balance line shows it too: the SMDs explode to −3.68 and −11.35, far from 0.

  • Users
  • Non-users
The propensity scores with the mediator pages_viewed added. The groups now spread from 0.06 to 1.0, and a few customers in each group have scores above 0.99 (24 users, 27 non-users). The non-users among them get huge weights. Synthetic data, seed 40.
Text description of the chart

A chart of the propensity score from 0 to 1, in bands of 0.05. Non-users peak at 0.25 to 0.30 with 21.7% of the group. Users peak at 0.35 to 0.40 with 18.3%. Both groups have long thin tails to the right. From 0.65 to 1.0, each band holds under 1% of each group, but the tails reach 1.0.

There are two errors here. pages_viewed is a mediator, so it does not belong in the model at all. And any column that predicts the treatment almost perfectly breaks overlap. A better propensity model is not one that predicts better. It is one that balances the confounders.

Regression gave +3.38, matching +4.43 and weighting +2.87. They roughly agree, and all three are too high. Why do they agree? Because they use the same six columns. They differ in how they use them: a formula for revenue, pairs, or weights. They do not differ in what they can see.

So agreement between the three methods is a check of the method: did the formula, the pairs and the weights work? It is not a check of the data: is a confounder missing? The garden is missing from all three, so it biases all three in the same direction. The next lesson tries to break the estimate on purpose, and asks how big a missing confounder would need to be.

When to use it, when not, and what it costs

Section titled “When to use it, when not, and what it costs”

Use propensity scores when:

  • you have logs with the confounders, and no test;
  • you understand who takes the treatment better than how the outcome works;
  • you want to show balance: a table of SMDs convinces people more than a regression formula.

Do not trust it when:

  • the scores reach 0 or 1: there is no overlap, and the weights explode. Look at the score chart first;
  • you added columns that the treatment changes: mediators, colliders;
  • a confounder is missing: balance looks perfect, and the answer is still biased.

Matching estimates the effect on the users (ATT); weighting estimates the effect on everyone (ATE). Choose the question first. Matching with one partner is noisy; weighting is sensitive to extreme scores. Both cost little to compute. The real cost is checking: the overlap chart and the balance table, every time.

  1. Open the notebook from the box Do this lesson in a notebook at the top of this page.
  2. Run the setup cells and the helper cell. The notebook reads the same logs as sim run --scenario logs (seed 40).
  3. In section 1, fit the propensity scores. You see users: 0.191 to 0.505; non-users: 0.191 to 0.508.
  4. In section 2, measure the balance before adjustment. You see 'tenure_days': 0.14 and 'desktop': 0.15.
  5. In section 3, weight the customers. You see weighting: +2.87 dollars (ATE); truth +2.37, and every SMD near 0.
  6. In section 4, match the customers. You see matching: +4.43 dollars (ATT).
  7. In section 5, add pages_viewed to the propensity model and weight again. Put the estimate in ipw_mediator. Run the check cell. It prints Check passed.
Show the answer to step 7
ps_m = propensity(CONFOUNDERS + ["pages_viewed"])
w_m = np.where(users, 1 / ps_m, 1 / (1 - ps_m))
ipw_mediator = round(float((w_m[users] * y[users]).sum() / w_m[users].sum()
- (w_m[~users] * y[~users]).sum() / w_m[~users].sum()), 2)
print(f"largest score {ps_m.max():.4f}; largest weight {w_m.max():,.0f}; weighting {ipw_mediator:+.2f}")

The value is −282.02: the same as xp adjust --add pages_viewed.

Mateo, the growth analyst, asks: “Is seed 40 a lucky one? Run another month.” Simulate the logs with seed 41, then run xp adjust and xp check. Before you run them, predict: which of the three methods will move the most?

Show the answer
Terminal window
sim run --scenario logs --seed 41
xp adjust
xp check
Scenario logs (seed 41): 28 days of logs after the recommendation block's launch: who used it and what they spent (no test).
Wrote runs/logs-s41: customers (40,000 rows). Synthetic data. 0.9 s
Run logs-s41: effect of used_block on revenue (dollars per customer, 28 days).
Covariates: pre_spend, pre_orders, visits, tenure_days, new_customer, desktop
naive (users − non-users) +8.90
regression +2.66 (95% interval +0.98 to +4.35; ATE)
matching (nearest neighbour) +6.20 (95% interval +3.94 to +8.46; ATT)
weighting (IPW) +2.64 (95% interval +0.53 to +4.58; ATE)
Overlap: propensity scores 0.204–0.515 (users), 0.204–0.507 (non-users).
Balance (standardised mean difference; below 0.1 is good): pre_spend +0.08 → -0.00, pre_orders +0.09 → -0.01, visits +0.09 → -0.01, tenure_days +0.13 → -0.00, new_customer -0.15 → +0.00, desktop +0.15 → +0.00
Run logs-s41: the last analysis was `xp adjust`.
True effect (from the simulator): ATE +2.16, ATT +2.46 dollars per customer. Naive: +8.90.
regression: +2.66 against ATE +2.16: the interval contains it.
matching (nearest neighbour): +6.20 against ATT +2.46: the interval does NOT contain it.
weighting (IPW): +2.64 against ATE +2.16: the interval contains it.
Hidden from the log: has_garden (45% of customers): users are more likely to have a garden, and gardeners spend more. It is not in the log.

Matching moves the most: +6.20 dollars against a true ATT of +2.46, and its interval misses the truth. Regression (+2.66) and weighting (+2.64) stay about 0.5 dollars above the true ATE of +2.16. Each user’s matched partner is one customer, so matching is noisy. It also carries the garden’s bias. So the course measured each method over 50 seeds. One run cannot show a method’s bias or its noise.

The next xp commands use the newest logs run. To go back to seed 40, run sim run --scenario logs again.

Common mistake

“Our propensity model is 99% accurate”

Symptom: a team builds a very good model of who used a feature, with every column it has. The model is almost always right. Then the weighted estimate is huge and negative, and the interval is enormous.

Cause: the model used columns that the feature itself changes, such as pages viewed. They predict the treatment almost perfectly, so the scores reach 0 and 1. Customers with a score near 1 who did not use the feature get weights in the thousands. Overlap is gone: some users have no comparable non-users.

Fix: put only the confounders into the propensity model: causes of the treatment and the outcome, measured before the treatment. Plot the scores of both groups and check their overlap. Check balance after weighting. A propensity model is good when it balances the groups, not when it predicts well.

Check your understanding

A keen customer has a propensity score of 0.75. What weight does she get if she used the shelf? And if she did not?

Show the answer

If she used it: 1 / 0.75 = 1.33. If she did not: 1 / (1 − 0.75) = 1 / 0.25 = 4. A keen non-user is rare, so she counts more: she stands for the keen customers who might have stayed away.

Check your understanding

After weighting, every confounder of Larkfield's log has a standardised mean difference near 0. Does that prove the estimate (+2.87 dollars) is right?

Show the answer

No. You can check balance only for the columns in the log. The garden is missing, and its gap stays at about 0.18 after weighting. Gardeners use the block more and spend more, so the estimate stays too high: +2.87 against a true +2.37. Over 50 seeds, weighting was on average 1.27 dollars too high.

Check your understanding

Regression (+3.38), matching (+4.43) and weighting (+2.87) roughly agree. Explain why they agree, and why they could still all be wrong.

Show the answer

They agree because they use the same six confounders: they differ only in how they use them (a formula for revenue, pairs, or weights). So their agreement checks the method, not the data. If a confounder is missing from the log, like the garden, all three miss it, and all three are biased in the same direction. Here all three are above the truth.

Recap

  • A propensity score is each customer’s chance of using the treatment, given the confounders. Customers with the same score are alike in those confounders.
  • Matching pairs each user with the closest non-user: the effect on the users (ATT). Weighting (IPW) counts rare customers more: the effect on everyone (ATE).
  • On Larkfield’s logs: matching +4.43 (true ATT +2.73), weighting +2.87 (true ATE +2.37). Over 50 seeds, bias +2.23 and +1.27 dollars.
  • Check balance (SMD below 0.1) and overlap (both groups in the same score range) every time. A mediator in the model broke overlap: scores reached 1.0 and weighting gave −282.02.
  • Balance cannot see a missing column: the garden’s gap stayed at 0.18. Three methods that agree can all be wrong together.

Next, we try to break our own answer, and ask how strong a hidden confounder would need to be.

Next: Test your own answer