Regression adjustment
- Lesson 21
- 35 min
- Module 6: Adjust for confounders in observational data
Where we are
Natural experiments need a lucky design. Often we only have logs: who used a feature, and what they spent. In this lesson, we add the confounders from our causal graph to a regression, and we see what a missing confounder and a mediator do to the answer.
Lesson 21 of 30 · Experiments and Causal Impact: Did Our Change Work?Regression adjustment
1. Why "it went up" is not an answer
2. Design an A/B test
3. Analyse the test
4. Traps in real tests
5. When you cannot randomise: natural experiments
6. Adjust for confounders in observational data
7. Uplift: who should get the treatment?
8. Experiments for AI features and in production
What you will be able to do
Estimate the effect of a feature from logs, when nobody chose at random who used it. Add the confounders to a regression, so that you compare like with like. Learn which columns belong in the regression, and which ones break it.
NotebookDo this lesson in a notebook
In the notebook, you fit the naive answer as a regression, add the confounders, leave one out, and add a mediator and a collider. Its setup cells download the logs data and install the packages, so it needs no local setup. About 35 minutes.
Colab and Kaggle open your own copy. They need a free Google or Kaggle account. Stuck? See the solution notebook.
Before you startWhat to know and to have ready
What you need to know
- Confounders, mediators and colliders, from Draw the causes.
- Why users and non-users of the block differ, from Causes and comparisons.
- A straight line with a slope. For a refresher, see Functions and change in the mathematics course.
- A 95% interval, from The result and its uncertainty.
What you need on your computer (only for the project steps)
- The project
larkfield-experimentsat the snapshotend-of-m05, from Difference-in-differences, with.venvactive. This lesson updates it toend-of-m06.
Tested on macOS with Python 3.12 and the end-of-m06 snapshot of the project. Windows was not tested.
A new shelf in a small shop
Section titled “A new shelf in a small shop”A small shop puts a new shelf near the door: “Our favourites this month”. At the end of the month, the owner looks at the till receipts. Customers who stopped at the shelf spent 25 dollars more than customers who walked past it.
Did the shelf bring 25 dollars per customer? The owner is not sure. The regulars stop at everything. They know the shop, they have time, and they buy a lot anyway. The people who walk past are often visitors in a hurry.
So the owner does a fairer thing. She compares regulars who stopped with regulars who did not. Then she compares visitors who stopped with visitors who did not. Each comparison is between similar people. The difference that is left is the shelf.
That is the whole idea of this lesson. The rest is how to do it with many columns at once, and which columns to leave out.
The problem: users of the block spent 9.70 dollars more
Section titled “The problem: users of the block spent 9.70 dollars more”TermLarkfieldA made-up Canadian online shop for home and garden products. It is not a real company. All Nextia courses use its help desk and data as the example.Learn more Meet Larkfield launched its recommendation block for everyone on 6 October 2025. There was no test. Customers chose to click the block or not. The logs scenario of the simulator has 28 days of these logs, for 40,000 customers. All of it is synthetic.
TermOmarLarkfield’s product manager. He asks the business questions: what a change costs, and what it gives the shop., the product manager, reads the dashboard. The 13,338 customers who used the block spent 39.19 dollars each. The 26,662 who did not spent 29.48 dollars. The difference is +9.70 dollars. “The block brings almost 10 dollars per customer,” Omar says.
You saw in Causes and comparisons why this number is too big. The users were different before the block existed. They spent 51.77 dollars in the 8 weeks before the launch, against 41.28 for the others. They visit more often. More of them use a computer. These columns are confounders: they cause both the use of the block and the spending.
The simulator knows the truth. If every customer used the block, each would spend +2.37 dollars more on average. For the customers who actually used it, the effect is +2.73 dollars. The naive +9.70 is more than three times too big. The gap of +9.70 against +2.73 is a practice problem we made on purpose.
A coin flip would fix this. But the block is already live for everyone. Larkfield has only the logs. Can the logs give a fair answer?
The idea: compare like with like
Section titled “The idea: compare like with like”Regression adjustment compares users and non-users who are alike in the confounders. “Adjust for past spending” means: compare customers with the same past spending. A regression does this for many columns at once.
A tiny example with every value visible
Section titled “A tiny example with every value visible”Ten customers of the small shop. The numbers are made up for this example. Four of the five keen customers used the shelf. Only one of the five casual customers did.
| Customer | Kind | Used the shelf? | Spent (dollars) |
|---|---|---|---|
| 1 to 4 | keen | yes | 60 each |
| 5 | keen | no | 57 |
| 6 | casual | yes | 23 |
| 7 to 10 | casual | no | 20 each |
The naive answer:
- users: (4 × 60 + 23) / 5 = 263 / 5 = 52.60 dollars;
- non-users: (57 + 4 × 20) / 5 = 137 / 5 = 27.40 dollars;
- difference: 52.60 − 27.40 = +25.20 dollars.
Now compare like with like:
- keen users against the keen non-user: 60 − 57 = +3 dollars;
- the casual user against casual non-users: 23 − 20 = +3 dollars.
The shelf adds 3 dollars in both kinds. The other 22.20 dollars of the naive answer came from who used the shelf, not from the shelf. The kind of customer is the confounder: keen customers use the shelf more, and they spend more.
What the regression does
Section titled “What the regression does”With 40,000 customers and six confounders, you cannot make a table for every kind of customer. A regression does the comparison for you. It fits one equation:
revenue = a + b × used_block + c × pre_spend + d × visits + … (one term for each confounder)Each letter is a number that the regression finds from the data. The number b is the difference in revenue between users and non-users with the same values of every other column. That is the adjusted effect. Write this equation for the small shop, with one column keen (1 or 0). The regression finds b = +3.00 dollars, exactly as the table did.
The regression assumes a simple shape: each column adds a straight-line amount, and the block adds the same b for everyone. When the true shape is not a straight line, the answer can be a little off.
Two kinds of effect: ATE and ATT
Section titled “Two kinds of effect: ATE and ATT”Our tools print two names for the effect. They answer two different questions.
- ATE, the average treatment effect: the average change if everyone used the block, against nobody. At Larkfield, the truth is +2.37 dollars.
- ATT, the average treatment effect on the treated: the average change for the customers who used it. The truth is +2.73 dollars.
The two differ because the users are more active customers. The block raises each customer’s chance to order on each visit, so it helps a customer who visits more. A regression like ours estimates the ATE.
Larkfield: adjust the logs
Section titled “Larkfield: adjust the logs”First, update the project to the snapshot end-of-m06. It adds xp adjust (this lesson and the next) and xp refute (the third lesson). It also adds two packages: scikit-learn and DoWhy. Your own runs and plans stay.
-
Open a terminal in the folder that holds
larkfield-experimentsandnextia-learning-labs. If you cloned the labs repository with Git, update it first.Terminal window git -C nextia-learning-labs pull -
Copy the snapshot over your project, and install the new packages.
Terminal window cp -R nextia-learning-labs/C40/snapshots/end-of-m06/. larkfield-experiments/cd larkfield-experimentssource .venv/bin/activatepython -m pip install -r requirements.txtTerminal window Copy-Item -Recurse -Force nextia-learning-labs\C40\snapshots\end-of-m06\* larkfield-experimentscd larkfield-experiments.venv\Scripts\Activate.ps1python -m pip install -r requirements.txtDoWhy brings several other packages, so the install takes a few minutes.
-
Run the tests.
Terminal window python -m pytest -q.......................... [100%]29 passed in 5.51sTwenty-nine tests pass. Your time will differ.
-
Simulate the logs.
Terminal window sim run --scenario logsScenario logs (seed 40): 28 days of logs after the recommendation block's launch: who used it and what they spent (no test).Wrote runs/logs-s40: customers (40,000 rows). Synthetic data. 0.5 s -
Adjust for the six confounders, then compare with the truth.
Terminal window xp adjustxp checkRun logs-s40: effect of used_block on revenue (dollars per customer, 28 days).Covariates: pre_spend, pre_orders, visits, tenure_days, new_customer, desktopnaive (users − non-users) +9.70regression +3.38 (95% interval +1.74 to +5.03; ATE)matching (nearest neighbour) +4.43 (95% interval +1.96 to +6.89; ATT)weighting (IPW) +2.87 (95% interval +0.98 to +5.16; ATE)Overlap: propensity scores 0.191–0.505 (users), 0.191–0.508 (non-users).Balance (standardised mean difference; below 0.1 is good): pre_spend +0.08 → -0.01, pre_orders +0.10 → -0.00, visits +0.09 → +0.00, tenure_days +0.14 → -0.00, new_customer -0.15 → -0.00, desktop +0.15 → +0.00Run logs-s40: the last analysis was `xp adjust`.True effect (from the simulator): ATE +2.37, ATT +2.73 dollars per customer. Naive: +9.70.regression: +3.38 against ATE +2.37: the interval contains it.matching (nearest neighbour): +4.43 against ATT +2.73: the interval contains it.weighting (IPW): +2.87 against ATE +2.37: the interval contains it.Hidden from the log: has_garden (45% of customers): users are more likely to have a garden, and gardeners spend more. It is not in the log.
Look at the regression line. The six confounders bring the answer from +9.70 down to +3.38 dollars (+1.74 to +5.03). That is much closer to the true +2.37. The TermConfidence intervalA range around a measured value, made by a method that holds the true value 95 times in 100. If the interval of a difference does not include zero, the difference is probably not chance.Learn more Estimate uncertaintyEssential Mathematics and Statistics for AI contains the truth.
But +3.38 is still about 1 dollar too high. The last line of xp check says why: whether a customer has a garden. Gardeners use the block more, and they spend more. So the garden is a confounder too. Larkfield never recorded it, so no regression can adjust for it.
The other lines of xp adjust are for the next lesson: matching, weighting, overlap and balance.
What the garden would change
Section titled “What the garden would change”Only a simulator can do this. Our measurement script replayed the same logs with the hidden garden column, and added it to the regression. The answer became +2.38 dollars (+0.74 to +4.01), against the true +2.37. With every confounder, regression adjustment works here. With one missing, it stays about 1 dollar too high.
A real shop never gets this check. It cannot add a column that nobody recorded.
Which columns go in?
Section titled “Which columns go in?”The causal graph of Draw the causes told us which columns to add: the six confounders. What happens if we get the list wrong? The chart shows seven regressions on the same logs.
Text description of the chart
A panel with a line at 0 (no effect) and a line at the true effect, +2.37 dollars. Nothing added (naive): +9.70 (+7.56 to +11.84). The 6 confounders: +3.38 (+1.74 to +5.03). Without pre_spend: +2.76 (+0.94 to +4.59). Without pre_spend and visits: +3.97 (+2.07 to +5.87). The 6 confounders and the mediator pages_viewed: +2.79 (+0.81 to +4.76). The 6 confounders and the collider left_review: −3.16 (−4.70 to −1.61), below zero. The 6 confounders and the hidden garden, only in the simulator: +2.38 (+0.74 to +4.01), on the truth.
A missing confounder
Section titled “A missing confounder”Leave out past spending and visits:
xp adjust --covariates pre_orders,tenure_days,new_customer,desktop regression +3.97 (95% interval +2.07 to +5.87; ATE)We show only the regression line of the output. The estimate rises to +3.97 dollars. Without past spending and visits, the regression cannot see part of what makes users different. That part goes back into the block’s “effect”. A missing confounder pushes the answer away from the truth, in the direction of the confounder. Here, users are the more active customers, so the answer goes up.
Closer by luck is not better
Section titled “Closer by luck is not better”Now leave out only past spending:
xp adjust --covariates pre_orders,visits,tenure_days,new_customer,desktop regression +2.76 (95% interval +0.94 to +4.59; ATE)The answer is +2.76 dollars: closer to the truth (+2.37) than the full list (+3.38). Does that mean pre_spend should go?
No. Two errors cancel here by luck. The hidden garden pushes the answer up. Dropping pre_spend changes how the other columns share the work, and here it happens to pull the answer down. On other data, it could push it up. In a real study, you do not know the truth, so you cannot see which errors cancel. Choose the columns from the causal graph, before you see the estimate. Never choose them by how the answer looks.
A mediator added by mistake
Section titled “A mediator added by mistake”pages_viewed counts the pages a customer saw in the 28 days. The block itself adds pages: customers click products in it. So pages_viewed is a mediator: it is on the path from the block to revenue.
xp adjust --add pages_viewed regression +2.79 (95% interval +0.81 to +4.76; ATE)The answer is +2.79 dollars: again closer to the truth, and again for a bad reason. Adjusting for pages compares users and non-users who saw the same number of pages. That removes the part of the effect that works through the pages. It also mixes in each customer’s wish to buy, which drives pages too. Here, these errors pull the answer down, and the hidden garden pushes it up. They happen to cancel.
The rule from Draw the causes still holds: do not adjust for anything that the treatment can change. A column recorded after the treatment started needs a careful look at the graph.
Over many seeds
Section titled “Over many seeds”One run can be lucky. The course’s measurement ran the regression on 50 simulated logs, each with 20,000 customers and its own true effect. On average, the regression was +1.19 dollars too high. Its 95% interval contained the truth in 86% of the runs, not 95%.
So the interval of seed 40 contained the truth, but that does not make the method unbiased. The interval measures the noise of the sample. It does not measure the bias of a missing confounder. More customers make the interval narrower around the same wrong answer.
When to use it, when not, and what it costs
Section titled “When to use it, when not, and what it costs”Use regression adjustment when:
- you cannot run a test, and you have logs with the confounders;
- you drew the causal graph first, and you know which columns are confounders;
- users and non-users overlap: for each kind of user, there are non-users of the same kind. The next lesson shows how to check this.
Do not trust it when:
- an important confounder is missing from the data, like the garden. The answer stays biased, and nothing in the output warns you;
- you are tempted to add every column you have. Mediators and colliders do damage;
- users and non-users are very different, for example all users are big spenders and no non-user is. Then the regression must guess, with its straight lines, what a big spender would do without the block.
It costs little: data you already have and one regression. The real cost is the thinking: a causal graph that you can defend, and a list of the confounders you could not measure.
Guided practice: build the adjustment step by step
Section titled “Guided practice: build the adjustment step by step”- Open the notebook from the box Do this lesson in a notebook at the top of this page.
- Run the setup cells and the helper cell. The notebook reads the same logs as
sim run --scenario logs(seed 40). - In section 1, fit the naive answer as a regression. You see
naive: +9.70 dollars (95% interval +7.56 to +11.84). - In section 2, add the six confounders. You see
adjusted: +3.38 dollars (95% interval +1.74 to +5.03). - In section 3, predict, then leave out past spending and visits. You see
+3.97 dollars. - In section 4, add the mediator. You see
with the mediator: +2.79 dollars. - In section 5, add the collider
left_review. Put the coefficient ofused_blockinwith_collider. Run the check cell. It printsCheck passed.
Show the answer to step 7
mc = smf.ols(f"revenue ~ used_block + {confounders} + left_review", data=logs).fit()with_collider = round(float(mc.params["used_block"]), 2)print(with_collider)The value is −3.16: the same as xp adjust --add left_review.
Your turn: a collider in the project
Section titled “Your turn: a collider in the project”TermTomásLarkfield’s AI engineer. He often suggests a quick shortcut, and the lesson shows why it fails., the AI engineer, says: “Customers who write reviews are our most engaged customers. Let us adjust for left_review too.” Before you run anything, predict: will the estimate go up, go down, or stay near +3.38? Then run xp adjust --add left_review and xp check.
Show the answer
xp adjust --add left_reviewxp checkRun logs-s40: effect of used_block on revenue (dollars per customer, 28 days). Covariates: pre_spend, pre_orders, visits, tenure_days, new_customer, desktop, left_review naive (users − non-users) +9.70 regression -3.16 (95% interval -4.70 to -1.61; ATE) matching (nearest neighbour) -3.87 (95% interval -6.27 to -1.47; ATT) weighting (IPW) -4.55 (95% interval -6.29 to -2.43; ATE) Overlap: propensity scores 0.187–0.628 (users), 0.187–0.630 (non-users). Balance (standardised mean difference; below 0.1 is good): pre_spend +0.08 → -0.04, pre_orders +0.10 → -0.01, visits +0.09 → -0.00, tenure_days +0.14 → -0.00, new_customer -0.15 → -0.00, desktop +0.15 → +0.00, left_review +0.26 → +0.00Run logs-s40: the last analysis was `xp adjust`.True effect (from the simulator): ATE +2.37, ATT +2.73 dollars per customer. Naive: +9.70. regression: -3.16 against ATE +2.37: the interval does NOT contain it. matching (nearest neighbour): -3.87 against ATT +2.73: the interval does NOT contain it. weighting (IPW): -4.55 against ATE +2.37: the interval does NOT contain it.Hidden from the log: has_garden (45% of customers): users are more likely to have a garden, and gardeners spend more. It is not in the log.The estimate goes down to −3.16 dollars: the block now seems to lose money. All three methods agree on the wrong sign, and no interval contains the truth.
Both the block and spending cause reviews, so left_review is a collider. Compare only reviewers: a reviewer who did not use the block probably spent a lot, because spending is the other reason to write a review. So among reviewers, non-users look like big spenders. The comparison is no longer fair. Engaged customers are a real thing, but “engaged” here is measured after the block, and the block changed it. Leave it out.
Common mistake
“More control variables are always better”
Symptom: an analyst adds every column of the log to the regression: past spending, visits, pages viewed, reviews, returns. The estimate of the block’s effect becomes negative, with a narrow interval. The team stops a feature that works.
Cause: some columns were results of the treatment. Pages viewed is a mediator: it carries part of the effect. Reviews are a collider: both the block and spending cause them. Adjusting for them takes away part of the effect and creates a false link. The extra columns made the interval narrower, so the wrong answer looked more certain.
Fix: draw the causal graph first, and adjust only for the confounders: causes of both the treatment and the outcome, measured before the treatment. Write the list down before you see the estimate. Leave out everything the treatment can change, even when it looks useful.
Check your understanding
In a shop, keen users spent 70 dollars and the keen non-user 66; casual users spent 25 and casual non-users 21. Most users are keen and most non-users are casual. What does adjusting for the kind of customer give, and why is the naive answer bigger?
Show the answer
Inside each kind, the difference is 70 − 66 = +4 dollars and 25 − 21 = +4 dollars. So the adjusted effect is +4 dollars. The naive answer is bigger because it compares mostly keen customers with mostly casual customers. Keen customers spend more anyway, and that difference is added to the effect. The kind of customer is a confounder.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
Larkfield's regression with the six confounders gives +3.38 dollars, and the true effect is +2.37. Its 95% interval (+1.74 to +5.03) contains the truth. Is the method unbiased? Explain with the measurement over 50 seeds.
Show the answer
No. One confounder, the garden, is missing from the logs, so the estimate is pushed up. Over 50 simulated logs, the regression was on average 1.19 dollars too high. Its interval contained the truth in only 86% of the runs, not 95%. The interval measures the noise of the sample, not the bias from a missing confounder.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
Leaving out pre_spend gives +2.76 dollars, and adding the mediator pages_viewed gives +2.79. Both are closer to the truth (+2.37) than the six confounders (+3.38). Should Larkfield report one of them? Why or why not?
Show the answer
No. Both are closer by luck: two errors cancel. The hidden garden pushes the answer up, and the wrong column pulls it down. On other data, the errors may not cancel. In a real study, nobody knows the truth, so nobody can see that they cancel. Choose the columns from the causal graph before you see the estimate: the confounders only, and no mediator.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Recap
- Regression adjustment compares users and non-users who have the same values of the confounders. The coefficient of the treatment is the adjusted effect.
- On Larkfield’s logs, the naive +9.70 dollars falls to +3.38 (+1.74 to +5.03) with the six confounders. The truth is +2.37 (ATE) and +2.73 (ATT).
- It is still too high because the garden is missing. With the garden, only possible in the simulator: +2.38. Over 50 seeds: bias +1.19 dollars, coverage 86%.
- A missing confounder pushes the answer away. A mediator (+2.79) or a dropped confounder (+2.76) can land closer by luck. A collider (−3.16) gives the wrong sign.
- Choose the columns from the causal graph, before you see the answer.
Next, a second way to adjust: model who used the block, then match or weight the customers.