Skip to content
Français

Regression adjustment

  • Lesson 21
  • 35 min
  • Module 6: Adjust for confounders in observational data

Where we are

Natural experiments need a lucky design. Often we only have logs: who used a feature, and what they spent. In this lesson, we add the confounders from our causal graph to a regression, and we see what a missing confounder and a mediator do to the answer.

What you will be able to do

Estimate the effect of a feature from logs, when nobody chose at random who used it. Add the confounders to a regression, so that you compare like with like. Learn which columns belong in the regression, and which ones break it.

NotebookDo this lesson in a notebookColab · Kaggle · GitHub

In the notebook, you fit the naive answer as a regression, add the confounders, leave one out, and add a mediator and a collider. Its setup cells download the logs data and install the packages, so it needs no local setup. About 35 minutes.

Colab and Kaggle open your own copy. They need a free Google or Kaggle account. Stuck? See the solution notebook.

Before you startWhat to know and to have readyAbout 35 minutes

What you need to know

What you need on your computer (only for the project steps)

  • The project larkfield-experiments at the snapshot end-of-m05, from Difference-in-differences, with .venv active. This lesson updates it to end-of-m06.

Tested on macOS with Python 3.12 and the end-of-m06 snapshot of the project. Windows was not tested.

A small shop puts a new shelf near the door: “Our favourites this month”. At the end of the month, the owner looks at the till receipts. Customers who stopped at the shelf spent 25 dollars more than customers who walked past it.

Did the shelf bring 25 dollars per customer? The owner is not sure. The regulars stop at everything. They know the shop, they have time, and they buy a lot anyway. The people who walk past are often visitors in a hurry.

So the owner does a fairer thing. She compares regulars who stopped with regulars who did not. Then she compares visitors who stopped with visitors who did not. Each comparison is between similar people. The difference that is left is the shelf.

That is the whole idea of this lesson. The rest is how to do it with many columns at once, and which columns to leave out.

The problem: users of the block spent 9.70 dollars more

Section titled “The problem: users of the block spent 9.70 dollars more”

TermLarkfieldA made-up Canadian online shop for home and garden products. It is not a real company. All Nextia courses use its help desk and data as the example.Learn more Meet Larkfield launched its recommendation block for everyone on 6 October 2025. There was no test. Customers chose to click the block or not. The logs scenario of the simulator has 28 days of these logs, for 40,000 customers. All of it is synthetic.

TermOmarLarkfield’s product manager. He asks the business questions: what a change costs, and what it gives the shop., the product manager, reads the dashboard. The 13,338 customers who used the block spent 39.19 dollars each. The 26,662 who did not spent 29.48 dollars. The difference is +9.70 dollars. “The block brings almost 10 dollars per customer,” Omar says.

You saw in Causes and comparisons why this number is too big. The users were different before the block existed. They spent 51.77 dollars in the 8 weeks before the launch, against 41.28 for the others. They visit more often. More of them use a computer. These columns are confounders: they cause both the use of the block and the spending.

The simulator knows the truth. If every customer used the block, each would spend +2.37 dollars more on average. For the customers who actually used it, the effect is +2.73 dollars. The naive +9.70 is more than three times too big. The gap of +9.70 against +2.73 is a practice problem we made on purpose.

A coin flip would fix this. But the block is already live for everyone. Larkfield has only the logs. Can the logs give a fair answer?

Regression adjustment compares users and non-users who are alike in the confounders. “Adjust for past spending” means: compare customers with the same past spending. A regression does this for many columns at once.

Top half, the naive comparison. A blue box: used the shelf, 5 customers, 4 keen who spent 60 dollars each and 1 casual who spent 23 dollars. A grey box: did not use it, 5 customers, 1 keen who spent 57 dollars and 4 casual who spent 20 dollars each. Red arrows lead to a red box: all users against all non-users, 52.60 minus 27.40 is plus 25.20 dollars, mostly keen against mostly casual. Bottom half, the adjusted comparison. Two green boxes: keen with keen, 60 minus 57 is plus 3 dollars; casual with casual, 23 minus 20 is plus 3 dollars. Blue arrows lead to a green box: adjusted for the kind of customer, plus 3 dollars; a regression with a keen column does this.
Compare like with like. The naive comparison puts mostly keen customers against mostly casual ones. Inside each kind, the shelf adds 3 dollars. The ten customers are made up for this example.

Ten customers of the small shop. The numbers are made up for this example. Four of the five keen customers used the shelf. Only one of the five casual customers did.

Customer Kind Used the shelf? Spent (dollars)
1 to 4 keen yes 60 each
5 keen no 57
6 casual yes 23
7 to 10 casual no 20 each

The naive answer:

  • users: (4 × 60 + 23) / 5 = 263 / 5 = 52.60 dollars;
  • non-users: (57 + 4 × 20) / 5 = 137 / 5 = 27.40 dollars;
  • difference: 52.60 − 27.40 = +25.20 dollars.

Now compare like with like:

  • keen users against the keen non-user: 60 − 57 = +3 dollars;
  • the casual user against casual non-users: 23 − 20 = +3 dollars.

The shelf adds 3 dollars in both kinds. The other 22.20 dollars of the naive answer came from who used the shelf, not from the shelf. The kind of customer is the confounder: keen customers use the shelf more, and they spend more.

With 40,000 customers and six confounders, you cannot make a table for every kind of customer. A regression does the comparison for you. It fits one equation:

revenue = a + b × used_block + c × pre_spend + d × visits + … (one term for each confounder)

Each letter is a number that the regression finds from the data. The number b is the difference in revenue between users and non-users with the same values of every other column. That is the adjusted effect. Write this equation for the small shop, with one column keen (1 or 0). The regression finds b = +3.00 dollars, exactly as the table did.

The regression assumes a simple shape: each column adds a straight-line amount, and the block adds the same b for everyone. When the true shape is not a straight line, the answer can be a little off.

Our tools print two names for the effect. They answer two different questions.

  • ATE, the average treatment effect: the average change if everyone used the block, against nobody. At Larkfield, the truth is +2.37 dollars.
  • ATT, the average treatment effect on the treated: the average change for the customers who used it. The truth is +2.73 dollars.

The two differ because the users are more active customers. The block raises each customer’s chance to order on each visit, so it helps a customer who visits more. A regression like ours estimates the ATE.

First, update the project to the snapshot end-of-m06. It adds xp adjust (this lesson and the next) and xp refute (the third lesson). It also adds two packages: scikit-learn and DoWhy. Your own runs and plans stay.

  1. Open a terminal in the folder that holds larkfield-experiments and nextia-learning-labs. If you cloned the labs repository with Git, update it first.

    Terminal window
    git -C nextia-learning-labs pull
  2. Copy the snapshot over your project, and install the new packages.

    Terminal window
    cp -R nextia-learning-labs/C40/snapshots/end-of-m06/. larkfield-experiments/
    cd larkfield-experiments
    source .venv/bin/activate
    python -m pip install -r requirements.txt

    DoWhy brings several other packages, so the install takes a few minutes.

  3. Run the tests.

    Terminal window
    python -m pytest -q
    .......................... [100%]
    29 passed in 5.51s

    Twenty-nine tests pass. Your time will differ.

  4. Simulate the logs.

    Terminal window
    sim run --scenario logs
    Scenario logs (seed 40): 28 days of logs after the recommendation block's launch: who used it and what they spent (no test).
    Wrote runs/logs-s40: customers (40,000 rows). Synthetic data. 0.5 s
  5. Adjust for the six confounders, then compare with the truth.

    Terminal window
    xp adjust
    xp check
    Run logs-s40: effect of used_block on revenue (dollars per customer, 28 days).
    Covariates: pre_spend, pre_orders, visits, tenure_days, new_customer, desktop
    naive (users − non-users) +9.70
    regression +3.38 (95% interval +1.74 to +5.03; ATE)
    matching (nearest neighbour) +4.43 (95% interval +1.96 to +6.89; ATT)
    weighting (IPW) +2.87 (95% interval +0.98 to +5.16; ATE)
    Overlap: propensity scores 0.191–0.505 (users), 0.191–0.508 (non-users).
    Balance (standardised mean difference; below 0.1 is good): pre_spend +0.08 → -0.01, pre_orders +0.10 → -0.00, visits +0.09 → +0.00, tenure_days +0.14 → -0.00, new_customer -0.15 → -0.00, desktop +0.15 → +0.00
    Run logs-s40: the last analysis was `xp adjust`.
    True effect (from the simulator): ATE +2.37, ATT +2.73 dollars per customer. Naive: +9.70.
    regression: +3.38 against ATE +2.37: the interval contains it.
    matching (nearest neighbour): +4.43 against ATT +2.73: the interval contains it.
    weighting (IPW): +2.87 against ATE +2.37: the interval contains it.
    Hidden from the log: has_garden (45% of customers): users are more likely to have a garden, and gardeners spend more. It is not in the log.

Look at the regression line. The six confounders bring the answer from +9.70 down to +3.38 dollars (+1.74 to +5.03). That is much closer to the true +2.37. The TermConfidence intervalA range around a measured value, made by a method that holds the true value 95 times in 100. If the interval of a difference does not include zero, the difference is probably not chance.Learn more Estimate uncertaintyEssential Mathematics and Statistics for AI contains the truth.

But +3.38 is still about 1 dollar too high. The last line of xp check says why: whether a customer has a garden. Gardeners use the block more, and they spend more. So the garden is a confounder too. Larkfield never recorded it, so no regression can adjust for it.

The other lines of xp adjust are for the next lesson: matching, weighting, overlap and balance.

Only a simulator can do this. Our measurement script replayed the same logs with the hidden garden column, and added it to the regression. The answer became +2.38 dollars (+0.74 to +4.01), against the true +2.37. With every confounder, regression adjustment works here. With one missing, it stays about 1 dollar too high.

A real shop never gets this check. It cannot add a column that nobody recorded.

The causal graph of Draw the causes told us which columns to add: the six confounders. What happens if we get the list wrong? The chart shows seven regressions on the same logs.

The block's effect on revenue from seven regressions on the same simulated logs (seed 40), with 95% intervals. The six confounders bring the naive answer close to the truth. Leaving confounders out, or adding a mediator or a collider, moves it. Only the regression with the hidden garden lands on the truth, and only the simulator can run it.
Text description of the chart

A panel with a line at 0 (no effect) and a line at the true effect, +2.37 dollars. Nothing added (naive): +9.70 (+7.56 to +11.84). The 6 confounders: +3.38 (+1.74 to +5.03). Without pre_spend: +2.76 (+0.94 to +4.59). Without pre_spend and visits: +3.97 (+2.07 to +5.87). The 6 confounders and the mediator pages_viewed: +2.79 (+0.81 to +4.76). The 6 confounders and the collider left_review: −3.16 (−4.70 to −1.61), below zero. The 6 confounders and the hidden garden, only in the simulator: +2.38 (+0.74 to +4.01), on the truth.

Leave out past spending and visits:

Terminal window
xp adjust --covariates pre_orders,tenure_days,new_customer,desktop
regression +3.97 (95% interval +2.07 to +5.87; ATE)

We show only the regression line of the output. The estimate rises to +3.97 dollars. Without past spending and visits, the regression cannot see part of what makes users different. That part goes back into the block’s “effect”. A missing confounder pushes the answer away from the truth, in the direction of the confounder. Here, users are the more active customers, so the answer goes up.

Now leave out only past spending:

Terminal window
xp adjust --covariates pre_orders,visits,tenure_days,new_customer,desktop
regression +2.76 (95% interval +0.94 to +4.59; ATE)

The answer is +2.76 dollars: closer to the truth (+2.37) than the full list (+3.38). Does that mean pre_spend should go?

No. Two errors cancel here by luck. The hidden garden pushes the answer up. Dropping pre_spend changes how the other columns share the work, and here it happens to pull the answer down. On other data, it could push it up. In a real study, you do not know the truth, so you cannot see which errors cancel. Choose the columns from the causal graph, before you see the estimate. Never choose them by how the answer looks.

pages_viewed counts the pages a customer saw in the 28 days. The block itself adds pages: customers click products in it. So pages_viewed is a mediator: it is on the path from the block to revenue.

Terminal window
xp adjust --add pages_viewed
regression +2.79 (95% interval +0.81 to +4.76; ATE)

The answer is +2.79 dollars: again closer to the truth, and again for a bad reason. Adjusting for pages compares users and non-users who saw the same number of pages. That removes the part of the effect that works through the pages. It also mixes in each customer’s wish to buy, which drives pages too. Here, these errors pull the answer down, and the hidden garden pushes it up. They happen to cancel.

The rule from Draw the causes still holds: do not adjust for anything that the treatment can change. A column recorded after the treatment started needs a careful look at the graph.

One run can be lucky. The course’s measurement ran the regression on 50 simulated logs, each with 20,000 customers and its own true effect. On average, the regression was +1.19 dollars too high. Its 95% interval contained the truth in 86% of the runs, not 95%.

So the interval of seed 40 contained the truth, but that does not make the method unbiased. The interval measures the noise of the sample. It does not measure the bias of a missing confounder. More customers make the interval narrower around the same wrong answer.

When to use it, when not, and what it costs

Section titled “When to use it, when not, and what it costs”

Use regression adjustment when:

  • you cannot run a test, and you have logs with the confounders;
  • you drew the causal graph first, and you know which columns are confounders;
  • users and non-users overlap: for each kind of user, there are non-users of the same kind. The next lesson shows how to check this.

Do not trust it when:

  • an important confounder is missing from the data, like the garden. The answer stays biased, and nothing in the output warns you;
  • you are tempted to add every column you have. Mediators and colliders do damage;
  • users and non-users are very different, for example all users are big spenders and no non-user is. Then the regression must guess, with its straight lines, what a big spender would do without the block.

It costs little: data you already have and one regression. The real cost is the thinking: a causal graph that you can defend, and a list of the confounders you could not measure.

Guided practice: build the adjustment step by step

Section titled “Guided practice: build the adjustment step by step”
  1. Open the notebook from the box Do this lesson in a notebook at the top of this page.
  2. Run the setup cells and the helper cell. The notebook reads the same logs as sim run --scenario logs (seed 40).
  3. In section 1, fit the naive answer as a regression. You see naive: +9.70 dollars (95% interval +7.56 to +11.84).
  4. In section 2, add the six confounders. You see adjusted: +3.38 dollars (95% interval +1.74 to +5.03).
  5. In section 3, predict, then leave out past spending and visits. You see +3.97 dollars.
  6. In section 4, add the mediator. You see with the mediator: +2.79 dollars.
  7. In section 5, add the collider left_review. Put the coefficient of used_block in with_collider. Run the check cell. It prints Check passed.
Show the answer to step 7
mc = smf.ols(f"revenue ~ used_block + {confounders} + left_review", data=logs).fit()
with_collider = round(float(mc.params["used_block"]), 2)
print(with_collider)

The value is −3.16: the same as xp adjust --add left_review.

TermTomásLarkfield’s AI engineer. He often suggests a quick shortcut, and the lesson shows why it fails., the AI engineer, says: “Customers who write reviews are our most engaged customers. Let us adjust for left_review too.” Before you run anything, predict: will the estimate go up, go down, or stay near +3.38? Then run xp adjust --add left_review and xp check.

Show the answer
Terminal window
xp adjust --add left_review
xp check
Run logs-s40: effect of used_block on revenue (dollars per customer, 28 days).
Covariates: pre_spend, pre_orders, visits, tenure_days, new_customer, desktop, left_review
naive (users − non-users) +9.70
regression -3.16 (95% interval -4.70 to -1.61; ATE)
matching (nearest neighbour) -3.87 (95% interval -6.27 to -1.47; ATT)
weighting (IPW) -4.55 (95% interval -6.29 to -2.43; ATE)
Overlap: propensity scores 0.187–0.628 (users), 0.187–0.630 (non-users).
Balance (standardised mean difference; below 0.1 is good): pre_spend +0.08 → -0.04, pre_orders +0.10 → -0.01, visits +0.09 → -0.00, tenure_days +0.14 → -0.00, new_customer -0.15 → -0.00, desktop +0.15 → +0.00, left_review +0.26 → +0.00
Run logs-s40: the last analysis was `xp adjust`.
True effect (from the simulator): ATE +2.37, ATT +2.73 dollars per customer. Naive: +9.70.
regression: -3.16 against ATE +2.37: the interval does NOT contain it.
matching (nearest neighbour): -3.87 against ATT +2.73: the interval does NOT contain it.
weighting (IPW): -4.55 against ATE +2.37: the interval does NOT contain it.
Hidden from the log: has_garden (45% of customers): users are more likely to have a garden, and gardeners spend more. It is not in the log.

The estimate goes down to −3.16 dollars: the block now seems to lose money. All three methods agree on the wrong sign, and no interval contains the truth.

Both the block and spending cause reviews, so left_review is a collider. Compare only reviewers: a reviewer who did not use the block probably spent a lot, because spending is the other reason to write a review. So among reviewers, non-users look like big spenders. The comparison is no longer fair. Engaged customers are a real thing, but “engaged” here is measured after the block, and the block changed it. Leave it out.

Common mistake

“More control variables are always better”

Symptom: an analyst adds every column of the log to the regression: past spending, visits, pages viewed, reviews, returns. The estimate of the block’s effect becomes negative, with a narrow interval. The team stops a feature that works.

Cause: some columns were results of the treatment. Pages viewed is a mediator: it carries part of the effect. Reviews are a collider: both the block and spending cause them. Adjusting for them takes away part of the effect and creates a false link. The extra columns made the interval narrower, so the wrong answer looked more certain.

Fix: draw the causal graph first, and adjust only for the confounders: causes of both the treatment and the outcome, measured before the treatment. Write the list down before you see the estimate. Leave out everything the treatment can change, even when it looks useful.

Check your understanding

In a shop, keen users spent 70 dollars and the keen non-user 66; casual users spent 25 and casual non-users 21. Most users are keen and most non-users are casual. What does adjusting for the kind of customer give, and why is the naive answer bigger?

Show the answer

Inside each kind, the difference is 70 − 66 = +4 dollars and 25 − 21 = +4 dollars. So the adjusted effect is +4 dollars. The naive answer is bigger because it compares mostly keen customers with mostly casual customers. Keen customers spend more anyway, and that difference is added to the effect. The kind of customer is a confounder.

Check your understanding

Larkfield's regression with the six confounders gives +3.38 dollars, and the true effect is +2.37. Its 95% interval (+1.74 to +5.03) contains the truth. Is the method unbiased? Explain with the measurement over 50 seeds.

Show the answer

No. One confounder, the garden, is missing from the logs, so the estimate is pushed up. Over 50 simulated logs, the regression was on average 1.19 dollars too high. Its interval contained the truth in only 86% of the runs, not 95%. The interval measures the noise of the sample, not the bias from a missing confounder.

Check your understanding

Leaving out pre_spend gives +2.76 dollars, and adding the mediator pages_viewed gives +2.79. Both are closer to the truth (+2.37) than the six confounders (+3.38). Should Larkfield report one of them? Why or why not?

Show the answer

No. Both are closer by luck: two errors cancel. The hidden garden pushes the answer up, and the wrong column pulls it down. On other data, the errors may not cancel. In a real study, nobody knows the truth, so nobody can see that they cancel. Choose the columns from the causal graph before you see the estimate: the confounders only, and no mediator.

Recap

  • Regression adjustment compares users and non-users who have the same values of the confounders. The coefficient of the treatment is the adjusted effect.
  • On Larkfield’s logs, the naive +9.70 dollars falls to +3.38 (+1.74 to +5.03) with the six confounders. The truth is +2.37 (ATE) and +2.73 (ATT).
  • It is still too high because the garden is missing. With the garden, only possible in the simulator: +2.38. Over 50 seeds: bias +1.19 dollars, coverage 86%.
  • A missing confounder pushes the answer away. A mediator (+2.79) or a dropped confounder (+2.76) can land closer by luck. A collider (−3.16) gives the wrong sign.
  • Choose the columns from the causal graph, before you see the answer.

Next, a second way to adjust: model who used the block, then match or weight the customers.

Next: Propensity scores, matching and weighting