Test your own answer
- Lesson 23
- 35 min
- Module 6: Adjust for confounders in observational data
Where we are
Regression, matching and weighting gave us similar answers. Similar answers can still all be wrong. In this lesson, we test our own estimate with DoWhy's refutation tests and ask how strong a hidden confounder would need to be to change the decision.
Lesson 23 of 30 · Experiments and Causal Impact: Did Our Change Work?Test your own answer
1. Why "it went up" is not an answer
2. Design an A/B test
3. Analyse the test
4. Traps in real tests
5. When you cannot randomise: natural experiments
6. Adjust for confounders in observational data
7. Uplift: who should get the treatment?
8. Experiments for AI features and in production
What you will be able to do
Try to break your own estimate with three refutation tests, and read what a pass means. Then ask the question the tests cannot answer: how strong would a hidden confounder need to be to change the decision?
NotebookDo this lesson in a notebook
In the notebook, you write the three refutation tests by hand. Then you check the rule for a hidden confounder on a made-up column, and build the sensitivity grid. Its setup cells download the logs data and install the packages, so it needs no local setup. About 35 minutes.
Colab and Kaggle open your own copy. They need a free Google or Kaggle account. Stuck? See the solution notebook.
Before you startWhat to know and to have ready
What you need to know
- Regression adjustment, from Regression adjustment.
- Balance and why three methods can agree and still be wrong, from Propensity scores, matching and weighting.
- A p-value. For a refresher, see Compare alternatives in the mathematics course.
What you need on your computer (only for the project steps)
- The project
larkfield-experimentsat the snapshotend-of-m06, from Regression adjustment, with.venvactive, and the logs run of seed 40.
Tested on macOS with Python 3.12 and the end-of-m06 snapshot of the project. Windows was not tested.
Checking your own homework
Section titled “Checking your own homework”The shop owner is proud of her answer: the new shelf adds 3 dollars per customer. Before she tells the bank, she checks it in three ways.
First, she picks five customers at random and pretends that they stopped at the shelf. Her method should find no effect for a made-up shelf. Second, she adds a meaningless column, the last digit of each receipt number, to her comparison. It should change nothing. Third, she repeats the analysis on half of the receipts. The answer should stay about the same.
All three checks pass. Is she right? Not necessarily. Her method works on the data she has. But she never wrote down which customers came by car. Drivers buy more, and the shelf is next to the car park. No check of her data can find a column that is not in her data.
The problem: three answers that agree
Section titled “The problem: three answers that agree”On TermLarkfieldA made-up Canadian online shop for home and garden products. It is not a real company. All Nextia courses use its help desk and data as the example.Learn more Meet Larkfield’s logs, the last two lessons gave three adjusted answers: regression +3.38 dollars, matching +4.43, weighting +2.87. They roughly agree. They are all much smaller than the naive +9.70.
TermOmarLarkfield’s product manager. He asks the business questions: what a change costs, and what it gives the shop. wants to report +3 dollars per customer and move on. In a real shop, nobody knows the truth. Here, the simulator does: +2.37 dollars if everyone used the block (ATE). All three answers are too high, because they share one blind spot: the garden.
Larkfield cannot ask the simulator. So how can a team find out whether its answer is fragile? There are two kinds of check. Refutation tests look for problems in the method and the data you have. Sensitivity analysis asks what a column you do not have could do.
The idea: try to break your own answer
Section titled “The idea: try to break your own answer”A refutation test changes the data in a way whose result you know in advance, then runs your method again. If the method does not give the known result, something is wrong. To “refute” means to prove wrong. You try to prove your own answer wrong, and you hope to fail.
DoWhy is a free Python library for causal questions (MIT licence). You give it the treatment, the outcome and the confounders. It estimates the effect, then runs refutation tests on that estimate. The project’s xp refute uses three of them.
A tiny example: a placebo at the small shop
Section titled “A tiny example: a placebo at the small shop”Take the ten customers of the small shop again. The numbers are made up for this example. The real users were customers 1 to 4 (keen) and 6 (casual), and the adjusted effect was +3 dollars.
Now pick five customers at random as fake “users”: 1 and 5 (keen), and 6, 7 and 8 (casual). Then compare like with like, as before:
- keen fake users (60 and 57: mean 58.50) against the other keen customers (60, 60, 60: mean 60): −1.50 dollars;
- casual fake users (23, 20, 20: mean 21) against the other casual customers (20, 20: mean 20): +1.00 dollar;
- the average of the two kinds: (−1.50 + 1.00) / 2 = −0.25 dollars.
The fake shelf gets about 0, much smaller than the real +3. That is a pass. Suppose the method had found +3 for a random group too. Then the real +3 would mean nothing: it would come from the method, not from the shelf.
Larkfield: run the refutation tests
Section titled “Larkfield: run the refutation tests”Your project should be at end-of-m06, with the logs of seed 40 as the newest logs run. If you ran seed 41 in the last lesson, run sim run --scenario logs first.
-
Run the refutation tests. DoWhy repeats each test 20 times, so this can take up to a minute.
Terminal window xp refuteDoWhy estimate (backdoor adjustment, linear regression): +3.38 dollars per customer.placebo_treatment_refuter new effect -0.07 (p = 0.47)random_common_cause new effect +3.38 (p = 0.50)data_subset_refuter new effect +3.45 (p = 0.42)Expected: the placebo gives about 0; the other two stay close to the estimate. Passing does not prove the estimate: these tests cannot see a confounder that is missing from the data.Hidden confounder: how big must it be to move the estimate to 0? (a 0/1 cause; its share among users minus non-users × its effect on revenue in dollars)share gap 0.05: 5 $ → +3.13 10 $ → +2.88 20 $ → +2.38 40 $ → +1.38share gap 0.10: 5 $ → +2.88 10 $ → +2.38 20 $ → +1.38 40 $ → -0.62share gap 0.20: 5 $ → +2.38 10 $ → +1.38 20 $ → -0.62 40 $ → -4.62share gap 0.30: 5 $ → +1.88 10 $ → +0.38 20 $ → -2.62 40 $ → -8.62 -
Compare with the truth.
Terminal window xp checkRun logs-s40: the last analysis was `xp refute`.True effect (from the simulator): ATE +2.37 dollars per customer; your DoWhy estimate +3.38.The refutation tests passed or failed without knowing this: has_garden (45% of customers): users are more likely to have a garden, and gardeners spend more. It is not in the log.
DoWhy’s estimate is +3.38 dollars: the same regression as xp adjust, with the same six confounders. “Backdoor adjustment” is DoWhy’s name for adjusting for the confounders.
Text description of the chart
A panel with a line at 0 and a line at the true effect, +2.37 dollars. Our estimate: +3.38. Placebo treatment: −0.07, at about 0. Random common cause: +3.38, the same as the estimate. Data subset of 80%: +3.45, close to the estimate. The estimate and the two tests that keep it sit about 1 dollar to the right of the truth.
Read each line:
- Placebo treatment: −0.07 dollars. With
used_blockshuffled, the method finds about nothing. Good: the +3.38 is not a side effect of the method. - Random common cause: +3.38. A column of noise changes nothing. Good: the estimate is not fragile to one extra column.
- Data subset: +3.45. On 80% samples, the answer stays close. Good: it does not rest on a few customers.
Each p-value asks: is the result what a sound estimate would give? A large p-value, here 0.42 to 0.50, means the test found no problem. A p-value below 0.05 would mean a failed test.
Why a pass is not proof
Section titled “Why a pass is not proof”All three tests pass, and the estimate is still 1.01 dollars too high. This is not a bug in DoWhy. It is what these tests can and cannot do.
Each test changes something inside the data: the treatment column, an added column of noise, the rows. Then it checks that the method reacts as it should. So the tests catch problems in the method and in the data you have. Examples: a coding error, an estimate that hangs on a few customers, a method that finds effects everywhere.
The garden is not inside the data. Every test runs on the same six columns, so every test inherits the same blind spot. The method is correct for those six columns. It is the list of columns that is incomplete.
It is not luck of one seed either. On the logs of seed 41, the estimate is +2.66 dollars, and the true ATE is +2.16. The tests pass again: placebo −0.14, random common cause +2.67, data subset +2.65.
So a pass means “no bug found”. It does not mean “the estimate is right”. This is also why regression, matching and weighting agree: the same columns, the same blind spot. Agreement and passed tests are worth having. They are not evidence about a missing confounder.
Sensitivity: how strong would a hidden confounder need to be?
Section titled “Sensitivity: how strong would a hidden confounder need to be?”If no test can find the missing column, ask a different question. Suppose a confounder is missing. How strong would it need to be to change our decision? This is sensitivity analysis.
The project uses the simplest version. Imagine a hidden yes-or-no cause, like “has a garden”. It has two sizes:
- the share gap: how much more common it is among users than among non-users;
- its effect in dollars: how much more its owners spend.
The bias it adds is about share gap × dollars. The true effect is then about our estimate minus that bias.
The rule, with every value visible
Section titled “The rule, with every value visible”Ten users and ten non-users. The numbers are made up for this example. Six users have a garden; five non-users do. A garden adds 10 dollars of spending, and the treatment does nothing.
- Users get 6 × 10 = 60 extra dollars, or 60 / 10 = 6 dollars each.
- Non-users get 5 × 10 = 50 extra dollars, or 50 / 10 = 5 dollars each.
- The users seem to spend 1 dollar more, because of the garden alone.
The share gap is 6/10 − 5/10 = 0.10. The bias is 0.10 × 10 = 1 dollar. The notebook checks the rule on a made-up column in Larkfield’s logs. For a regression with other columns, the rule holds about, not exactly.
The grid
Section titled “The grid”The grid in the output of xp refute applies this rule to the estimate of +3.38 dollars:
| Share gap | Worth 5 $ | Worth 10 $ | Worth 20 $ | Worth 40 $ |
|---|---|---|---|---|
| 0.05 | +3.13 | +2.88 | +2.38 | +1.38 |
| 0.10 | +2.88 | +2.38 | +1.38 | −0.62 |
| 0.20 | +2.38 | +1.38 | −0.62 | −4.62 |
| 0.30 | +1.88 | +0.38 | −2.62 | −8.62 |
Each cell is the effect that would remain if a hidden confounder had that size. Small or weak confounders leave a positive effect. Only a confounder that is both common among users and worth a lot of money flips the sign.
The simulator’s garden has a share gap of 0.087 and is worth 10.27 dollars. That is close to the cell (0.10, 10 dollars), which leaves +2.38: almost exactly the true +2.37. In a real study, you do not know the garden’s size. You ask the people who know the customers: “Is there a cause this strong that we did not measure?”
The chart: where the answer flips
Section titled “The chart: where the answer flips”- Brings the estimate to 0
- Explains the real bias
- The garden
Text description of the chart
A chart with the share gap from 0 to 0.3 across and the hidden cause’s effect in dollars up. An orange curve falls from 67.6 dollars at a gap of 0.05 to 33.8 dollars at 0.10 and 11.3 dollars at 0.30. Any confounder on or above it would bring the estimate to 0. A dashed blue curve falls from 33.7 dollars at 0.03 to 10.1 dollars at 0.10 and 3.4 dollars at 0.30. Any confounder on it explains the real bias of 1.01 dollars. A point marks the garden at a share gap of 0.087 and 10.27 dollars, close to the blue curve and far below the orange one.
The garden adds 0.087 × 10.27 = 0.89 dollars of bias by the rule. The real bias of this run is 1.01 dollars. The rule is close, not exact, because the garden is a little related to the other columns.
Could a garden-like cause bring the answer to 0? With the garden’s share gap, it would need to be worth about 3.38 / 0.087 = 39 dollars. That is almost four times the garden. So the sign of the answer is safe: the block helps. Its size is not: a garden-like cause can move it by a dollar. Larkfield should report “about +2 to +3 dollars per customer, likely a little less than our estimate”, and not “+3.38”.
When to use it, when not, and what it costs
Section titled “When to use it, when not, and what it costs”Use refutation tests:
- every time you estimate an effect without a test. They are cheap. They catch real bugs: a wrong column, a method that finds effects in noise, an answer that rests on a few customers;
- as a record: a report that lists the tests run is easier to trust.
Use a sensitivity analysis:
- whenever a confounder could be missing, which is almost always with logs;
- to turn “it could be biased” into a number. For example: “to change the decision, a cause must be worth X dollars and be Y points more common among users”.
Do not treat either one as proof. Refutation tests cannot see missing columns. A sensitivity grid does not tell you whether such a cause exists: people who know the customers must judge that. Our grid also assumes a simple yes-or-no cause; DoWhy and other libraries have more general methods.
The cost: under a minute of computing for the refutation tests on 40,000 customers, and a conversation about what could be missing. That conversation is the valuable part.
Guided practice: refute and measure by hand
Section titled “Guided practice: refute and measure by hand”- Open the notebook from the box Do this lesson in a notebook at the top of this page.
- Run the setup cells and the helper cell. The notebook reads the same logs as
sim run --scenario logs(seed 40). It writes the three tests by hand, without DoWhy, so its random numbers differ a little. - In section 1, fit the estimate. You see
estimate: +3.38 dollars per customer. - In sections 2 to 4, run the three tests. You see
placebo: mean +0.03,random common cause: mean +3.38anddata subsets: mean +3.31. - In section 5, check the rule on a made-up column. You see
share gap 0.103; the naive difference moved by +1.03 (rule: +1.03). - In section 6, find the share gap that brings the estimate to 0 for a 10-dollar confounder. Put it in
gap_needed. Run the check cell. It printsCheck passed.
Show the answer to step 6
gap_needed = round(est / 10, 3)print(gap_needed)The value is 0.338. The bias is share gap × 10 dollars, and it must equal the estimate, so the share gap is 3.38 / 10.
Your turn: does the decision hold?
Section titled “Your turn: does the decision hold?”Omar says: “The block costs us about 2 dollars per customer to run. We keep it if it adds more than 2 dollars.” Use the grid above. Which hidden confounders would change his decision? Is the garden one of them? Answer before you open the solution.
Show the answer
The decision changes where a cell falls below +2.00. In the grid, these are:
- a gap of 0.05 with 40 dollars (+1.38);
- a gap of 0.10 with 20 or 40 dollars;
- a gap of 0.20 with 10 dollars or more;
- a gap of 0.30 with any of the four sizes (even 5 dollars leaves +1.88).
A garden-sized cause (0.087 share gap, 10.27 dollars) leaves about 3.38 − 0.89 = +2.49 dollars by the rule. The true ATE is +2.37. Both are above 2, so the decision holds. But the margin is small: a second hidden cause of the garden’s size would bring the answer below 2.
So the honest report is: “The block adds about 2.5 to 3.4 dollars per customer. A second missing cause as strong as a garden would change the decision. A test would settle it.” Then plan a test: in Randomise, a coin balanced the garden without seeing it.
Common mistake
“DoWhy says the estimate is robust”
Symptom: a team estimates a feature’s effect from logs, runs DoWhy’s refutation tests, and every test passes. The report says “robust: validated by refutation tests”. Later, a real A/B test finds an effect a third smaller.
Cause: the tests only change the data that the team has. They cannot see a confounder that is missing from it. The estimate was correct for the columns in the log, and biased by a column that was not there.
Fix: report what the tests show: no bug was found in the method. Add a sensitivity analysis: how strong would a hidden confounder need to be to change the decision? Ask the people who know the customers whether such a cause exists. When the decision matters, run a test.
Check your understanding
A placebo test on your estimate gives a new effect of +3.10 dollars, close to your estimate of +3.38. What does that tell you?
Show the answer
The test failed. With the treatment shuffled at random, a sound method should find about 0. Finding +3.10 means the method produces an “effect” even when the treatment means nothing. So the +3.38 likely comes from the method or the data, not from the treatment. Examples: a coding error, or a column that leaks the outcome.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
All three refutation tests pass on Larkfield's logs, but the estimate (+3.38) is 1 dollar above the truth (+2.37). Explain why the tests could not catch this.
Show the answer
The tests change things inside the data: the treatment column, an added noise column, the rows. They check that the method reacts correctly to those changes. The bias comes from the garden, a confounder that is not in the data at all. Every test runs on the same six columns, so every test has the same blind spot. A pass means no bug was found, not that the estimate is right.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
A hidden confounder is 20 points more common among users than non-users (a share gap of 0.20) and adds 15 dollars of spending. By the rule, what effect would remain from an estimate of +3.38?
Show the answer
The bias is about 0.20 × 15 = 3.00 dollars. The remaining effect is about 3.38 − 3.00 = +0.38 dollars: still positive, but close to 0. A confounder of this size would change most decisions, so the team must ask whether such a cause could exist.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Recap
- A refutation test changes the data in a known way and checks the method’s answer. Placebo treatment should give about 0; a random common cause and a data subset should leave the estimate about the same.
- On Larkfield’s logs, DoWhy’s three tests pass: −0.07, +3.38 and +3.45 for an estimate of +3.38. The truth is +2.37. A pass means “no bug found”, not “right”.
- No test of the data can see a missing column. Three methods that agree, and tests that pass, share the same blind spot.
- Sensitivity analysis asks how strong a hidden confounder must be: bias ≈ share gap × dollars. The garden (0.087 × 10.27 = 0.89) explains most of the real bias (1.01). To bring the answer to 0, it would need to be worth about 39 dollars.
- Report the effect as a range with its weak point, and run a test when the decision matters.
Next, a new question: not whether a change works on average, but for whom.